Rolle Laukkarinen on Nostr: The downtime was more than just a minute. It lasted 16 minutes in total, during which ...
The downtime was more than just a minute. It lasted 16 minutes in total, during which mementomori.social was completely inaccessible and timing out.
What happened: during today's routine server updates the main server was rebooted for a new kernel. After the reboot it had two routes to the internet and picked the wrong one, so requests arrived but the replies never made it back out.
mementomori.social was unreachable over IPv4 from about 14:40 to 14:56. Over IPv6 it worked the whole time. Nothing was lost: posts, media and the database were fine throughout.
What went wrong: both the server's private and public networks assign a default route with the same priority, so after a reboot, it is completely random which one takes over. It has been like this since the server was migrated to a new cluster a week ago, and this was the first maintenance window for routine updates. I thought it was reboot-safe, but it turns out it was not.
Fix: the private route now has a lower priority on both servers, so the public route always wins, including after reboots. Sorry for the outage.
It was a bit scary to get completely locked out for a moment, especially since my household only has IPv4. I managed to get back in after realizing it randomly chooses the interface, so I sent Ctrl+Alt+Del through the admin console, hoping the coin toss would go my way...
#MementomoriSocial #SysAdmin #SysOp
Published at
2026-10-03 12:09:26 UTCEvent JSON
{
"id": "ea7c284cffaaeeb8ce79c2c411379788a3c95cae08d2bab41905663925cb3dc9",
"pubkey": "118cd39da270a800372ab7276a46b488cca3c40dd2b34f73b857fc8f72fae0f8",
"created_at": 1791029366,
"kind": 1,
"tags": [
[
"t",
"mementomorisocial"
],
[
"t",
"sysadmin"
],
[
"t",
"sysop"
],
[
"proxy",
"https://mementomori.social/users/rolle/statuses/117376900590070947",
"activitypub"
],
[
"client",
"Mostr",
"31990:6be38f8c63df7dbf84db7ec4a6e6fbbd8d19dca3b980efad18585c46f04b26f9:mostr",
"wss://relay.ditto.pub"
]
],
"content": "\n\nThe downtime was more than just a minute. It lasted 16 minutes in total, during which mementomori.social was completely inaccessible and timing out.\n\nWhat happened: during today's routine server updates the main server was rebooted for a new kernel. After the reboot it had two routes to the internet and picked the wrong one, so requests arrived but the replies never made it back out.\n\nmementomori.social was unreachable over IPv4 from about 14:40 to 14:56. Over IPv6 it worked the whole time. Nothing was lost: posts, media and the database were fine throughout.\n\nWhat went wrong: both the server's private and public networks assign a default route with the same priority, so after a reboot, it is completely random which one takes over. It has been like this since the server was migrated to a new cluster a week ago, and this was the first maintenance window for routine updates. I thought it was reboot-safe, but it turns out it was not.\n\nFix: the private route now has a lower priority on both servers, so the public route always wins, including after reboots. Sorry for the outage.\n\nIt was a bit scary to get completely locked out for a moment, especially since my household only has IPv4. I managed to get back in after realizing it randomly chooses the interface, so I sent Ctrl+Alt+Del through the admin console, hoping the coin toss would go my way...\n\n#MementomoriSocial #SysAdmin #SysOp",
"sig": "48c686324ef3492cb8ebc5cca64e19fff4c71627ec55f74ff0b5bd9c01fe1da1adf2bd24cce1057b5b7f772b872b3b50c5c6f5070014948ea004a2828bf30ba3"
}
