Nostr notes by Stuart Longland (VK4MSL)

Oookay, let's see if she starts…

2026-08-22T10:38:23Z

In reply to nevent1q…rve6
_________________________

Oookay, let's see if she starts…

`bundle install` and `yarn install` have run now… just ...

2026-08-22T10:31:51Z

In reply to nevent1q…jlh4
_________________________

`bundle install` and `yarn install` have run now… just compiling the assets before I start database migrations.

Oookay, time for Mastodon 4.7.0… #MastoAdmin First step, ...

2026-08-22T09:56:51Z

Oookay, time for Mastodon 4.7.0… #MastoAdmin

First step, updating `ruby`. I just did a clean up of several old builds of Ruby in `rbenv`.

> An unplanned outage is affecting the network Ahh, that ...

2026-07-30T04:24:34Z

In reply to nevent1q…c4q4
_________________________

> An unplanned outage is affecting the network

Ahh, that explains it. I'm not going crazy.

Hmmm great… seems the NBN link at home has dropped out and the ...

2026-07-30T03:56:22Z

Hmmm great… seems the NBN link at home has dropped out and the 4G fail-over hasn't cut across.

Not much I can do for the next 2½ hours while I'm stuck at work.

Oookay, a kernel got pulled down in the `dist-upgrade`, guess ...

2026-06-28T10:24:28Z

In reply to nevent1q…83na
_________________________

Oookay, a kernel got pulled down in the `dist-upgrade`, guess that's a reboot…

Since I'm a glutton for punishment… maybe I'll do that ...

2026-06-28T10:13:45Z

In reply to nevent1q…9e0n
_________________________

Since I'm a glutton for punishment… maybe I'll do that update to Mastodon 4.6.2 that I was originally going to do last night. `ffmpeg` is being dealt with.

Nostr event nevent1qqsgq55dkmngj5r2853q60n24kndc2a7kcvlh0qj6y04jvww4689e8qzyp2ukpfkr69q6zt09c949uttgpl87u656mfde64c5ak9g8mskc8fuznn8pn

2026-06-28T10:09:12Z

In reply to nevent1q…9e0n
_________________________

RE: https://mastodon.longlandclan.id.au/@stuartl/116826959111257560

Found the culprits… one was the border router being power cycled unexpectedly, leaving filesystem corruption in `/var` that forced single-user mode manual `fsck` (a big #OpenBSD anti-feature here).

The other, was this little gem. Two M.2 SATA SSDs in two different servers deciding that last night would be their last ever. These are 240GB WD Green SSDs, bought purely as boot drives back around 2020 or so. One installed in the M.2 slot on a Intel NUC, the other into a M.2 USB enclosure. They've run these machines almost 24/7 for about 5 years.

I've held my nose and bought some Silicon Power 128GB drives, as that's all Umart has in M.2 SATA form factor. (Well, I can go bigger, but I barely use 12GB.) Reviews were terrible, lots of reports of early failures, maybe I'll be lucky, or maybe these will have the life expectancy of a tray of Tim Tams at a weight-watchers weigh-in. We'll see.

quoting
note1j5k…04zs

Well, you picked a fine time to leave me boot disk…
Along with your comrade, blowing my OS away…
My pair of downed servers, 12TB of replicas…
now my Ceph cluster won't heal…
…you picked a fine time to leave me, boot disk!

#HomeServer #Ceph

Not quite sure why, but the home Internet link seems to have gone ...

2026-06-27T20:40:23Z

Not quite sure why, but the home Internet link seems to have gone belly-up.

I'll swap over here while that happens.

2026-06-17T22:24:36Z

In reply to nevent1q…m6xz
_________________________

I'll swap over here while that happens.

So, I've mounted my snapshot from the 11th read-only. ...

2026-04-13T10:45:15Z

In reply to nevent1q…ezty
_________________________

So, I've mounted my snapshot from the 11th read-only.
I've re-formatted the volume XFS and have it mounted read-write.

`rsync` is copying the files. I see there's remnants of the old "combined" PostgreSQL + Mastodon system + redis + backups directories there (as btrfs subvolumes).

Some time ago I split it out to separate RBD volumes for performance reasons. So only the back-ups remained on the disk. Not quite sure why that brought the machine down, but we might get through this unscathed.

`rsync` of course is complaining about I/O errors transferring some of the back-up files, but whatever… the other volumes are good, I can recover from this. Back-ups will be on XFS from now on.

I've done a `fsck` of the other volumes, they're okay, but I've snapshotted them just in case. I can migrate those to XFS later. We'll let this run overnight and see how things are tomorrow.

eff it… we're going XFS.

2026-04-13T10:14:54Z

In reply to nevent1q…0muu
_________________________

eff it… we're going XFS.

``` tree extent[30474240, 16384] parent 78826651648 has no ...

2026-04-13T10:05:51Z

In reply to nevent1q…kf9k
_________________________

```
tree extent[30474240, 16384] parent 78826651648 has no backref item in extent tree
backpointer mismatch on [30474240 16384]
adding new tree backref on start 30474240 len 16384 parent 78826651648 root 78826651648
Repaired extent references for 30474240
bad full backref, on [30490624]
ERROR: commit_root already set when starting transaction
ERROR: failed to start transaction: Invalid argument
failed to repair damaged filesystem, aborting
beryllium:~#
```
… bugger!

Still going this afternoon, at extent #170752147456… I note the ...

2026-04-13T04:23:23Z

In reply to nevent1q…m6xn
_________________________

Still going this afternoon, at extent #170752147456… I note the pace has quickened quite a bit, so maybe that's a good thing?

I wake up to the server fiddling extents at extent #128730550272. ...

2026-04-12T20:43:21Z

In reply to nevent1q…wz0k
_________________________

I wake up to the server fiddling extents at extent #128730550272. The hardest bit is knowing what the progress is… the number alone is meaningless unless I know what the total number of extents is.

Some might suggest at this point I hit ^C, reformat the volume and restore from a back-up. If I have to do that, I think I'll be re-formatting XFS not BTRFS.

Well, no such luck… still going. Somewhere around extent ...

2026-04-12T08:30:33Z

In reply to nevent1q…3r9m
_________________________

Well, no such luck… still going. Somewhere around extent 107335499776 now.

Not sure what the uptime stats are for my server. I don't actually measure uptimes, but I first noticed it offline yesterday morning around 6AM.

Last stretch of downtime was on the 28th March for about 10 minutes.

This is now about 36 hours. Assuming no downtime between those points, I'm at about 89.8% uptime availability.

That puts me just a smidge above Github: https://mrshu.github.io/github-statuses/

As some might've guessed… that `fsck` is still grinding ...

2026-04-11T21:17:08Z

In reply to nevent1q…vgxl
_________________________

As some might've guessed… that `fsck` is still grinding away.

24+ hours. I thought large EXT2 volumes took a long time to check, but it seems BTRFS can be just as bad.

```
ref mismatch on [77402689536 8192] extent item 1, found 0
data extent[77402689536, 8192] bytenr mimsmatch, extent item bytenr 77402689536 file item bytenr 0
data extent[77402689536, 8192] referencer count mismatch (root 256 owner 227855 offset 33595392) wanted 1 have 0
backpointer mismatch on [77402689536 8192]
owner ref check failed [77402689536 8192]
repair deleting extent record: key [77402689536,168,8192]
Repaired extent references for 77402689536

```

Not sure what number that's expected to climb to. Maybe with some luck I might be "back home" this afternoon, but at this rate, who knows?

And no, I don't know what a "mimsmatch" is either…

2026-04-11T10:50:06Z

In reply to nevent1q…xjru
_________________________

And no, I don't know what a "mimsmatch" is either…

Still going… seeing a lot of messages like this: ``` ref ...

2026-04-11T10:48:37Z

In reply to nevent1q…u0el
_________________________

Still going… seeing a lot of messages like this:

```
ref mismatch on [35132342272 8192] extent item 1, found 0
data extent[35132342272, 8192] bytenr mimsmatch, extent item bytenr 35132342272 file item bytenr 0
data extent[35132342272, 8192] referencer count mismatch (root 256 owner 227508 offset 149012480) wanted 1 have 0
backpointer mismatch on [35132342272 8192]
owner ref check failed [35132342272 8192]
repair deleting extent record: key [35132342272,168,8192]
Repaired extent references for 35132342272
ref mismatch on [35132350464 8192] extent item 1, found 0
data extent[35132350464, 8192] bytenr mimsmatch, extent item bytenr 35132350464 file item bytenr 0
data extent[35132350464, 8192] referencer count mismatch (root 256 owner 227508 offset 149020672) wanted 1 have 0
backpointer mismatch on [35132350464 8192]
owner ref check failed [35132350464 8192]
repair deleting extent record: key [35132350464,168,8192]
Repaired extent references for 35132350464
```

Whatever `btrfs` did, it did a proper job of it. 10 hours later ...

2026-04-11T05:53:33Z

In reply to nevent1q…80ak
_________________________

Whatever `btrfs` did, it did a proper job of it.

10 hours later and `btrfsck --repair` still grinding away. Yes I took a snapshot of the disk before making changes.

Over here for a bit, because right now my Mastodon server is busy ...

2026-04-10T20:59:01Z

Over here for a bit, because right now my Mastodon server is busy undergoing a `btrfsck --repair` on a key volume.

Swapping over here as I jump the home server from Mastodon 4.5.5 ...

2026-03-28T10:34:59Z

Swapping over here as I jump the home server from Mastodon 4.5.5 to 4.5.8.

Let's see if we can force-recompile the offender. Guessing ...

2026-02-06T22:12:04Z

In reply to nevent1q…cqhx
_________________________

Let's see if we can force-recompile the offender. Guessing here, but let's try:

```
mastodon@mastodon:~/live$ env RAILS_ENV=production bundle install --force
```

Ohh wonderful, so `apt-get dist-upgrade` has managed to break the ...

2026-02-06T22:09:57Z

Ohh wonderful, so `apt-get dist-upgrade` has managed to break the custom Ruby build that runs Mastodon. Or rather, one of its "gems".

```
==> /var/log/syslog <==
2026-02-07T08:08:31.370873+10:00 mastodon bundle[9097]: There was an error while trying to load the gem 'charlock_holmes'.
2026-02-07T08:08:31.371202+10:00 mastodon bundle[9097]: Gem Load Error is: libicudata.so.72: cannot open shared object file: No such file or directory - /home/mastodon/live/vendor/bundle/ruby/3.4.0/gems/charlock_holmes-0.7.9/lib/charlock_holmes/charlock_holmes.so
```

#MastoAdmin

Oookay… booting back up.

2026-01-20T20:57:33Z

In reply to nevent1q…0kx6
_________________________

Oookay… booting back up.

I notice `pg_dump` is still complaining about the collation: ``` ...

2026-01-20T20:51:47Z

In reply to nevent1q…sp9j
_________________________

I notice `pg_dump` is still complaining about the collation:

```
WARNING: database "mastodon_production" has a collation version mismatch
DETAIL: The database was created using collation version 2.36, but the operating system provides version 2.41.
HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE mastodon_production REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
```

… but it seems to be running okay. I don't feel like doing a `DROP DATABASE` and restore at this time to rebuild the objects, unless there's some simpler way to do a rebuild.

Anyway, the build is started, we should be back soonish.

Swapping over here during the update.

2026-01-20T20:45:55Z

In reply to nevent1q…6awk
_________________________

Swapping over here during the update.

Are we back yet? Let's see.

2026-01-08T02:32:27Z

In reply to nevent1q…u95j
_________________________

Are we back yet? Let's see.

Oookay, Mastodon rebuild done, updates applied… another reboot.

2026-01-08T02:30:06Z

In reply to nevent1q…eqt8
_________________________

Oookay, Mastodon rebuild done, updates applied… another reboot.

This would explain it, I forgot about `/etc/apt/sources.list.d` ...

2026-01-08T02:26:09Z

In reply to nevent1q…sl00
_________________________

This would explain it, I forgot about `/etc/apt/sources.list.d` … so another update pass.

Pulled latest stable Mastodon, and I'm currently building ...

2026-01-08T02:24:36Z

In reply to nevent1q…slsd
_________________________

Pulled latest stable Mastodon, and I'm currently building that.

I notice this message:

```
WARNING: database "mastodon_production" has a collation version mismatch
DETAIL: The database was created using collation version 2.36, but the operating system provides version 2.41.
HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE mastodon_production REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
```

Oookay, `reboot` time… at last.

2026-01-08T02:18:37Z

In reply to nevent1q…32nc
_________________________

Oookay, `reboot` time… at last.

Did an `apt-get --fix-broken install` which succeeded, and re-ran ...

2026-01-08T02:14:37Z

In reply to nevent1q…py0g
_________________________

Did an `apt-get --fix-broken install` which succeeded, and re-ran `apt-get dist-upgrade -y`.

It seems to be installing without incident, and most crucially, without this `sse4.2-support`. So clearly it wasn't needed, makes me wonder why it was picked up as an update in the first place.

``` Preparing to unpack .../0-sse4.2-support_27_amd64.deb ... ...

2026-01-08T01:48:31Z

In reply to nevent1q…xl5j
_________________________

```
Preparing to unpack .../0-sse4.2-support_27_amd64.deb ...
This machine doesn't support SSE4.2, sorry.
Aborting.
dpkg: error processing archive /tmp/apt-dpkg-install-e5FGqW/0-sse4.2-support_27_amd64.deb (--unpack):
new sse4.2-support package pre-installation script subprocess returned error exit status 2
```

Not a promising message. The CPUs in the VM report themselves as "QEMU Virtual CPU version 2.5+", the physical CPU is a Intel Atom C2000 series.

Download is finished and database back-ups taken. The packages ...

2026-01-08T00:34:29Z

In reply to nevent1q…kw2y
_________________________

Download is finished and database back-ups taken. The packages are being installed now.

After that, I'll reboot (to activate the new kernel and `glibc`), then I can tackle Mastodon's update to 4.5.4.

I'll swap over here for the moment.

2026-01-08T00:19:59Z

In reply to nevent1q…ttc8
_________________________

I'll swap over here for the moment.

``` FATAL ERROR: Reached heap limit Allocation failed - ...

2025-12-12T03:05:44Z

In reply to nevent1q…0qcw
_________________________

```

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

1: 0xe38a0e node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/usr/bin/node]
2: 0x120a5a0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/bin/node]
3: 0x120a877 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/bin/node]
4: 0x1438155 [/usr/bin/node]
5: 0x14519e9 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/bin/node]
6: 0x1426098 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
7: 0x1426fc5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
8: 0x13ffc9e v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/bin/node]
9: 0x18612fc v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/bin/node]
10: 0x1dbf476 [/usr/bin/node]

pid 1614196 exit 1
Build with Vite failed! ❌
```

!@#$ JavaScript!

Starting up…

2025-11-07T00:06:20Z

In reply to nevent1q…dtk6
_________________________

Starting up…

Failed a second time, apparently it now works _without_ that ...

2025-11-07T00:04:29Z

In reply to nevent1q…mdxa
_________________________

Failed a second time, apparently it now works _without_ that option set.

It used to be it'd fail to build unless I did set the "max old space size". This is a VM with 8GB of RAM… it did get an upgrade when the Atom C2000 issue was resolved on a couple of old boards meant I gained some extra compute capacity… but yeah… JavaScript drives me nuts.

Anyway, managed to get the assets compiled… onward with database migration.

Trying again with `--max-old-space-size=512`.

2025-11-06T23:31:12Z

In reply to nevent1q…wfn6
_________________________

Trying again with `--max-old-space-size=512`.

``` <--- JS stacktrace ---> FATAL ERROR: Reached heap limit ...

2025-11-06T23:29:55Z

In reply to nevent1q…2006
_________________________

```
<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

1: 0xe38a0e node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/usr/bin/node]
2: 0x120a5a0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/bin/node]
3: 0x120a877 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/usr/bin/node]
4: 0x1438155 [/usr/bin/node]
5: 0x14519e9 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/bin/node]
6: 0x1426098 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
7: 0x1426fc5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
8: 0x13ffc9e v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/bin/node]
9: 0x18612fc v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/bin/node]
10: 0x1dbf476 [/usr/bin/node]

pid 2154020 exit 1
Build with Vite failed! ❌
```

… I f'ing hate #JavaScript.

``` root@mastodon:~# nodejs --version v22.21.0 ``` …let's ...

2025-11-06T23:23:10Z

In reply to nevent1q…lh4s
_________________________

```
root@mastodon:~# nodejs --version
v22.21.0
```

…let's try that assets compile step again!

So the issue was `liburing1` not being available on Debian 12… ...

2025-11-06T23:22:24Z

In reply to nevent1q…v40w
_________________________

So the issue was `liburing1` not being available on Debian 12… and also `/etc/apt/sources.list.d/postgresql.list` still referencing Debian 11 repositories.

Not so much I'm "holding" broken packages, just the distribution has moved on. Now PostgreSQL v18 is installed and it'll attempt an update of the cluster (from v17). I've got a full database dump.

``` root@mastodon:~# apt-get install -y postgresql Reading ...

2025-11-06T23:18:48Z

In reply to nevent1q…vhl8
_________________________

```
root@mastodon:~# apt-get install -y postgresql
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
postgresql-18 : Depends: liburing1 (>= 0.7) but it is not installable
Recommends: postgresql-18-jit but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
```

…excuse me, *I* have held broken packages? *I*? You're operating in Australia, we're presumed innocent until found guilty by a jury here… Evidence please!

In the midst of picking apart NodeSource's `setup_22.x` ...

2025-11-06T23:17:10Z

In reply to nevent1q…2qzx
_________________________

In the midst of picking apart NodeSource's `setup_22.x` script so I can manually apply the changes without running a script as `root`… I see this:

```
The following packages have been kept back:
postgresql
```

I hate it when Debian does things without explaining what's going on.

``` Building with Vite ⚡ You are using Node.js 20.5.1. Vite ...

2025-11-06T23:07:06Z

In reply to nevent1q…pj26
_________________________

```
Building with Vite ⚡
You are using Node.js 20.5.1. Vite requires Node.js version 20.19+ or 22.12+. Please upgrade your Node.js version.
failed to load config from /home/mastodon/live/vite.config.mts
error during build:
TypeError: crypto.hash is not a function
at file:///home/mastodon/live/node_modules/@vitejs/plugin-legacy/dist/index.js:584:21
at Array.map (<anonymous>)
at file:///home/mastodon/live/node_modules/@vitejs/plugin-legacy/dist/index.js:584:3
at ModuleJob.run (node:internal/modules/esm/module_job:192:25)
at async DefaultModuleLoader.import (node:internal/modules/esm/loader:228:24)
at async loadConfigFromBundledFile (file:///home/mastodon/live/node_modules/vite/dist/node/chunks/config.js:36128:12)
at async bundleAndLoadConfigFile (file:///home/mastodon/live/node_modules/vite/dist/node/chunks/config.js:36014:17)
at async loadConfigFromFile (file:///home/mastodon/live/node_modules/vite/dist/node/chunks/config.js:35981:42)
at async resolveConfig (file:///home/mastodon/live/node_modules/vite/dist/node/chunks/config.js:35632:22)
at async createBuilder (file:///home/mastodon/live/node_modules/vite/dist/node/chunks/config.js:34104:19)

pid 2145854 exit 1
Build with Vite failed! ❌
```
… bugger.

Nostr event nevent1qqswcrr5rjqafe4w57jc4wzls6adc8sjzyrg3c22pj8pzv5hucv0nfczyp2ukpfkr69q6zt09c949uttgpl87u656mfde64c5ak9g8mskc8fu982eat

2025-11-06T23:06:17Z

In reply to nevent1q…tr43
_________________________

`NODE_OPTIONS="--max-old-space-size=768" RAILS_ENV=production bundle exec rails assets:precompile`

The `--max-old-space-size=768` is needed here because `nodejs`' heap management sucks, it just `malloc`s until it hits `ENOMEM` then dies.

Oookay… onto the next step. `bundle install && yarn ...

2025-11-06T22:51:33Z

In reply to nevent1q…e3xk
_________________________

Oookay… onto the next step. `bundle install && yarn install --immutable`

To be fair, I saw the `rbenv install` and missed the other bit, ...

2025-11-06T22:23:22Z

In reply to nevent1q…au78
_________________________

To be fair, I saw the `rbenv install` and missed the other bit, due to the way the text wrapped.

Crap, forgot `RUBY_CONFIGURE_OPTS=--with-jemalloc`…

2025-11-06T22:21:16Z

In reply to nevent1q…yvrl
_________________________

Crap, forgot `RUBY_CONFIGURE_OPTS=--with-jemalloc`…

``` mastodon@mastodon:~/live$ rbenv install ==> Downloading ...

2025-11-06T22:19:13Z

In reply to nevent1q…v4c3
_________________________

```
mastodon@mastodon:~/live$ rbenv install
==> Downloading ruby-3.4.7.tar.gz...
-> curl -q -fL -o ruby-3.4.7.tar.gz https://cache.ruby-lang.org/pub/ruby/3.4/ruby-3.4.7.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 22.1M 100 22.1M 0 0 2503k 0 0:00:09 0:00:09 --:--:-- 3143k
==> Installing ruby-3.4.7...
-> ./configure "--prefix=$HOME/.rbenv/versions/3.4.7" --enable-shared --with-ext=openssl,psych,+
-> make -j 4
-> make install
```
… took its time as usual… but we should soon be moving on to the next phase.

Errm, the original Pi is ARMv6: `armhf` architecture. `armel` is ...

2025-11-06T22:04:44Z

In reply to nevent1q…9ekc
_________________________

Errm, the original Pi is ARMv6: `armhf` architecture.

`armel` is the software floating-point ABI.
`armhf` is the hardware floating-point ABI.

`armhf` isn't going anywhere for a little while yet, your Pi v1s are still good for now

There are times I wonder why I run Debian for a Mastodon host… ...

2025-11-06T21:53:27Z

In reply to nevent1q…q36v
_________________________

There are times I wonder why I run Debian for a Mastodon host… I compile that much of it from source I might as well be running Gentoo.

Ahh, `rbenv` my nemesis, we meet again… time to pull v3.4.7 ...

2025-11-06T21:52:45Z

In reply to nevent1q…vlq5
_________________________

Ahh, `rbenv` my nemesis, we meet again… time to pull v3.4.7 (from v3.2.3).

Oookay… I'll swap over here for a bit while I get my ...

2025-11-06T21:40:42Z

Oookay… I'll swap over here for a bit while I get my instance updated.

Swapping over here for a bit… my server's behind a few ...

2025-10-24T02:44:17Z

Swapping over here for a bit… my server's behind a few versions so I better bump Mastodon up to the latest on the 4.4 branch ahead of a move to 4.5 when it's finally released. #MastoAdmin

``` ✓ built in 9m 52s PWA v1.0.0 Building ...

2025-09-28T00:40:46Z

In reply to nevent1q…49pt
_________________________

```
✓ built in 9m 52s

PWA v1.0.0
Building mastodon/service_worker/sw.js service worker ("es" format)...
vite v6.3.6 building for production...
transforming...
✓ 73 modules transformed.
rendering chunks...
computing gzip size...
../../public/packs/sw.mjs 188.78 kB │ gzip: 60.27 kB │ map: 1,065.79 kB
✓ built in 11.28s
Build with Vite complete: /home/mastodon/live/public/packs
mastodon@mastodon:~/live$
```
…restarting.

``` ✓ 1940 modules transformed. rendering chunks... ``` … ...

2025-09-28T00:38:57Z

In reply to nevent1q…3lze
_________________________

```
✓ 1940 modules transformed.
rendering chunks...
```
… nearly there.

While I'm fine with quote posts in general, can't say ...

2025-09-28T00:36:38Z

While I'm fine with quote posts in general, can't say I'm excited by the "boost" button essentially being a menu button. Ohh, and I'll have to whitelist quote-posts for UNOFFICIAL Queensland fire bot (npub1s35…d4qd). #QuotePost #Mastodon #mastodon45

``` I, [2025-09-28T10:29:20.664134 #3384849] INFO -- : [dotenv] ...

2025-09-28T00:31:56Z

In reply to nevent1q…ca9r
_________________________

```
I, [2025-09-28T10:29:20.664134 #3384849] INFO -- : [dotenv] Loaded .env.production
Building with Vite ⚡
vite v6.3.6 building for production...
transforming...
```

Little bit of downtime on my home server while I update to ...

2025-09-28T00:30:00Z

Little bit of downtime on my home server while I update to Mastodon 4.4.5. #MastoAdmin

WICEN OwnCloud also got knocked out… seemingly couldn't run ...

2025-08-31T03:44:40Z

In reply to nevent1q…a8fs
_________________________

WICEN OwnCloud also got knocked out… seemingly couldn't run `e2fsck` (it seems to be using EXT4). A map of the RBD and a manual run has sorted that one out.

Actually it's taken out 3 of the 4 VMs with filesystem ...

2025-08-31T03:40:18Z

In reply to nevent1q…9m3z
_________________________

Actually it's taken out 3 of the 4 VMs with filesystem corruption… so much for btrfs being self-healing.

…and of course the Mastodon host running there has just crapped ...

2025-08-31T03:38:22Z

In reply to nevent1q…a54q
_________________________

…and of course the Mastodon host running there has just crapped itself. btrfs is failing to mount the filesystem and Debian + systemd is incapable of dropping to a root shell when things go pear shaped.

Nostr event nevent1qqsz239lffscmv0vnje88ykhc90uw3azrle06xz2vd0v9tguhtxvxjczyp2ukpfkr69q6zt09c949uttgpl87u656mfde64c5ak9g8mskc8fukel4dc

2025-08-31T03:29:24Z

Yaaay, machine check exception bringing a node down… what fun!

``` ✓ built in 18.20s Build with Vite complete: ...

2025-08-30T08:50:19Z

In reply to nevent1q…060p
_________________________

```
✓ built in 18.20s
Build with Vite complete: /home/mastodon/live/public/packs
```

…ya huh, that was a long 18.2 seconds.

``` Building with Vite ⚡ vite v6.3.5 building for production... ...

2025-08-30T08:48:50Z

In reply to nevent1q…k427
_________________________

```
Building with Vite ⚡
vite v6.3.5 building for production...
transforming...
✓ 1940 modules transformed.
rendering chunks...
```
…don't rush yourself…

Waiting for `vite` to weave its magic… then we should be able ...

2025-08-30T08:41:37Z

In reply to nevent1q…k2z4
_________________________

Waiting for `vite` to weave its magic… then we should be able to re-start.

Quick update of the server to Mastodon 4.4.3… since I notice ...

2025-08-30T08:35:12Z

Quick update of the server to Mastodon 4.4.3… since I notice 4.4.2 isn't sending email notifications and can't show videos full-screen. (It'll open the video then immediately close it.)

Migrations done… let's see if she'll start up.

2025-07-09T03:45:59Z

In reply to nevent1q…jj2s
_________________________

Migrations done… let's see if she'll start up.

``` bin/rails aborted! LoadError: Could not open library ...

2025-07-09T03:08:53Z

In reply to nevent1q…wce5
_________________________

```
bin/rails aborted!
LoadError: Could not open library 'vips.so.42': vips.so.42: cannot open shared object file: No such file or directory. (LoadError)
Could not open library 'libvips.so.42': libvips.so.42: cannot open shared object file: No such file or directory.
```

`apt-get install -y libvips` duh! Forgot that.

``` ==> Installed ruby-3.4.4 to ...

2025-07-09T02:36:02Z

In reply to nevent1q…ry09
_________________________

```
==> Installed ruby-3.4.4 to /home/mastodon/.rbenv/versions/3.4.4

```

First hurdle cleared. Now for the next steps…

```
env NODE_OPTIONS="--max-old-space-size=768" RAILS_ENV=production sh -c 'bundle install && yarn install --immutable && bundle exec rails assets:precompile' && SKIP_POST_DEPLOYMENT_MIGRATIONS=true RAILS_ENV=production bundle exec rails db:migrate
```

Hi ho hi ho, it's Mastodon update time I go… you'll ...

2025-07-09T02:16:07Z

Hi ho hi ho, it's Mastodon update time I go… you'll find me over here in the interim.

https://mastodon.social/@Mastodon/114818205314251502

#MastoAdmin

8 of the 10 OSDs have updated their metadata… waiting on the ...

2025-06-30T22:45:43Z

In reply to nevent1q…jwqv
_________________________

8 of the 10 OSDs have updated their metadata… waiting on the last two which are taking a while.

In the meantime, the RBD volume that runs PostgreSQL is blocked… so I'll hang around on here for a hour or two.

Enough power to be annoying, but not useful. ...

2025-04-05T06:50:59Z

In reply to nevent1q…5f86
_________________________

Enough power to be annoying, but not useful.

Nostr event nevent1qqsvg8lj6kxy2v2an7266krg0kvrn24xh2xq98dne66h3w5wv65a2ugzyp2ukpfkr69q6zt09c949uttgpl87u656mfde64c5ak9g8mskc8fuenjpmt

2025-04-05T06:42:38Z

In reply to nevent1q…da8u
_________________________

Brilliant, power outage. Just what I need. ...

2025-04-05T06:26:23Z

Brilliant, power outage. Just what I need.

Managed to get my firewall to fail over to 4G… but geez… ...

2025-03-28T08:55:37Z

In reply to nevent1q…d64w
_________________________

Managed to get my firewall to fail over to 4G… but geez…

Friday evening habit is to stream a 50s and 60s radio show… well… with my shitty line-of-sight to the Telstra towers, that just ain't happening.

Glitching left, right and centre… and no wonder… look at the ping time:

```
--- 8.8.8.8 ping statistics ---
50 packets transmitted, 43 received, 14% packet loss, time 49504ms
rtt min/avg/max/mdev = 161.328/762.185/1152.049/286.621 ms, pipe 2
```

…and down she goes again. Seriously, the HFC network around ...

2025-03-12T22:37:37Z

In reply to nevent1q…jpdj
_________________________

…and down she goes again.

Seriously, the HFC network around here really is terrible just now.

A Brunswick street bird's underpants stay up longer!

> A woman takes a selfie as she walks along flooded ...

2025-03-12T20:03:47Z

In reply to nevent1q…r7cp
_________________________

> A woman takes a selfie as she walks along flooded Edmondstone Street in the suburb of Newmarket.

Bloody hell, lots would see that and think "only in America", but that photo was taken a couple of suburbs over from where I am.

``` 2025-03-10T09:23:01Z>2025-03-10T10:41:01Z PADI=17 PADR=0 ...

2025-03-10T10:52:50Z

In reply to nevent1q…km6v
_________________________

```
2025-03-10T09:23:01Z>2025-03-10T10:41:01Z PADI=17 PADR=0 STATUS="no carrier">"active"
2025-03-10T10:41:01Z>2025-03-10T10:42:01Z PADI=18 PADR=0 STATUS="active">"no carrier"
```

NBNco, you taunt me so.

…and down it goes again… ``` ...

2025-03-10T09:26:08Z

In reply to nevent1q…4mwa
_________________________

…and down it goes again…

```
2025-03-10T09:14:01Z>2025-03-10T09:23:01Z PADI=49 PADR=0 STATUS="active">"no carrier"
```

``` 2025-02-17T15:54:01Z>2025-03-08T13:46:01Z PADI=73 PADR=0 ...

2025-03-10T08:55:29Z

In reply to nevent1q…v4tm
_________________________

```
2025-02-17T15:54:01Z>2025-03-08T13:46:01Z PADI=73 PADR=0 STATUS="active">"no carrier"
```

Now that I'm home… I can see exactly down to the second, when the Internet link last went down.

I might manually grab the emails from the VPS, pull those down, then feed them into the server here to see what people have been sending me.

Not certain, but it might have peaked. ...

2025-03-10T04:19:47Z

In reply to nevent1q…vnl0
_________________________

Not certain, but it *might* have peaked.

http://www.bom.gov.au/fwo/IDQ65389/IDQ65389.540604.plt.shtml

#CycloneAlfred

Alert is back a few pages now. ...

2025-03-10T02:10:56Z

In reply to nevent1q…vdac
_________________________

Alert is back a few pages now. https://www.theguardian.com/australia-news/live/2025/mar/10/australia-live-news-ex-tropical-cyclone-alfred-flood-evacuation-queensland-nsw-lismore-brisbane-wa-election-roger-cook-anthony-albanese-peter-dutton-ntwnfb?CMP=share_btn_url&page=with%3Ablock-67cdf9c18f08d2770c97ddd6#block-67cdf9c18f08d2770c97ddd6

Some tree carniage in our street already. Thankfully only a ...

2025-03-09T00:28:11Z

Some tree carniage in our street already. Thankfully only a little one. #CycloneAlfred

NBN outage confirmed, I just had this message in from Internode. ...

2025-03-08T23:43:31Z

NBN outage confirmed, I just had this message in from Internode. No doubt a legacy of #CycloneAlfred

*.longlandclan.id.au
*.vk4msl.id.au
and *.vk4msl.com are going to have major outages for a little while.

Getting closer now… #CycloneAlfred ...

2025-03-05T07:37:32Z

In reply to nevent1q…rrnx
_________________________

Getting closer now… #CycloneAlfred

The updated forecast map. #CycloneAlfred ...

2025-03-05T03:33:14Z

In reply to nevent1q…mu33
_________________________

The updated forecast map. #CycloneAlfred

Well hello #CycloneAlfred … we've been expecting you! ...

2025-03-05T01:54:39Z

In reply to nevent1q…rnew
_________________________

Well hello #CycloneAlfred … we've been expecting you!

Another forecast… as #CycloneAlfred wobbles itself to the ...

2025-03-05T00:49:38Z

In reply to nevent1q…m59w
_________________________

Another forecast… as #CycloneAlfred wobbles itself to the coast.

For those in the Northern Rivers NSW region (and those living ...

2025-03-04T23:15:51Z

In reply to nevent1q…u63j
_________________________

For those in the Northern Rivers NSW region (and those living close to the Queensland border… maybe one of these is closer to you):

The following SES evacuation centres will open as of 4pm this afternoon, according to Chris Minns.

Terranorra Public School — 650 Terranora Rd, Terranora
Tyalgum Hall — Coolman St, Tyalgum
Mullumbimby RSL — 58 Dalley St, Mullumbimby
Kingscliff TAFE — 806 Cudgen Road, Kingscliff
Murwillumbah TAFE — 146 Murwillumbah Street, Murwillumbah
Ocean Shores Country Club —113A Orana Road, Ocean Shores
Coraki Pubic School — 48-52 Adams Street, Coraki
Kyogle Memorial Hall — Stratheden Street, Kyogle
Southern Cross University — Military Road, East Lismore
Evans Head RSL — 11-13 McDonald Place, Evans Head
Burringar Sports Club — Fourth Ave, Burringbar
Uki Hall — 1462 Kyogle Rd, Uki

https://www.abc.net.au/news/2025-03-05/tropical-cyclone-alfred-brisbane-forecast-queensland-nsw-live/105010498?utm_campaign=abc_news_web&utm_content=link&utm_medium=content_shared&utm_source=abc_news_web#live-blog-post-156886

#CycloneAlfred

Forecast updates are coming through thick and fast … and of ...

2025-03-04T23:06:26Z

In reply to nevent1q…lzwg
_________________________

Forecast updates are coming through thick and fast … and of course it has to happen while my server is busy running the JavaScript build tools that build Mastodon's UI.

#CycloneAlfred has a definite kink in the predicted path as it crosses the coast.

Seriously Alfred, go home, you're drunk!