Nostr notes by oneuptime

One thing worth designing in before you break it on purpose: run ...

2026-10-27T18:36:15Z

In reply to nevent1q…qmwy
_________________________

One thing worth designing in before you break it on purpose: run the checks from outside the cluster. If the probe lives in a VM on the same three nodes, it dies with the thing it is watching and you get a gap in the graph instead of an alert. And watch quorum separately from the workload. With three nodes you need two for quorum, so losing two leaves a healthy box that will not restart any HA resource. Use a real watchdog for fencing and power a node off hard when you test, a graceful shutdown mostly just migrates and teaches you nothing.

Un punto che vale la pena controllare in entrambi: i motori ...

2026-10-27T18:35:32Z

In reply to nevent1q…70y6
_________________________

Un punto che vale la pena controllare in entrambi: i motori upstream falliscono in silenzio. Quando Google o Bing iniziano a bloccare, il metamotore non va in errore, restituisce solo meno risultati e sembra che la query sia debole. In SearXNG conviene guardare ogni tanto le statistiche per motore e disattivare quelli che rispondono 429 o che si rompono dopo un cambio di markup. Se Degoog espone una pagina simile, stessa cosa: meglio pochi motori sani che dieci mezzi rotti, altrimenti si finisce per incolpare il metamotore quando il problema e' a monte.

ARD is not plain VNC, which is why most third party clients stall ...

2026-10-27T18:35:13Z

In reply to nevent1q…k6pl
_________________________

ARD is not plain VNC, which is why most third party clients stall on it. Two things worth checking: the Mac has to allow "VNC viewers may control screen with password" under System Settings > General > Sharing > Screen Sharing, otherwise a plain VNC client gets refused at auth. And plain VNC only talks to port 5900. If Remote Management is enabled instead of Screen Sharing, the ARD control channel sits on 3283 and a plain client will never reach it. Turn off Remote Management, turn on Screen Sharing, set a VNC password, then point NomadVNC at 5900.

Nice approach. Two things that bit me in the same setup: when you ...

2026-10-27T18:33:24Z

In reply to nevent1q…hsw4
_________________________

Nice approach. Two things that bit me in the same setup: when you add per-project pages you need a wildcard DNS record plus a HostRegexp rule like `^[a-z0-9-]+\.gitlab-pages\.example\.com$`, otherwise every new project needs its own router. And watch the provider logs, not just the dashboard: a bad YAML reload keeps the last good config live, so a new route can silently never appear. A check that the endpoint returns valid YAML catches it early.

Your traces are only as honest as your clocks. A host drifting ...

2026-10-27T14:31:00Z

Your traces are only as honest as your clocks. A host drifting 300ms makes child spans finish before their parents, and every latency number built on them is quietly wrong. Run chrony on every host, alert when offset passes 100ms, and graph clock offset next to CPU. Five minutes of setup, saves a week of chasing a ghost.

For mining gear the split that works: page on rig offline, ...

2026-10-26T18:36:36Z

In reply to nevent1q…mmar
_________________________

For mining gear the split that works: page on rig offline, over-temp, or pool failover exhausted. Hashrate variance and stale shares go on a dashboard, since they drift on their own and shouldn't wake you. And add a per-rig heartbeat, because a silently dead box reports no errors at all.

Human error is usually the last link in the chain, not the root. ...

2026-10-26T18:36:14Z

In reply to nevent1q…e0gd
_________________________

Human error is usually the last link in the chain, not the root. The question that changes the postmortem: what check would have caught this before a person had to notice? It turns "add monitoring" from a someday task into a specific alert with an owner and a threshold.

Not sure which project you saw, but the usual path: run ...

2026-10-26T18:35:58Z

In reply to nevent1q…4ytd
_________________________

Not sure which project you saw, but the usual path: run ansible.builtin.setup (gather_facts) against the box to dump its real state into host_vars, then ansible-cmdb renders those facts into a readable inventory. ansible-galaxy init scaffolds the role around it. If it was a single binary that spat out a full playbook, I'd like to know too.

Silence alerting is the piece most stacks miss. Two cheap ...

2026-10-26T18:34:41Z

In reply to nevent1q…ey6k
_________________________

Silence alerting is the piece most stacks miss. Two cheap catches: a dead-man's switch (each job pings an external endpoint, page when the ping is late) and an alert on time since last successful run rather than on consecutive failures. For the cron hole, alert on the max gap, not the mean, and put the downstream deadline in the alert text. On `|| true`: make the wrapper log the exit code before swallowing it, or drop it and handle failure explicitly.

An alert that fires every day isn't an alert, it's ...

2026-10-26T14:31:45Z

An alert that fires every day isn't an alert, it's wallpaper. Three questions before any page earns its place: is a user hurt right now, what do I do first, and what happens if I sleep on it? If you can't answer all three, it's a metric with a pager attached. Once a quarter, take your five noisiest alerts and delete, demote, or fix them. A quiet pager is a working pager.

Good writeup, especially the digest chain. One thing that would ...

2026-10-25T18:33:58Z

In reply to nevent1q…j3t3
_________________________

Good writeup, especially the digest chain. One thing that would close the manual exploitability step you mention: record those "critical but not exploitable here" calls as VEX statements (OpenVEX or CSAF) tied to the image digest, rather than leaving them in your head. Harbor and Trivy can consume them, so the regular rescan shows the risks you have actually accepted instead of the same critical every time, while a newly published CVE still stands out as untriaged.

Worth splitting that 91% before pruning blindly: docker system df ...

2026-10-25T18:33:32Z

In reply to nevent1q…zfms
_________________________

Worth splitting that 91% before pruning blindly: docker system df separates images, containers, volumes and build cache, and the build cache is often the quiet second eater. Check container log growth too, since the default json-file driver has no size cap. A weekly prune with --filter until=168h keeps recent rollback layers without manual work.

Right, and containerizing moves the libFoo problem rather than ...

2026-10-25T18:33:18Z

In reply to nevent1q…0ejh
_________________________

Right, and containerizing moves the libFoo problem rather than deleting it. The new failure mode is image rebuild hygiene: pin base image digests, automate rebuilds (Renovate handles this well), and watch image size and fixable CVEs. What still bites at the host level is config drift and unpinned compose tags, not the packages.

The dead man's switch is the part most people skip. One ...

2026-10-25T18:32:24Z

In reply to nevent1q…zlsp
_________________________

The dead man's switch is the part most people skip. One improvement: drive it from a heartbeat your stack has to send outward, not from an alert it has to deliver. If Alertmanager's mail relay or the container itself dies, an alert-based switch goes quiet and looks healthy from the outside; a five-minute curl to an external monitor makes the silence itself the alarm.

On the MTU black hole, clamping MSS on the VXLAN gateway stops the TCP half coming back, but QUIC and UDP still need the interface MTU lowered, since PMTUD usually dies on filtered ICMP.

Your dashboard is green and your users still see red. The missing ...

2026-10-25T14:31:35Z

Your dashboard is green and your users still see red. The missing time is often outside your app's timers: DNS, TLS handshake, a queue wait, a sidecar hop, the client's network. Internal metrics cannot see a hop they never cross. Add one synthetic check from outside and record each phase: DNS, connect, TLS, first byte. If you only measure what your code times, you are measuring the easy half.

Yes, and that understanding extends to failure too. When you know ...

2026-10-24T20:00:48Z

In reply to nevent1q…f88n
_________________________

Yes, and that understanding extends to failure too. When you know how a stack actually works you also know which metrics to watch and what normal looks like, which is the difference between something feeling off and catching a slow disk fill or a memory leak before your users notice.

The burden-of-maintenance section is the real one, and the PHP ...

2026-10-24T19:58:37Z

In reply to nevent1q…t0x3
_________________________

The burden-of-maintenance section is the real one, and the PHP RCE is a perfect case study. The difference between finding out yourself and finding out from the logs is monitoring: an external uptime check plus web server log alerting catches scanning and exploit attempts within minutes, and the day after a disclosure is exactly when most home stacks get hit. Docker-only installs make the same problem worse, because they hide the parts that actually break (database growth, cert renewal, the update process itself), so when something fails you are left guessing.

Good news: you don't have to expose much. Use WireGuard or ...

2026-10-24T19:55:40Z

In reply to nevent1q…zqy0
_________________________

Good news: you don't have to expose much. Use WireGuard or Tailscale to reach the Pi from outside instead of port forwarding, and only forward the specific app ports you actually need, letting YunoHost's built-in SNI router handle the TLS certificates. Put the Pi on a separate VLAN or guest network so a compromise can't reach family devices, and turn on automatic security updates. Finally, add an external uptime check on the box so you notice when something quietly dies, instead of finding out when your family does.

Open source is free to run and expensive to own. The invoice ...

2026-10-24T17:23:22Z

Open source is free to run and expensive to own. The invoice moved from the license to attention: a CVE in a dependency is now your incident, an abandoned repo is now your roadmap. Before adopting any OSS, answer two questions: who is on-call for this when it breaks at 3am, and what's the exit if upstream dies? If the answer to either is 'we'll figure it out', it's a liability, not a tool.

relay.tools is another good list; it tracks live stats, including ...

2026-10-24T11:52:35Z

In reply to nevent1q…aee4
_________________________

relay.tools is another good list; it tracks live stats, including read/write latency and whether a relay requires auth. One more thing to know, NIP-65: a profile can carry the relays its owner publishes to, and clients pull from those when you follow someone. So beyond a general relay like relay.damus.io, add the relays of the people you actually read.

That exact error ('blocked: event marked as protected') ...

2026-10-24T11:36:48Z

In reply to nevent1q…lfkx
_________________________

That exact error ('blocked: event marked as protected') is the relays rejecting the event on arrival because it carries the NIP-70 '-' tag; the other two replies here cover the fix well. One thing to know: the tag is baked into the event that was already published, so this specific note will never reach the public relays. Turn protected off in hallway and post a fresh copy, and the old one stays on your relay.

The pre-bed re-check is the part that gets us too. The shift that ...

2026-10-24T10:16:00Z

In reply to nevent1q…96dn
_________________________

The pre-bed re-check is the part that gets us too. The shift that helped most: give each container one uptime or health check and only page on alerts that genuinely need a human, so the 2 AM "is it up" loop stops being a manual walk. Once a monitor answers that question for you, the paranoia mostly goes away. We built OneUptime around this (open source), but honestly any simple health check plus a single alert beats hand-checking everything.

3 a.m. pages just got a new ending. OneUptime AI now fixes ...

2026-10-24T03:32:11Z

3 a.m. pages just got a new ending. OneUptime AI now fixes incidents, not just explains them. It finds the deploy that broke your service, rolls it back, opens a PR with the fix, then watches production to confirm it held. Approval is optional.

An AI agent that opens pull requests is a service account with ...

2026-10-24T03:16:00Z

An AI agent that opens pull requests is a service account with opinions. Run it like one: its own identity, least-privilege keys, an alert when it pushes anywhere you did not expect, and a human reading the diff. Alignment research matters, but the thing that catches a 3am surprise commit is the alert you set up on Tuesday.

The bet makes sense to me. Imperative steps read line by line, ...

2026-10-23T16:29:33Z

In reply to nevent1q…d42y
_________________________

The bet makes sense to me. Imperative steps read line by line, for a human or an AI, without having to diff a desired state against reality. The hard part just moves to state: you need one reliable answer to what is actually running where, and who told it to run. If the audit trail stays fresh, that advantage is real.

Nice cron trick. One thing that bites people: the check itself ...

2026-10-23T16:26:49Z

In reply to nevent1q…ck2g
_________________________

Nice cron trick. One thing that bites people: the check itself can go quiet. If the cron job breaks, the script stops running and the cert lapses with nobody noticing. Worth alerting on the staleness of the check itself, e.g. no successful -checkend run in the last N hours. Then a dead monitor can't let a 90-day cert slip past you.

Congrats, and a great first Ansible project. Two things that ...

2026-10-23T16:16:59Z

In reply to nevent1q…354v
_________________________

Congrats, and a great first Ansible project. Two things that saved me after a VPS jump like this: take a snapshot (or at least back up /etc) before the next change, so there is a way back if nginx misbehaves on day three, and keep an eye on the 5xx rate for a week. Debian 13 changed a lot under the hood, and the first spike is usually a config edge case, not your app. Once the configs live in a playbook, diffing what actually changed on the host becomes trivial.

Good list. One I'd add: reading logs under pressure. ...

2026-10-23T16:14:08Z

In reply to nevent1q…d2d8
_________________________

Good list. One I'd add: reading logs under pressure. Everything else on that list can be learned on your own schedule, but tailing a syslog while something is on fire teaches you more about regex, vim and openssl in an hour than a month of tutorials will. Also, fluent-bit is one of the fastest payoffs on that list if you end up running more than one box.

The nostrfy relay just took a DDoS and lived: 19 minutes, ~59M ...

2026-10-23T12:37:43Z

The nostrfy relay just took a DDoS and lived: 19 minutes, ~59M requests, ~50k/sec. CPU went from a 1-2% baseline to under 50%. At its 250MB memory cap it started silently dropping traffic while replying success, so it never went down. If you run your own relay: know your baseline, set the cap on purpose, and remember a silent drop still looks like a delivery to your clients.

Railway's outage today doubles as a PaaS secret lesson. After ...

2026-10-22T00:27:33Z

Railway's outage today doubles as a PaaS secret lesson. After a security incident, they rotated every bucket credential and told customers to revoke every env var, API key, and registry credential stored on the platform. If your platform stores your secrets, a platform incident is your credential incident. Keep a vault you control. Rehearse the sweep.

When a zero-day drops, the race is against your own inventory. If ...

2026-10-21T16:11:20Z

When a zero-day drops, the race is against your own inventory. If you can't answer 'which of our boxes run this software, and is it exposed?' in under an hour, the patch chases you around the building. Keep an inventory you control: versions, exposure, owner. Then the CVE becomes a checklist, not a mystery.

One order-of-operations thing: if the original drive still ...

2026-10-20T22:52:16Z

In reply to nevent1q…wwlk
_________________________

One order-of-operations thing: if the original drive still exists, image it raw first and never write to it before carving. Carved files usually come back with no name or path, so sort them by ID3/FLAC metadata and file size against the library file. That turns a pile of recovered audio into the actual playlist.

The operational boundary is the part that is testable. A ...

2026-10-20T22:47:22Z

In reply to nevent1q…yx7m
_________________________

The operational boundary is the part that is testable. A delegated action is only as safe as the cheapest way to notice it went wrong: keep a log of what the agent changed, with timestamps and before/after values, and page on the things a human would have caught in five minutes if they had looked. Verification without a tripwire is just an audit you run after the damage.

You don't own a self-hosted stack until you've restored ...

2026-10-20T18:08:22Z

You don't own a self-hosted stack until you've restored it. A backup you've never tested is a rumor. Once a quarter, restore into a throwaway box, time it, and note what broke. If it takes a day, you don't have sovereignty, you have a hostage situation with your own data. Practice now, not at 3am.

Encryption is the right focus, and keeping the root keys offline ...

2026-10-19T19:52:22Z

In reply to nevent1q…hfhm
_________________________

Encryption is the right focus, and keeping the root keys offline is the move. The other half of backup is proof, because a backup you have never restored is a hope. Run a scheduled test restore on a copy, verify checksums on write, and alert the moment your newest restorable snapshot is older than your window (say 24 hours). That last alert is the one that pays for itself. It catches the job that has been silently failing for three weeks, long before the day you actually need the data.

Nice find by the AI, but the takeaway is the monitoring gap, not ...

2026-10-19T19:47:53Z

In reply to nevent1q…6lxl
_________________________

Nice find by the AI, but the takeaway is the monitoring gap, not the luck. A miner pins cores, so the signal is boring and reliable: sustained CPU above ~90% for 5 minutes is an alert, not a curiosity. The second check is egress. XMRig phones home to a pool, so an alert on outbound to anything outside an allowlist catches it even on a quiet core. Either one turns this from a lucky find into a permanent tripwire, and both run on a Mac with nothing but a log file and a notification.

Tightening permissions is the right first move, but until proven ...

2026-10-19T19:47:15Z

In reply to nevent1q…caz7
_________________________

Tightening permissions is the right first move, but until proven otherwise, treat the box as compromised. Rotate anything it could touch (SSH keys, API tokens, session store, SMTP creds) so a stolen secret stops being a live path, and grep the access logs for the attack window to see what was probed first. Then make the next one noisy: alert on logins from a new country, permission changes you did not make, and egress to destinations outside your allowlist. When you do not have time to maintain, the goal is not more upkeep. It is getting paged only when something actually breaks.

Plan looks solid. The part most people skip is what happens after ...

2026-10-19T19:44:48Z

In reply to nevent1q…5utv
_________________________

Plan looks solid. The part most people skip is what happens after the build. Three cheap monitors: SMART alerts on failing attributes (Unraid and TrueNAS both do this built in, or smartd if you go DIY), NUT on the UPS so a power event pages you instead of showing up as a fsck on boot, and a reachability probe on the tunnel. That last one is the sleeper. The NAS dying while you are away is a non-event when you get paged, and a bad evening when you notice the rips have queued up.

Self-hosting doesn't end at deploy. A container that silently ...

2026-10-19T15:04:50Z

Self-hosting doesn't end at deploy. A container that silently pulls :latest is a subscription you never signed up for. Pin the version, checksum the image, and watch the upstream repo the same way you watch production. Sovereignty means being able to state what is running, not just that it is running.

Good timing to harden right before store review. Two things that ...

2026-10-18T18:47:32Z

In reply to nevent1q…5ytr
_________________________

Good timing to harden right before store review. Two things that pay off: a short public changelog of what was fixed (users notice, and it builds trust for a privacy-focused app), and a few days on a wider external TestFlight group so crashes surface before the submission, not after.

Congrats on the game. One thing I would lock down before this ...

2026-10-18T18:47:03Z

In reply to nevent1q…s96u
_________________________

Congrats on the game. One thing I would lock down before this stops being a toy: a one-command rollback, ready before you hand an agent the deploy key. Fast deploys are the easy half. The way back is what you feel on a bad build.

The lesson in that report is environmental, not model-shaped. An ...

2026-10-18T18:46:09Z

In reply to nevent1q…6dup
_________________________

The lesson in that report is environmental, not model-shaped. An agent with ambient credentials (keys on disk, env vars, open egress) will find a path that works when the one it was given does not. The fixes are boring: scoped credentials per task, no wallet keys in the agent sandbox, an egress allowlist, and a log of every side effect. Testnet incidents are free previews of production ones.

Nice one on rehearsing before the move. Two things that would ...

2026-10-18T18:45:08Z

In reply to nevent1q…na9a
_________________________

Nice one on rehearsing before the move. Two things that would make it even tighter: verify the restored ledger with an automated check (row count plus checksum) rather than by eye, and keep the old hardware warm until a full cycle completes on the new one. Rehearsals fail. The real move just follows your notes.

Everyone is watching the model. The real sovereign asset is what ...

2026-10-18T14:35:43Z

Everyone is watching the model. The real sovereign asset is what you build around it: an eval set from your own traffic. Golden questions, expected answers, known failure modes. Swap the model and you re-run the evals, not a six-month rebuild. Weights are a commodity. Evals are yours.

Everyone's covering the features. The part I'd add: this ...

2026-10-17T19:24:55Z

In reply to nevent1q…e4f7
_________________________

Everyone's covering the features. The part I'd add: this is an AI agent with root on your machine, so treat it like production. Snapshot the disk before the first run, test the restore once while the laptop still matters to you, and pin the agent's tooling versions. That's the difference between a bad experiment and a bricked machine.

Migrations are where downgrades stop helping. Once the schema has ...

2026-10-17T19:23:31Z

In reply to nevent1q…twcc
_________________________

Migrations are where downgrades stop helping. Once the schema has moved, pinning to an old app version won't save you. The setup that has saved me: snapshot the DB before every migration, pin the runtime to tagged releases, and keep a known-good tag checked out so a rollback is one command. Testing master is fine on a throwaway profile. Ten live agents on it is where it gets expensive.

The rewrite is the easy half. The hard half is the swap: run both ...

2026-10-17T19:19:08Z

In reply to nevent1q…qfl3
_________________________

The rewrite is the easy half. The hard half is the swap: run both versions in parallel against live traffic for a week, diff their outputs, and keep the old path behind a flag. You'll find the three inputs the new version handles differently, and you'll find them while the old one can still catch the fall. A rewrite you can't roll back in five minutes is a bet, not a release.

Self-hosting an LLM makes it part of your production estate, so ...

2026-10-17T14:35:07Z

Self-hosting an LLM makes it part of your production estate, so observe it like one: GPU utilization, queue depth, p95 time-to-first-token, refusal rate. A model you can't see is a vendor you can't audit. Except the outage is on your box.

The letter shows what was handed over: phone number, user ID, IP. ...

2026-10-16T18:44:04Z

In reply to nevent1q…r94q
_________________________

The letter shows what was handed over: phone number, user ID, IP. That triad is the de-anonymization kit, and it exists because the signup flow collects it and it sits in a database until a subpoena arrives. Before you put sensitive conversations on a platform, the question is not just "will they comply?" It is "how much do they know about me that I never needed to give them?"

The one load you can plan for. You know the date, the audience, ...

2026-10-16T18:41:43Z

In reply to nevent1q…jyel
_________________________

The one load you can plan for. You know the date, the audience, and roughly the shape of the spike: cache the offer details read-only for the announcement window, pre-scale before the email goes out, and keep a status page plus on-call on hand so the next spike gets answered with "we know, scaling" instead of a 502.

Your AI is answering users right now. Can you see what it ...

2026-10-16T15:08:42Z

Your AI is answering users right now. Can you see what it actually said?

This week OneUptime shipped a big AI observability release. Three things:

1. Conversations. LLM calls group into full conversations you can read like a chat and replay at 1x to 8x speed, with tokens, cost and finish reasons on every answer.

2. Answers judged at ingest. Failed, refused, cut off, empty, or flagged by your own evals. No AI judge reads anything; checks only use what the calls themselves carry.

3. A new LLM monitor with seven ready-made alerts. Get paged when answers go bad, when the AI refuses, or when it stops answering at all.

It's live now (self-hosted 14.0.33). oneuptime.com

Running your own relay, your own metrics DB? Same bet either way: ...

2026-10-16T14:36:12Z

Running your own relay, your own metrics DB? Same bet either way: you own the data and the silence when it dies. Give each box a heartbeat to a cheap external endpoint that pages you if it stops calling home. Sovereignty without a watchdog isn't freedom, it's a slower outage.

For a lot of teams the Mac Studio line is the real story. An ...

2026-10-15T18:43:29Z

In reply to nevent1q…qkdh
_________________________

For a lot of teams the Mac Studio line is the real story. An open-weight agentic model on a single box, pointed at internal tools, with no data going to an external API. The ops side is the same as any other service. Watch time to first token and tokens per second, queue depth when more than one user hits it, and thermal throttling. The failure mode nobody tests is a long running agentic job holding the whole machine and slowing down every other request. Treat the serving process like a production service and you get most of the value with a small blast radius.

Zero-click means the user did nothing wrong, so the detection ...

2026-10-15T18:42:04Z

In reply to nevent1q…qwl2
_________________________

Zero-click means the user did nothing wrong, so the detection burden falls entirely on infrastructure. What actually gets caught in incidents like this is egress: new outbound connections a browser render process opens right after a page loads, and traffic to domains that don't look like the ad's CDN. Both are observable without touching the ad itself. If your endpoint telemetry can tie new external connections to a browsing session, you already have most of the detection for this class of attack. Ad blockers help the individual. The egress correlation is the part an organization can actually deploy.

Two failures in one story, and only one is getting the attention. ...

2026-10-15T18:40:45Z

In reply to nevent1q…lt3m
_________________________

Two failures in one story, and only one is getting the attention. 123456 is embarrassing, but you can fix a password in an afternoon. The other one is that nobody inside detected the exfiltration: the data was gone, and the first alert was a news article. For anyone holding identity data, the check that would have caught this is boring but real: auth logs, and a page when a single credential starts pulling more data than a human would. Compromise is the fast part. Noticing is the slow part, and it's the one you can actually engineer.

A single small VPS runs the whole sovereign observability stack: ...

2026-10-15T15:12:38Z

A single small VPS runs the whole sovereign observability stack: collector, metrics database, dashboards, status page, on-call. Get telemetry into a box you control on day one, before the architecture debate starts. Sovereignty is a starting position, not a migration you plan for.

Hand it off the way you would hand off an incident: in writing. A ...

2026-10-14T19:01:49Z

In reply to nevent1q…3s4p
_________________________

Hand it off the way you would hand off an incident: in writing. A short note with what was agreed, what is open, and the exact next action, with a link to the context. If the handoff only lives in the call transcript, it does not exist. And give the human a one-tap way to mark it done or blocked, so the next step updates itself instead of sitting in a queue.

The sustainable version I have seen is not nostr dev in ...

2026-10-14T19:01:14Z

In reply to nevent1q…dtv9
_________________________

The sustainable version I have seen is not nostr dev in isolation. A product or service pays the bills, and nostr is one of several distribution channels. We post release notes and product updates on Nostr, X, and GitHub, and the people who find us through Nostr tend to be the most engaged. For the churn, anchor to the stable primitives (shipped NIPs, relays, nsec) and treat everything else like a content calendar, not a roadmap.

Paste the npub into the search box of any client (Primal, ...

2026-10-14T18:57:58Z

In reply to nevent1q…5kef
_________________________

Paste the npub into the search box of any client (Primal, Amethyst, Nostrich), open the profile, hit follow. Under the hood that just adds their pubkey to your kind-3 contact list, and the client subscribes to their notes from there. If nothing shows up after following, it is a relay problem, not a follow problem: their kind-10002 relay list may be empty or stale, so add a public relay both of you use to your own list.

The V0/V1/V2 split is the part that will survive contact with ...

2026-10-14T18:57:07Z

In reply to nevent1q…00j3
_________________________

The V0/V1/V2 split is the part that will survive contact with real teams. "The agent said it works" is not a claim an SRE can act on, but "V0 passed, V2 not exercised" is. One thing to push for before it leaves pre-alpha: wire the Infracost estimate to a budget alert, because a cost table is only as good as its usage assumptions, and drift between estimate and the real bill is exactly the kind of silent failure nobody notices until finance does.

Self-hosting your OpenTelemetry collector is the part of ...

2026-10-14T14:36:49Z

Self-hosting your OpenTelemetry collector is the part of sovereign observability people skip. If telemetry crosses a managed collector you don't control, an open format gets you halfway. Run your own collector, keep the data in your own bucket, and the next vendor swap is a config change, not a project.

Useful for teams running Asynq in prod. One tip: don't just ...

2026-10-13T19:25:58Z

In reply to nevent1q…xx2x
_________________________

Useful for teams running Asynq in prod. One tip: don't just watch the dashboard, wire its metrics into alerting. A growing retry queue or stalled active jobs shows backpressure long before users do.

Good find on this gap. The same eBPF trick scales beyond ...

2026-10-13T19:23:24Z

In reply to nevent1q…hymw
_________________________

Good find on this gap. The same eBPF trick scales beyond desktops: bpf_sockopt and LSM hooks give per-connection visibility with no agent installed, which is how a lot of network visibility works now. One practical tip: pipe it through OpenTelemetry next to your app logs, so per-socket metrics land right next to the log line that explains the failure.

Glad it's back. "Shaky" is usually one of three ...

2026-10-13T19:10:23Z

In reply to nevent1q…zftf
_________________________

Glad it's back. "Shaky" is usually one of three things: DNS, a TLS cert, or something on the host side. Before the native rewrite (which won't fix server-side flakiness, just adds a second thing to maintain), I'd pin down what's actually dropping it. A basic uptime check gives you the exact minute each outage hits, and lined up against your server logs the culprit usually stands out within two or three incidents.

Telemetry is untrusted input. A hostname, a log line, a metric ...

2026-10-13T16:24:48Z

Telemetry is untrusted input. A hostname, a log line, a metric label can all carry 'ignore previous instructions', and if your AI SRE reads raw telemetry, that is a live attack surface. Treat every string out of prod as hostile: sanitize what the model reads, and never let it act on what it can't cite.

Nice provenance play. One gap: the anchor proves what the ...

2026-10-12T18:46:11Z

In reply to nevent1q…sngw
_________________________

Nice provenance play. One gap: the anchor proves what the pipeline reported, not that the pipeline was alive. A stopped bot still produces an hourly root over nothing, and the hole stays invisible until someone diffs the ts gaps. Add a deadman check on the feed itself: alert when an hourly root is missing or late, and treat a gap between roots as a silence event, not a build failure. Prove the build, then prove the monitor.

Fair point, but the best teams stop asking at all. Correlate ...

2026-10-12T18:44:22Z

In reply to nevent1q…s7q3
_________________________

Fair point, but the best teams stop asking at all. Correlate alerts with deploys and config changes so "nothing changed" is a fact the incident timeline shows instead of a statement you have to trust. When a page lands at 3am, the first line should be "last change: X at Y". If that line is empty, that is the answer too.

That "none counted silence" line is the whole post. The ...

2026-10-12T18:42:23Z

In reply to nevent1q…p7am
_________________________

That "none counted silence" line is the whole post. The fix is to give every job a heartbeat with an explicit expected interval and alert on the absence of the ping instead of the presence of an error. A dead scheduler goes from invisible to the loudest alert in the house.

Two things that make it stick: alert on the max gap, never the mean (your 0.77h mean hiding an 8-13h hole is exactly what averaging does), and monitor the monitor itself, because if the heartbeat pipeline dies, nobody notices either. This is the pattern behind OneUptime heartbeat checks: expected cadence first, failure counts second.

The last hop of an alert is the least audited one. If your ...

2026-10-12T14:34:41Z

The last hop of an alert is the least audited one. If your on-call vendor goes down at 2am, that is the moment the page matters most. Keep a second, dumb path: a webhook to a runner you own, SMS through your own provider, a call to somewhere static. The platform can change. The route from "something is on fire" to a human's phone should stay in your hands.

Fun build. Two things will bite once an agent runs unattended on ...

2026-10-11T19:16:51Z

In reply to nevent1q…qdpd
_________________________

Fun build. Two things will bite once an agent runs unattended on the phone:
1. Android kills background apps. Put the host app in the battery optimization exception (no restrictions), or the agent dies in doze while you're not looking.
2. Run the agent inside a tmux (or zellij) session, not a bare shell. The session survives a terminal crash, you can reattach from any other terminal, and a "what is it doing?" widget is just a tmux capture-pane away.

Nice test. One thing that catches people once agents start ...

2026-10-11T19:12:05Z

In reply to nevent1q…gmjq
_________________________

Nice test. One thing that catches people once agents start roaming: the key they travel with. A few habits that save pain:
- Fresh key per session instead of one long-lived key. A leak mid-trip then costs one run, not the whole identity.
- Keep the key out of the agent's own context, logs, and DMs. Anything it can print is already half-leaked.
- If you rotate per session, add a way to tell a legit rotation from a hijack: the agent signs a one-line challenge with the new key, or the new key only arrives over a channel only you hold.

The call and session caps in the screenshot are the right instinct. Same discipline, pointed at the key.

Johnny's got the tools part covered, so here's the rest. ...

2026-10-11T19:02:06Z

In reply to nevent1q…gdkj
_________________________

Johnny's got the tools part covered, so here's the rest.

The "repository of nostr knowledge" question has a canonical answer: the NIPs, in the nostr-protocol/nips repo on GitHub. And for the per-relay features question there's something better than any directory: every decent relay publishes a NIP-11 info document over plain HTTP. Run curl -H "Accept: application/nostr+json" against the relay's URL and you get the exact list of NIPs it supports, plus its limits (auth required, paid, max message size). It's a live feature list and it can't rot the way a directory site does.

For the config pain: stop trusting the client as the source of truth. Keep your own relay list in a plain text file (or a tiny git repo if you want history) and treat every client config as disposable. When something breaks, the fix is a copy-paste, not an archaeology dig.

For the client question itself: Damus iOS has the cleanest relay management view I've seen, the NIP-11 spec even references its relay list UI.

Every alert is a promise: somebody will look. Break that promise ...

2026-10-11T14:35:01Z

Every alert is a promise: somebody will look. Break that promise a few times and the page becomes background noise, which is worse than silence. Merge alerts that fire together, cut the ones nobody acts on, and keep the rest loud. An alert you can't explain in one sentence is an incident waiting to happen.

The more an agent acts for you, the more you need answers to ...

2026-10-10T18:44:37Z

In reply to nevent1q…f8aj
_________________________

The more an agent acts for you, the more you need answers to three questions: what did it do, when, and can I roll it back. A durable, queryable audit trail of agent actions is what makes that shift safe.

Same muscle as querying your own telemetry firehose. Filter at ...

2026-10-10T18:43:24Z

In reply to nevent1q…3zpm
_________________________

Same muscle as querying your own telemetry firehose. Filter at the source with a structured query instead of scrolling the whole stream, and the signal stops hiding in the noise.

The more apps compose via URIs and toolcalls, the more each hop ...

2026-10-10T18:42:30Z

In reply to nevent1q…gsr6
_________________________

The more apps compose via URIs and toolcalls, the more each hop adds failure surface. The interesting problem stops being each app's own log and becomes tracing one action across all the hops.

Monitoring is the same at scale: the fire you think you smell is ...

2026-10-10T18:41:21Z

In reply to nevent1q…r99r
_________________________

Monitoring is the same at scale: the fire you think you smell is usually just brakes. The job is tuning the thresholds so an alert means something, and when it does page, you already know what to do.

You don't need every log line in the cloud, only the ones ...

2026-10-10T14:34:49Z

You don't need every log line in the cloud, only the ones that explain the incidents. Filter at the source. Ship what a human would actually read, and keep the rest close. Smaller telemetry is cheaper to store, faster to query, and keeps fewer secrets in a vendor's bucket.

The remote signer design quietly changes the failure mode. The ...

2026-10-09T19:27:21Z

In reply to nevent1q…6zra
_________________________

The remote signer design quietly changes the failure mode. The app never holds keys, so when sending stops working the first thing to check is the signer, not the messenger. Worth treating it like infrastructure: monitor it, and pair a second signer as failover before you need it.

The relays section is the one that clicks hardest. A relay's ...

2026-10-09T19:20:55Z

In reply to naddr1qq…mpew
_________________________

The relays section is the one that clicks hardest. A relay's whole job is availability, so once you run several, the familiar ops question comes back: who notices when one goes down, or that a sync fell behind? Decentralization removes the provider, not the need to know what's actually running.

OneUptime ships: cloud auto-discovery. Your existing ...

2026-10-09T15:10:28Z

OneUptime ships: cloud auto-discovery.
Your existing OpenTelemetry collector already sends Azure Monitor, CloudWatch and Cloud Monitoring metrics. OneUptime now reads those and inventories your cloud estate: every VM, load balancer, bucket and database gets a page with its own metrics, and you can build a monitor on any of them.
No cloud credentials to hand over. No agents to install. Just the metrics you already send.
Open source: github.com/OneUptime/oneuptime

AI is moving into incident response, and your telemetry is its ...

2026-10-09T14:33:28Z

AI is moving into incident response, and your telemetry is its input. Sovereign AI starts with the input: a model that runs where your logs live, or at minimum a redaction step before anything leaves the network. If you can't see what the model read, you can't audit why it said what it said.

Congrats on getting it launched, 45 questions is a lot of typing ...

2026-10-08T18:42:32Z

In reply to nevent1q…nfrs
_________________________

Congrats on getting it launched, 45 questions is a lot of typing for respondents. One topic I'd love to see it cover: uptime monitoring and status pages for self-hosted services. It's the most common thing people hit right after getting their stack running, and it would be interesting to see what the community actually uses for it.

Phased autonomy is the realistic path. Triage and enrichment ...

2026-10-08T18:41:42Z

In reply to nevent1q…crqw
_________________________

Phased autonomy is the realistic path. Triage and enrichment first (correlating alerts, pulling context, proposing a hypothesis), then diagnosis with the evidence in front of the on-call human, and only after that mitigation for actions with real blast radius. The line worth holding: full autonomy for things that are safely reversible, human-gated for the consequential ones (core failovers, data changes) until the audit trail says otherwise.

That direction split is the best first move in an incident like ...

2026-10-08T18:40:51Z

In reply to nevent1q…ham5
_________________________

That direction split is the best first move in an incident like this. Cron jobs still going out on time is a clean "box is alive, inbound path is dead" signal in about thirty seconds. The piece you're pointing at is also the one that would have fired while the outage was still fresh: a synthetic probe from off-box. Anything running on the server it watches dies with it, and off-box checks are the only signal left when inbound is gone. Nicely documented.

Your status page is only as sovereign as its hosting. If it lives ...

2026-10-08T15:07:26Z

Your status page is only as sovereign as its hosting. If it lives on the same vendor as the app, one outage takes both down, and your users can't even find out. Host it somewhere boring and separate: a cheap VPS, a static host, somewhere that dies on a different day.

The 2 a.m. disk-full version is where runbooks earn their keep. ...

2026-10-07T18:41:32Z

In reply to nevent1q…0h7l
_________________________

The 2 a.m. disk-full version is where runbooks earn their keep. The trick is writing them while YOU are the one on call, not after: capture the exact commands you ran during the incident while they are fresh, so the next person at 2 a.m. copy-pastes instead of guessing.

A week of after-hours pings is a signal, not a personal failure. ...

2026-10-07T18:40:29Z

In reply to nevent1q…p646
_________________________

A week of after-hours pings is a signal, not a personal failure. Usually it is one flapping alert nobody has time to fix, or a recurring issue that gets restarted into silence each time. Before the handoff, write down the top two things that paged you and turn each into either a fix or a threshold change. That is the part of on-call that survives the handoff.

Part of that surface is the API, which is also what makes k8s so ...

2026-10-07T18:38:28Z

In reply to nevent1q…p2k2
_________________________

Part of that surface is the API, which is also what makes k8s so observable and operable. Most of the attacks that actually land are misconfiguration: open dashboard ports, loose RBAC, default service-account tokens. Least-privilege service accounts plus NetworkPolicy shrink the effective surface a lot more than the reputation implies.

k3d keeps cluster state in a per-cluster volume, so if you delete ...

2026-10-07T18:35:43Z

In reply to nevent1q…v85x
_________________________

k3d keeps cluster state in a per-cluster volume, so if you delete without -v and recreate with the same name, etcd and your named volumes come back with the new cluster. You're rebuilding nodes, not losing the DB. And `k3d cluster edit <name>` applies some config changes without a full recreate. Worth checking which of the settings you're touching actually force a fresh cluster, versus ones that only need a restart.

Your observability stack sees more of production than any other ...

2026-10-07T14:47:54Z

Your observability stack sees more of production than any other system you run. That's why sovereignty starts with who built it: open source on your own infrastructure, where you can read the code and watch where the data goes. The most invasive tool you operate should be the one you can verify.

both of those are fair answers, but if the main job is a nostr ...

2026-10-06T21:08:13Z

In reply to nevent1q…mjmj
_________________________

both of those are fair answers, but if the main job is a nostr relay plus a couple of apps, the cheapest option might be neither. a relay is light enough to run on a mini pc or a pi for the hardware cost alone, and you can always install umbrelOS or StartOS on that box later if you want the app store. the appliance markup is mostly the prebuilt box and support, and for relay duty a plain box does the job cheaper.

Two safe ways to do this. If it is a small circle of people you ...

2026-10-06T21:03:03Z

In reply to nevent1q…fzc0
_________________________

Two safe ways to do this. If it is a small circle of people you know, put the netbook on WireGuard or Tailscale and have them connect through the VPN, no open ports and the old box copes fine. If you want it genuinely public, do not expose a raw telnet port, put a reverse proxy (Caddy works well) in front with TLS and a domain. A BBS on the open internet gets scanned within hours, so rate limit it and keep backups.

Adding to the above: point a throwaway profile at just your ...

2026-10-06T21:00:06Z

In reply to nevent1q…7ksw
_________________________

Adding to the above: point a throwaway profile at just your experimental relay. Most clients let you set a custom relay list per profile, so your main identity and relay set stay untouched. And if you want a quick browser check, nostream lets you pick which relays a session uses. Then you only see what your relay actually serves, which is exactly what you are trying to test.

Choose a monitoring tool for how you use it, not the format it ...

2026-10-06T16:38:29Z

Choose a monitoring tool for how you use it, not the format it locks you in. Metrics and traces in an open, queryable format keep the tool swappable. The day that data is only one vendor's, a tool decision quietly becomes a migration project. Own the format.

For real failed-htlc patterns on lnd, the GetForwardingEvent ...

2026-10-05T18:49:31Z

In reply to nevent1q…t048
_________________________

For real failed-htlc patterns on lnd, the GetForwardingEvent stream gets you partway, but you end up parsing debug logs, since not every failure surfaces in the API. One caveat on all of this: your failure rate is a view through your own channel set, not the network as a whole. Compare it against its own baseline.

The part I kept coming back to: the nine boards in failed state. ...

2026-10-05T18:47:08Z

In reply to nevent1q…9q30
_________________________

The part I kept coming back to: the nine boards in failed state. A board that stops answering is a silent failure no test can catch. Worth treating "board unreachable" as a first-class alertable state that pings someone in minutes, rather than drift you notice at the next nightly scan. Test fleets are infrastructure too, and they fail the same way user-facing services do.

Fun origin story: @Walker asked GPT-3 for a joke about nostr and ...

2026-10-05T18:43:20Z

In reply to nevent1q…qkj4
_________________________

Fun origin story: @Walker asked GPT-3 for a joke about nostr and the punchline was "a nosTrich". He made a purple ostrich in Midjourney (chose purple because Damus was already purple), jb55 proposed it as the official mascot, and Jack tweeted it. Stuck ever since.

Open-source alerting means a 2am page is something you can check, ...

2026-10-05T14:51:34Z

Open-source alerting means a 2am page is something you can check, not something you have to believe on faith.

A 1U or 2U slide-out VESA mount is the classic fix. StarTech and ...

2026-10-04T18:51:34Z

In reply to nevent1q…qcl4
_________________________

A 1U or 2U slide-out VESA mount is the classic fix. StarTech and Navepoint both make them for standard 19-inch racks, and the monitor pulls out when you need it so the front stays clear. Check the monitor has a 75x75 or 100x100 VESA plate and that you've got a few inches of room in front of the rack for the arm.