lwadmin

[](https://mastodon.world/@LemmyWorld )

#

Outages 🔥
##

[status.lemmy.world](https://status.lemmy.world )

💓 [fedihosting.foundation](https://fedihosting.foundation )

aspe:keyoxide.org:EFWMPEN4N66VBK5WGFM5FOZXXQ

Last Notes

2026-04-30 20:18:42 UTC

Hello World,
we have just defederated from lemmy.org due to lack of moderation on their side.
This instance has open registrations, seemingly zero abuse prevention on signup, including not requiring emails. This leads to a small number of people constantly creating new disposable accounts for various types of abuse with zero consequences.
Their admin is the CEO of an ad company, which surely doesn't affect incentives. They do virtually zero moderation, allowing all these abusive accounts to rampantly harass other instances. The very rare cases when moderation actually happens is when it affects local communities on that instance, anything else seems to be getting ignored. The admin is also frequently posting to their local communities, but doesn't at all comment or interact with remote communities in any way.
We, as well as other people, including at least one admin of another instance, have attempted to reach out to them about their lack of moderation, but we haven't received any response at all.
At this point there isn't really anything left besides defederating them, which several other instances have already done as well.

2026-04-10 20:20:26 UTC

Calling for donations! We're running Lemmy.World for you to use for free, however the running costs aren't for free... If you appreciate our services and are willing and able to help out paying the bills, please consider a donation on one of these platforms:
* https://ko-fi.com/fhfworld - Preferred.
* https://liberapay.com/fhf
* https://opencollective.com/mastodonworld - (Either recurring or one-time donations).
* https://patreon.com/mastodonworld - (Recurring donations only)

2025-05-04 13:14:10 UTC

Hello world,
as many of you probably already know, Lemmy is an open source project and its development is funded by donations.
Unfortunately, as is often the case, donations amounts are often going down over time if people are not aware of their necessity. When older users leave the platform they may stop donating, while new users joining will typically not be aware of this and won't start donating to even things out or even go towards an overall increase in donations.
All of the services provided by our non-profit Fedihosting Foundation are dependent on the development of FOSS platforms, which we can host without paying any licensing or other fees, instead only being required to pay for the infrastructure cost. We are currently investing a small part (€50 each) of the donations we receive in development of Lemmy and Mastodon, but the majority of the donations we receive are used for covering infrastructure costs. We're currently just about breaking even with the donations we receive, but it's certainly not enough to cover a large part of Lemmy or other software development costs.
We're looking to support sustainable software development for all the services we provide and will post similar announcements on our other platforms to promote donations towards the respective development teams in the coming days.
You can find the original announcement by @npub17g6…xrv5 below:
cross-posted from: https://lemmy.ml/post/29579005
> An open source project the size of Lemmy needs constant work to manage the project, implement new features and fix bugs. Dessalines and I work full-time on these tasks and more. As there is no advertising or tracking, all of our work is funded through donations. Unfortunately the amount of donations has decreased to only 2000€ per month. This leaves only 1000€ per developer, which is not enough to pay my bills. With the current level of donations I will be forced to find another job, and drastically reduce my contributions to Lemmy. To avoid this outcome and keep Lemmy growing, I ask you to please make a recurring donation:
>
> [Liberapay](https://liberapay.com/Lemmy) | [Ko-fi](https://ko-fi.com/lemmynet) | [Patreon](https://www.patreon.com/dessalines) | [OpenCollective](https://opencollective.com/lemmy) | [Crypto](https://join-lemmy.org/crypto)
>
> If you want more information before donating, consider the comparison with Reddit. It began as startup funded by rich investors. The site is managed by corporate executives who over time have become more and more disconnected from normal users. Their main goal is to make investors happy and to make a profit. This leads to user-hostile decisions like firing the employee responsible for AMAs, blocking third-party apps and more. As Reddit is a single website under a single authority, it means all users need to follow the same rules, including ridiculous ones like censoring the name "Luigi".
>
> Lemmy represents a new type of social media which is the complete opposite of Reddit. It is split across many different websites, each with its own rules, and managed by normal people who actually care about the users. There is no company and no profit motive. Much of the work is carried out by volunteer admins, mods and posters, who contribute out of enthusiasm and not for money. For users this is great as there is no advertising nor tracking, and no chance of takeover by a billionaire. Additionally there are no builtin political or ideological restrictions. You can use the software for any purpose you like, add your own restrictions or scrutinize its inner workings. Lemmy truly belongs to everyone.
>
> Dessalines and I work fulltime on Lemmy to keep up with all the feature requests, bug reports and development work. Even so there is barely enough time in the day, and no time for a second job. Previously I sometimes had to rely on my personal savings to keep developing Lemmy for you, but that can't go on forever. We partly rely on NLnet for funding, but they only pay for development of new features, and not for mandatory maintenance work. The only available option are user donations. To keep it viable donations need to reach a minimum of 5000€ per month, resulting in a modest salary of 2500€ per developer. If that goal is reached Dessalines and I can stop worrying about money, and fully focus on improving the software for the benefit of all users and instances. Please use the link below to see current donation stats and make your contribution! We especially rely on recurring donations to secure the long-term development and make Lemmy the best it can be.
>
>
> [Donate](https://join-lemmy.org/donate)

2025-04-11 10:41:48 UTC

Hello world,
as many of you may already be aware, there is an ongoing spam attack by a person claiming to be Nicole.
It is very likely that these images are part of a larger scale harassment campaign against the person depicted in the images shared as part of this spam.
Although the spammer claims to be the person in the picture, we strongly believe that this is not the case and that they're only trying to frame them.
Starting immediately, we will remove any images depicting "Nicole" and information that may lead to identifying the real person depicted in those images to prevent any possible harassment.
This includes older posts and comments once identified.
We also expect moderators to take action if such content is reported.
While we do not intend to punish people posting this once, not being aware of the context, we may take additional actions if they continue to post this content, as we consider this to be supporting the harassment campaign.
Discussion that does not include the images themselves or references that may lead to identifying the real person behind the image will continue to be allowed.
If you receive spam PMs please continue reporting them and we'll continue working on our spam detections to attempt to identify them early before they reach many users.

2025-03-28 03:21:15 UTC

Hello,
as some of you may have noticed we just had about 25 minutes of downtime due to the update to Lemmy 0.19.10.
Lemmy release notes: https://join-lemmy.org/news/2025-03-19_-_Lemmy_Release_v0.19.10_and_Developer_AMA
This won't fix YouTube thumbnails for us, as YouTube banned all IPs belonging to our hosting provider.
We were intending to apply this update without downtime, as we're looking to apply the database migration that allows marking PMs as removed due to the recent spam waves.
Although this update contains database migrations, we expected to still be able to apply the migration in the background before updating the running software, as the database schema between the versions was backwards compatible. Unfortunately, once we started the migrations, we started seeing the site go down.
In the first minutes we assumed that the migrations contained in this upgrade were somehow unexpectedly blocking more than intended but still processing, but it turned out that nothing was actually happening on the database side. Our database deadlocked due to what appears to be an orphaned transaction, which didn't die even after we killed all Lemmy containers other than the one running the migrations.
While the orphaned transaction was pending, a pending schema migration was waiting for the previous transaction to complete or be rolled back, so nothing was moving anymore. As the previous transaction also didn't move anymore everything started to die. We're not entirely sure why the original transaction broke down, as it was started about 30 seconds before the schema migration query, which seems like that shouldn't have been broken by that happening at the same time.
Lemmy has a "replaceable" schema, which is applied separately from the regular database schema migrations, which runs every time a DB migration occurs. We unfortunately did not consider this replaceable schema migration in our planning, as we would otherwise have realized that this would likely have larger impact on the overall migration.
After we identified that the database had deadlocked, we resorted to restarting our postgres container, then run the migration again. Once we restarted the database, everything was back online in less than 30 seconds, which includes first running the remaining migrations and then starting up all containers again.
When we tested this process on our test instance prior to deploying this to the Lemmy.World production environment we did not run into this issue. Everything was working fine with the backend services running on Lemmy 0.19.9 and the database being upgraded to Lemmy 0.19.10 schema already, but the major difference here is the lack of user activity during the time of the migration.
Our learning from this is to always plan for downtime for Lemmy updates if any database migrations are included, as it does not appear to be possible to "safely" apply them even if they seem small enough to be theoretically doable without downtime.

2025-03-15 17:41:53 UTC

Hello,
we will be performing the long awaited update to Lemmy 0.19.9 tomorrow.
We are planning for around 1 hour of downtime between 16:00-17:00 UTC on 16th of March.
You can convert this to your local time here: https://inmytime.zone/?iso=2025-03-16T16%3A00%3A00.000Z
You can find an overview of the changes in our previous announcement [here](https://lemmy.world/post/23471887) and in the Lemmy release notes:
- [Lemmy v0.19.4 Release](https://join-lemmy.org/news/2024-06-07_-_Lemmy_Release_v0.19.4_-_Image_Proxying_and_Federation_improvements)
- [Lemmy v0.19.5 Release](https://join-lemmy.org/news/2024-06-19_-_Lemmy_Release_v0.19.5_-_A_Few_Bugfixes)
- [Lemmy v0.19.6 Release](https://join-lemmy.org/news/2024-11-08_-_Lemmy_Release_v0.19.6)
- [Lemmy v0.19.7 Release](https://join-lemmy.org/news/2024-11-15_-_Lemmy_Release_v0.19.7)
- [Lemmy v0.19.8 Release](https://join-lemmy.org/news/2024-12-12_-_Lemmy_Release_v0.19.8)
- [Lemmy v0.19.9 Release](https://join-lemmy.org/news/2025-02-10_-_Lemmy_Release_v0.19.9)
---
Update 16:50 UTC:
The upgrade was successfully completed at around 16:27 UTC, but we're still fighting with some performance issues after the upgrade. Our database and the outbound federation container are currently using significantly higher CPU than expected, which is still being investigated to identify the root cause.