Outages again?
1mon 4d ago by kbin.earth/u/haverholm in kbinEarth@kbin.earth
I've had a couple of cloudflare errors logging on today, errors that were supposedly on the server end.
I also notice that my feed hasn't been updated for 8 hours (at least that's the time passed since the latest thread was published). Visiting other instances, it appears there have been several updates to my subscribed communities that aren't federated to KbinEarth. Threads, comments, votes, etc.
Is everything alright over there? Or have I just been shadowbanned 😓
Lol, shadowbanning's not a thing here. Anyway, the issue was just brought to my attention an hour ago, so I am still trying to figure out what's wrong.
Good stuff, I'll let you get back to it 👍
(Also, the shadowban thing was meant as a joke, I'm not too worried 😉)
While inward federation seems to have an issue since yesterday (but at some point content came in again until it stopped later), outward federations works without any delay.
It turns out the issue is simply due to the server's CPU being under heavy load, and it is unable to keep up with the federation queues. I will need to investigate where the increased load is coming from; it's possible we are being DDoSed.
I had the same problem for 4 days this week on feddit.online (PieFed instance). My CPUs were pegged suddenly, and the database load that is normally like 1.5 spiked to 15. The UI became unresponsive, and gateway timeouts became common. After 4 days, yesterday morning, it just suddenly stopped. I didn't have the opportunity to investigate.
I'm hearing about scraper attacks from other instances too.
I'm going to try to do some forensic analysis this weekend. If you find a way to identify these scrapers, please share. If I learn anything I'll share too.
Good luck!
My instance, also Mbin-powered, logged me off some days ago and stayed some 4 hours without updates during the day today. Also dunno if related either, but piefed.ca has been having some stability issues.
Auster@thebrainbin.org in our instance (tardigram running Mbin 1.9.1) we haven,t noticed any login issue yet recently, also have cloudflare but our server is personal.