Strange World Amnesia

This page is about Heyuri or Heyuri Culture

This page is under construction or broken...

Strange World Amensia

Length of outage	2-4 hours
Affected boards	/sw/
Casualties	120GB error log file
	Recent 5000 messages (sorta)

On November 30th, 2025. Heyuri's /sw/ board disappeared. All posts suddenly went out of existence, and for seemingly no reason. In actuality, error logs from NGNIX had filled the disk of the server /sw/ was being hosted on to such a degree that any further writes corrupted files. Luckily, thanks to the wise monks at HARD, alongside some help from a suprise guest, next to no data was fully lost, only the front-end cache.

TRAGEDY!

At some point in time (i dunno... i don't work there!), updates to the error log file in the nginx directory had ballooned up to a size reaching above 120+ GB. The exact cause is unknown, but this maxed out the disks that /sw/ was hosted on. (i'm assuming /sw/ is on a seperate disk to the imgboard...)

As a result, /sw/ suddenly no longer worked properly. The strange world script relies on a 5000-post long message log (I'll call it the warmlog) that's used for the frontend. Due to there being literally zero disk space, any further writes corrupted this frontend log. The script would (presumably) then try to load the recently corrupted log as a person connected, fail, and then error out; resulting in weird behaviour when trying to connect. This would vary from getting a white screen, to getting some of the site to work, to seeing weird errors. I'd have to scroll further than 1000 lines on Chat@Heyuri to find the people complaining about it though...

This got reported quite quickly, and HARD devs quickly got to work fixing it. kuz also came in to assist, helping to find the bloated nginx error log.

Oh, it's over? That was kinda fast.

This was solved kind of quickly. I think it was only a downtime of a few hours, and people just used Chat@Heyuri to whine about it while HARD got to work. All that needed to happen to fix the issue was to free up disk space, and restart the scripts. It was as simple as kaguya deleting the old inflated error log, and kuz and him throw away and replace the corrupted warmlog. The script initially starts entirely new posts, as if the instance was completely new, but this is quickly corrected to have their posts line up correctly with the message log.

No data was actually lost. This is because the backend, KuzuhaScriptPHP+, writes posts to the warmlog, and then also to cold, long term logs that aren't limited by lines, so every message dating back to 2022 was still there. You can access them here.

However, the warmlog was lost. I'm not sure if it means any posts were lost, and all it now means that you can't just flip pages on the frontend all the way back 5000 posts, until the warmlog fills back up.

Kuz says that they'll be doing smart server admin stuff to make sure it doesn't happen again, which is probably just cleaning the dust off his 5 year old server configs. Not like I'd know, I haven't hosted shit.

OC

screenshot from the kuz flash OC

As with most downtime incidents, this got some OC as well. OC that i forgot to add... and was also made by Kuz.

There was a poem written:

Watching the bare log, alone, how can I bare the darkness?

Staring blankly at the empty board, words fail me, my voice choked with sobs.

It's just like birds that have eaten and flown back to their nest, a vast expanse of dark green, pure and, truly clean!

and a little flash made in commermoration. Both were made by Kuz.