2024-02-17 Systemwide Outage

[blantonl]

We're aware of the site being down. If you are reading this, we're back up.

Our NFS server died. This server had been running, I'm not joking, for 1120 days without a reboot. It's really the only component of the site that doesn't have easy failover capability. Think of it as this little brain that hosts all the code for all the servers, web, archive, audio, everything. It's a small server. Doesn't need to be large. But it's important, and if it has a problem, everything has a problem.

I wouldn't be surprised if the server had some sort of hardware failure and just locked up. I really don't know what happened to it. That's the beauty of the cloud. We launched it on a new server somewhere out there in the data centers across N America. All should be well now.

Thanks for your patience, and sorry for the short down time of about 30 minutes this Saturday.

 

[vagrant]

No problem. It’s Saturday and I was taking a nap. I’m glad my subscription helped with the backup and failover costs.

 

[sallen07]

This server had been running, I'm not joking, for 1120 days without a reboot.

Not sure what OS it was running, but I bet I know what it WASN'T running.

 

[vagrant]

Unix is beautiful in its many flavors.

 

[blantonl]

It's running linux. It hosts a Network File System protocol that was literally designed in the 80's by Sun Microsystems. But it does it's job faithfully.

It's really the only single point of failure in our entire system, and designing a failover solution for this thing is actually really complex, expensive, and a pain in the butt.

 

[sallen07]

It's running linux. It hosts a Network File System protocol that was literally designed in the 80's by Sun Microsystems. But it does it's job faithfully.

It's really the only single point of failure in our entire system, and designing a failover solution for this thing is actually really complex, expensive, and a pain in the butt.

My post was a feeble attempt at a joke.

I have been a system and network administrator for 40+ years, including a stint supporting HP and then Sun workstations. I use NFS from my home NAS to access files on some of my numerous Raspberry Pi's.

In my current job we have lots of systems that have been up for YEARS.

None of them run Windows.

 

[vagrant]

Some of us knew what OS you meant

Anyways, Pi‘s are so far ahead of the hardware I was using almost 30 years ago. They’re fun for home/amateur projects and inexpensive. I have to add an RTC though on those without Internet access, plus a cron job to reboot before the RAM goes. Still, fun if one knows of shortcomings.

 

[Starcom21]

All good, had a bunch of things to do that were more important that I keep putting off lol!

 

[gmclam]

- Network File System protocol that was literally designed in the 80's
- None of them run Windows
- This server had been running, I'm not joking, for 1120 days without a reboot.

Recipe for success. I can't keep Win10 up for a week b4 MS wants to reboot, patch, update or something.