Ok, I found the problem. The archive server maintenance process got hung up on an orphaned archive file: which spawned multiple instances of the archive maintenance process, which in turn queued up hundreds of queries on the database. This resulted in the filesystem on archive4 getting full, the orchestrator not spawning any additional records, and the database got hammered.
I'm manually running maintenance on archive4 as we speak and it should be back up shortly.
I'll also look at implementing some fixes this week that would keep this from happening again.