Archive problem

[GTR8000]

Seems like the archive4 server has been acting up since around 4am CST this morning. Unable to access any archive files on that server all day long. (The archive2 server seems to be working fine)

 

[kendrik578]

I have notified Lindsay.

 

[blantonl]

Ok, I found the problem. The archive server maintenance process got hung up on an orphaned archive file: which spawned multiple instances of the archive maintenance process, which in turn queued up hundreds of queries on the database. This resulted in the filesystem on archive4 getting full, the orchestrator not spawning any additional records, and the database got hammered.

I'm manually running maintenance on archive4 as we speak and it should be back up shortly.

I'll also look at implementing some fixes this week that would keep this from happening again.

Thanks,