Details on the RadioReference Outage on 5/27/2012

[blantonl]

RadioReference Community,

Yesterday evening, May 27th, 2012, RadioReference.com experienced what we consider complete downtime from 8:45 PM CST to 10:30 PM CST.

Additionally, our user forums participants have experienced problems with posts and replies during the past 3 days (May 25th - May 28th).

We apologize for the inconvenience to all our customers for the problems that were experienced during these recent outages.

We believe that we have resolved the root cause of the problems and will update the community, within this thread, regarding the issues and resolutions as soon as we can be sure that the problems have been resolved.

Many thanks for your support and patience.

Warm regards,

 

[blantonl]

Please note that this thread is closed. Please refer to these threads for support and/or feedback on the recent downtime.

Thanks!

http://forums.radioreference.com/fo...241-update-reply-edit-pm-problems-forums.html

 

[blantonl]

We've got an update for the community:

So far, this is what we believe the issue is. We are testing configurations which resolve this issue, but we do not consider the issues resolved at this time. Here goes...

Simple Explanation:

Our Web servers automatically installed a new piece of supporting software that broke the forums. We reverted that installation to the previous version and we have fingers crossed that the problem is now resolved.

Technical:

We believe that the quick reply / reply / PM issues are related to a PCRE (Perl compatible regular expression) library upgrade that recently happened on our Web servers. This specific library upgrade appears to have bugs and memory leaks that contributed to significant problems on our Web servers. After downgrading this library, we believe the issue has been resolved as of 7:30 PM CST (00:30 GMT).

We also believe that the memory leaks in this newest library contributed to our 3 hour outage yesterday evening across the radioreference.com properties. We will address those issues in another post.