Emergency Maintenance

March 28th, 2011: Filed under  by Alexander Catapang @ 3:59 pm

One of the servers became unresponsive yesterday. Since that is where the entire database was located, the site and all the games have been affected. I estimate total downtime to be around 30 minutes or so, before I got things back to normal.

I have been experiencing this issue in the past, however it usually resolves by itself within a minute or two, and it only occurs every couple of weeks. I haven’t been able to figure out the exact cause of the problem, since it occurs very briefly, so I assumed it was just some intermittent network issues with my host.

Yesterday, the server was just being unresponsive on and off, so I had no choice but to just reboot the machine, to see if it will fix things (it did). Fortunately, I was able to catch a glimpse of what’s possibly causing this issue, which looks like database related. I have not really made optimizations since I ordered the new servers last December, so it is likely causing some bottlenecks given the site’s regular traffic.

I have made some optimization changes yesterday and today, so hopefully the problem is fully resolved. As usual, I will be monitoring things just in case the same issue re-appears.