Game Server Update

March 25th, 2009: Filed under  by Alexander Catapang @ 2:00 pm

During the server migrations, I have updated the database driver, and it is now causing some mysql errors (in a few specific instances) that were not happening in the previous driver version. This caused some of the automated detection scripts that I did to incorrectly flag that a game server needed to be restarted, because of these mysql “errors”. End result is that sometimes, a game server will be auto-restarted, when it’s not really necessary.

I have just made a fix to prevent this from happening, so only valid game server restarts will occur. The fix will take effect in about 2-3 hours, just in time for the scheduled daily restarts of the game servers. I will monitor everything afterwards, to see if the fix works as expected.

Migration Update: it’s been 3 days, and the new servers have so far been stable. I have also been turning off a few more unnecessary services on the new servers, to make things more efficient. I will continue fine-tuning things as the need arises. One other good thing coming out from this migration (despite the sudden additional work it required on my part) is that I have learned a couple of new things again.

Two of the old servers will already be taken offline tomorrow. The other old server will be migrated on Thursday (with RAM upgrade), however that is not being used in production anymore, it’s basically just a server for testing purposes now. I’m keeping it until the next billing cycle only.

So far, the 3-server setup has been doing great, so that means starting next month, it will be a little cheaper for me to maintain this site. At least there are some positive things to come out from this migration that my hosting provider forced to all its clients.


Server Migration: Phase 2 Complete

March 22nd, 2009: Filed under  by Alexander Catapang @ 5:57 pm

All server migrations have now been completed. IP propagation may take an hour or so to take effect, but if you are reading this post, it means you are now viewing the site on the new servers.

I will be monitoring things in the next couple of hours to see if there will be new issues, but so far, things are working fine already. Game servers were moved first, then the rest of all the site elements followed. Things were generally smooth going, until I made the switch for the web site itself. I pointed it to the wrong location! This made the site go in an endless loop, not showing the site contents at all.

It was really a major lapse on my end. I made all known precautionary measures, in fact almost all potential errors have already been detected prior to the big switch. And then I made a “simple” typing mistake (entering the wrong IP address, which is basically just a set of numbers)! Ouch. The site may have been down for around 45 minutes to some people, while those who are already playing when I made the error should not have been affected. Anyway, things are all back to normal now.

The 2 new servers have faster CPUs as well. This should be helpful, since I noticed (while migrating things over) that I can actually get the old servers to slow down a lot just by accessing some administrative reports.

Overall, the entire site is now hosted on just 3 dedicated servers (instead of 4). I got rid of 1 server, partly to save on monthly costs, so I just distributed its load to the rest of the 3 servers. One server got RAM upgrade, while the other 2 are new servers with significantly faster CPUs.

I had to allot a significant amount of time working on the migration this month. I’m just glad that it is now fully completed! Now I can relax a bit…

UPDATE: 6 hours after the big “switch”…

Upon further inspection, I realized that the 2 new servers (with new CPUs) aren’t running optimally. Basically, the CPU speed is adjusted dynamically, depending on the server load. However, it appears to be ALWAYS running at the minimum speed. I do not feel very comfortable that way, so I have made some adjustments to make sure the CPU minimum speed setting is set somewhere in the middle, to give things a speed bump (and at the same time, not take chances of overheating, if it’s set to maximum speed all the time).

I’ve also noticed some minor e-mail issues which have been corrected now. I have uninstalled some unnecessary services on the main server as well. So far, I am not seeing any other problem, although I am still monitoring the servers for any unusual occurences.




Site Being Moved

March 21st, 2009: Filed under  by Alexander Catapang @ 8:41 pm

I am almost done setting up the 2 new servers. The applications I need are all ready. Today, I have already copied and setup all the web site files to the new server as well, so I can do some tests before doing the big switch. So far, everything is working ok.

This message is being posted on the new server already. The blog engine has also been updated with the latest version. This post won’t really be readable by others until I have switched to the new server. This also serves as a test to make sure all areas are working properly.

I expect all the migration to be finished within the next 48 hours.


Server Migration: Phase 2

March 20th, 2009: Filed under  by Alexander Catapang @ 3:04 am

I am currently setting up 2 new servers, to replace 2 older servers that are currently in production. This is part of my host’s migration efforts. I don’t really have a choice, so I am forced to do all these unnecessary work, and with short notice at that.

Anyway, I am not done setting up everything on the 2 servers yet. I’ve been working on 1 server the last couple of hours. The other server was just deployed less than an hour ago, so I’d be working on that tomorrow more closely. I have 6 days to wrap up all these, the old servers will be taken offline after that.

The site and the games will continue to work. If all goes well, I don’t really expect any downtime. Well, perhaps 5-10 minutes at most when I do the switch from the old servers to the new ones. I still have to figure out the best way to handle this though. I am hoping that before the weekend is over, I’ll have everything settled already, so I can continue working on my new project.


Server Migration: Phase 1 Complete

March 12th, 2009: Filed under  by Alexander Catapang @ 8:18 pm

The server has finally been moved. It took a lot longer than I was expecting: instead of 2 hours, it took almost 4 hours, because of some mistakes done by my service provider. The worse part is no one has been updating me what’s going on, so for about 1 1/2 hours I’ve been very stressed, not knowing what’s happening with the server.

Once the server was back online and I was given the new information I needed, it took me about 30 minutes more to re-configure all the other affected things. My service provider has not setup everything right, but it’s a good thing that I did some “homework” ahead of this migration schedule, so I was able to fix things faster.

So far, things are back to normal. All the games are now back online. Hopefully, there won’t be new issues arising from this migration, since the server itself is now on a new location.

The only good thing coming out from this migration: the server has been upgraded with more RAM. Technically, it hasn’t really been doing memory swaps anyway, but it feels safer to know there’s more RAM available than what’s really needed at this point, since I have plans of changing server configurations later. The added RAM will likely be helpful.

That’s 1 server, 3 more servers to go! There will be a different migration path for these remaining servers though, one I plan on not using anymore, the other two I’ll have to set-up new servers from scratch! There’s gonna be a lot more work involved, I don’t like setting up new servers at all. However, I am hoping these new server migrations will be done more transparently, i.e. I don’t really expect any down time anymore.

Anyway, I’m just glad the 1st server migration is finished. Now, I can relax a bit. I’ll still be monitoring this new server the next few days to see if any new issue pops up, but so far it’s looking stable.




Server Migration: Phase 1

March 5th, 2009: Filed under  by Alexander Catapang @ 7:44 pm

My server host informed me that all my dedicated servers will be physically moved into another location. This will result to some site down time. Not much I can do, so hopefully the process will be quick and hassle-free.

I have a total of 4 dedicated servers to be moved. One of the servers is scheduled to be moved next week, March 12 at around 2:00am CDT. The physical migration will take about 2 hours, then I will need an hour or so to fix things on my end, since there will also be a change of server IP address allocations.

This server contains the entire database, so while the site pages may be accessible, all the game rooms will be down during the migration. Hopefully, the games will be back online around 5:00am CDT.

Again, this is still on March 12, so just posting to give everyone an early notice.