BlogShares

BlogShares is a very large, very complex game. At the front - the bit the players see - it all seems relatively simple. A few dozen screens are available to the player, and the game-play concept sits within these 30-40 screens you can access via buttons and links. On the back end though there is a database server handling upwards of 1,000 queries per second, holding 50gb of table data, 300-odd tables and 300-million rows of information. There's the web server, servicing Player and Non-Player requests, parsing and caching 80,000 lines of game code. And there's the Spider server handling 'the java spiders' that can consume anywhere between 5-15MBit/s of bandwidth alone. On or around December 19, 2011, the database server crashed. This was a machine that had been in service since 2007, and it has ultimately reached the end of its serviceable life. We were placing more and more demands on it with new game tools and code, and it had run out of room to expand. Eventually, on that fateful night, it hung up its will to operate any longer and summarily crashed dead in the water. After a couple of weeks of no news, I (Lee / The Architect) made an offer to Jay to take over hosting the game. It was clear that he no longer had the time to be able to fix and run BlogShares along with everything else going on at Santa Cruz Tech and it was likely 2011 would be the end of BlogShares for good. This isn't the first crash in the 9 year history of the game. This might be the 5th of any great note in terms of 'time taken to fix'. We are always breaking it subtly when making upgrades or introducing new tools, but I don't count those! After a few days, Jay accepted my offer but could not immediately proffer access to the system as he was a 2-hour round trip away from the data center where the game was hosted. There was not even any guarantee that there was any data left to salvage. Eventually he did find the time to travel to San Jose, and the database server was limped back into life, temporarily. It had a failed drive in the RAID array, and the crash had badly damaged the database.