User:Emily Diehl/Journal/November 2008

November Journal Archive

Tuesday, November 5, 2008

Icky wiki issues

So I am sure that all of you have noticed the poor wiki having some problems. I just wanted to pop a quick journal entry down so you all know that we are all aware of the issues, and that many people are working diligently to figure out what's going on.

In a nutshell, last night's problems were triggered when I flipped the logo from the Halloween one back to the original site logo. Like any large website, our wiki uses a relatively robust deployment system to ensure that the site remains safe from errors and other random misfortunes. We have a development environment for testing things, a staging environment that sits between dev and live, and then the live wiki (which you see here). Both dev and stage are copies of live, and are synced to regular backups done of the live site. After our recent upgrade, these backups are done in a rolling fashion, so they are regular and invisible to you guys as you're working (when all if going well).

As you can imagine, keeping all of these sites in sync is a pretty delicate process. If one configuration file is somehow off and that gets pushed to the next step in the chain, bad things happen.

Since the wiki is a dynamic and ever-changing site (unlike our static game site), we rarely need to make changes that cause us to have to "deploy" content from dev to stage to live. Most of the stuff that happens here happens directly in the live environment...that's what makes a wiki so unique from other kinds of web environments. For the wikis, we have to run custom deploy scripts only for configuration changes that can't be done from an admin page on the live site, for upgrades, and for things like main logo swaps/core skin revisions, etc.

I'm rambling, but long story short, one of our deployment scripts was not reflecting some of the new database settings that changed after our big upgrade. It was a matter of us not realizing that x needed to be changed in y file, and missing that when we updated things. When I ran the script, it was telling the live wiki to look in the wrong place for data. Obviously, that confused the site ;) Performance slowed to a crawl as the incorrect information started to propagate across our many wiki servers, and before you knew it...we had some issues :)

My main partner in crime on the IT side has been out on leave because of a new addition to their family, so it took a little while for everything to get tracked down and figured out. We were definitely on it, though. I sat and watched the election in a window on my desktop while all of my friends watched it on my TV at my house...lol.

Long story short is that we're aware of what caused the problem and are taking steps to resolve the remaning issues that are springing up around the site. Keep reporting stuff you see to me, and I'll keep shuttling the issues over to the rest of the team so they can get looked at.

Thanks for your patience in all of this! It's great to have a wiki community that understands when things get a little bumpy :)

Journal Archives