It is LIVE! As I write to you it is Friday the 13th and 28,000 players are online rocking out to a music concert inside of our web-based multiplayer game, AdventureQuest Worlds. In a few moments the song will end, and they will all put on giant “deady” bear costumes and destroy an island full of undead. The reward for teaming up and bringing down the box will be a key that opens a treasure chest full of unlucky weapons, pets and items. However… if you asked me what we were going to release just 4 days ago on Monday… I would have said, “No idea, let me see if there are any good ideas on the forums.” This is the secret, behind-the-scenes story of a real, regular release week at the studio of Artix Entertainment.
My name is Artix Krieger (Or in that real-world thing people keep talking about -- Adam Bohn). On a release week we drop all the ego-fluffing titles and are ready to do whatever it takes to make the best game for our players. Note: For over 6 years and for all 4 of our online games… every week is a release week.
This story starts last weekend when I put up a post saying, “OMG! Next week is Friday the 13th!” Superstitious gamers from all over the world know Friday the 13th is the most unlucky day of the year. We are anxious to celebrate and throw a special event for every special day we can. Needless to say the players responded with an endless sea of great ideas. Some of them were things we have done in the past. Some of them were brilliant but would require re-coding our entire game. (Probably worth it.) Last time, we threw a live event with musical guest star Voltaire. Why not turn this into a tradition?
Voltaire was on tour. He had just finished playing a show in Mexico when I talked to him on the phone. He said he was interested and offered to go to the recording studio on Tuesday when he got back. It is amazing that he will make up all new lyrics based on the game and re-sing the song with it. All we had to do was pick a song. We chose "Day of the Dead" since we had just missed the holiday of the same name on the 1st.
Later on Monday, I met with Cysero, who is currently in charge of AQWorlds. We brain stormed and reviewed the player responses. Then he put his headphones on and starting writing. Less than 2 hours letter he sent out a battle plan to the team. Meanwhile, this is an important week. We just acquired another gaming company and the new team was visiting the studio for the first time. With only 3 weeks left to launch their game we had to do a lot. This happened to include the creation of our new master account system… which would unify all of our games. One login and password to rule them all! If that was not enough, we also were going to create our first television commercial that week. Our controller (I do prefer calling him supreme ultimate accountant lord) asked, “Wow, great. What company did you hire to make the commercial?” There was an awkward pause. He suddenly looked panicked and said, “Oh no….”. I smiled. We do everything ourselves. Galanoth, the head of the original AdventureQuest was in town. We stayed up until 2am in front of my home computer writing a script and doing a mockup video (Also with the help of player suggestions and fan videos – thanks Skyline). To have the commercial air on Network TV we needed to get an ESRB rating. So we had to make that video too and submit it too.
Tuesday descended into chaos. At 9am, four of us met at my house, fired up Pro Tools and proceeded to do professional voice recording for the commercial in my guest bathroom. Meanwhile Miltonius began work on all of the monsters, weapons, pets and items for the release. This included the boss monster and a super unlucky rabbit who was missing all four of his lucky rabbit’s feet. We need enough items so that our free players and our paid members all get something new. J6 also had four days to create a skull themed stage in our main town, an animated band and an entire new jungle zone with destructible terrain. We also needed marketing materials… because we really wanted more than my Mom to show up for the event. This may sound easy so far… but perhaps you forgot about the programming part.
We did not have the song or art done yet. So programming had to be done “blind”. Zhoom is one of our two legendary super coders. He effortlessly pulls of miracles every week so this would be a no brainer for him. Except he had to deadline to finish embedding AQWorlds into the Tagged social networking site by Tuesday night, and then he had to switch gears and work on the newly acquired game. He was converting their entire database from MySQL to MSSQL and all of their code pages from PHP to ASP. Then move on to helping rig their system to support multiple servers. So, he was out. Fortunately, just like in Yugioh, we always keep our strongest card until the last moment. Er… scratch that. Unfortunately, the other legendary super coder, Minimal, was in the middle of a complete game-rewrite (see, we could have used some of those player suggestions after all.) He just launched our first Test Server where the players got to try out PvP and Stats. He was not planning on opening it until Thursday, but it opened up in a 3 hour special test event after being coerced by a flood of LOL-catz posted by the players.
That left Lucien. He is a hybrid artist, animator, programmer. No problem. All he had to do was figure out to get the animated event to run every hour, on the hour for all players. (Which we suddenly decided we needed on Wednesday.) We also needed a system wide message to warn players when it was about to start. Um, and add an unhackable button to automatically equip the Deady bear costumes. Not to mention the code to make the quests, destructible terrain, concert special effects, and other little things work… like being able to make the NPC’s in the concert, across all of the servers, say something in all of the rooms at the exact same time.
The good news is a new episode of the show “V” would be on my DVR we got done with Mauy Thai Kickboxing that night. We have a deal with the martial arts school that allows anyone on our team to train there if they want. The sensei is brutal and merciless… and cracks funny comments while torturing us. I think taking this sort of beating prepares us for the rest of the week.
On Wednesday morning I got a PM from Reens. She asked how everything was going. She already knew the answer before she asked. Which is good, because Wednesdays was so crazy it took me over an hour to respond. Nothing was done. Nothing was testable. And any questions about how any piece of this release will specifically work would be met with, “Um… I am not sure yet, but we will think of something!” The teams for all four games are in the same situation with their releases. Wednesday has a funny feeling to it. Sort of like throwing a ball into the air as far as you can throw it. There is a moment when it hits the highest point that it seems to almost float. You do not know where it is going to land… or whose window it is going to break… but you sure know it is coming down! There is only one day left between us and Friday… and then a player showed up at the studio.
We have had a few visitors at our office. I have been shadowed by young players whose parents flew them in from New York and California. We even had an entire school come visit once. (Yes, it was VERY cramped.) One of my favorite memories was a young girl who was brought by her mother on her birthday. This is not the sort of thing we normally do. But they were visiting from out of town… her mother called every day for a week. When I finally heard about this, we dropped everything we were doing and got her a cake, card and inevitably went over-board hurling our remaining out-of-print swag at her. Which reminds me, we celebrated a double birthday on this day too -- Zazul from player support, and Nightwraith from the new game. As per usual we fired up the amps in the band room and sang a heavy metal happy birthday while serving cake. The cake had game characters printed on it. Wow, did I get off topic.
THURSDAY is crunch day. It is the most vital and critical day of the week. We cannot afford to lose a moment of time. So naturally I had to leave for an all-day meeting.
Meanwhile, back at the studio, Cysero put on the Kamikaze headband that I bought him in Japan. He does this every Thursday. “It gets me through the week. I don’t know what I would do without it”, he says. The stress levels are running high as Thursday night approaches without a working prototype of the event. With less than 24 hours left on the clock things were looking very bad. Thursday night is mostly about faith. Beleen had just finished our email newsletter and turned in 3 articles for game related websites. She made the mistake of asking me to proof-read them. I do not know why I am compelled to rewrite everything I touch (For example: you should see how many times I rewrote this story). We started posting what we had available to promote the event on Twitter, Facebook, Myspace. If you had seen how little was actually working went we left to go get beat up at Martial Arts class… you would have blinked twice, allowed for an awkward pause, then burst out laughing. There was good news though. Voltaire called. We have a song!
During my meetings today I met with our marketing guru and business advisors to work on “big picture” items. Like buying more ads for the games, turning down more investors, coming up with next year’s project list, and drawing little stick figures. It takes a lot of strength to get excited about tomorrow’s completely uncompleted release while listening to high level execs from competing game companies explain why you will be crushed like a grape in the oncoming wave of annihilation. That is why I like the show, “The Office”. Thursday night’s was a pretty good episode too! But my favorite line was from the episode where Michael was negotiating an offer for Dunder Mifflin to buy his newly formed Michael Scott Paper company. “Our company is worth nothing. That’s the difference between you and I. Business isn’t about money to me, David. If tomorrow my company goes under I will just start another paper company. And then another and another and another. I have no shortage of company names.” I love that statement, because it has so much heart. Just like us… we are addicted to the players, games and the lifestyle. We would work out of a cardboard box to keep doing this… well, as long as the box had a high speed internet connection… and suddenly, it was Friday morning.
The first unlucky thing about Friday the 13th – was having to wake up early. I brought Starbucks into the studio for the early birds. We started a “play by play” on the homepage. Updated hourly or faster, it told the players every horrifying train-wreck quality mess-up we made as we struggled to make release live by 6pm. We had a lot to write about. The concert was not functioning, none of the quests worked, the player’s hair would pop through the giant Deady costume’s head. It looked like someone put a wig on its eye. We dubbed it the “Eye-wig feature.” My job was to fix the concert and add “things” that would happen as Voltaire played his song. J6 made me some last minute smoke FX and an explosion. Swarmed by bugs I was not able to add my first effect, which was Voltaire and his skeletal band poofing onto the stage, until 1pm EST. Cysero was having similar luck on his quest. 5 hours remained until launch.
We played musical chairs. Up until today the team had been pretty spread out working wherever they sat down with their laptop. Zhoom, Cysero, Llusien, and Beleen all moved into the War Room with me. Minimal, J6 and Miltonius would have sat with us too, but it is a long drive for them from Philadelphia, Canada and LA. They and roughly 60% of our 46 team members telecommute from all over the US, Canada and Europe.
After plugging my laptop into the 74” TV at the end of the conference table that we use for a monitor, we wished we had invested in some headphones instead. We listened to the same 11 seconds of the song starting a maddening gazillion times. Somehow Zhoom kept his focus through the noise and was able to setup two new servers for the game. We could not choose between the names Kabroz or Truffle for our newest moglin safe safer. These ultra safe canned-chat only servers are always named after our popular in-game mascot creatures called Moglins. So we let the players vote. It was really close. Also, Zhoom rigged prepared our Content Delivery Network (CDN) to offload the strain of tens of thousands of players loading the giant concert/music file… at the exact same time… every hour….
At 3:30pm, Safiria and Nythera finished answering thousands of player emails. They brought their laptops into the war room to help. We heard a cheer in the other room. DragonFable’s release had just gone live. MechQuest was about to go live too. I was biting into a luke-warm piece of non-dairy pizza as Adobe Flash found new creative ways to try and break both my file and my spirit. What we were doing was pretty crazy in the first place. We load a map file. Then we load players into that map. We also have game characters built into that map. THEN… we load a music file along with a script that makes everyone sing along and dance to the song. Needless to say, it all broke.
The room was as busy as a military command bunker in the middle of World War 3. The phone rang with Voltaire’s name on the caller ID at 5pm. I remember clearly because Safiria answered and I could hear her guiding him to the development server right as we crashed it. Someone hit a button and wiped out the game in a wave of errors. From the last few paragraphs you might have noticed – I have no idea what anyone else is doing. There is not a lot of idle time while broadcasting the countdown date to all of the players, updating the homepage, tweeting, and trying to resist the urge to see if I can buy my Prime Downtown Lot in Mafia wars yet. We have been doing this together for a long time though. If there is one thing I know, it is… WHAT DO YOU MEAN IT IS 5:45!? Prepare to roll all files live! Server restart in 5 minutes!
That is exactly how it happens.
See, the concert event is set for every hour on the hour. So if we are not up at 5:50, the players will not have time to get back online before 6:00pm when the first concert begins. There is exactly 1 minute and 15 seconds of testing on the staging server before we “roll” the files live. Worse yet, for obvious reasons, the final version of the file cannot have any of the developer test buttons still on it. So there is not actually anyway to see if that fancy CDN stuff works. Or… to see if ANYTHING would work. Except to go to the concert alongside the players at 6pm… and find out together! Exciting? Definitely. This is my favorite part of every week!
Zhoom says, “Alright. Bringing the servers back up.” At 5:59 Voltaire’s message to greet the players is broadcast to all of the servers. There was moment of silence….
Two hours later I got a call from my girlfriend who drove over to my place, and then accidentally locked herself out of my house while walking my undead slaying battle-Pomeranian, Daimyo. She is so sweet. She knew I would be working late with the team. In fact, she was here during the last Friday the 13th event where we crashed not only the AQWorlds servers but ALL the games on our network on that fateful launch 6 months ago.
Not tonight! The players cheered and messaged us with “Best Friday the 13th ever”! They loved the concert and even more people showed up for the later shows. We have never had a flawless, lagless release before. (Not counting that, um, “Eye wig feature” of course.) The entire team did such an amazing job. Including those mentioned in this article and all of the folks you have not heard of… yet. Voltaire, had not seen the event until it went live either. He was amazed we pulled it off in such little time and congratulated the team for all their hard work. Cysero, at long last took off his Kamikaze headband and he pressed the power button on his laptop. At 5:30am the next morning a few of us were still online. Rocking out with the players at the hourly concert.
What are we going to do next week in AdventureQuest Worlds? No idea, let me see if there are any good ideas on the forums. Did I mention we have been doing this for 6 years