A Glance At Audio Sprites In 1,000 Words Or Less!

In working on a current project, the Twitter-fueled HTML5 -powered game Squirrel Sqript (Which is almost ready to launch, by the way!), I’ve learned a lot about cross-functionality. I acted as a programmer on this team, and as all programmers must do, I had to overcome certain unique problems presented by the platform and the project. Because the game is HTML5, our team encountered an issue in that browser-based games (particularly mobile browser-based games, and especially mobile browser based games) don’t necessarily support audio in the way you want them to. And no single codec is accepted by every browser. AND the performance hits are dramatic for even some of the simplest of audio related functions. AND the list of quirks goes on. It’s maddening! Not even sites specifically built for audio like SoundCloud offer great usability on mobile because putting your phone to sleep not only kills playback, but also the player itself in many instances on awake, forcing a refresh of the entire page. The logic is that most mobile users pay for data per gigabyte/kilobyte, and overage gets expensive, so the browser will take any chance it gets to kill your audio. That’s where audio sprites come in.

There are a myriad of solutions to browser game audio, and the audio sprite is one of the more popular. It’s not unlike the classic sprite sheet, which is still prevalent even in modern games. A sprite sheet has, as the name implies, all of the individual frames of an animation laid out next to one another, and the program then parses the images, sets them into an array, and flips through them to create a simple (or even complex) animation. There’s a pretty cool examples on the net such as this fan-created sheet of Super Metroid’s Samus Aran, hosted at SpriteDatabase.net. Like sprites, a sound designer can create a single concatenated file containing all of the sound effects to be used in the game.  On the first button press of the game (say, “Start”), the audio sprite manager loads the file and loops a section of silence on the sprite file. Then, at key points in the game you can jump to other points in the file – through code and without user interaction (!) – to play sound effects as you need them. It’s quite beautiful in how simply it handles this, and the best part is that most mobile platforms are pretty good about holding entire files in RAM for you to jump around in.

There are a few drawbacks, however, as with anything. A huge sprite file takes up a lot of precious ram – audio in .wav format at 44.1kHz is the same size whether it’s silent or white noise, if the file sizes are the same length. So it’s best to make them short. If you have many sound effects, you may want to group them up by level and load them one by one (Just remember, you need an initial button press to load them!). It’s also important to be mindful of performance impacts – unlike desktops, mobile does not typically have designated VRAM and RAM. It all goes into the same pile, so if you’re displaying cool 3D graphics on mobile, make sure you test and test again, because you will find performance chokes quite often. And of course, browsers all have their own quirks to them.

Now unfortunately, the code we used for this game was originally published here on the Zynga Engineering and Code Blog, and has since been taken down for whatever reason (I’ll leave the link here for posterity, and in case it ever comes back!). So I have but only a short example and description to give, because I do not feel comfortable explicitly posting their hundreds of lines of work to paste-bin. All I can offer is a “This is how it works” and leave it for you to maybe engineer something better. 🙂

I’ve got a simple audio sprite here. Just a few recordings. There is silence until exactly 2 seconds into the file, and one sound effect every second thereafter. To fire them, you would first lay them out in your JavaScript file as if it were an animation, which you can see on this pastebin post here, and then call the JavaScript through HTML button functions. It’s pretty simple, really!

If one were inclined to do this in a game, it gets trickier, because you don’t always have explicit buttons or key-presses, so this is a small walk-through of a portion of Zynga’s code, in lieu of pasting it. A portion of the code checks an array for playable resources (e.g.: an array of files, “file.ogg”, “file.mp3”, “file.wav”, etc.) and adds a playable resource once it finds it to the Jukebox Manager’s loop, which is described in the encapsulated portion of the code that we on our teem did not need to interact with. Once this was was done, we set the background audio of the game to play as part of the document (in other words, the same way you’ve always gotten your JavaScript audio to play and loop), and defined ONLY the interactive parts on a spritemap in the declaration of the new Jukebox. The new jukebox itself has encapsulated functions pause, play, and resume functions which trigger when the appropriate function is called. So for example, we called “squirrelBox.play(‘optionclick’, true)” on a few of the button classes to define click sounds for buttons. The “play” function passes in two arguments – the pointer which defines the start and end of the portion, in seconds, of the resource to play, and an “enforce” bool which provides for queuing or directly playing a sound (playing them immediately, or forcing one to finish before the next one). “play” then provides for starting and ending the playback stream. It’s really simple and elegant, actually!

If you know of a great jukebox system like the one described, please link it! 🙂

Happy Coding
–Chris Prunotto