WebRTC Sprites Generator

WebRTC and Sprites

Yes. The computer graphic, not the soda.

Last weekend I traveled to Punta Del Diablo for a bit of code in the sun and to remotely participate in Music Hack Day Stockholm. As per usual, Johannes and I decided to build a video game, BeatStriker, in under 24 hours and submit it to the competition.

Today, I’d like to share some technical details on a simple client side WebRTC sprite sheet generator I built. We used it to define a simple nine grid sprite sheet consisting of a user’s mug facing nine directions, using a WebCam and a tiny bit of client side JavaScript.

You can expect another blog regarding Punta Del Diablo and the pros and cons of hacking from a hammock soon.

Sprites #

Street Fighter

A sprite sheet is a series of images (sprites) that usually correspond to individual animation frames which are combined into a larger image (or images.) Game designers have used these since 1974 to limit the amount of assets required to make players and elements move, and web developers have used them since… well, I guess since the CSS background property. Animations are represented by displaying one frame at a time in a pre-defined sequence.

My first run in with sprites was in 1999 when I participated in the Mugen gaming community. Mugen is basically an open-source Street Fighter which allows developers to define their own sprite sheets of playable characters and levels. This allowed for some pretty weird match-ups like Ryu vs Homer Simpson aka copyright infringement, but what the hell else was I going to do while waiting for all my Napster downloads to finish?

A new challenger appears!


Prerequisites #

The only HTML and CSS we need to pull this off is the following three elements and a defined size for the #sprite div.

<div id="webcam"></div>
<div id="instructions"></div>
<div id="sprite"></div>

<style type="text/css">
  #sprite {
    background-size: 450px;
    height: 150px;
    width: 150px;

You’ll also want to include Addy Osmani’s excellent getUserMedia.js library and Underscore because I was lazy about some selectors.

Variables #

Next we’ll need to define a few variables. photo keeps track of which picture we are currently taking. size defines the size of each sprite image, and should match the #sprite div sizing above. The directions array defines how each sprite should be placed on the sheet given the grid sizing. Each hash also includes an i (info) variable which will be used to display user instructions during the picture taking process.

photo      = 0
size       = 150
directions =
  center: { x: size, y: size, i: "straight ahead" }
  north: { x: size, y: 0, i: "up" }
  northwest: { x: 0, y: 0, i: "up over your right shoulder" }
  west: { x: 0, y: size, i: "right" }
  southwest: { x: 0, y: size * 2, i: "at your right shoulder" }
  south: { x: size, y: size * 2, i: "at your chest" }
  southeast: { x: size * 2, y: size * 2, i: "at your left shoulder" }
  east: { x: size * 2, y: size, i: "left" }
  northeast: { x: size * 2, y: 0, i: "up over your left shoulder" }

WebRTC #

If you read this blog, you’ll know I’m no stranger to WebRTC or getUserMedia.js as I’ve used it before in my Foo Fighters In Rock We Trust hack. Sadly, there hasn’t been great leaps in adoption since that post and we’re still stuck in Google Chrome only. Unless of course, your user is running Firefox nightly. Hint: he isn’t. So if you plan on going into production, you’ll need to utilize the getUserMedia.js flash fallback function. I tend to think you’re a dummy for not using Chrome (and this was a hack) so I won’t be covering that here.

Let’s start by configuring our getUserMedia shim, mapping a few jQuery selectors, and defining what happens when the WebCam successfully streams or throws an error.

App =
  init: ->
    @canvas = document.createElement("canvas")
    @canvas.setAttribute("width", size * 3)
    @canvas.setAttribute("height", size * 3)
    @context = @canvas.getContext("2d")

    @instructions = $("#instructions")
    @sprite = $("#sprite")

    getUserMedia(@options, @success, @error)

    audio      : false
    video      : true
    el         : "webcam"
    extern     : null
    append     : true
    noFallback : true
    width      : 320
    height     : 240
    mode       : "callback"

success: (stream) ->
  # see full gist for cross-browser stream declaration
  App.instructions.text("Look straight ahead and press spacebar.")
error: (error) ->
  console.error "An error occured: [CODE #{ error.code }]"

takePhoto() #

Next we’ll define a function which takes each photo by resizing and cropping the captured WebCam video and then drawing it onto a canvas at the relavant location using the directions array. The photo variable is then incremented to allow for the next picture to be taken.

Once all eight photos are taken, we can use the toDataURL() function canvas provides to apply the sprite sheet to our #sprite element’s CSS background.

takePhoto: () ->
  if photo < 8
    v = document.getElementsByTagName("video")[0]
    h = v.videoHeight
    w = v.videoWidth
    x = (w - h) / 2
    d = directions[ _.keys(directions)[photo] ]

    App.context.drawImage(v, x, 0, h, h, d.x, d.y, size, size)

    if photo isnt 8
      next = directions[ _.keys(directions)[photo + 1] ]
      App.instructions.text("Look #{ next.i } and press spacebar.")

    photo += 1

      "background": "url(#{ App.canvas.toDataURL('image/png') })"
    App.instructions.text("Done! Try pressing your arrow keys.")

Controls #

Now that we’ve created our sprite sheet and associated it with our #sprite div, we can define some basic controls to animate the face into pointing a particular direction. First let’s write a simple function, which takes a d (direction) and shifts the sprite sheet’s CSS background positioning given the appropriate coordinates from the directions array.

pointFace: (d) ->
    "background-position": "-#{ directions[d].x }px -#{ directions[d].y }px"

Then we’ll write a simple jQuery keydown function which calls the takePhoto() function each time Spacebar is pressed, and the pointFace() function each time Up, Right, Down, or Left is pressed. When the user let’s go of a key, the face returns to center.

$(window).keydown (e) ->
  switch e.keyCode
    when 32
    when 38
    when 39
    when 40
    when 37

$(window).keyup (e) ->

Note: We ended up rewriting this to detect mouse direction instead which allowed us to call the diagonal points also.

Finish Him! #

If you made it this far, you’re ready to initialize your app:


Refresh your browser and here’s what should happen:

  1. Google Chrome will prompt you to allow WebCam access.
  2. Once allowed, your beautiful face will appear in the #webcam div.
  3. Follow the #instructions by facing the appropriate direction and clicking Spacebar to generate your sprite sheet.
  4. Once that’s done, you should be able to control your face with your keyboard arrows.

From here, you may want to save your newly created sprite sheet to the server for future use or add some gaming elements like we did for BeatStriker.

Check out the full source on Gist and as always, let me know if you have any questions or suggestions on Twitter. Happy coding.


Now read this

In Rock We Trust

Minting $1,500,000 at the Federal Reserve of Foo Fighters A little over a week ago, the Foo Fighters played both Reading and Leeds festivals in the UK. Each of these shows ended in a confetti cannon launch of several hundred thousand... Continue →