Making Contact with Trivium

Vengeance Falls

Every time I get to premiere some guitars, a metallic tear of joy sheds from my eye. Luckily for me, I have colleagues like Harold Gutierrez whom keep the metal projects coming at a fierce and unrelenting pace which the genre would approve of. I swear, I just clicked send on my Amon Amarth invoice, when Harold shot me over the unreleased artwork from the upcoming Trivium release, “Vengeance Falls.” Invasion of earth by Trivium “T” shaped ships which open into metallic praying mantis robots? Check.

What Harold didn’t know, is that I’ve been watching an unhealthy amount of X-Files, Aliens, and Prometheus, and what was supposed to be an exploratory call turned into me nerding out about the various aspects of “making contact” with the unknown and all the 80s era futuristic interfaces that go with that.

I want to believe.

Harold was also a fan of the Clown Dialer I built for Queens of the Stone Age, because it provided a synced mobile/desktop experience that seemed to feel real. So I used that project and our artwork as a basis to propose the following concept:

What if your mobile phone existed as a sort of FM radio transmitter which can receive noisy communications from Trivium that must then be interpreted by the desktop experience both visually and textually so they made sense? And what if, post transmission, that mobile experience could then hijack the controls of the desktop’s visual experience? That could be ridiculous…

Here’s how we built Trivium FM:


 Sync Devices

Programming began by working out the tech of two mobile experiences running simultaneously, as we provided visitors two options for syncing a mobile device to their desktop browser: Telephony Call (if they were in US, CA, UK, DE) and Mobile “Web App” Pairing. This will be referenced throughout this case study as “Telephone” or “Mobile Web App” respectively. Regardless of the path that was chosen, we required a unique ID to represent the user and we needed to store that ID somehow on both clients.

 Telephone

If the visitor chose to be called, they simply input their phone number into a form the desktop website, which would asynchronously run a Twilio call request on the server using the Twilio-Ruby gem.

call = twilio.account.calls.create(
  from: '+14075456854',
  to: params[:phone_number],
  application_sid: 'abcdefghijklmnopqrstuvwxyz',
  if_machine: 'Hangup'
)

{ sid: call.sid }.to_json

Twilio provides a unique string id for each call (sid) which is passed back to the client as JSON, and stored locally using the localStorage wrapper Store.js:

store.set('call', data.sid)

 Mobile Web App

Visitors who could not participate in the phone syncing were provided a code which they could enter on the mobile web app version of Trivium.FM to complete the pairing process. This was done simply by provisioning the Heroku RedisToGo addon and continually incrementing a “code_count” integer on the server as needed.

{ code: REDIS.incr("code_count") }.to_json

This was also sent to the client using JSON and stored locally.

store.set('code', 12345)

 Event Distribution

So now when any future events came in, I would simply ask, “Is this a call or a code and is this the same as the unique ID I have stored locally? If so, run the event. If not, this action is not from the synced user.”


 Send Transmission

With our mobile devices synced to our desktop browser, our next step is to send the noisy transmission Trivium provided on each of our mobile experiences.

 Telephone

As soon as a call is placed, Twilio runs the following TwiML application which plays the transmission back and begins to gather key presses which will be used in the controller explained later.

twiml = Twilio::TwiML::Response.new do |r|
  r.Play "http://01.trivium.fm/sounds/transmission.wav"
  r.Say "Use your phone's number keys to navigate the invasion."
  r.Gather action: "/gather", timeout: 10, numDigits: 1
end

Note that the audio provided should be lossless because MP3 files will take longer to transcode and potentially sound worse.

 Mobile Web App

On the mobile app, I use the light-weight and mobile friendly Javascript audio library, Howler.js to play the audio and show our controls when it has completed.

sound = new Howl
  urls: ["/sounds/transmission.mp3", "/sounds/transmission.ogg", "/sounds/transmission.wav"]
  onend: ->
    # show controls

I usually provide MP3, OGG, and WAV for maximum compatibility.


 Receive Transmission

Of course, sending the transmission is only 50% of the experience. So as soon as the mobile device is synced and sending, the desktop browser should begin interpreting the incoming transmission using a bit of smoke and mirrors and a lot of Dancer.js.

Dancer.js is another JavaScript audio library which allows me to make sweet visualizations using the audio APIs provided by modern browsers, kick detection, and real-time frequency data. However, I think the real magic this library provides is that it can do all of this while the audio is muted, so what looks like an analysis is actually just a synced playback (with mobile) and visualization. The interpretation on the screen as a waveform line and lyrics is enough to make it feel real. Let’s setup the library and go through a few of those functions.

dancer = new Dancer()

The after event will fire continuously as soon as the track begins so I use it to draw my waveform line using Kinetic.js by defining a new array of points given the value of the provided waveform data and drawing a Kinetic.Line with those points.

dancer.after 0, () ->
  drawWaveform @getWaveform()

drawWaveform = (data) ->
  for point in data
    points.push x: _i, y: point * 50 + 50

  line = new Kinetic.Line
    points: points
    stroke: '#FEC45E'
    strokeWidth: 2

Dancer.js kicks are used to detect peaks in frequency (or heightened points in the track) and I’ve used it to begin teasing the Invasion artwork simply by adjusting the opacity of that layer given the magnitude provided.

kick = dancer.createKick
  frequency: [0, 400]
  onKick: (mag) ->
    $("#invasion").css opacity: mag
  offKick: ->
    $("#invasion").css opacity: 0
kick.on()

You can also use the library to fire an event onceAt a specified time, which is perfect for lyrics visualization and could be expanded to show various changes in the experience around major track sections.

dancer.onceAt 1.445, ->
  $(".lyric").text "I am outnumbered"

Finally, the same sound is loaded in various formats, the volume set to mute, and is played back.

dancer.load({ src: "/sounds/transmission", codecs: ['mp3','ogg','wav']})
dancer.setVolume(0)
dancer.play()

 Control Invasion

Once mobile has transmitted the message and the desktop browser has interpreted it, the invasion artwork is faded in and the mobile device is now able to control the artwork’s position because we still have our synced connection in place. Let’s take a look at how this is achieved differently within a call and the mobile web app.

 Telephone

Since we can’t exactly do anything “visual” within a phone call, we’ll need to turn our Keypad into a 9 axis controller. Earlier we saw that as soon as the audio is played back, we begin gathering key presses. These are sent to a gather function which loops continuously, listening for key presses and sends those to the desktop browser in real-time using Pusher.

Pusher['private-trivium'].trigger('client-key_press', { key: params["Digits"], call: params["CallSid"] })

twiml = Twilio::TwiML::Response.new do |r|
  r.Gather action: '/gather', timeout: 10, numDigits: 1
end

The desktop client then translates the numerical key press to a CSS position which is passed using jQuery to the Invasion artwork div.

keyPress = (key) ->
  direction = switch parseInt(key)
    when 1 then "top left"
    when 2 then "top center"
    when 3 then "top right"
    when 4 then "center left"
    when 5 then "center center"
    when 6 then "center right"
    when 7 then "bottom left"
    when 8 then "bottom center"
    when 9 then "bottom right"

  $("#invasion").css backgroundPosition: direction

Throw in a bit of CSS3 transitioning and you’ve got a smooth artwork explorer.

transition(2s background-position)

 Mobile Web App

The mobile web app experience is mostly similar, except we need to define the view for the keypad. I use Symbolset in combination with a directions array in my Haml to make quick work of this:

directions = ["UPLEFT", "UP", "UPRIGHT", "LEFT", "CROSSHAIR", "RIGHT", "DOWNLEFT", "DOWN", "DOWNRIGHT"]
directions.each_with_index do |direction, index|
  %a.ss-icon{href: "#", data: {key: index + 1}}>= direction

I then listen for clicks on these buttons and pass the corresponding key values over to the same Pusher client event listening for key presses.

$("nav a").click ->
  channel.trigger "client-key_press", key: $(this).data("key"), code: store.get("code")
  false

Note that in both of these examples, we’re passing the unique ID we established earlier (as code or call) to make sure these are being fired on the appropriate client.


 Reception

The response to this campaign was amazing, leading to something like 200 million Pusher events, thousands of phone calls, and critics asking themselves, “Who do Trivium think they are, Jay-Z, using a mobile phone to deliver music?” Fans were also compelled enough to record the experience on YouTube, which I especially enjoyed, because it shows off all the initial bugs.

If I had to summarize what impressed me most about this campaign, it’s that the entire concept came from one image which we developed a storyline around over a single phone call. We we so happy with how Part 1 went, that we quickly flipped the website into Transmission 2 and premiered the whole Brave This Storm track using some heavy visual lifting by HTML5 canvas.

Don’t bet your whole campaign on a singular idea. Instead break your experience into interesting bite sized segments you can analyze and roll out over the length of your cycle, subscribing emails along the way.

Part 3? You’ll just have to sign-up, wait, and see.

 Thanks

Of course, we couldn’t have made contact in the first place without the help of all these incredible libraries and technologies:

Redis to Go
Store.JS
Phony
Pusher
Twilio
Dancer.JS
Howler.JS
isMobile
Kinetic.JS
Symbolset

As always, thanks for reading. Drop me a line on Twitter or over email if you have any questions or comments.

 
57
Kudos
 
57
Kudos

Now read this

An Artist’s Guide to Reddit’s AMA Questions and Answers Series featuring Tenacious D

This is not The Greatest AMA in the World, no. This is just a tribute. Reddit, the self-described front page of the Internet, is a social network in which users submit links to anything interesting on the web and then vote (up or down)... Continue →