Sinatra on Sinatra

Sinatra on Sinatra

May you live to be 100 and may the last website you see be mine.

Four years ago my niece received a microphone and stand, preset with a few classic diva tracks for Christmas. It also just so happened to be the “Christmas the TV broke” stranding us without entertainment. She was’t able to sing a single song before my little sister, Mallory and I commandeered the present and began performing an impromptu holiday set in the living room for friends and family. My brother, Leslie, who plays music for a living, reluctantly decided to accompany us on piano. It was legendary.

This production has been revisited every year since in several different forms:

Well, It’s that time of the year again. The time to stop working and let a little liquid confidence and family cheer bring out the crooner in all of us. So it goes without saying, that I was quite pleased when my friend Leda asked me if I could sneak in one last project this year for the chairman of drunken holiday sing-a-longs: Frank Sinatra.

As it turned out, it was the 20th anniversary of Frank’s compilation, Duets, and the estate wanted to build an app that allowed fans to duet with old blue eyes on Jingle Bells. Here’s how I built it, on a Sinatra powered web stack no less:

 Prerequisites

In an effort to expand my programming knowledge, I always pitch my projects with at least one new technical problem or concept. The pressure of an actual job forces me to learn and this project’s homework assignment was the Web Audio API. Even though I worked at SoundCloud for three years as their Experimental Developer, I had yet to dive into browser-based audio APIs. Luckily for me, Boris Smus had written the excellent Getting Started with Web Audio API article on HTML5 Rocks. As a prerequisite for me and you, I suggest that as a starting point so you can understand contexts, buffers, and sources below.

 Karaoke

A sing-along application requires lyrics synced to sound in order to emulate a karaoke experience. I began this process by creating a JSON subtitles file for Frank’s rendition of Jingle Bells, time stamping each of the lyrical phrases with their timed position in the song. This could be expanded to syllables if I wanted to add in a bouncing ball.

lyrics = [
  {
    timestamp: 4.88,
    lyric: 'I love those J-I-N-G-L-E bells'
  },{
    timestamp: 10.05,
    lyric: 'Those holiday J-I-N-G-L-E bells'
  },{
    timestamp: 15.93,
    lyric: 'Those happy J-I-N-G-L-E B-E, double-L-S'
  },{
    timestamp: 21.66,
    lyric: 'I love those J-I-N-G-L-E bells'
  }
]

Once the BufferLoader loaded my master Jingle Bells MP3, playing the song was easy enough:

sinatra = context.createBufferSource()
sinatra.buffer = jingle_bells
sinatra.connect(context.destination)
sinatra.start(0)

I then used a 50 millisecond interval event to constantly check the track’s current position. If a lyric was found that existed before the current position, the lyric would be displayed and then dropped from my lyrics array for future searches.

setInterval ->
  position = context.currentTime
  result   = _.find lyrics, (l) -> l.timestamp < position

  if result
    $('.lyrics').text result.lyric
    lyrics.shift()
, 50

 Recording

All of my previous attempts at recording in the browser utilized a Flash based solution for capturing the microphone input. Not this time, bro. I decided to take this project to the edge by bringing in Matt Diamond’s excellent Recording.JS library. Not to be confused with Jo’s library of the same name. Following Matt’s provided example, I quickly got it configured and combined it with my karaoke solution.

One cool thing about Matt’s library, is that it plays back what it hears from the microphone through the computer’s current output. This can lead to some ear bleeding feedback through your speakers OR an excellent monitor effect in your headphones. I made sure the instructions on the website reflected this. This led to a nice recording booth simulator.

Anyway, we’ll start the recorder as soon as our master track plays:

sinatra.start(0)
recorder.record()

And as soon as it finishes:

setTimeout ->
  sinatra.stop(0)
  recorder.stop()    
, jingle_bells.duration

Yep, that simple.

 Mixing

Using this solution, we’ve successfully recorded our user vocals, but they are separate from Sinatra’s Jingle Bells master track.

But Lee, why don’t you just record both simultaneously from the microphone?

Well, you… because that would sound like shit. Instead, I wanted to mix the pure recorded vocal with the original master creating a higher quality end product.

I originally proposed this as a server side merge using FFMPEG but then realized in my research that I could actually do this client side with OfflineAudioContext in less than a second.

Once the recording process finishes, I immediately get the left and right channel buffers from Recorder.JS. I then create a new offline audio context and connect new buffer sources for both the vocals and Sinatra master. These sources are also told to start playing but are actually processed by the OfflineAudioContext’s aptly called startRendering function, which renders a new buffer as quickly as possible rather than at a real time rate.

recorder.getBuffer (buffers) ->
  offline = new webkitOfflineAudioContext(2, duration, 44100)

  vocal_source = offline.createBufferSource()
  vocal_buffer = offline.createBuffer(2, buffers[0].length, 44100)
  vocal_buffer.getChannelData(0).set(buffers[0])
  vocal_buffer.getChannelData(1).set(buffers[1])
  vocal_source.buffer = vocal_buffer
  vocal_source.connect(offline.destination)

  sinatra_source = offline.createBufferSource()
  sinatra_source.buffer = jingle_bells
  sinatra_source.connect(offline.destination)

  vocal_source.start(0)
  sinatra_source.start(0)

  offline.oncomplete = (event) ->
    event.renderedBuffer

  offline.startRendering()

I then use this renderedBuffer to allow my users to preview the final version of their duet with Frank. If they like what they hear, submitting is just one click away.

 Saving

The last piece of tech needs to create a WAV version of the final duet and upload it to my Heroku server for saving. Matt’s Recorder.JS library actually comes packaged with a worker that includes this exact function. Simply provide the worker with the appropriate commands and wait for it’s onmessage response.

worker = new Worker('js/recorderWorker.js')

worker.postMessage
  command: 'init'
  config:
    sampleRate: context.sampleRate

worker.postMessage
  command: 'record'
  buffer: [
    recording.getChannelData(0),
    recording.getChannelData(1)
  ]

worker.postMessage
  command: 'exportWAV'
  type: 'audio/wav'

worker.onmessage = (event) ->

This response includes an event.data which I must pass one last time to the FileReader API using the readAsDataURL function in order to turn it into a Blob which my server can process. I then setup an XMLHttpRequest which accepts a simulated form data file and sends it to my server.

reader = new FileReader()

reader.onloadend = (event) ->
  file_contents = event.target.result.split(',')[1]

  data = new FormData()
  data.append('filename', 'JingleBells.wav')
  data.append('mimetype', 'audio/wav')
  data.append('data', file_contents)
  data.append('size', e.data.size)

  xhr = new XMLHttpRequest()

  xhr.open 'POST', '/duets'
  xhr.send(data)

reader.readAsDataURL(e.data)

The server then decodes this file using Base64 and writes it to a temporary file. The file is then finally uploaded to SoundCloud using their provided Ruby Gem.

duet = Duet.new

File.open(".tmp/#{ duet.id }-jingle-bells.wav", 'wb') do |f|
  f.write(Base64.decode64(params[:data]))
end

client = SoundCloud.new(access_token: ENV["SOUNDCLOUD_TOKEN"])

track = client.post('/tracks', track: {
  title: "Jingle Bells",
  asset_data: File.new(".tmp/#{ duet.id }-jingle-bells.wav")
})

duet.soundcloud = track.id
duet.save

 Happy Holidays

As soon as you’re ready, head over to duets.sinatra.com and give it your best, but for goodness sake: don’t forget to wear headphones.

I’m writing this on a rainy New Orleans night in the same cafe I spent most of my time this year. To say it’s been an odd year is an understatement. I actually spent less time worrying about work and more time worrying about myself. Something I haven’t done ever.. really. I can confidently say that I’m a better person now than I was at the beginning of the year, but there’s still more work to be done. To all of you who helped me through the year both personally and in my career: Thank You.

I’m signing off for the year as I’ll be heading down to Chauvin for the latest installment of my family’s sing-along tradition. I hope you also find yourself in a place filled with love and cheer.

Merry Christmas and Happy New Year.

 
102
Kudos
 
102
Kudos

Now read this

Intimacy Doesn’t Scale

For years artists have been using the Internet as a channel of communication to talk to their fans. We send out newsletters via email and keep our fans informed of our every move on Twitter and Facebook. We perform live stream concerts,... Continue →