Face detection with getUserMedia

This post is more than 2 years old.

There are quite a few interesting APIs evolving in the "modern web", but not all of them are going to be things you would use in most projects. I've been very public about my feelings concerning canvas for example. Great for games and charting - but not much else. That doesn't make it a bad feature. It just makes it one I won't use terribly often. Whenever I read about some new cool feature being developed, my mind starts trying to figure out what they could be used for in a practical sense. Obviously what's practical to you may not be practical to me, but figuring out how I would actually use a feature is part of how I learn it.

One such feature is getUserMedia (W3C Spec). This is a JavaScript API that gives you access to (with permission) the user's web cam and microphone. getUserMedia is currently supported in Opera and Chrome (I believe it is in version 18 now, but you may need to grab Canary. You also need to enable it. Instructions on that here.) Once you get past actually enabling it, the API is rather simple. Here's a quick request for access:

The first argument to getUserMedia is the type. According to the spec, this is supposed to be an object where you enable audio, video, or both, like so: {audio:true, video:true}. However in my testing, passing a string, "video", worked fine. The demo you will be seeing is based on another demo so that line possibly came from an earlier build that still works with Chrome. The second and third arguments are your success and failure callbacks respectively.

You can see in the gist where the success handler assigns the video stream to an HTML5 video tag. What's cool then is that once you have that running you can use the Canvas API to take pictures. For a demo of this, check out Greg Miernicki's demo:

http://miernicki.com/cam.html

If this demo doesn't work for you - then stop - and try following the instructions again to enable support. (Although I plan on sharing a few screen shots so if you just want to keep reading, that's fine too.)

Based on Greg's demo, it occurred to me that there is something cool we can do with pictures of our web cams. (Cue the dirty jokes.) I remembered that Face.com had a very cool API for parsing pictures for faces. (I blogged a ColdFusion example back in November.) I wondered then if we could combine Greg's demo with the Face.com API to do some basic facial recognition.

Turns out there are a few significant issues with this. First - while Face.com has a nice REST API, how would we use it from a JavaScript application? Secondly - Face.com requires you to either upload a picture or give it a URL. I know I could send a canvas picture to a server and have my backend upload it to Face.com, but is there a way to bypass the server and send the picture right to the API?

The first issue actually turned out to be a non-issue. Face.com implements CORS (Cross-Origin Resource Sharing). CORS basically allows a server to expose itself to Ajax calls from documents on other domains. It's a great feature and I hope more services enable it.

The more complex issue then was taking the canvas data and sending it to Face.com. How can I fake a file upload? Turns out there's another cool new trick - FormData. Fellow ColdFusion blogger Sagar Ganatra has an excellent blog entry on the topic. Here's how I used it:

Let's look at this line by line. First off - I need to get the binary data from the canvas object. There's a few ways of doing this, but I wanted a Binary Blob specifically. Notice the dataURIToBlob method. This comes from a StackOverflow post I found a few weeks back.

I create a new FormData object and then simply begin setting my values. You can see I pass in a few API requirements but the crucial parts are the filename and file object itself.

Below that you can see the simple jQuery Ajax call. Face.com has a variety of options, but I basically just asked it to return an estimated age, gender, mood, and whether or not the person was smiling and wearing glasses. That's it. I get a nice JSON packet back and format it.

Now obviously no API is perfect. I've had different levels of results from using the API. Sometimes it's pretty damn accurate and sometimes it isn't. Overall though it's pretty cool. Here are some scary pictures of yours truly testing it out.

Ok, ready to test it yourself? Just click the demo button below. For the entire source, just view source! This is 100% client-side code.

For another look at getUserMedia, check out these examples:

Edit on May 23: Chrome recently modified the getUserMedia API to match the spec (I believe) which requires you to pass an object of media you want, so instead of "video", I used {video:true}.

Raymond Camden's Picture

About Raymond Camden

Raymond is a senior developer evangelist for Adobe. He focuses on document services, JavaScript, and enterprise cat demos. If you like this article, please consider visiting my Amazon Wishlist or donating via PayPal to show your support. You can even buy me a coffee!

Lafayette, LA https://www.raymondcamden.com

Archived Comments

Comment 1 by andy matthews posted on 4/7/2012 at 12:00 AM

I'm sueing! Your demo says that I'm 47 while yours only says 22.

Comment 2 by Raymond Camden posted on 4/7/2012 at 12:03 AM

I built it to reduce my age. ;)

Comment 3 by Tim Leach posted on 4/7/2012 at 9:27 PM

Never a bad idea to compliment your maker.

Comment 4 by Terry Sampson posted on 4/9/2012 at 3:17 PM

I've heard of women faking their age, but web programmers?

Actually, I would have put 19 myself :>)

Comment 5 by Raymond Camden posted on 4/16/2012 at 7:07 PM

Another great example of this from Daniel Goodwin. He actually adds training support to his demo.

http://vimeo.com/26850592

Comment 6 by Sagar Ganatra posted on 5/23/2012 at 7:11 PM

Demo is not working in Chrome 21. I think they have changed the API to accept a map instead of a comma separated list i.e. {video: true} instead of 'video'

Comment 7 by Raymond Camden posted on 5/23/2012 at 7:12 PM

Yep, I saw that recently when I tested my code. I will update the blog entry today. Thanks!

Comment 8 by Raymond Camden posted on 5/23/2012 at 7:24 PM

Post + demo updated. Thanks Sagar.

I should now post the other one I did. I was going to show it at cfObjective but never got around to it.

Comment 9 by Chad Gray posted on 8/14/2012 at 7:44 PM

I tweaked your code to send the picture to a CF page that saves it to the server and sends back the file name.

Thanks for the blog post!

https://github.com/rchadgra...

Comment 10 by Raymond Camden posted on 8/14/2012 at 7:50 PM

Glad to help - but please note that this API is going away. :( There are two new APIs out there that I know of. (Don't have em handy right now though.)

Comment 11 by Raymond Camden posted on 8/14/2012 at 7:51 PM

I don't have the URLs, but they are ReKognition and Lambda.

Comment 12 by Chad Gray posted on 8/14/2012 at 9:07 PM

I thought i would ditch the function dataURItoBlob() in the JS and use binaryDecode() in CF. I get "The input and output encodings are not same".

I am taking the raw data from canvas.toDataURL() and sending it to my CF page.

Any ideas to try to get CF to Decode this base64 data?

This is beginning of the data being sent:
image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQA

Comment 13 by Raymond Camden posted on 8/14/2012 at 9:09 PM

Did you try toBinary?

Comment 14 by Chad Gray posted on 8/14/2012 at 10:11 PM

ToBinary() results in "The parameter 1 of function ToBinary, which is now data:image/jpeg;base64,/9j/4AAQSk... must be a Base-64 encoded string."

cfset image = BinaryDecode(tobase64(file), "Base64") AND
cfset image = tobase64(file)

does not throw an error, but the JPG that is written by CFFile action=write is corrupt.

Comment 15 by Raymond Camden posted on 8/14/2012 at 10:13 PM

Can you shoot me (via email) a file with JUST the data?

Comment 16 by Raymond Camden posted on 8/14/2012 at 10:50 PM

So it looks like base64 sometimes includes clues in front of it - like

data:image/jpeg;base64,

I'm assuming that is part of the spec, but CF doesn't like it when using toBinary. I believe - stress believe - you are safe to just remove it:

contents = fileRead("/Users/ray/Downloads/test.txt");
contents = replace(contents, "data:image/jpeg;base64,","");
b = toBinary(contents);

This worked for me. Well, it didn't throw an error. I didn't try displaying it. ;)

Comment 17 by Raymond Camden posted on 8/14/2012 at 10:51 PM

Ok, I did, and it worked.

<cfimage action="writetobrowser" source="#b#">

Comment 18 by Chad Gray posted on 8/14/2012 at 10:56 PM

Yay! That fixed it.

Good thing you did not view the image. It is my messy office and ugly mug.

Thanks Ray!

Comment 19 by Adrian Lamberts posted on 4/24/2013 at 2:10 PM

I've tried your demo,but it doesn't work, it keeps saying: "working hard for the money.." is it my pc or the code? because I would really love to try it out!

Comment 20 by Raymond Camden posted on 4/24/2013 at 2:46 PM

Face.com shut down their API.

Comment 21 by Raymond Camden posted on 4/24/2013 at 3:04 PM

Although that may not be it. What error do you see in the console?

Comment 22 by Raymond Camden posted on 4/24/2013 at 3:17 PM

It is BlobBuilder - that isn't supported anymore. I'll try to get a fix later today.

Comment 23 by Raymond Camden posted on 4/24/2013 at 3:56 PM

So I fixed the BlobBuilder thing (it is easy to work around), but the API still won't respond, so I'm pretty sure that the API itself is dead. There are alternatives out there though.

Comment 24 by Raymond Camden posted on 4/24/2013 at 4:04 PM

Btw, an alternative: http://www.raymondcamden.co...

Comment 25 by Adrian Lamberts posted on 4/25/2013 at 11:44 AM

Well, that's a shame. But thanks for the extra info!

Comment 26 by Adrian Lamberts posted on 4/25/2013 at 12:08 PM

http://www.seeingmachines.c... is this a possibility?

Comment 27 by Raymond Camden posted on 4/25/2013 at 5:29 PM

Well, did you look at the site and try to figure it out? :) That link appears to be for Windows code. If you don't see a REST or web service option, then you are out of luck.

Comment 28 by Rg posted on 1/14/2015 at 6:29 PM

Hi,
can you b able to suggest how u have made this web api

Comment 29 (In reply to #28) by Raymond Camden posted on 1/14/2015 at 6:30 PM

I didn't make the API.