Using the new Bluemix Visual Recognition service in Cordova

This post is more than 2 years old.

Before I begin, a quick disclaimer. I've been at IBM for a grand total of five days. Considering three were taken up by travel and orientation, I'm very much the new kid on the block here. I've only begun to look into MobileFirst and Bluemix so you should take what I show here with the same confidence you would give anyone using a new technology for two days. In other words - proceed with caution! ;)

Bluemix is a cloud platform that offers both Platform as a Service (PaaS) and MBaaS (Mobile Backend as a Service). The PaaS offerings let you deploy web applications using a variety of engines, including Node. I'm currently use AppFog as my PaaS for a few sites so I'm already a bit familiar with the concept. The MBaaS side are services that integrate with your web or mobile apps. These include things like NoSQL datastorage and access to IBM Watson.

Yesterday we announced five new services that make use of the Watson. They include: Speech to Text and Text to Speech, Visual Recognition, Concept Insights, and Tradeoff Analytics.

As soon as I saw the Visual Recognition API (demo link) I thought - this would be cool in Cordova!

One thing I wasn't sure of though was how to use the service by itself, and not with a particular application. I checked the docs and came across this Enabling external applications and third-party tools to use Bluemix services:

You might have applications that were created and run outside of Bluemix, or you might use third-party tools. If Bluemix services provide endpoints that are accessible from the internet, you can use those services with your local apps or third-party tools.

To enable an external application or third-party tool to use a Bluemix service, complete the following steps:

  1. Request an instance of the service for an existing Bluemix application. The credentials and connection parameters for the service are created. For more information about requesting a service instance, see Requesting a new service instance.
  2. Retrieve the credentials and the connection parameters of the service instance from the VCAP_SERVICES environment variable of the Bluemix application.
  3. Specify the credentials and the connection parameters in your external application or third-party tool.

My understanding of that was that I needed to make a new Node application and simply create a view that would dump out VCAP_SERVICES. I wished it were easier, but I figured if I did this once, I could reuse the same Node code in the future when I needed to test out another service. Turns out I didn't need to do all those steps. This next section will focus on what you have to do on the Bluemix side to prepare the service. I'll then switch to the Cordova side.

After signing up for Bluemix (you can get a free, 30 day trial), you want to first create an application. You won't actually be using the application, but it is necessary to get the proper credentials to use the service.

bm1

On the next screen I selected Web. This may be something I change next time I do this.

bm2

And then I selected the Node option. Since I really wasn't using the app, I probably could have selected "I Have Code Already."

bm3

For the application name, pick anything. If you pick "RayIsTheBestNewIBMer" you get double the time for your free trial. (Note - the preceding statement may not be exactly true.)

bm4

On the next screen, select Add a Service:

bm5

And then - obviously - select the Visual Recognition service. Note the beta label. Results may vary. Yada, yada, yada.

bm6

Since you added this service from an app, it will be automatically selected in the next screen. (You may not have a "Space" though. I made a few while testing and I don't remember if it is required or not.)

bm7

You'll get a warning about needing to restage the application, but since you don't have anything there anyway you can just go ahead and let it restart the app. Woot - almost done. Now we need the authentication and API info. Back on the app dashboard, note there is a link to show credentials for the service:

bm8

Clicking that will expose properties for the service including the ones we care about: url, username, and password:

bm9

Ok, we've got what we came for, let's talk about the Cordova side. I began by creating a simple application that would make use of the Camera API. I created a web page with two buttons - one to source from the device camera and one from the photo gallery. Once I had the image file, I would then make use of the File Transfer plugin to post to the API.

Documentation for the Visual Recognition API may be found here: http://www.ibm.com/smarterplanet/us/en/ibmwatson/developercloud/apis/#!/visual-recognition/. I focused my attention on the POST call to /v1/tag/recognize. I saw there that I needed to send the image with a file name of img_file. Let's take a look at the code (this portion and the rest may be found in the Github URL I'll share towards the end):



var API_URL = "https://gateway.watsonplatform.net/visual-recognition-beta/api";
var API_USER = "supersecret";
var API_PASSWORD = "anothersecret";

$(document).on("deviceready", function() {

    function uploadWin(res) {
        var data = JSON.parse(res.response);
        var labels = data.images[0].labels;
        var result = "<p>Detected the following possible items:<br/>";
        for(var i=0, len=labels.length; i<len; i++) {
            result += "<b>"+labels[i].label_name + "</b><br/>";   
        }
        $("#status").html(result);
    }
    
    function uploadFail() {
        console.log('uploadFail');
        console.dir(arguments);
    }
    
    //Credit: http://stackoverflow.com/a/14313052/52160
    function authHeaderValue(username, password) {
        var tok = username + ':' + password;
        var hash = btoa(tok);
        return "Basic " + hash;
    };
    
    function onCamSuccess(imageData) {
		var image = document.getElementById('myImage');
        $("#imgDisplay").attr("src", imageData);

        $("#status").html("<i>Uploading picture for BlueMix analysis...</i>");
        
        var options = new FileUploadOptions();
        options.fileKey = "img_file";
        options.fileName = imageData.substr(imageData.lastIndexOf('/') + 1);
        
        options.headers = {'Authorization': authHeaderValue(API_USER, API_PASSWORD) };
        
        var ft = new FileTransfer();
        ft.upload(imageData, encodeURI(API_URL+"/v1/tag/recognize"), uploadWin, uploadFail, options);

    }

    function onCamFail(message) {
	alert('Failed because: ' + message);
    }	
    
    //Touch handlers for the two buttons, one uses lib, one uses cam
    $("#cameraButton, #galleryButton").on("touchend", function() {
        var source = ($(this).prop("id")==="cameraButton")?Camera.PictureSourceType.CAMERA:Camera.PictureSourceType.PHOTOLIBRARY;
        
		navigator.camera.getPicture(onCamSuccess, onCamFail, { 
			quality: 50,
			sourceType: source,
			destinationType: Camera.DestinationType.FILE_URI
		});

    });
    
});

I assume the camera usage is not necessarily new for my readers. All I do "fancy" here is switch my source based on the button clicked. As soon as I have the image I render it on the app so the user can see it. I then begin my file transfer. I use the URL I got from the service and ensure I include my authorization info. I'm using fake values in the code above of course. (More on that in a minute.)

Once done, I take the result and simply output the labels to the app. The service returns labels with an underscore between words and that could be cleaned up, but I didn't really bother. Here are a few samples. Of course, the results are not perfect, but close, and can be improved. First, a scary picture of myself.

bm10

I'm not quite sure where Combat Sport came from, but I'm all about the combat sport. Meat Eater is also dead on too. On a serious note, human, indoors and person view were perfect.

Each label returns a score and you could use that to filter out items that appear to be too low. I'm not doing that in this demo but that would help improve the results shown to the user. Ok, another test.

bm11

Personally I think toothed whale is pretty damn funny. It didn't recognize shark, but it got close, which I think is pretty good.

Another option the API supports is including labels to narrow down the search. One could use that to help direct the results as well.

So - the obvious issue we have with this demo is that the username and password are available in the source code. One way around that would be to actually use the Node application and have it work as a proxy. That would let me log, monitor, restrict to authorized users, etc.

I've put the source code for this up on my Github repo of Cordova demos. You can find it here: https://github.com/cfjedimaster/Cordova-Examples/tree/master/imagerecognitionbluemix

Raymond Camden's Picture

About Raymond Camden

Raymond is a senior developer evangelist for Adobe. He focuses on document services, JavaScript, and enterprise cat demos. If you like this article, please consider visiting my Amazon Wishlist or donating via PayPal to show your support. You can even buy me a coffee!

Lafayette, LA https://www.raymondcamden.com

Archived Comments

Comment 1 by OscarB posted on 2/6/2015 at 6:29 PM

Obviously, combat sports is a clear hint that AI see us as rivals. Awesome work, thanks.

Comment 2 by Robert Zehnder posted on 2/6/2015 at 8:01 PM

Looks like a pretty awesome platform. I dig it already

Comment 3 by jcesar posted on 2/9/2015 at 7:48 AM

whaaaaat? you didn't use a cat picture for the examples? shame on you

Comment 4 by Jon Camilleri posted on 2/10/2015 at 1:34 PM

Can you help with this please?

Your environment has been set up for using Node.js 0.12.0 (x64) and npm.

C:\Users\slyth_000>dir

Volume in drive C has no label.

Volume Serial Number is F6C8-3FA8

Directory of C:\Users\slyth_000

10/02/2015 13:38 <dir> .

10/02/2015 13:38 <dir> ..

23/12/2014 14:36 <dir> .android

11/12/2014 19:35 <dir> .AndroidStudio

03/06/2014 18:41 <dir> .AndroidStudioPreview

27/11/2014 09:36 <dir> .atom

10/02/2015 13:38 <dir> .cordova

23/12/2014 18:00 <dir> .dartium

10/02/2015 10:30 <dir> .eclipse

20/03/2014 08:22 <dir> .freemind

24/11/2013 18:43 <dir> .gfclient

04/12/2014 10:34 <dir> .gimp-2.8

03/06/2014 20:11 <dir> .gradle

09/02/2015 14:24 <dir> .jmc

31/05/2014 10:33 <dir> .nbi

13/11/2013 08:59 <dir> .netbeans-derby

17/05/2014 08:05 <dir> .thumbnails

21/12/2014 15:47 <dir> .VirtualBox

09/02/2015 15:40 <dir> AndroidStudioProjects

24/11/2013 18:35 <dir> Application Data

31/05/2014 12:02 <dir> Aptana Rubles

13/11/2014 20:55 <dir> Contacts

01/06/2014 09:50 <dir> dart

01/06/2014 09:50 <dir> DartEditor

10/02/2015 14:17 <dir> Desktop

10/02/2015 11:09 <dir> Documents

10/02/2015 14:07 <dir> Downloads

17/04/2014 19:12 <dir> Dropbox

13/11/2014 20:55 <dir> Favorites

21/12/2014 08:23 <dir> GameMaker-Studio 1.4

21/12/2014 08:23 <dir> GameMakerPlayer

13/12/2014 19:57 <dir> git

10/02/2015 13:38 <dir> invoicing-app

13/11/2014 20:55 <dir> Links

13/11/2014 20:55 <dir> Music

23/07/2014 11:43 <dir> PhpstormProjects

18/01/2015 16:18 <dir> Pictures

13/11/2014 20:55 <dir> Saved Games

13/11/2014 20:55 <dir> Searches

10/02/2015 14:31 <dir> SkyDrive

26/05/2014 21:05 <dir> Source

13/11/2014 20:55 <dir> Videos

15/12/2014 17:48 <dir> VirtualBox VMs

08/01/2015 16:28 <dir> workspace

0 File(s) 0 bytes

44 Dir(s) 259,190,337,536 bytes free

C:\Users\slyth_000>cd invoicing-app

C:\Users\slyth_000\invoicing-app>dir

Volume in drive C has no label.

Volume Serial Number is F6C8-3FA8

Directory of C:\Users\slyth_000\invoicing-app

10/02/2015 13:38 <dir> .

10/02/2015 13:38 <dir> ..

10/02/2015 13:38 <dir> .cordova

10/02/2015 13:38 4,881 config.xml

10/02/2015 13:38 <dir> hooks

10/02/2015 13:38 <dir> platforms

10/02/2015 13:38 <dir> plugins

10/02/2015 13:38 <dir> www

1 File(s) 4,881 bytes

7 Dir(s) 259,178,242,048 bytes free

C:\Users\slyth_000\invoicing-app>phonegap run android

[phonegap] executing 'cordova platform add android'...

Creating android project...

C:\Users\slyth_000\.cordova\lib\npm_cache\cordova-android\3.6.4\package\bin\node

_modules\q\q.js:126

throw e;

^

Error: Failed to run "android". Make sure you have the latest Android SDK instal

led, and that the "android" command (inside the tools/ folder) is added to your

PATH.

at C:\Users\slyth_000\.cordova\lib\npm_cache\cordova-android\3.6.4\package\b

in\lib\check_reqs.js:43:27

at ChildProcess.exithandler (child_process.js:751:5)

at ChildProcess.emit (events.js:110:17)

at maybeClose (child_process.js:1008:16)

at Socket.<anonymous> (child_process.js:1176:11)

at Socket.emit (events.js:107:17)

at Pipe.close (net.js:476:12)

Error: C:\Users\slyth_000\.cordova\lib\npm_cache\cordova-android\3.6.4\package\b

in\create.bat: Command failed with exit code 1

at ChildProcess.whenDone (C:\Users\slyth_000\AppData\Roaming\npm\node_module

s\phonegap\node_modules\cordova\node_modules\cordova-lib\src\cordova\superspawn.

js:135:23)

at ChildProcess.emit (events.js:110:17)

at maybeClose (child_process.js:1008:16)

at Process.ChildProcess._handle.onexit (child_process.js:1080:5)

[phonegap] executing 'cordova run android'...

No platforms added to this project. Please use `cordova platform add <platform>`

.

C:\Users\slyth_000\invoicing-app>cordova platform add

'cordova' is not recognized as an internal or external command,

operable program or batch file.

C:\Users\slyth_000\invoicing-app>dir

Volume in drive C has no label.

Volume Serial Number is F6C8-3FA8

Directory of C:\Users\slyth_000\invoicing-app

10/02/2015 13:38 <dir> .

10/02/2015 13:38 <dir> ..

10/02/2015 13:38 <dir> .cordova

10/02/2015 13:38 4,881 config.xml

10/02/2015 13:38 <dir> hooks

10/02/2015 13:38 <dir> platforms

10/02/2015 13:38 <dir> plugins

10/02/2015 13:38 <dir> www

1 File(s) 4,881 bytes

7 Dir(s) 258,813,038,592 bytes free

C:\Users\slyth_000\invoicing-app>

Comment 5 (In reply to #4) by Raymond Camden posted on 2/10/2015 at 2:44 PM

This comment is not really on topic for the blog post. For generic Bluemix support, you should consider our developerworks site: https://developer.ibm.com/a...

Also, in the output above, I see * multiple* messages telling you that you have things missing. I'd suggest reading it closely.

Comment 6 by Matt Hill posted on 3/7/2015 at 5:58 AM

Hi Ray - great post! And welcome to IBM! I'm part of the team that developed the Visual Recognition service and it's great to see people using it! If there's anything you'd like to see in there, I'd be happy to chat sometime.

Comment 7 (In reply to #6) by Raymond Camden posted on 3/7/2015 at 12:01 PM

Glad you liked it. I don't honestly remember any issues I had with it. It just plain worked, which is nice. :)