Adding voice-based search to a PhoneGap app

This post is more than 2 years old.

A few weeks back I blogged about plugins under PhoneGap. If you haven't read that blog entry, take a quick look at it. For the most part, plugins in PhoneGap are pretty simple. Download a Java file. Download a JavaScript file. Make one tweak to an XML file and you're good to go. In that entry I made use of the SpeechRecognizer plugin for Android. Since that blog entry was kind of a joke (ok, most of my blog entries contain code that is a kind of a joke - but let's forget about that for a moment), I thought it might be nice to demonstrate a more real world use for the plugin. With that in mind here's a simple application built without and with speech recognition.

I began with a simple idea - an application that would allow you to quickly search for images. Turns out the Google Image API is deprecated, but Bing still has a valid API. Not only that, it works really well! About the only weird thing i had with the API was Microsoft's... unusual... capitalization of stuff. That being said, it didn't take long to build up a simple form and tie it to their API. Here is the HTML for the page.

<!DOCTYPE HTML>
<html>
<head>

<meta name="viewport" content="width=320; user-scalable=no" />
<meta http-equiv="Content-type" content="text/html; charset=utf-8">
<title>PhoneGap</title>
<script type="text/javascript" charset="utf-8" src="js/phonegap-1.2.0.js"></script>
<script type="text/javascript" charset="utf=8" src="js/jquery-1.7.min.js"></script>
<script type="text/javascript" charset="utf=8" src="js/main.js"></script>
<style>
input {
	width:100%;
	padding: 10px;
}

img {
	display:block;
	margin-left:auto;
	margin-right:auto;
}
p {
	display:block;
	margin-left:auto;
	margin-right:auto;
	width: 80%;
}
</style>
</head>

<body onload="init()">

<input type="search" id="searchField" value="star wars">
<input type="button" id="searchButton" value="Search">

<div id="results"></div>

</body>
</html>

Nothing special there - just a form field and a button. Btw - Pro Tip here - when working on web applications do not be shy about defaulting form fields as I've done above. You get real sick and tired of re-entering text input after you've run your application a few hundred times. Obviously the default value there would be removed before release. Here's a quick shot of how it looks.

Now let's look at the code....

var appid = "5252D701A7CE4B4F3C190F1403D2181F2C330F2E";

function init() {
	document.addEventListener("deviceready", deviceready, true);
}

function deviceready() {
	console.log('loaded');
	
	$("#searchButton").bind("touchstart",function() {
		var s = $.trim($("#searchField").val());
		console.log("going to search for "+s);

		$.getJSON("http://api.search.live.net/json.aspx?Appid="+appid+"&query="+escape(s)+"&sources=image&image.count=20", {}, function(res) {
			var results = res.SearchResponse.Image.Results;
			if(results.length == 0) {
				$("#results").html("No results!");
				return;
			}
			var s = "";
			for(var i=0; i<results.length; i++) {
				s+= "<p><img src='"+results[i].Thumbnail.Url+"'><br/><a href='"+results[i].Url+"'>"+results[i].DisplayUrl+"</a></p>";				
			}
			$("#results").html(s);
		});

	});
}

Again - nothing terribly complex here. Bind to the button - check the value - and hit Bing. Here's an example of the result.

Ok, so far so good. So let's add speech recognition! As I said above, there is a process that plugins follow. It involves getting a Java file, a JavaScript file, and editing your plugins.xml file. The SpeechRecognizer page actually documents this well. Before I go into the code though I ran into a brick wall.

How do I handle the UI?

That's a pretty important question, and I won't pretend to know the best answer to this. I decided to add a button to the left of the text field. This way the user could click the button or the text field if they didn't want to use the recognizer. I'm not saying this is the best answer but it seemed to work ok. Here's the updated HTML:

<!DOCTYPE HTML>
<html>
<head>

<meta name="viewport" content="width=320; user-scalable=no" />
<meta http-equiv="Content-type" content="text/html; charset=utf-8">
<title>PhoneGap</title>
<script type="text/javascript" charset="utf-8" src="js/phonegap-1.2.0.js"></script>
<script type="text/javascript" charset="utf=8" src="js/jquery-1.7.min.js"></script>
<script type="text/javascript" charset="utf-8" src="js/SpeechRecognizer.js"></script>
<script type="text/javascript" charset="utf=8" src="js/main.js"></script>
<style>
#micButton, #searchField {
	padding: 10px;
}

input[type=button] {
	width: 100%;
}

img {
	display:block;
	margin-left:auto;
	margin-right:auto;
}
p {
	display:block;
	margin-left:auto;
	margin-right:auto;
	width: 80%;
}
</style>
</head>

<body onload="init()">

<button id="micButton" disabled="disabled">SPEAK!</button> <input type="search" id="searchField" value="">

<input type="button" id="searchButton" value="Search">

<div id="results"></div>

</body>
</html>

And the new UI....

Ok, now let's take a look at the code.

var appid = "5252D701A7CE4B4F3C190F1403D2181F2C330F2E";

function init() {
	document.addEventListener("deviceready", deviceready, true);
}

function deviceready() {
	console.log('loaded');
	
	window.plugins.speechrecognizer.init(speechInitOk, speechInitFail);
	
	function speechInitOk() {
		$("#micButton").removeAttr("disabled");
	}
	
	function speechInitFail(e) {
		//Since this isn't critical, we don't care...
	}
	
	$("#micButton").bind("touchstart", function() {		
		var requestCode = 4815162342;
		var maxMatches = 1;
		var promptString = "What do you want?";
		window.plugins.speechrecognizer.startRecognize(speechOk, speechFail, requestCode, maxMatches, promptString);
	});

	function speechOk(result) {
		var match, respObj;
		if (result) {
			respObj = JSON.parse(result);
			if (respObj) {
				var response = respObj.speechMatches.speechMatch[0];
				$("#searchField").val(response);
				$("#searchButton").trigger("touchstart");
			}        
		}
	}

	function speechFail(m) {
		navigator.notification.alert("Sorry, I couldn't recognize you.", function() {}, "Speech Fail");
	}

	$("#searchButton").bind("touchstart",function() {
		var s = $.trim($("#searchField").val());
		console.log("going to search for "+s);

		$.getJSON("http://api.search.live.net/json.aspx?Appid="+appid+"&query="+escape(s)+"&sources=image&image.count=20", {}, function(res) {
			var results = res.SearchResponse.Image.Results;
			if(results.length == 0) {
				$("#results").html("No results!");
				return;
			}
			var s = "";
			for(var i=0; i<results.length; i++) {
				s+= "<p><img src='"+results[i].Thumbnail.Url+"'><br/><a href='"+results[i].Url+"'>"+results[i].DisplayUrl+"</a></p>";				
			}
			$("#results").html(s);
		});

	});
}

So first off, we have to see if we can even do recognition. Therefore I've got a call to init it in my device ready block. If it works, we remove the disabled attribute from the button. The button has an event handler that will fire off the request to the device. The request code is random so don't worry about it too much. Once clicked, the device will prompt you to speak:

In the result handler, we can simply then grab the response object and assume the first result is what we want. Notice I automatically trigger the search. That may or may not be a good idea. You may want the user to have a chance to confirm the recognition first. I thought it was kind of cool to have it automatically search though. Here's an example - and yes - I did say "red banana":

That's it. If you want to play with this, I've included a zip of the Eclipse project. You will find the first draft in the assets folder as "www - Copy". You will also find an APK you can install. Note - I had JDK issues with the SpeechRecognizer Java file. So the version in my zip is slightly edited to get around that.

Download attached file.

Raymond Camden's Picture

About Raymond Camden

Raymond is a senior developer evangelist for Adobe. He focuses on document services, JavaScript, and enterprise cat demos. If you like this article, please consider visiting my Amazon Wishlist or donating via PayPal to show your support. You can even buy me a coffee!

Lafayette, LA https://www.raymondcamden.com

Archived Comments

Comment 1 by Mark Fuqua posted on 12/3/2011 at 4:02 AM

I guess the Air examples are over? PhoneGap or bust?

Comment 2 by Raymond Camden posted on 12/3/2011 at 4:04 AM

I don't think AIR is over, but I know AIR. Right now my focus is on something new. :)

Comment 3 by Alain posted on 12/5/2011 at 1:13 PM

Nice example Ray, thanks for sharing it.
I think you can use input type text with x-webkit-speech in order to get a nice HTML5 form suited for speech input.

Comment 4 by Raymond Camden posted on 12/5/2011 at 4:39 PM

Holy crap - for real? I didn't even know that existed. Going to test it today.

Comment 5 by Raymond Camden posted on 12/13/2011 at 3:40 AM

I finally got around to looking at this - looks to be Android only though.

Comment 6 by Martin posted on 1/27/2012 at 12:23 AM

hi - nice tutorial, but I am already running into speechInitFail at the initialization

window.plugins.speechrecognizer.init(speechInitOk, speechInitFail);

I configured everything described on github, using android 2.2.1 - phonegap 1.3 - best regards

Comment 7 by Martin posted on 1/27/2012 at 12:24 AM

any ideas?

Comment 8 by Raymond Camden posted on 1/27/2012 at 12:28 AM

Heh, I love how you give me one minute to respond. ;) (Just joking - looking at your comments above though it looks like you got a bit impatient. ;) So what do you see in the object passed to speechInitFail?

Comment 9 by Martin posted on 1/27/2012 at 3:32 PM

Hey Raymon,

haha, sorry, I thought I just add any ideas because I forgot it in the first post, - the alert tells me,

function speechInitFail(e) {
alert(e);

-> Speech recognition is not present or enabled

regards

Comment 10 by Martin posted on 1/27/2012 at 3:51 PM

Hey Raymond,

it's me again - did you try the idea of Alain with x-webkit-speech? looks like it's just working on chrome?

martin

Comment 11 by Raymond Camden posted on 1/27/2012 at 4:54 PM

So unless I'm missing something, isn't the error pretty obvious? Your device is saying it doesn't have that ability.

To your second - yes - I too believe it is only Chrome.

Comment 12 by Martin posted on 1/27/2012 at 6:48 PM

hey Raymond,

Thx for the answer, - I thought if I can record sound over the record functionality this plugin is also working, so you think on 2.2.1 is not possible..
regards

Comment 13 by Raymond Camden posted on 1/27/2012 at 7:08 PM

It may be less OS and more hardware. Do you have that feature on your phone? On mine, it is tied to Google search.

Comment 14 by Martin posted on 1/27/2012 at 7:14 PM

it's on the tablet: http://www.archos.com/produ...

it's kind of old but should work normally ;)

g - search is installed

regards

Comment 15 by Raymond Camden posted on 1/27/2012 at 7:19 PM

Then it may indeed be a version thing. I don't remember what I used, but I think I normally use 2.3.

At the end of the day, this is not going to work for you unfortunately. You may want to reach out to the plugin author. If the plugin says no, I can't help you there. ;)

Comment 16 by Martin posted on 1/27/2012 at 9:08 PM

alright, thx for helping !

regards

Comment 17 by quickfix posted on 5/21/2012 at 3:05 PM

Hi, have you been able to use the SpeechRecognizer plugin with recent versions of phonegap?

I'm trying with no success to make it work with Phonegap 1.7.
I tried to change the references to "Phonegap" to "cordova" both in the js and the java files but I still have no answer from the init function.

Is https://github.com/phonegap... the right place to find plugins since the project name change?

Comment 18 by Raymond Camden posted on 5/21/2012 at 3:13 PM

I forwarded your message to Scott Stroz who updated my Eliza demo (I believe) for PG 1.6.

Comment 19 by Raymond Camden posted on 5/21/2012 at 5:53 PM

Ok, he was NOT successful. Right now the best I can recommend is reaching out to the author of the plugin.

Comment 20 by quickfix posted on 5/21/2012 at 6:03 PM

Ok, thanks for your help and for your very interesting blog.

It's quite sad that most of the phonegap plugins seem unmaintained since a few months and very fewq use the cordova namespace (actually I only found one for android). Maybe they're waiting for the 2.0.

Comment 21 by Raymond Camden posted on 5/21/2012 at 6:05 PM

Ah, the joy of open source. ;) If you are familiar with Java I'm sure the author would take a pull request. :)

Comment 22 by quickfix posted on 5/21/2012 at 10:03 PM

Eventually I managed to make it work with phonegap 1.7.
Once the name change and a function call change done, the last thing I was missing was also the dumbest one : in the readme of the plugin they instruct to add <script type="text/javascript" charset="utf-8" src="speechrecognizer.js"></script> in the html file and the js name is SpeechRecognizer.js (the case did not match).
Stupid me not to have noticed earlier!

Comment 23 by Raymond Camden posted on 5/21/2012 at 10:09 PM

Don't feel stupid - I think we've all done that a thousand times. I absolutely loathe case-sensitive file systems. Who would want to have two files in one directory with the same name, but different case?

Comment 24 by Hafeez posted on 8/6/2012 at 6:36 AM

I was wondering if we want to do a location based search ie. searching "chinese food" yielding results what is the closest restaurant to the current phone geolocation and moving further away. any hints or ideas how to go implement it. Thanks for any help.

Comment 25 by Raymond Camden posted on 8/6/2012 at 3:28 PM

Simple - check out Google Places: https://developers.google.c...

I've used their API before - and in a PhoneGap app (Ineedit).

Comment 26 by Hafeez posted on 8/6/2012 at 4:24 PM

Thanks for the link ... was good help

Comment 27 by Hafeez posted on 8/9/2012 at 10:34 PM

Hi Raymond.. I have another question.. I am using the code as for google places with out voice recognition. Tested the API key it was returning json. problem is I am not able to get that data to display to the user.

$.getjson("https://maps.googleapis.com...""+escape(s)+""&sensor=true&key=AIzaSyBspIWJUaqb79ZIJG9QQFDSh8ZM89FxbG4&rankby=distance", {}, function(res) {
var results = res.SearchResponse.Results;
if(results.length == 0) {
$("#results").html("No results!");
return;
}
var s = "";
for(var i=0; i<results.length; i++) {
s+= results[i].formatted_address+<br/><a href='"+results[i].Url+"'>"+results[i].DisplayUrl+"</a>";
}
$("#results").html(s);
});

It does nothing. I am a novice in phonegap develpoment. Can you please help me where I am going wrong..

Comment 28 by Raymond Camden posted on 8/9/2012 at 10:50 PM

Are you getting results? Have you tried doing a console.log on res?

Comment 29 by Hafeez posted on 8/9/2012 at 11:02 PM

Sorry I might becoming a bother for you but it is really important for me to earn.. when I use the API key in a browser i am getting a page with json.
sorry it was an old key..
this is the new url
https://maps.googleapis.com..."

I really appreciate your help.. your are like a guru for me.

Comment 30 by Hafeez posted on 8/9/2012 at 11:03 PM

typo ..it was learn.. not earn..

Comment 31 by Raymond Camden posted on 8/9/2012 at 11:05 PM

As I said though - you want to check in your console to see what results - if any - are coming back.

Comment 32 by Hafeez posted on 8/9/2012 at 11:21 PM

Checked in the console.. got an error
uncaught syntax error: unexpected string
below the getjson

$.getjson("https://maps.googleapis.com...""+escape(s)+""&sensor=true&key=AIzaSyB9jAgjipRv4yG1Jpn95acRzcqR94nZrLo&rankby=distance", {}, function(res) {
Uncaught SyntaxError: Unexpected string
var results = res.SearchResponse.Results;

Comment 33 by Raymond Camden posted on 8/9/2012 at 11:22 PM

Posting code here is probably a bad idea. I'd suggest pastebin. It looked like you had two quotes though. Try pastebin and then post back the URL.

Comment 34 by Hafeez posted on 8/9/2012 at 11:42 PM

the html code is basically what have for just the search no voice elements added. I am sorry for pasting the code like that.. this is the pastebin http://pastebin.com/61UAftgc
I really appreciate you taking time from your busy schedule..

Comment 35 by Raymond Camden posted on 8/9/2012 at 11:46 PM

In your pastebin 21 line is:

Uncaught SyntaxError: Unexpected token <

Is that really in your source code?

Comment 36 by Hafeez posted on 8/9/2012 at 11:49 PM

it was from the console..

Comment 37 by Raymond Camden posted on 8/9/2012 at 11:51 PM

Um - ok - but you need to post your _code_. Not the output from the console. That's helpful, but we need to look at the code first.

Comment 38 by Raymond Camden posted on 8/9/2012 at 11:52 PM

Oh I see it:

s+= results[i].formatted_address+<br/><a href='"+results[i].Url+"'> "+results[i].DisplayUrl+"</a>";

This is wrong. No " in front of <br/>

Comment 39 by Hafeez posted on 8/10/2012 at 12:25 AM

Thanks for the help. I corrected it. No errors in the console now but still nothing seems to happen.. this is the html code.. http://pastebin.com/QdYD98Gz
please if you have time can you see where I am making a mistake..I really appreciate your help.

Comment 40 by Raymond Camden posted on 8/10/2012 at 12:36 AM

Go back to my original suggestion - do a console.log on the result from the ajax call.

Comment 41 by Hafeez posted on 8/10/2012 at 2:33 AM

Thanks for your valuable input Raymond..It seems the google places API does'nt work this way with jscript. In the google docs they mentioned we need to load the places library and make service call.. back to the drawing board... Thanks again for your help.

Comment 42 by Marcio posted on 11/10/2012 at 3:14 AM

I'm newbie on PhoneGap... It's possible to run a mobile web app directly on my url, using an Android tablet?

Thanks

Comment 43 by Raymond Camden posted on 11/10/2012 at 3:17 AM

By definition, "web app" implies a web page, so yes, but you may not be grokking how PhoneGap is different. PhoneGap packages up HTML (plus JS and CSS of course) into a native application you install on your device.

Comment 44 by luca posted on 10/30/2013 at 2:19 PM

Thanks for sharing this, interesting&useful.
Is there any update to this method with iOS7 (and recent versions of PhoneGap/Cordova)?

Cheers

Comment 45 by Raymond Camden posted on 10/30/2013 at 2:24 PM

You would need to check with the plugin author to see if they plan to support Cordova 3. I believe Simon Macdonald is making a compatible plugin. I'm not seeing his Github repo at the moment though.

Comment 46 by Kresten posted on 12/17/2013 at 7:10 PM

I stumbled upon this blog only recently and it helped me a lot in the end so thanks! At first it didn't work for me, because "module was undefined" with Cordova 3. I changed the .js file to fix this with the help of this post: https://github.com/phonegap....

Hope it will save somebody some time!

Comment 47 by Babulal rathod posted on 9/13/2014 at 10:48 AM

i am new to phonegap app what is that "var appid = "5252D701A7CE4B4F3C190F1403D2181F2C330F2E"?
how to get it?

Comment 48 by Raymond Camden posted on 9/13/2014 at 5:21 PM

That's a key for the Bing API.

Comment 49 by Stanly Stephen posted on 5/14/2020 at 11:56 AM

This don't work , can you confirm since.that plugin for GitHub is 404

Comment 50 (In reply to #49) by Raymond Camden posted on 5/14/2020 at 1:27 PM

Looks like it is gone. I'd recommend what I'd do - Google for it.

Comment 51 (In reply to #50) by Stanly Stephen posted on 5/14/2020 at 1:54 PM

:)