Interestng CFHTTP issue

This post is more than 2 years old.

Here is an interesting issue that came from a reader. He is using Yahoo's stock quote system to get data using CFHTTP. Here is the code he used:

<cfhttp name="Stocks" method="get" delimiter="," firstRowAsHeaders="false" url="http://quote.yahoo.com/d/quotes.csv" columns="stockID,IndexValue,TradeTimeDate,TradeTimeTime,ChangePoint,OpenValue,En dingDayRange,OpeningDayRange,DoNotKnow1,MarketCap,PreviousClose,ChangePercent,Ye arRange,EarningPerShare,EarningRatio,CompanyName" throwonerror="yes"> <cfhttpparam type="Header" name="charset" value="ISO-8859-1" /> <cfhttpparam name="s" value="#form.symbol#" type="formfield" encoded="no" /> <cfhttpparam name="f" value="xxx" type="formfield" /> </cfhttp>

I've not used the Yahoo API before, but it should be pretty obvious what the above does. Pass in a symbol via a form variable and parse the result into a query. Note - the "f" param was a longer string and I was concerned that it may be a private key. I changed it to xxx just to be safe.

So what's the problem? When he used ^DJI as a symbol, the value ended up being escaped. Yahoo's result indicated that it didn't recognize the value, which makes sense as it was no longer ^DJI, but had changed to %5EDJI. As you can see, he did pass encoded="no", but this was ignored. I thought perhaps it was the use of GET as the cfhttp operation. Changing this to POST did not help. Nor did changing the cfhttpparam type to URL. Nothing seemed to work right.

I tried a few different combinations, and finally got it working, but to be honest, I have no idea why this works. Consider the new version (and again, 'f' has been changed in case it is a private variable):

<cfhttp method="post" url="http://quote.yahoo.com/d/quotes.csv?s=#urlEncodedFormat(form.symbol)#&f=xxx" throwonerror="false" columns="stockID,IndexValue,TradeTimeDate,TradeTimeTime,ChangePoint,OpenValue,EndingDayRange,OpeningDayRange,DoNotKnow1,MarketCap,PreviousClose,ChangePercent,YearRange,EarningPerShare,EarningRatio,CompanyName" delimiter="," name="Stocks" firstRowAsHeaders="false" > <cfhttpparam type="Header" name="charset" value="ISO-8859-1" /> <cfhttpparam name="s" value="#form.symbol#" type="formfield" /> <cfhttpparam name="f" value="xxx" type="formfield" /> </cfhttp>

Notice I've supplied the s and f values both in the URL and in the form field. This worked perfectly, but as I said, I've got not idea why. Anyone want to take a guess?

Raymond Camden's Picture

About Raymond Camden

Raymond is a senior developer evangelist for Adobe. He focuses on document services, JavaScript, and enterprise cat demos. If you like this article, please consider visiting my Amazon Wishlist or donating via PayPal to show your support. You can even buy me a coffee!

Lafayette, LA https://www.raymondcamden.com

Archived Comments

Comment 1 by Joshua Cyr posted on 4/30/2009 at 9:40 PM

so if you remove the f and s form params and keep just in the URL sting it doesn't work?

I would have guessed that the URL string forces it to decode and thus work properly vs form which does not. but it would be odd that it would require both url and form to force that behavior.

Comment 2 by Raymond Camden posted on 4/30/2009 at 9:44 PM

I may not have made it clear. I needed both args, in both places, for it work. It didn't work w/o them both there.

Comment 3 by Ben Nadel posted on 4/30/2009 at 9:47 PM

That's bananas.

Comment 4 by Joshua Cyr posted on 4/30/2009 at 9:56 PM

My bet is that they do a check for form fields in their code or reject it, but actually let the URL string work.

Try putting junk in the form field part.

Comment 5 by Mike posted on 4/30/2009 at 9:57 PM

The "f" param tells Yahoo! what information you want back from the query. More information here: http://www.gummy-stuff.org/...

Also to be clear this is not part of Yahoo!'s API, instead it is a free service as long as you state that the stock quotes are via Yahoo!

Comment 6 by Raymond Camden posted on 4/30/2009 at 9:58 PM

@JC: Nope, no go.

@Mike:Ah, sorry then. For folks who want to try my code, change F to: sl1d1t1c1ohgvj1pp2wern

Comment 7 by Rob Brooks-Bilson posted on 4/30/2009 at 10:06 PM

This is the same "service" that the cf_stockgrabber tag I wrote back in 1997 uses. Interestingly enough, the MX version of the tag still works today.

My code is a little different, but the CFHTTP call looks like this:

<code>
<cfhttp
method="get"
url="http://download.finance.yah..."
name="myQuery"
columns="Symbol,Company_Name,Last_Traded_Price,Last_Traded_Date,Last_Traded_Time,Change,Opening_Price,Days_High,Days_Low,Volume"
delimiter=","
textqualifier=""""
firstRowAsHeaders="no">
</code>

Going this way, there's no problem with escaped characters.

Comment 8 by Raymond Camden posted on 4/30/2009 at 10:11 PM

Interesting. Rob, did you notice your URL was different? When you switch the URL to Mikes, and use the rest of your code, it breaks again.

Comment 9 by Rob Brooks-Bilson posted on 4/30/2009 at 10:17 PM

Interesting indeed. I'm not sure where Mike's url came from. I've been using the one I posted forever. Tell him to switch ;-)

Comment 10 by Eric Belair posted on 4/30/2009 at 10:35 PM

Try changing the "type" attribute in teh CFHTTPPARAM tags to "url". This is how my code to retrieve Yahoo! Stock Quotes is written (also, it is sent as a "GET"), and I've never had a problem with any symbols being sent.

Comment 11 by Raymond Camden posted on 4/30/2009 at 10:49 PM

Eric, I'm pretty sure I tried that and it failed. Does it work for you using the code above (fixing 'f' of course).

Comment 12 by Dan G. Switzer, II posted on 5/1/2009 at 12:02 AM

It's because your using the POST method in the 2nd example. In the first sample, you're using GET--which will escape the variables automatically (since GET is sent with the URL--not in the body of the request.)

Comment 13 by Raymond Camden posted on 5/1/2009 at 12:05 AM

Dan, trust, I tried switching to POST in the first example. I think I even say that in the blog post, don't I?

Comment 14 by Dan G. Switzer, II posted on 5/1/2009 at 12:23 AM

Perhaps the private key is required via the URL scope and the stock is expected in the FORM scope.

It might be the URL is just supposed to be:
http://quote.yahoo.com/d/qu...

Comment 15 by Raymond Camden posted on 5/1/2009 at 12:24 AM

Nope, see above comments. I was wrong about f. F helps define the results.

Comment 16 by JC posted on 5/1/2009 at 12:39 AM

That "F" string is actually telling it which fields you want... there's no special codes, you can get the URL just by going to the yahoo finance site and saying you want to download the info.

same service, this code works and has for ages:

<cfparam name="attributes.symbols" default="^DJI">
<cfhttp URL="http://finance.yahoo.com/d/..."
METHOD="GET"
PORT="80"
RESOLVEURL="true"
timeout="120">

I'll cut most of the rest out to keep it short.

<cfset detailarray = listtoarray(cfhttp.filecontent)>
<cfscript>
symbol = detailarray[1];
date = detailarray[2];
tradetime = detailarray[3];
lasttrade = detailarray[4];
askprice = detailarray[5];
bidprice = detailarray[6];
start52 = detailarray[7];
end52 = detailarray[8];
daystart = detailarray[9];
dayend = detailarray[10];
volume = detailarray[11];
previousclose = detailarray[12];
change = detailarray[13];
lastopen = detailarray[14];
</cfscript>

Comment 17 by JC posted on 5/1/2009 at 12:41 AM

err

URL="http ://finance.yahoo.com/d/quotes.... s=#attributes.symbols# &f=sd1t1l1abjkghvpc1o &e=.csv"

strip the spaces out

Comment 18 by JC posted on 5/1/2009 at 12:44 AM

sorry to post too much... but here:

http://finance.yahoo.com/q?s=^DJI

righthand side, beneath the chart, it says "Download Data" and that's where this link comes from:
http://download.finance.yah...

it's not a post, it's a get. Just a plain old link on a webpage.