What follows comes from a lot of sources, and is not something I'm 100% solid on mentally, so if I screw anything up here, blame me, not the group of guys who helped me out. A few weeks ago I looked into writing an API for a very cool service called Zencoder. Zencoder is an API based video encoding service. Basically think of it as an easier way to do ffmpeg type work. They aren't free, but from what I can tell of their API, it seems extremely well thought out and like a great service to consider. I began working on a ColdFusion wrapper (which, by the way, will be happening now that I've gotten past my issues) for their service. I immediately noticed a problem. Whenever I did a cfhttp against their API URL I got:
At this point I remembered that you sometimes have to do some command line work to get HTTPS sites working with ColdFusion. I followed some tutorials on Steven Erat's blog but this tech note is the one you want: How to import certificates into certificate stores (ColdFusion).
So I tried that - on multiple machines - but it never actually worked. At this point I did what all good developers do when stuck (and after Googling) - I punted. I posted on a private list serv and got some great responses. First, Simon Free suggested the following code:
Do your call <cfset objSecurity.insertProviderAt(storeProvider, 1) />
<cfset objSecurity = createObject("java", "java.security.Security") />
<cfset storeProvider = objSecurity.getProvider("JsafeJCE") />
<cfset objSecurity.removeProvider("JsafeJCE") />
I had no idea what this was doing, but it worked! Turns out - this comes down to a problem with the security provider used by ColdFusion. Nathan Mische has an indepth blog entry detailing how he ran into the issue and how he got around it. If you read his blog post you will also find out that this only impacts ColdFusion 9 Enterprise. I confirmed this myself. Switching from the Developer edition (which acts like Enterprise) to a Standard license makes the issue go away. Jason Dean then followed up with this good explanation.
This is definitely a bug in the BSafe CryptoJ libraries and one that I have been meaning to report to ASSET and PSIRT for some time. It is a tough one to articulate and demonstrate. Though I have suspected for a while that this would only affect CF Ent, this is the first time i have seen it confirmed.
One thing I did not see anyone point out here, so I will do it, is that the way that the Java Crypto Architecture (JCA) and Java Crypto Extensions (JCE) work is through a "Provider List" (Note my terminology may be off). When you request a crypto method from JCA/JCE it looks though the provider list to find the first provider that supports the algorithm you have requests. So, if I ask for CF to encrypt something using AES it will start search the providers to find one that supports AES.
In CF Enterprise, BSafe CryptoJ is the first provider in the list and it has almost EVERYTHING one would need so it is pretty much always the one used. Which makes CF FIPS-140 compliant by default (nice!).
But sometimes the BSafe library has issues with certain certs. Dunno why, but it does. It cannot deal with the BaseCamp cert, the Zencoder cert, an some others. I have not been able to find a pattern.
As Ray saw, when he dumped the list of providers available to JCA/JCE, the list was HUGE. I think CF Enterprise and the JVM have a combined totals of 11 or more crypto providers. The second provider in the list for CF Enterprise is the Sun Provider (Which is the first one in CF Standard) that Nathan mentioned. It is a perfectly good provider and will do everything we need it to, it's just not FIPS-140 compliant.
So when we disable the BSafe library using the JVM arg or with Simon's hack, we are not necessarily removing any security. Java will simply use the next available provider, which is fine.
The only place this would not be acceptable is when FIPS-140 compliance is mandatory.
So... interesting, right? My take from this is that a) you have multiple security providers in CF9/Ent and b) the default one is great 90% of the time... except when it isn't. ;) So my plans for the Zencoder API are to abstract out the HTTP calls and wrap them in Simon Free's fix when CF9/Ent or Dev is encountered. I will also document some of this in the package so folks can remove my workaround if they wish to do the change at the JVM level. Pete Freitag expressed concerned that this workaround might have drawbacks in production.
As always - thank to you my smart friends for the help (they are listed below), and hopefully this will help others if they run into it. Oh - and one more quick thing. I worked with Zencoder while debugging this and found them to be very responsive. If anyone is considering purchasing their service, I think this is a great sign.
Credit
- Simon Free
- Nathan Mische
- Jason Dean
- Brian Meloche
- Pete Frietag
- Todd Sharp
Archived Comments
We started encountering the same error with code that was working fine up to that point. It was to connect to 37Signal's Highrise API in our case. I never found a solution, and we ended up discontinuing our connection to that service.
I was hopeful for a moment that this could be the cause/solution at first, until I got to the part about it being CF9 ent/dev only. This was CF8 standard.
NBD since we've already killed that offering in our code, but I would have been interested to know what went down.
We had the same issue with cfhttp having problems authenticating and we ended up using a 3rd party C++ tag called cfx_httpd which has been working great.
Correction to my last post, the tag is called cfx_http5
Hey Ray,
Awesome! Thanks for posting that. I actually had that same problem with another API I was using, which I used the provider removal solution. I didn't think to link it to the same problem when I was trying to use the Zencoder API.
With the Zencoder API, I actually wrote it to use cURL; which works fine, but not a very nice solution to something that should just work. The call itself is abstracted out, so it will be an easy change to use your fix.
Since you're working on a Zencoder API, I have quite a bit of one already written. So it would be nice to combine efforts.
Shoot me an email if you're interested in working on the Zencoder API together.
Todd
No reason to duplicate efforts. I don't have your email address so use my contact form and I'll reply to that. Thanks!
@Ray
Had the same problem on CF7 with paypal.
Those errors are tough for me to debug because I don't have so many smart friends and the error is pretty terse.
PS - New gravatar is awesome. Love the beard.
I've narrowed this down a bit further, and removing the KeyAgreement.DiffieHellman from the RSA JsafeJCE provider (which causes the default sun implementation to be used instead) seams to work, and probably has less of an effect on your server than removing the entire provider would. Here's how you do it:
<cfset objSecurity = createObject("java", "java.security.Security") />
<cfset storeProvider = objSecurity.getProvider("JsafeJCE") />
<cfset dhKeyAgreement = storeProvider.getProperty("KeyAgreement.DiffieHellman")>
<!--- dhKeyAgreement=com.rsa.jsafe.provider.JSA_DHKeyAgree --->
<cfset storeProvider.remove("KeyAgreement.DiffieHellman")>
Do your http call, but pack the key agreement if you want:
<cfset storeProvider.put("KeyAgreement.DiffieHellman", dhKeyAgreement)>
I figured this out by using the SSLSocketFactory to create a https connection, which provided a bit more details in the stack trace, than when using cfhttp:
yadayadayada
Caused by: java.security.InvalidKeyException: Cannot build a secret key of algorithm TlsPremasterSecret
at com.rsa.jsafe.provider.JS_KeyAgree.engineGenerateSecret(Unknown Source)
at javax.crypto.KeyAgreement.generateSecret(DashoA13*..)
at com.sun.net.ssl.internal.ssl.DHCrypt.getAgreedSecret(DHCrypt.java:166)
Would be great if the exception thrown from ColdFusion was a bit less generic.
After speaking with Adam Lehman (The ColdFusion Product Manager) about this issue, he encouraged me to log a detailed bug. I have done so. Please take the time to go vote on it and provide additional comments.
http://cfbugs.adobe.com/cfb...
"At this point I did what all good developers do when stuck (and after Googling) - I punted. I posted on a private list serv and got some great responses."
What you mean you don't do what most other CF devs do - send you an emai? ;)
Anyhow thanks for the excellent post, it came in handy today - I was having a similar problem.
regards,
larry
I tried emailing me but I took too long to respond.
May have something to do with the beard. He probably didn't recognize you...
Thank you so much for sharing this! I cannot tell you how long I've pulled my hair out over this. I used this solution to fix a broken Basecamp integration.
Thank you thank you thank you!
Has anyone heard of any proper fixes for this?
Adding the cert to the CF cert store is not the best solution for one of my apps.
Our API provider just updated their cert causing our app to break.
I am using cfhttp5 as a temp solution. Any experiences with cfhttp5?
David,
Needing to have the cert in the store is not a "bug", per se. It is a security feature of Java. There will not be a fix for it because a "fix" would work around the security.
I suppose one way you could fix it is by getting your certificate from a Certificate Authority that already has their root cert in the JVM keystore.
NOTE: Needing the keystore in the runtime CA certs file is a COMPLETELY different issue fromt he bug that I reported (linked in my comment above) where certain certs do not work even if you have added them to the cacerts file. That issue is a bug and has been reported.
Jason,
That makes sense that it is NOT a bug. However still creates the problem I just faced.
We have a partner that we use an API with and by renewing their cert... it breaks our app.
Is there a way to check/add using cf that you know of?
This would be a great benefit to this app in particular.
What I would encourage you to do is to note the expiration date of that cert and mark your calendar.
I would also consider seeing if that API provider offers a notification service to those that use it so they can be informed of such changes. If not, scold them and tell them they should offer one.
This is most definitely not a CF9 only bug. I experience this on CF8 both developer and enterprise editions.
You can vote up this bug in Adobe's tracker: http://cfbugs.adobe.com/cfb...
And Jason Dean's later, more specific bug:
http://cfbugs.adobe.com/cfb...
Guys, none of this worked for me UNTIL I saved the certificate as a DER-encoded binary file...as per this instruction http://www.java-samples.com...
Ray, thank you for sharing this, but I am getting VERY odd results. I am literally going crazy as a result of this bug.
For an entire day I was trying to integrate with the Spreedly.com API, and no matter what I did (importing certificates, etc.) I got the original "PEER could not authenticate" message.
Then, I finally came across this post, plugged in the workaround (tried both Simon Free's and Pete Freitag's), and it worked great for 1 day exactly.
Then, the workaround stopped working again. Then it started again. Then it stopped again. I cannot get any consistent results from this thing! I don't know if my requests are being routed to different servers on Spreedly's end that are somehow configured differently, or what, but I know this is the bug.
I would just leave the "fix" in all the time, but when I do and it's not required, it breaks the application. When I take it out, and it is required, it breaks the application.
I have setup the simplest possible test page here, without workaround: [can't post link, thinks it's spam]
And with workaround: [can't post link, thinks it's spam]
On my production server it seems to always work with the workaround, but on my development machine, it swaps back and forth, but now more often than not it works without the workaround. This stuff is over my head, unfortunately. Does anyone have any sugggestions?
Thanks,
Brian
Not sure what to suggest. As I said - this was all a bit confusing. If the work around causes an issue when it's used all the time, what about wrapping the work around in a try/catch where you ignore the error (if and only if it the 'error cuz i wasnt neeeded' type error).
Thank you for the suggestion, Ray. No worries, what had me all confused was that it works one way on my production server and another way on my dev machine. But hey, as long as it works on my production server that's what matters, right?
Hopefully Adobe will address the bug report to avoid future hassles. Thanks again for the reply.
Brian
fwi, I'm not sure why BlogCFC is flagging my comments as spam, but for some reason it won't let me put a website in the web site field
Sorry about the issues you have with your URL. You are the second person to note that. My spam checking is effective - but a bit aggressive. I use Honeypot and Akismet so it's pretty strong.
i'm trying this method with cfhttp but i'm getting error "variable storeprovider is undefined..." please assist
You need to provide more detail. I assume you followed Pete's comment, since his mentions that variable. Perhaps you can use Pastebin to share the code you tried.
argh.. sorry for the repetitive posts
http://pastebin.com/raw.php...
(I deleted your mistake comments.)
Well it looks like the API returning storeprovider returned null instead. I'll have to lean on Pete, but if he can't respond, then I'd suggest looking at the JavaDocs and see if they explain why the method would return null.
@Jason:
I'm struggling here with the same problems and have a hint for you're missing storeProvider.
If you have set the java.args Flag "-Dcoldfusion.disablejsafe=true" in your jvm.config then the JsafeJCE Provider its not loaded and therefore will be missing. See the following page in the docs talking about "Disable the FIPS-140-only cryptography setting": http://livedocs.adobe.com/c...
I'm working here on my local machine with the ColdFusion 9.0.1 Built-In Webserver and SSL and had to add this flag since a https-request without this setting seems to be possible in combination with the built-in web server :(
I'm having a problem with this.
Trying to do a CFHTTP to a https URL, but the security certificate is not valid.
I keep getting: I/O Exception: peer not authenticated
I added the lines you mentioned above and it now give me:
Variable STOREPROVIDER is undefined. at variables.objSecurity.insertProviderAt(storeProvider, 1)
I have the same issue. STOREPROVIDER is undefined.
I changed objSecurity.getProvider("JsafeJCE") to objSecurity.getProviders() and dumped it. There is no JsafeJCE. I'm running CF 9,0,1,274733 Standard Edition. Also tried to import my cert into /opt/coldfusion9/runtime/jre/lib/security/cacerts. Still get I/O error. Any idea?
Having looked at the cacerts file with the free gui tool KeyStore Explorer, I can't see any good reason why I shouldn't import the certificates on my Dev system, then copy the cacerts file across to Test and Production. It would be nice if you didn't have to restart Coldfusion though.
Hello,
trying to run the given code snippet, but:
<cfset storeProvider = objSecurity.getProvider("JsafeJCE")/>
seems to be doing nothing as I get "storeProvider undefined" on trying to add the provider back;
<cfset objSecurity.insertProviderAt(storeProvider, 1)/>
Any help or suggestion please ...
Also, can I work around by attaching the cert/pfx file with my CFHTTP request, if the above method fails on my machine.
<cfhttp url="https://..." clientsert="path/to/pfx">
Thanks
Hello,
I have to call both https://buy.itunes.apple.com and https://sandbox.itunes.appl... via CF 8 with CFHTTP. The first URL is working fine, because I imported the corresponding certificate into the keystore. The second one does not word, but gives this error:
"Connection failure - I/O Exception: Name in certificate `buy.itunes.apple.com' does not match host name `sandbox.itunes.apple.com'"
As the same certificate is being used by Apple for both Subdomains, I have no idea how to overcome this. Is there any possibility to fix this issue?
Best regards,
Rüdiger
Anyone know if this issue is still an issue in CF 10?
Thanks,
Matthew
I'm having the same problem as Patrick -- basically the opposite of the initial post. I have a development version of CF9 on my desktop. I imported the certs using keytool. Everything works great. I do the same on the "closer to production" server and I get the peer not authenticated issue. Both are patched to the same version but the failed server is running CF standard.
Patrick, did you ever solve your issue. Or does anyone have further ideas?
I found my issue. As per usual, a site or two after a made this post had the answer.
For my issue the fix was simple. Look at the Setting Summary page within the ColdFusion administrator and verify where the JRE home directory is. In my case, our production servers are not using the default CF9 JRE. Once I found this out, I used the keytool to import my certs, restarted CF, and presto - success.
@Pete - your solution on your comment here worked like a charm on CF 8 Developer edition!
http://www.raymondcamden.co...
Thanks, and see you around!
I added -Dcoldfusion.disablejsafe=true to my JVM Arguments, and no longer needed the other code provided. The statuscode returned as OK. Thanks for all of the help guys.
I've also found this error can occur from a wrong server date / time. In our case, the date was off by 2.5 years after a hardware swap. The date didn't get set b/c of strict PCI controls blocking outbound access.
Once we set the proper date and time, it started working again.
Thanks for this article. I added -Dcoldfusion.disablejsafe=true to JVM arguments but it didn't help. Then I opened C:\CF9\runtime\jre\lib\security\java.security and changed ssl.TrustManagerFactory.algorithm=PKIX to ssl.TrustManagerFactory.algorithm=SunX509
Now it works.
This started happening on my CF10 production servers a few days ago with Stripe connectivity which was working without any issue previously. I could not replicate on any of my development servers, so I looked a little deeper to see what was different on my CF installations other than platform and on the production server my JVM was at 1.7.0_17 while my development servers are at 1.7.0_15 and 1.6.0_29 and both worked without a problem. I updated my production server which is now at 1.7.0_45 and it started working again.
Figured I'd add my two cents.
A week ago, this started happening with the Citrix API to me...and apparently to other CF installations as well, which suggests *something* changed on the Citrix end, even though they didn't know of anything...
Anyway, the interesting wrinkle for me is that it broke not only on my local Developer Edition, but also my CF9 Standard Edition, which runs against the detail that this problem only shows up on Developer and Enterprise editions. I cribbed my solution from here:
http://naveenchhabra.wordpr...
...which worked on both the Developer and Standard Editions. One change was that I had Citrix give the certificate file to me, which came to me as a .crt file. The instructions state that the file extension is .cer. I decided to leave it as .crt, and that appeared to work.
I did *not* have to put in any of the code Raymond lists above as coming from Simon Free. I tried that initially, without doing the command line stuff, and that failed. After installing the certificate in CF's keystore, there was no need for the code from Simon.
So, I post this in the hopes that it might help others, but also because it's a little different from what others have seen, which may mean the problem has morphed, or it may mean that the root cause is a little bit different, even though the initial error message is the same.
Thanks for this thread. We had the same issue. We started off by adding the cert (http://www.talkingtree.com/..., but that didn't seem to help. Then I noticed Giancarlo Gomez's comment above about updating Java. That did the trick. Thanks Giancarlo - this helped us out tremendously.
Thanks for posting this Ray. In my case, I've been getting a "Connection Failure: Status code unavailable" message in scheduler.log on cfhttp calls to the same server after installing SSL certs. I've tried, it seems, the whole gamut of suggestions I've found. Upgrading the JVM solved another https issue I was having sending emails via Amazon's SMS service (after they dropped support for SSLv3 in response to Poodle), but it a combination of JVM upgrade and installing certs to the java keystore didn't do the trick for me. Adding the following to onApplicationStart() worked tho'!
if ( not structKeyExists(application, "sslfix") ) {
var abjSecurity = createObject("java", "java.security.Security");
abjSecurity.removeProvider("JsafeJCE");
application.sslfix = true;
}
(Source http://forum.hostek.com/sho...
The strange thing about this code snippet is it seems to work at server level, across all applications. Since my (vital) cfhttp calls started working again, I've left it as is, but the if() block seems a little strange to me.
Hi Raymond,
I hate to dig up a dinosaur here, but we are currently running ColdFusion 9 and are trying to update our other servers to protect from the POODLE SSL vulnerability.
A colleague of mine said that using what you mentioned in your write-up should bypass the problem that we're experiencing in ColdFusion.
However, I am hesitant because I feel like removing JSafeJCE will only disable the security rather than resolve it. Am I correct in this sense? I want to make sure that removing the provider on a specific basis won't open us up to further vulnerabilities.
Please let me know if this makes sense.
Thanks,
No worries on digging it up - but I honestly have no idea. :\ Sorry.
Alright thanks anyways :)
I actually managed to resolve this issue by upgrading the JRE from 6 to 7. This became available starting in Hotfix 3.
What if you ran the first 3 lines of the fix and removed the provider, but did not run the 4th line to put it back. Is it now permanently gone? Is there any way to restore that provide once it's removed but after the original call?
Restarting CF worked for me on CF 9 dev
The "hack" worked for me, just put the 3 lines of code before my cfhttp call and viola, it started working again, THANKS!!!
Just to say Ray, that the header Authorization part is correct, but the Disqus text editor scrambles it a little...
Afaik that was the idea. Honestly though I haven't thought of this in the past 5 years. Sorry. :(
OK. This did not work for me. I will try restarting CF10, and then I will turn off the firewall. Desperate measures but I have no idea why this issue suddenly started out of the blue...
Just out of interest why does this CFHTTP call need to use a security provider in the first place. Is it to do with SSL?
As far as I know. This area of Java is a bit fuzzy for me.
Just to let you know Ray. I have an answer but not a solution.
First of all, all my certificates in cacerts are up-to-date.
PayPal's API security updates to iits certificate & cipher requirements, require that Railo/Lucee use the JSafeJCE cipher suite. ACF11 has an updated JsafeJCE suite and so my API operations now work locally again, after I installed CF11. I use Lucee 4.5 on my production server, and now require an update to the security provider list. I have put in a request. I await with baited breath. Until then, my production PayPal API operations are dead...
Sorry to hear that - but I hope it works out for you.
Thanks Ray...
And finally after a week of blood, sweat & tears. Locally, removed CF10 and installed CF11. Issue solved. Remotely, removed Railo 4.2 and installed Lucee 4.5. Issue solved. Just thought I would let you know, in case you ever do any work with the Paypal API...
Good to hear!
Cheers Ray!
I am fighting this issue and quick application of the various techniques isn't working so I am starting at the very top, with a fine-tooth comb.
You wrote:
"This tech note is the one you want: How to import certificates into certificate stores (ColdFusion). So I tried that - on multiple machines - but it never actually worked."
I discovered that above article has a bug in the example code:
keytool -list -v -keystore JAVA_HOME\jre\lib\security\cacert -alias myServer-cert -storepass changeit
"cacert" should be "cacerts". With this change, the cert was recognized and the Adobe tech note works. I've commented on it to let them know as well, hopefully they'll change it.
Of course, I have to do this with two different URLs, and only the first one worked, the second is giving the same error even after adding to the cert store. Back to the drawing board....
Good luck!
The Api I was connecting to does not support TLS 1.0. While Java 1.7 states that it supports TLS 1.1 and 1.2, ColdFusion 10 requires Java 1.8 to connect using TLS 1.1 or 1.2 - it's a combination of the remote api service and the jvm.
Patch your jvm's now, as june 30 2018 is around the corner...