Is it time to stop using ColdFusion for remote APIs?

This post is more than 2 years old.

So, let me begin by saying I'm a bit frustrated, and so this blog post may be one I regret later on, but, I try to be as honest as possible here and right now, I'm kinda ticked off about something and I want to get it off my chest. For a long time now I've had an incredible amount of respect for how ColdFusion makes it easy to access data from client-side code (or remote servers). As much as I'm digging Node.js these days, the fact that I can write up a CFC and get an API that can be used by JavaScript is pretty darn powerful. This feature has come a long way. When it first came out, the only option for output was WDDX. You can now output anything, from WDDX, to SOAP, to XML, plain strings, and of course, JSON.

In fact, I like this feature so much that I proposed a topic for it at this years ColdFusion Summit. My session will be an overview of how to generate output for remote consumption and cover everything from the beginning (ColdFusion MX and earlier) to the upcoming release (12).

However...

Since the beginning of native JSON support in ColdFusion (ColdFusion 8), there have been consistent issues with serialization. This all drives from the fact that ColdFusion variables are typeless and therefore the server has to (or does it?) make guesses as to how to convert values into JSON. Over the past three releases, I've seen multiple bugs, and multiple fixes, and while I had no real proof (more on that in a minute), my gut told me that things had simmered down a bit and that ColdFusion 11 had this problem licked.

Or so I thought.

Turns out bug 3337394, created nearly three years ago, oh one marked closed and fixed, is still very much an issue for serialization. If you have data in a struct, and it has the string value "No", ColdFusion will convert it to false. Here is a sample:

x = {"name":"No"};
y = queryNew("id,name", "integer,varchar", [{"id":1, "name":"ray"},{"id":2, "name":"No"}]); 
writeoutput(serializeJSON(x));
writeoutput("

"); writeoutput(serializeJSON(y,"struct"));

Which gives you:


{"name":false}

[{"ID":1,"NAME":"ray"},{"ID":2,"NAME":"No"}]

As you can see, the struct is broken, the query works fine. (As reported in the bug itself.)

Another issue involves strings that contain numbers. Consider these two examples:

z = {"productkey":"89900909130939081290830983019819023"};
z2 = queryNew("id,name", "integer,varchar", [{"id":1, "name":"ray"},{"id":2, "name":"89900909130939081290830983019819023"}]);

writeoutput("

"); writeoutput(serializeJSON(z)); writeoutput("

"); writeoutput(serializeJSON(z2));

This returns:


{"productkey":89900909130939081290830983019819023}

{"COLUMNS":["ID","NAME"],"DATA":[[1,"ray"],[2,"89900909130939081290830983019819023"]]}

As you can see, productkey is now a number, and one that will be converted to 8.990090913093909e+34 in JavaScript.

So.... yeah. In all cases, there are workarounds. But let me ask you this. Would you use a database that randomly changed values on you?

aw_hell_no

It seems ridiculous that this is still a problem now. It may be incredibly difficult. Heck, I won't pretend to be able to solve it. But here are some suggestions:

  • Obviously the ColdFusion team has a unit test for this. They must. So when the unit test is updated for the items found in the bug, share it with us. I know there are multiple people in the community who would give their time to help flesh out the unit test, or heck, just have it locally and test when new versions come up. So that's my first request - let us see the tests for JSON serialization. Speaking of unit tests, the last time this bug was fixed, why not share the test immediately? I mean, we won't see it working, but I promise you people who have asked why a bug involving data serialization had a test that only tested queries (and CFCs I believe). I don't want to harp on the engineer who fixed this - we all make mistakes - but why not post the test when you write it and share it with us?

    Of course, one could write essays on the lack of communication that goes on sometimes on the bug tracker, which is a real shame. You know, I get that some people don't like to engage or are basically shy. But there's no excuse for it anymore. I'm very shy too. When someone speaks to me, I have to work hard to respond to their questions with questions of my own. I recognize my lack of engagement in basic human communication and force myself to hack around my lack of social skills. Make a list and put it on a PostIt next to your monitor. "When I fix a bug, respond with details about how I fixed it, how I tested it, and what I may be concerned about. The reporter may have good input!"

  • Given that the "glue" aspect of ColdFusion is one of its greatest selling points, make JSON serialization a priority for ColdFusion 12. Obviously the issue isn't licked yet, which, ok, fine, it's been years but fine. The ColdFusion 12 road map says this: "Ability to manage, monitor, regulate, secure REST and SOAP web services – API management". Before I manage my APIs, I want 100% certainty that they actually process data correctly.
  • And hey - how about the nuclear option? If the very nature of ColdFusion variables means the problem is not 100% solvable, then remove the feature. Seriously. Ok, maybe that's overkill, but there are things in ColdFusion now that haven't been updated in a while, look abandoned, and maybe should be dropped. So if they can't fix it, remove it.

So I started this blog entry with a somewhat bold title - is it time to stop using ColdFusion for remote APIs? Many of us in the community have already argued against using ColdFusion UI features. If we can't trust the data coming out of CF's built-in JSON serialization than is it time to stop using it completely? What do you think?

p.s. Don't forget you can switch to Nadel's excellent JSONSerializer.cfc.

p.s. On top I made the point that ColdFusion has to guess at the data type. Adam Cameron makes a great comment in the bug about this: "You speak good sense Peter: it's inconvenient but understood that CF can't infer data types in situations like this, so - on that basis - don't try. If you know you can't do something... don't then go ahead and try do it anyhow." Given that "No" is a string, don't convert it. If someone is using No for false in their data, then why not punish them instead of everyone else?

p.s. As I was writing this, it occurred to me that we have other places where things get a bit fuzzy with data. Math, for example, breaks down in large numbers. I get that that's an issue across other languages, and certainly we shouldn't remove arithmetic from ColdFusion because of it, but it just feels like a different domain than what I'm seeing in JSON serialization.

Raymond Camden's Picture

About Raymond Camden

Raymond is a senior developer evangelist for Adobe. He focuses on document services, JavaScript, and enterprise cat demos. If you like this article, please consider visiting my Amazon Wishlist or donating via PayPal to show your support. You can even buy me a coffee!

Lafayette, LA https://www.raymondcamden.com

Archived Comments

Comment 1 by Tim Brown posted on 9/4/2015 at 10:15 PM

Couldn't agree more with how frustrating JSON is in ACF. I love Coldfusion, but wish they could just get this part right. If I have to resort to looping my property names and extending Nadel's serializer.cfc like in this gist ( https://gist.github.com/tim... ) just to get dynamic proper casing and data typing of my JSON returns, it becomes a question of why am I working so hard and makes me think about other technologies. It looks like Lucee has addressed this for quite sometime ( http://trycf.com/gist/9277c... )... I guess I need to be more agressive in my switch to Lucee for all my applications as I'm tired of fighting against the grain on my REST API development on ACF...

Comment 2 by roger tubby posted on 9/4/2015 at 10:40 PM

Unit tests? If so, you would think they would like to publish them so we can all run on our instances. (This is part of the same company that also doesn't test its other products [Flash, Acrobat] very well.)

Comment 3 by Dana Kowalski posted on 9/5/2015 at 11:18 AM

"Before I manage my APIs, I want 100% certainty that they actually process data correctly."

-- this

Comment 4 by Gary F posted on 9/5/2015 at 1:48 PM

When I first read about serialization problems many, many years ago (just after CF8's release?) I thought "oh dear, I don't trust that function". So ever since I've rolled my own json output from CF. It doesn't do Adobe's reputation any good and if there are any newbies to CF they'll wonder why their json is broken and go through the same pain that others did many years before them.

I recommend they create a new function called toJSON() which is 100% new, bug-free, and doesn't matter if it's incompatible with serializeJSON() as this won't break anyone's code using the "legacy" function. This would seam so logical in order to move forwards. Start fresh.

Comment 5 by Carl Von Stetten posted on 9/5/2015 at 7:06 PM

One gripe I have is that if you specify JSON as the returntype for remote CFCs so that the proper mime type is set, you are locked into implicitly executing SerializeJSON on the return variable. You can't use a roll-your-own serializer or your return variable will get double-serialized.

If Adobe can't figure out how to get serialization to work in a safe and sane manner, then I agree they should just stop trying and deprecate the mess they created.

Comment 6 (In reply to #5) by Raymond Camden posted on 9/6/2015 at 9:58 AM

Actually, you can. You can specify a custom serializer in CF11.

Comment 7 by Peter Tilbrook posted on 9/9/2015 at 10:44 AM

Just started using the Stripe payment gateway. All sorts of complicated CF "solutions" out there. I ended up ussing CFHTTP over SSL. Works a treat. In three lines of code and no MAPPINGS or Application.cfc bullshit.

Comment 8 (In reply to #7) by Clayton Reeves posted on 9/10/2015 at 8:53 PM

You should look at https://github.com/philcruz..., there is a lot of functionality in there and works very quickly and easily.

Comment 9 by Matt Woodward posted on 9/11/2015 at 1:05 PM

I made changes in OpenBD a few years back to make JSON serialization work as it always should have in CFML (and apparently still doesn't in ACF), and the changes were not difficult to make at all. Fixing the case issues and the problems with numbers made things SO much nicer. Moot point to me now since I don't do CF anymore, but kinda hard to believe this is still a thing.

Comment 10 by Ben Nadel posted on 10/29/2015 at 12:58 PM

I've been feeling a lot of the same frustrations as well. From a "render this page," standpoint, ColdFusion is still just unbelievably powerful. You can literally create a working page with a single .CFM file that includes the text "Hello World.". While a trite example, the same with something like Node.js requires many more steps - probably using express, configuring an HTTP service, port forward, etc.

But, when it comes to JSON, which we use a LOT these days, ColdFusion is just one headache after another. ColdFusion is at the point where it makes some things super easy, and some things super hard.

That said, it's still my go-to tool anytime I want to spin up something quick and have it just "work" out of the box.

Comment 11 (In reply to #10) by Raymond Camden posted on 10/29/2015 at 1:20 PM

I think your point about "just making a page" is very valid. In Node, that means me making a route first and then adding the code I want to test.

However...

I think this is a mental... um... "mistake" (not the best word) on my part. I'm so used to building CF pages where I write a CFM under web root and hit it with my browser.

I forget that I can just write foo.js and node it. I don't need a web server. I can just execute it as is.

I mean I *know* this, but I swear it just doesn't seem to click in my brain sometimes.

Of course, CF has a CLI now with CommandBox so it has something similar.

Comment 12 (In reply to #11) by Ben Nadel posted on 10/29/2015 at 1:44 PM

Yeah, that's true. I definitely catch myself making a cookie-cutter HTML page sometimes when I just want to test something in JavaScript, because I forget I can just `node thing.js` to test.

But, really, what I mean is that if I wanted to just make a silly page where it was like a count-down clock until the new Star Wars movie, I could literally create a .cfm file with a dateDiff() call and a writeOuptut() call and then point IIS or Apache at it, and its done. There is something so powerful about that.

Of course, I'm skimming over the fact that ColdFusion had to be installed and configured to work with something like IIS or nginx. But, once you eat that upfront cost, there is next to no cost for adding 2-N sites to the same box. No worrying about IP forwarding or all that stuff.

I don't know.... maybe this is all just the "curse of knowledge" and ColdFusion just feels easier because I know it and it's already installed. Maybe Node.js people feel the same way about spinning up a new Express instance and configuring Nginx.

Comment 13 (In reply to #12) by Raymond Camden posted on 10/29/2015 at 1:48 PM

Heh, I'd just use JavaScript and create a static site and host it via Surge or S3. No servers at all. Nothing to break. ;)

Comment 14 (In reply to #13) by Richard Spector posted on 3/23/2016 at 2:07 PM

webstorm editor is awesome you can build and edit the js then just run it from the editor without having to node js it