During Peter Farrell's cfObjective presentation on front end optimizations, one of the many tips he had involved minimizing the amount of HTTP requests your HTML makes. As a simple, and somewhat contrived example, consider the following HTML:
<script>
$(document).ready(function() {
$("body").append("<p>Loaded jQuery")
})
</script>
<link rel="stylesheet" href="/cfdocs/newton_ie.css" type="text/css" />
<link rel="stylesheet" href="/cfdocs/newton_ns.css" type="text/css" />
<link rel="stylesheet" href="/cfdocs/toc.css" type="text/css" /> </head>
<body>
Hello World.
</body>
</html>
<html>
<head>
<script src="/jquery/jquery.js"></script>
<script src="/jquery/jquery.cookie.js"></script>
<script src="/jquery/jquery.flot.js"></script>
<script src="/jquery/jquery.validate.js"></script>
<script src="/jquery/jquery.selectboxes.js"></script>
As you can see, I've got jQuery and 4 plugins being loaded. I've also got 3 different style sheets. All in all this reflects 8 additional HTTP requests the page has to make while parsing the HTML, and that's not counting the images that would normally make up an average web page.
As a graphical example of the impact of these scripts, check out the YSlow report:
As you can see, I got a C and an overall score of 70. The first thing YSlow points out to me are the number of HTTP requests. I decided to write my own CFML code to see if I can address this. My code would act as a simple service. I'd pass it a list of files and the code would return all the resources in one request. Before going any further, please note I did this as an experiment. There is a supported, and more full featured, open source project out now you should use instead of my code: combine. I just wrote this for fun - so please keep that in mind. Ok, with that out of the way, let's go through what I built step by step.
<cfsetting enablecfoutputonly="true" showdebugoutput="false">
I begin by enabling cfoutoutput only. This will reduce the whitespace generated by the request. Since I'm serving up JavaScript and CSS files, it also makes sense to disable debug output.
<cfset variables.rootfolder = ["/Library/WebServer/Documents/jquery/", "/Library/WebServer/Documents/cfdocs/"]>
Next we have a root folder setting. You must edit this line before using the script. I felt bad about this at first because it felt like it should be something I externalize - but then I remembered - I'm not building a custom tag here. I'm building a service. So one small amount of setup isn't so bad. Why the array? Well, this (and the next block) are the one part I'm most unsure of. For security reasons, I didn't want you to pass in full paths to files. Therefore, it makes sense to embed a root folder in the service itself. However, many people keep their CSS and JS files separate. I could allow folks to simply use their web root for the root but then they would need to pass in the subfolder for every resource requested. Ie, something like /js/foo.js, /js/goo.js, /js/zoo.js. My solution was to allow for either a simple string value or an array. If you use a string, well, it's used as is. If you use an array, I allow you to pass which index to use for the root in the URL string. More on that in a second. If you don't specify one, then the first array element is used.
<cfparam name="url.root" default="">
<!--- Only care about url.root if passed and if variables.rootfolder is an array --->
<cfif len(url.root) and isArray(variables.rootfolder) and isNumeric(url.root) and
url.root gte 1 and url.root lte arrayLen(variables.rootfolder) and round(url.root) is url.root>
<cfset variables.folder = variables.rootfolder[url.root]>
<cfelseif isArray(variables.rootfolder)>
<cfset variables.folder = variables.rootfolder[1]>
<cfelse>
<cfset variables.folder = variables.rootfolder>
</cfif>
So yeah, here is the part I really don't like. As I said, you can use an array of root folders, and you can ask for a specific one via the URL. I use url.root for that value. I didn't want you to pass in a string value of course, so instead, I simply let you pass in the index. This requires some knowledge of the configuration. All in all, this feels a bit wonky. You've never had to really hide the paths of resources before, so why bother? If I were to rewrite this, I'd probably suggest folks use the web root and simply use the subfolders when requesting resources. Actually, I don't have to rewrite it - that works already. So um - consider it deprecated. ;) Ok, carrying on....
<!--- Set our content type based on roottype --->
<cfif url.roottype is ".js">
<cfset variables.contenttype = "text/javascript">
<cfelse>
<!--- I set url.roottype just to be anal since we use it again later for file security --->
<cfset url.roottype = ".css">
<cfset variables.contenttype = "text/css">
</cfif>
<!---
Root Type: This should be .js or .css.
--->
<cfparam name="url.roottype" default=".js">
The next block of code handles setting up a requirement for the type of file being requested. This will allow us to do a security check later on, and it also allows us to use the right content type. I could have simply looked at the first resource requested (is it something.js or something.css), but I felt like being anal about allowed me to really lock it down to *.js or *.css.
<!--- If blank, quickly leave. --->
<cfif url.list is "">
<cfabort>
</cfif>
<cfparam name="url.list" default="">
Here we define the URL parameter that will contain the requested resources.
<cfparam name="url.refreshcache" default="0">
Here is our hook to let us refresh the cache. More on that in a bit.
<cfset variables.scope = "application">
The service will use a cache for it's file work. I can't imagine needing a different scope than Application, but if you need to - you can tweak it.
<cfset variables.cacheRoot = "_multiloadres2">
And there is the key used for the cache.
<cfset cacheScope = structGet(variables.scope)>
<cfset needInit = false>
<cflock scope="#variables.scope#" type="readOnly" timeout="10">
<cfif not structKeyExists(cacheScope, variables.cacheRoot)>
<cfset needInit = true>
</cfif>
</cflock>
<cfif needInit>
<cflock scope="#variables.scope#" type="exclusive" timeout="10">
<cfif not structKeyExists(cacheScope, variables.cacheRoot)>
<cfset cacheScope[variables.cacheRoot] = {}>
</cfif>
</cflock>
</cfif>
There we have the cache set up routine. Notice I use structGet (remember it?) to create a pointer to the scope holding my cache. I do the necessary locks and create the root structure for my cache if I need to.
<cfif structKeyExists(cacheScope[variables.cacheRoot], url.list)>
<cfheader name="expires" value="#getHTTPTimeString("1/1/2032")#">
<cfcontent type="#variables.contenttype#"><cfoutput>#cacheScope[variables.cacheRoot][url.list]#</cfoutput><cfabort>
</cfif> </cfif>
<cfif isBoolean(url.refreshcache) and not url.refreshcache>
Now that we have a cache, we can actually use it - if it exists in the cache. Notice too the use of the expires header. YSlow pointed this out to me during my development. To my readers in 2032, I apologize. Also, please tell the alien overlords to be nice to my kids.
<cfset buffer = "">
<cfloop index="res" list="#url.list#">
<!--- For each file, if it contains .., assume it is a hack attempt and immediately barf. --->
<cfif find("..", res)>
<cfabort>
</cfif>
<!--- For each file, if it does not end in js, assume it is a hack attempt and immediately barf. --->
<cfif right(res, len(url.roottype)) is not url.roottype>
<cfabort>
</cfif>
<cfset trueFile = variables.folder & "/" & res>
<!--- If the file doesn't exist, we skip. Don't throw an error because we don't want to be used to scan the system. --->
<cfif fileExists(trueFile)>
<cfset buffer &= fileRead(trueFile)>
</cfif>
</cfloop>
Woot! Finally some real work. Here you can see how we loop over the list of requested files. For each, I'm going to do a quick extension check, and if it passes, and if the file exists, I read the contents into a buffer variable. That's really it. Pretty simple, right?
<cfif len(buffer)>
<cfset cacheScope[variables.cacheRoot][url.list] = buffer>
<cfif isBoolean(url.refreshcache) and not url.refreshcache>
<cfheader name="expires" value="#getHTTPTimeString("1/1/2032")#">
<cfelse>
<cfheader name="expires" value="#getHTTPTimeString(now())#">
</cfif>
<cfcontent type="#variables.contenttype#"><cfoutput>#cacheScope[variables.cacheRoot][url.list]#</cfoutput>
</cfif>
The final bits then simply handle storing the buffer into the cache and finally returning it to the client. Again, note the user of the expires header. Ok, so how does it look when I use it? Here is a modified form of the original HTML:
</head>
<body>
Hello World.
</body>
</html>
<html>
<head>
<script src="multiload.cfm?list=jquery.js,jquery.cookie.js,jquery.flot.js,jquery.validate.js,jquery.selectboxes.js"></script>
<script>
$(document).ready(function() {
$("body").append("<p>Loaded jQuery")
})
</script>
<link rel="stylesheet" href="multiload.cfm?root=2&roottype=.css&list=newton_ie.css,newton_ns.css,toc.css" type="text/css" />
As you can see, each of my multiple JS and CSS requests have been turned into one. For the CSS one I have to specify a bit more as most of my defaults are for JS. But really, it isn't that difficult to use. And the result?
Woot! An A! Those of you who know how fragile my ego is will not be surprised to hear this made me do a quick little dance of joy. Anyway, it was really fun to build this, but again, I'll point people to the combine project by Joe Roberts. His also adds compression to the mix for even more performance. The complete code for the script is below. Enjoy.
Note - this is the only line you need to edit (most likely!)
--->
<!---
<cfset variables.rootfolder = "/Library/WebServer/Documents/jquery/">
--->
<cfset variables.rootfolder = ["/Library/WebServer/Documents/jquery/", "/Library/WebServer/Documents/cfdocs/"]> <cfparam name="url.root" default="">
<!--- Only care about url.root if passed and if variables.rootfolder is an array --->
<cfif len(url.root) and isArray(variables.rootfolder) and isNumeric(url.root) and
url.root gte 1 and url.root lte arrayLen(variables.rootfolder) and round(url.root) is url.root>
<cfset variables.folder = variables.rootfolder[url.root]>
<cfelseif isArray(variables.rootfolder)>
<cfset variables.folder = variables.rootfolder[1]>
<cfelse>
<cfset variables.folder = variables.rootfolder>
</cfif> <!---
Root Type: This should be .js or .css.
--->
<cfparam name="url.roottype" default=".js"> <!--- Set our content type based on roottype --->
<cfif url.roottype is ".js">
<cfset variables.contenttype = "text/javascript">
<cfelse>
<!--- I set url.roottype just to be anal since we use it again later for file security --->
<cfset url.roottype = ".css">
<cfset variables.contenttype = "text/css">
</cfif> <!---
List of resources to load.
--->
<cfparam name="url.list" default=""> <!--- If blank, quickly leave. --->
<cfif url.list is "">
<cfabort>
</cfif> <!---
If true, we don't use a cached version of the resource.
--->
<cfparam name="url.refreshcache" default="0"> <!---
Default scope for the cache. No reason normally to tweak this.
--->
<cfset variables.scope = "application"> <!---
Key used to store cached info
--->
<cfset variables.cacheRoot = "_multiloadres2"> <!---
Ok, begin working.
---> <!--- I handle creating initial cache struct --->
<cfset cacheScope = structGet(variables.scope)>
<cfset needInit = false>
<cflock scope="#variables.scope#" type="readOnly" timeout="10">
<cfif not structKeyExists(cacheScope, variables.cacheRoot)>
<cfset needInit = true>
</cfif>
</cflock>
<cfif needInit>
<cflock scope="#variables.scope#" type="exclusive" timeout="10">
<cfif not structKeyExists(cacheScope, variables.cacheRoot)>
<cfset cacheScope[variables.cacheRoot] = {}>
</cfif>
</cflock>
</cfif> <!--- I handle caching concerns --->
<cfif isBoolean(url.refreshcache) and not url.refreshcache> <cfif structKeyExists(cacheScope[variables.cacheRoot], url.list)>
<cfheader name="expires" value="#getHTTPTimeString("1/1/2032")#">
<cfcontent type="#variables.contenttype#"><cfoutput>#cacheScope[variables.cacheRoot][url.list]#</cfoutput><cfabort>
</cfif> </cfif> <!--- I handle loading my files --->
<cfset buffer = "">
<cfloop index="res" list="#url.list#">
<!--- For each file, if it contains .., assume it is a hack attempt and immediately barf. --->
<cfif find("..", res)>
<cfabort>
</cfif>
<!--- For each file, if it does not end in js, assume it is a hack attempt and immediately barf. --->
<cfif right(res, len(url.roottype)) is not url.roottype>
<cfabort>
</cfif>
<cfset trueFile = variables.folder & "/" & res>
<!--- If the file doesn't exist, we skip. Don't throw an error because we don't want to be used to scan the system. --->
<cfif fileExists(trueFile)>
<cfset buffer &= fileRead(trueFile)>
</cfif>
</cfloop> <!--- All done - if we actually have content, cache it and store it. --->
<cfif len(buffer)>
<cfset cacheScope[variables.cacheRoot][url.list] = buffer>
<cfif isBoolean(url.refreshcache) and not url.refreshcache>
<cfheader name="expires" value="#getHTTPTimeString("1/1/2032")#">
<cfelse>
<cfheader name="expires" value="#getHTTPTimeString(now())#">
</cfif>
<cfcontent type="#variables.contenttype#"><cfoutput>#cacheScope[variables.cacheRoot][url.list]#</cfoutput>
</cfif>
<cfsetting enablecfoutputonly="true" showdebugoutput="false">
<!---
Root Folder:
This may be a simple path (full path!) or an array. Using an array allows you to use
one instance of this script and N root paths. If an array is used, you specify a specific
root folder by using root=N. It may be confusing to have to specify the array index, but it
also means you aren't passing a real path along the query string. You can also leave it out
for the 1st item in the array.
Archived Comments
Cool! And it's just steps away from some document parsing with OnRequest() in Application.cfc to convert an existing app without having to even change the code! I don't know if it'd be worth it performance-wise, but that'd be cool... :-D
Dude, I'm not normally a onRequest fan, but that's a cool freaking idea. You mind if I roll with it for another blog post?
I have always been opposed to this method. IIS/Apache is way faster at delivering content than ColdFusion. I do like your file caching mechanism, I just don't think it can compete with what already exists in Apache or IIS.
The other downside, I can't offload my static resources to a CDN. I don't see any websites using their Application engine to create combined resources. What I do see is rake/make files for combining/minifying resources, and then calling that combined file from the HTML page.
Side note, when I added my website to the website field. My comments were always flagged as spam. Weird? http://drew wells.net/blog spam site?
@Drew: Good point there - does Apache offer a way to combine N JS files into one? I've not seen that. I do know it offers gzip compression but I have not heard of it having a way to handle getting N files into one HTTP request.
@Drew 2: Obviously your comment was flagged as spam since you disagreed with me. ;) Seriously though - no idea. I use cfFormprotect for my spam protection. Sorry it blocked your host!
Hey Drew - get this. Even my emails from you for the comments were marked as spam.
There is an Apache module called mod_concat which combines multiple files for a single request. Haven't used it yet myself, but it looks promising. http://code.google.com/p/mo...
Interesting. I wonder though if it is still doing file io for each request. I'd think that would make it slower than a solution that returns from RAM.
I'm using similar approach for quite a while, there are a couple of things though you might want to watch out when combining CSS files:
- relative urls in css rules, most probably result path won't be the same as for original css files, hence all nice "background: url(images/sprite.png)" rules could end up broken. Fix: all those rules have to be parsed and relative paths replaced with the absolute - extracted from the original url
- beware of "@import" rules - according to the w3c standard they are allowed at the top of the css file only, therefore if you are combining multiple files you have to catch all @import-s and move them to the top of the result file.
Ah good catch there Jura!
@Ray - Sent ya an email, but just wanted to respond in the blog comments -- Go for it!
<--- Spammer!
I was thinking more combining them as part of your deployment. For instance, the new High Performance JS book describes some ANT scripts for combining your requests which is simple enough. I'm still trying to wrap my head around how you manage calls to these files in your HTML. Apache wouldn't know anything about this besides referring to the files you create with your ANT scripts.
The js libraries that I use are either in virtual directories or hosted on third-party websites.
Here are some non-server options:
http://labjs.com/ (I'm using this one currently.)
http://requirejs.org/
http://code.google.com/p/ja...
Another option would be to move all your js to the bottom of the webpage (this is what Facebook is doing.)
Hi Ray,
Couldn't you just use one of the minify toolkits out there to compress and "minify" your CSS and JS?
YUI Compressor and similar tools reduce bandwidth AND can combine files into one.
This is what I mean: http://scriptalizer.com/
Hi Ray,
Great post, and thanks for the shout out to Combine.cfc!
I really like the way you cache the js/css to memory. Combine caches to file, but I'm going to consider adding some in-memory cache, maybe an LRU cache to stop the memory usage from getting out of control!
@Sebastian: I'm sure it wouldn't take much more - but I just wasn't interested in taking it that far. Joe's work already does.
Well it does the compression, not sure if it does the minification. This was brought up in Farrell's presentation. The bonus you get from minification doesn't really add a lot and when you consider how it can lead to issues with debugging, it wasn't recommended. (That's what I remember, but I could be wrong.)
@Joe: Glad you liked it!
what about forms that use built in js files, such as cflayout or cfmenu, this example or the RiaForge.org code will not work with those files, this I will still get a lower grade with yslow, correct?
Correct. If you have no control over the JS being used, then there ya go.
I wrote a couple of custom tags that make using the "Combine" project that you mention in your post really simple. Full source and examples on my blog: http://fusiongrokker.com/po...
A while back, we created a ColdFusion based framework/cms based upon what we liked in other such frameworks. In modules, you can call add_js with a variable of where the module lives, since we store our modules (above) the webroot, meaning you can not access them using the browser.
Our function does this exact thing, except it also writes a copy to disk. Storing everything in RAM was impossible because of each user role / permission combo *might* require drastically different files.
The file is then served from the file system, with IIS and some settings for file system cache. All in all, it can be an extremely efficient solution when combined with other performance tuning.
(we do this with CSS too)
Is your product OS? Commercial? Available at all?
<CFAjaximport> loads a bunch of scripts at the top of the document, e.g.:
<script type="text/javascript" src="/CFIDE/scripts/ajax/package/cfajax.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/ajax/yui/yahoo-dom-event/yahoo-dom-event.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/ajax/yui/animation/animation-min.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/ajax/ext/adapter/yui/ext-yui-adapter.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/ajax/ext/ext-core.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/ajax/ext/package/resizable.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/ajax/ext/package/dragdrop/dragdrop.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/ajax/ext/package/util.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/ajax/ext/build/state/State-min.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/ajax/ext/package/widget-core.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/ajax/ext/package/dialog/dialogs.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/ajax/package/cfwindow.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/cfform.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/masks.js"></script>
<script type="text/javascript" src="/CFIDE/scripts/cfformhistory.js"></script>
Is there any way, we can get these to load at the bottom of the document, or do they NEED to be at the top?
You can't (afaik). Keep in mind - most of the Ajax stuff (front end stuff) in CF is for folks _not_ familiar with Ajax. If you are experienced enough to know you prefer this differently, then you are probably not the target audience for those tags. Make sense?
What are you trying to say, punk? ;-). Thanks to your very informative post here http://www.coldfusionjedi.c..., I now use cfwindow in my app.
Heh, well, to be honest, I hear complaints from time to time about the CF/Front End Ajax stuff. Complaints about the size mainly. Questions about doing some particular customization. Etc. My typical reaction is - well - what I typed above. :) Things like cfwindow make it -very- easy for folks, but certainly could be done with other code, like jQuery UI for example. Just something I try to get people to remember.
Hi, I have been looking into a way of combining css and java calls and have had a look at the combine.cfc project. However i have inherited a 64 bit CF server and am not sure combine will run on a 64bit version? I was trying to utilise your script, but it does not appear to load the css or jquery. I have changed the root folder to "/css/" which is the correct root, but no styles can be seen on the page. multiload.cfm is in the root, along with index.cfm which uses it, css folder is next level up? Any ideas, I can give a testing url if needed.
Hi @jason. Combine will run fine on a 64bit server. This sounds like a paths problem. To be perfectly honest, I've pretty much retired the project as a) I think it's better practice to combine assets at build time (using something like Sprockets, Gruntjs, etc) b) I don't write any CFML any more, so I'm way out of touch.
So, I don't think I can be of much help, but I do suggest fiddling with the paths (and maybe try a mapping - But I can't remember if this is an appropriate solution!. Also, you could try out the example in the repo to see if that works as expected.
Hi @Joe
Thanks for your reply, I must have been having a bad day when I initially tried combine on a 64bit server. I could not get it to work whatsoever! Whet back and tried it yesterday and all works fine?? But I will defiantly look into combing assets at build time. Thanks.