ColdFusion Logs to RSS (with a quick sidetrack into zombies)

This post is more than 2 years old.

A reader on Friday asked a pretty cool question:

I really haven't looked into this, but was hoping that someone had already thought of it: I'd like to take a ColdFusion log file (like app.log) and transform to an RSS feed. Any kind of weird parsing that goes on there? I would assume you'd have to convert the log file to some sort of structure... Text Transformation is certainly not my strong suit, maybe one of the reasons I failed that perl class in college.

Didn't the designers of Perl want folks to fail? No? Ok, moving on. So I thought this was a great idea. RSS is a great way to receive updates, and monitoring a log file via RSS could be an awesome way to keep up on things... just as long as the log file isn't going crazy with updates.

So let's take a quick look at how you can accomplish this. The task really isn't that difficult. We need to:

a) Read the log file (easy!)
b) Parse the string (kinda easy!)<br/ > c) Convert the string into a query object (not so tough!)
d) Convert the query into RSS (simple with CFFEED!)

One at a time, here we go.

<cfinclude template="udf.cfm">

<!--- log file ---> <cfset logfile = "/Applications/ColdFusion8/logs/application.log">

<cfif not fileExists(logfile)> <cfoutput>Log file "#logfile#" does not exist.</cfoutput> <cfabort> </cfif>

I begin by including a UDF library, which in this case is one UDF, CSVtoArray. I define my log file with a hard coded pointer to the file I want to parse and my local CF install. (Yes, there is a way to find that path dynamically.) I then do a quick sanity check to ensure the file actually exists. Moving on...

<!--- our data, in line form ---> <cfset lines = []>

<!--- flag to let us ignore the first line ---> <cfset doneOne = false> <cfloop index="l" file="#logfile#"> <cfif doneOne> <cfset arrayAppend(lines, l)> <cfelse> <cfset doneOne = true> </cfif> </cfloop>

<!--- reverse it. This code is the suck. Please be aware it is the suck. ---> <cfset latestLines = []> <cfloop index="x" from="#arrayLen(lines)#" to="#max(1,arrayLen(lines)-20)#" step="-1"> <cfset arrayAppend(latestLines, lines[x])> </cfloop>

Ok, now before going on, I know some of you are screaming at your monitor now. This is not a good way to get the end of the file. Remember we want to show the latest updates in our RSS feed. My code loops through the file, adding each line to an array. I then reverse it by grabbing the last 20 lines. This works, but let me stress. It sucks. It can be done better. I'm going to leave that for tomorrow. For now though, just remember that this is not the most efficient way to get the end of a file.

I now have an array of 20 lines (at most) of text. These are in the format of:

"Information","jrpp-18","12/23/08","15:21:20",,"/Applications/ColdFusion8/logs/application.log initialized"
"Error","jrpp-26","12/23/08","17:02:44","ApplicationName","Variable X is undefined. The specific sequence of files included or processed is: /Library/WebServer/Documents/test4.cfm, line: 6 "

The first line is a header of columns (which I skipped when reading in the file) and each line is comma delimited and wrapped in quotes. Luckily I can parse this format using the csvToArray UDF mentioned earlier. The only issue I have with the UDF is that it expects a file, not one line. You will see how I handle that in the code below.

<!--- query to send to cffeed ---> <cfset q = queryNew("publisheddate,title,content")>

<!--- for each line, parse it into an array, and add to our query ---> <cfloop index="l" array="#latestLines#"> <cfset data = csvToArray(l)> <!--- udf expected N lines, we sent it one, so quickly copy over itself ---> <cfset data = data[1]>

&lt;cfset queryAddRow(q)&gt;
&lt;!--- array pos 3 and 4 is date and time ---&gt;
&lt;cfset querySetCell(q, "publisheddate", data[3] & " " & data[4])&gt;
&lt;!--- pos 6 is the full string ---&gt;
&lt;cfset querySetCell(q, "content", data[6])&gt;
&lt;!--- make a title from the full string ---&gt;
&lt;cfset title = left(data[6],100)&gt;
&lt;cfif len(data[6]) gt 100&gt;
	&lt;cfset title &= "..."&gt;
&lt;cfset querySetCell(q, "title", title)&gt;


The first line above creates the query that I'll give to CFFEED. Well don't forget that when creating RSS feeds with a query, you need to either a) name your columns right or b) use the columnMap feature to tell the feed generator to map certain RSS items to particular query columns. Since I'm building the query from scratch I'll just use the columns that make sense for RSS. I parse the line (and notice how I copy the array over itself, I explained why above), and then I simply copy relevant data items over into my query.

I made the call that I'd use the date, time, and message values from the ColdFusion log. That made the most sense for this type of log. Other log files would probably warrant different logic. I combine the date and time when adding it to the query. Message comes in as is, and I reuse a portion of the message for my title. 100 characters isn't an RSS requirement - it just felt right for the feed.

We have our data, but there is one more step before we can create an RSS feed. We have to give some metadata about the feed. The values I chose here were completely arbitrary and really don't matter that much, but they are required by the CFFEED tag.

<cfset meta = { version="rss_2.0", title="ColdFusion Application Logs", link="http://null", description="Latest log items." }>

Woot! That's it. Now to just actually convert the query into RSS and serve it up.

<cffeed action="create" query="#q#" properties="#meta#" xmlVar="rss">

<cfcontent type="text/xml; chartset=utf-8"><cfoutput>#rss#</cfoutput>

You can see the output here, but note that it is a static export, not a live copy:

As I said, woot! Nothing says excitement like ColdFusion log files. Maybe we can kick it up a notch? For fun, I decided to build a simple little zombie infestation simulator. It takes a pool of 100 people. Every 'round' there is a chance the infestation will start. Once it done, every round will end up with either a dead zombie or a dead human and one more zombie. (Guess who tends to win?) I wrote the simulator to not only output to the screen but also log the values as well. Check it out:

<cffunction name="logit" returnType="void" output="true" hint="Handle cflogging and printing."> <cfargument name="str" type="string" required="true"> <cfargument name="color" type="string" required="false">
&lt;cfif structKeyExists(arguments,"color") and len(arguments.color)&gt;
	&lt;cfoutput&gt;&lt;span style="color: #arguments.color#"&gt;&lt;/cfoutput&gt;
&lt;cfoutput&gt;#arguments.str# [Humans: #humanpop# / Zombies: #zombiepop#]&lt;/cfoutput&gt;
&lt;cfif structKeyExists(arguments,"color") and len(arguments.color)&gt;
&lt;br /&gt;
&lt;cflog file="zombie" text="#arguments.str# [Humans: #humanpop# / Zombies: #zombiepop#]"&gt;


<!--- initial human pop ---> <cfset humanpop = 100>

<!--- initial zombie pop ---> <cfset zombiepop = 0>

<!--- has the infection began? ---> <cfset infectionOn = false>

<!--- sanity check, if we hit this, abort ---> <cfset y = 1>

<!--- checks to see if we are done ---> <cfset simDone = false>

<cfloop condition="not simDone && y lt 1000">

&lt;!--- ok, if no infection, we have a 5% chance of starting it. ---&gt;
&lt;cfif not infectionOn&gt;
	&lt;cfif randRange(1,100) lte 5&gt;
		&lt;cfset infectionOn = true&gt;
		&lt;cfset humanpop--&gt;
		&lt;cfset zombiepop++&gt;
		&lt;cfset logit("Zombie infestation begun.")&gt;	
		&lt;cfset logit("All good!")&gt;		

	&lt;!--- Ok, we either kill a human or a zombie. 
	The more zombies, the greater chance a human dies.
	Zombies only have a chance to die when the grow in #, since people won't worry about them until they mass up. 
	It is a bit too late when the zombies mass up, but guess what, thats how these things work.
	Tweak the +/- to help/hinder the survival rate
	&lt;cfset chanceHumanDied = randRange(1,zombiePop) + 4&gt;
	&lt;cfset chanceZombieDied = randRange(1,zombiePop) - 0&gt;
	&lt;cfif randRange(1,100) lte chanceHumanDied&gt;
		&lt;cfset humanpop--&gt;
		&lt;cfset zombiepop++&gt;
		&lt;cfset logit("Zombie killed a human.","red")&gt;
	&lt;cfelseif randRange(1,100) lte chanceZombieDied&gt;
		&lt;cfset zombiepop--&gt;
		&lt;cfset logit("Human killed a zombie.","green")&gt;
		&lt;cfset logit("Nothing happend this time.")&gt;

&lt;!--- end the sim if humanpop is 0 or infection started and the zombies were slain ---&gt;
&lt;cfif humanpop is 0&gt;
	&lt;cfset logit("All humans killed. Sim done.")&gt;
	&lt;cfset simDone = true&gt;

&lt;cfif infectionOn and zombiepop is 0&gt;
	&lt;cfset logit("All zombies killed. Sim done.")&gt;
	&lt;cfset simDone = true&gt;

&lt;!--- This is my sanity check in case my logic is crap ---&gt;	
&lt;cfset y++&gt;


You can run this yourself here: Have fun. I know I ran it about a hundred times or so. I then used the same code above (and attached to this entry), with a slight modification to the RSS properties:

<cfset meta = { version="rss_2.0", title="FunCo Mall Security Logs", link="http://null", description="Latest log items." }>

You can view that XML here: (Again, not 'live', just a save from my machine.)

Enjoy. I attached all the files to the blog entry.

Download attached file.

Raymond Camden's Picture

About Raymond Camden

Raymond is a senior developer evangelist for Adobe. He focuses on document services, JavaScript, and enterprise cat demos. If you like this article, please consider visiting my Amazon Wishlist or donating via PayPal to show your support. You can even buy me a coffee!

Lafayette, LA

Archived Comments

Comment 1 by Jules Gravinese posted on 3/29/2009 at 5:48 AM

Well this is sure food for thought! The current project I am wrapping up has daily import/export routines. The output is emailed to me and gmail does a great job of filing that for me.

But if I were to log them instead, I can use a special character that I pick to be the end of each line (like tab, tilde, or pipe). Using that, it's easy to count the number of log entries, and also pick out the last 20.

Great post! Definitely has me inspired.

Comment 2 by Mike Whitby posted on 3/29/2009 at 2:33 PM

This is a great example of how to read logs, I love the sound of it - might have to give this a go! I previously have made a script to download the CF logs into a local Splunk, anyone done that before?

Comment 3 by Dan G. Switzer, II posted on 3/30/2009 at 5:36 PM

One thing to keep in mind regarding this project, is that potentially would be very easy to miss entries if you're only grabbing the last XX records.

Depending on your code, if a log file is growing rapidly (such as the mail log) unless you're checking the RSS feed pretty frequently, it would be very easy to miss chunks of data.

Just something to keep in mind.

Comment 4 by Raymond Camden posted on 3/30/2009 at 5:40 PM

Absolutely good point Dan. For real time notifications you really want to look at something else, like SMS.

Comment 5 by JC posted on 3/30/2009 at 6:58 PM

hmm. We used to do something like this with Jabber. It would IM us with errors. Just from our development box though.

Comment 6 by Robbie Byrd posted on 3/31/2009 at 2:00 AM

Thanks, Ray! This is exactly what I was looking for.

Comment 7 by David Jacobson posted on 11/16/2011 at 12:07 AM

Ray, this is very interesting and I would like to try it, however, I think I'm not 100% sure how to implement.

I have all the code on one page (tstRSS.cfm) but I don't see what I see in your example xml page. I think I am missing a step. Can you clairfy for me?

Comment 8 by Raymond Camden posted on 11/16/2011 at 12:20 AM

So what -are- you seeing?

Comment 9 by David Jaocbson posted on 1/24/2012 at 7:48 PM


Our xml file stopped updated and I am wondering if you have an updated way to do this?


Comment 10 by Raymond Camden posted on 1/24/2012 at 8:01 PM

Well, the XML comes from the log file. Did the log file stop updating? You need to provide a bit more detail.

Comment 11 by David Jaocbson posted on 1/24/2012 at 8:07 PM

Sorry about that Ray. Nope the log file has about 10 entries from 1/10/12 through 1/22/2010 and the RSS feed shows the last entry as 12/5.

Now when I go the to URL that creates the XML file I am getting the following error.

The element at position 0 of dimension 1, of array variable "LINES," cannot be found.

If you want I can email you the file in question if that would help?

Comment 12 by Raymond Camden posted on 1/24/2012 at 8:19 PM

Well look at the error. What line of code is that?

Comment 13 by David Jaocbson posted on 1/24/2012 at 9:04 PM

<!--- our data, in line form --->
<cfset lines = []>

<!--- flag to let us ignore the first line --->
<cfset doneOne = false>

<cfloop index="i" file="#logfile#">
<cfif doneOne>
<cfset arrayAppend(lines, i)>
<cfset doneOne = true>

Comment 14 by Raymond Camden posted on 1/24/2012 at 9:09 PM

And what line is throwing the error? Does logfile have content even?

Comment 15 by David Jaocbson posted on 1/24/2012 at 9:17 PM

Sorry Ray, having a senior moment. The applciation file says the error is line 44:

<cfloop index="x" from="#arrayLen(lines)#" to="#max(0,arrayLen(lines)-25)#" step="-1">
44 ----> <cfset arrayAppend(latestLines, lines[x])>

Comment 16 by Raymond Camden posted on 1/24/2012 at 9:23 PM

Ok, so what is the size of the array? Again, did you check to see if the log actually has content? If it's blank, it is best to stop processing.

<cfif arrayLen(lines) is 0>

My code assumed an active log file filled with content.

Comment 17 by David Jaocbson posted on 1/24/2012 at 9:37 PM

Well I guess I am lost. :)
If I dump out the ArrayLen of lines I get 8712.

Looking at the log file in CFadmin (Cannot usually but asked our IT for a peak) I see 9 rows of "Error".
I think it may be time for lunch! :D


Comment 18 by Raymond Camden posted on 1/24/2012 at 9:44 PM

At this point it may make sense for us to take it offline. It is throwing an error trying to read lines[0], but if arraylen(lines) is 8712, that shouldn't happen. You may want to add a quick debug line before it:

<cfoutput>x is #x#<br/></cfoutput>

Just to see whats going on.

Comment 19 by Raymond Camden posted on 1/25/2012 at 12:14 AM

Ugh. Ok, found it. Thanks David for sending me his log file which made it really clear. I've already fixed the code above, but the issue was here:

<cfloop index="x" from="#arrayLen(lines)#" to="#max(0,arrayLen(lines)-30)#" step="-1">

The logic was meant to be this, in English:

Go from the number of lines to the row 30 below that, unless that puts us below 0 and then use 0 instead.

The right logic is this:

<cfloop index="x" from="#arrayLen(lines)#" to="#max(1,arrayLen(lines)-30)#" step="-1">

In other words, make 1 the lowest we can go. -sigh-

Comment 20 by David Jaocbson posted on 1/25/2012 at 12:37 AM

Should the URL be the cfm file that creates the XML or the XML file itself? Also, do I need to the cffile create at the bottom of the getlog.cfm page as I didn't see that in your code?

I guess I'm trying to figure out how I get the xml file updated and pushed to subscribers?

Comment 21 by Raymond Camden posted on 1/25/2012 at 1:07 AM

Well, in my demo, I output the results to screen. That works, but would be a bit slow under load. Anything file related isn't going to be terribly quick. So normally I'd write it out to an XML file and share THAT url.

Now obviously the XML file will not be up to date. You can use the CF Scheduler to schedule an update every N minutes or hours to your liking.