A reader on Friday asked a pretty cool question:
I really haven't looked into this, but was hoping that someone had already thought of it: I'd like to take a ColdFusion log file (like app.log) and transform to an RSS feed. Any kind of weird parsing that goes on there? I would assume you'd have to convert the log file to some sort of structure... Text Transformation is certainly not my strong suit, maybe one of the reasons I failed that perl class in college.
Didn't the designers of Perl want folks to fail? No? Ok, moving on. So I thought this was a great idea. RSS is a great way to receive updates, and monitoring a log file via RSS could be an awesome way to keep up on things... just as long as the log file isn't going crazy with updates.
So let's take a quick look at how you can accomplish this. The task really isn't that difficult. We need to:
a) Read the log file (easy!)
b) Parse the string (kinda easy!)<br/ > c) Convert the string into a query object (not so tough!)
d) Convert the query into RSS (simple with CFFEED!)
One at a time, here we go.
<cfinclude template="udf.cfm"><!--- log file ---> <cfset logfile = "/Applications/ColdFusion8/logs/application.log">
<cfif not fileExists(logfile)> <cfoutput>Log file "#logfile#" does not exist.</cfoutput> <cfabort> </cfif>
I begin by including a UDF library, which in this case is one UDF, CSVtoArray. I define my log file with a hard coded pointer to the file I want to parse and my local CF install. (Yes, there is a way to find that path dynamically.) I then do a quick sanity check to ensure the file actually exists. Moving on...
<!--- our data, in line form ---> <cfset lines = []><!--- flag to let us ignore the first line ---> <cfset doneOne = false> <cfloop index="l" file="#logfile#"> <cfif doneOne> <cfset arrayAppend(lines, l)> <cfelse> <cfset doneOne = true> </cfif> </cfloop>
<!--- reverse it. This code is the suck. Please be aware it is the suck. ---> <cfset latestLines = []> <cfloop index="x" from="#arrayLen(lines)#" to="#max(1,arrayLen(lines)-20)#" step="-1"> <cfset arrayAppend(latestLines, lines[x])> </cfloop>
Ok, now before going on, I know some of you are screaming at your monitor now. This is not a good way to get the end of the file. Remember we want to show the latest updates in our RSS feed. My code loops through the file, adding each line to an array. I then reverse it by grabbing the last 20 lines. This works, but let me stress. It sucks. It can be done better. I'm going to leave that for tomorrow. For now though, just remember that this is not the most efficient way to get the end of a file.
I now have an array of 20 lines (at most) of text. These are in the format of:
"Severity","ThreadID","Date","Time","Application","Message"
"Information","jrpp-18","12/23/08","15:21:20",,"/Applications/ColdFusion8/logs/application.log initialized"
"Error","jrpp-26","12/23/08","17:02:44","ApplicationName","Variable X is undefined. The specific sequence of files included or processed is: /Library/WebServer/Documents/test4.cfm, line: 6 "
The first line is a header of columns (which I skipped when reading in the file) and each line is comma delimited and wrapped in quotes. Luckily I can parse this format using the csvToArray UDF mentioned earlier. The only issue I have with the UDF is that it expects a file, not one line. You will see how I handle that in the code below.
<!--- query to send to cffeed ---> <cfset q = queryNew("publisheddate,title,content")><!--- for each line, parse it into an array, and add to our query ---> <cfloop index="l" array="#latestLines#"> <cfset data = csvToArray(l)> <!--- udf expected N lines, we sent it one, so quickly copy over itself ---> <cfset data = data[1]>
<cfset queryAddRow(q)> <!--- array pos 3 and 4 is date and time ---> <cfset querySetCell(q, "publisheddate", data[3] & " " & data[4])> <!--- pos 6 is the full string ---> <cfset querySetCell(q, "content", data[6])> <!--- make a title from the full string ---> <cfset title = left(data[6],100)> <cfif len(data[6]) gt 100> <cfset title &= "..."> </cfif> <cfset querySetCell(q, "title", title)> </cfloop>
The first line above creates the query that I'll give to CFFEED. Well don't forget that when creating RSS feeds with a query, you need to either a) name your columns right or b) use the columnMap feature to tell the feed generator to map certain RSS items to particular query columns. Since I'm building the query from scratch I'll just use the columns that make sense for RSS. I parse the line (and notice how I copy the array over itself, I explained why above), and then I simply copy relevant data items over into my query.
I made the call that I'd use the date, time, and message values from the ColdFusion log. That made the most sense for this type of log. Other log files would probably warrant different logic. I combine the date and time when adding it to the query. Message comes in as is, and I reuse a portion of the message for my title. 100 characters isn't an RSS requirement - it just felt right for the feed.
We have our data, but there is one more step before we can create an RSS feed. We have to give some metadata about the feed. The values I chose here were completely arbitrary and really don't matter that much, but they are required by the CFFEED tag.
<cfset meta = { version="rss_2.0", title="ColdFusion Application Logs", link="http://null", description="Latest log items." }>Woot! That's it. Now to just actually convert the query into RSS and serve it up.
<cffeed action="create" query="#q#" properties="#meta#" xmlVar="rss"><cfcontent type="text/xml; chartset=utf-8"><cfoutput>#rss#</cfoutput>
You can see the output here, but note that it is a static export, not a live copy: http://www.coldfusionjedi.com/demos/cfrss/test.xml
As I said, woot! Nothing says excitement like ColdFusion log files. Maybe we can kick it up a notch? For fun, I decided to build a simple little zombie infestation simulator. It takes a pool of 100 people. Every 'round' there is a chance the infestation will start. Once it done, every round will end up with either a dead zombie or a dead human and one more zombie. (Guess who tends to win?) I wrote the simulator to not only output to the screen but also log the values as well. Check it out:
<cffunction name="logit" returnType="void" output="true" hint="Handle cflogging and printing."> <cfargument name="str" type="string" required="true"> <cfargument name="color" type="string" required="false"><cfif structKeyExists(arguments,"color") and len(arguments.color)> <cfoutput><span style="color: #arguments.color#"></cfoutput> </cfif> <cfoutput>#arguments.str# [Humans: #humanpop# / Zombies: #zombiepop#]</cfoutput> <cfif structKeyExists(arguments,"color") and len(arguments.color)> </span> </cfif> <br /> <cflog file="zombie" text="#arguments.str# [Humans: #humanpop# / Zombies: #zombiepop#]"> </cffunction>
<!--- initial human pop ---> <cfset humanpop = 100>
<!--- initial zombie pop ---> <cfset zombiepop = 0>
<!--- has the infection began? ---> <cfset infectionOn = false>
<!--- sanity check, if we hit this, abort ---> <cfset y = 1>
<!--- checks to see if we are done ---> <cfset simDone = false>
<cfloop condition="not simDone && y lt 1000">
<!--- ok, if no infection, we have a 5% chance of starting it. ---> <cfif not infectionOn> <cfif randRange(1,100) lte 5> <cfset infectionOn = true> <cfset humanpop--> <cfset zombiepop++> <cfset logit("Zombie infestation begun.")> <cfelse> <cfset logit("All good!")> </cfif> <cfelse>
<!--- Ok, we either kill a human or a zombie. The more zombies, the greater chance a human dies. Zombies only have a chance to die when the grow in #, since people won't worry about them until they mass up. It is a bit too late when the zombies mass up, but guess what, thats how these things work.
Tweak the +/- to help/hinder the survival rate --->
<cfset chanceHumanDied = randRange(1,zombiePop) + 4> <cfset chanceZombieDied = randRange(1,zombiePop) - 0>
<cfif randRange(1,100) lte chanceHumanDied> <cfset humanpop--> <cfset zombiepop++> <cfset logit("Zombie killed a human.","red")> <cfelseif randRange(1,100) lte chanceZombieDied> <cfset zombiepop--> <cfset logit("Human killed a zombie.","green")> <cfelse> <cfset logit("Nothing happend this time.")> </cfif>
</cfif>
<!--- end the sim if humanpop is 0 or infection started and the zombies were slain ---> <cfif humanpop is 0> <cfset logit("All humans killed. Sim done.")> <cfset simDone = true> </cfif>
<cfif infectionOn and zombiepop is 0> <cfset logit("All zombies killed. Sim done.")> <cfset simDone = true> </cfif>
<!--- This is my sanity check in case my logic is crap ---> <cfset y++> </cfloop>
You can run this yourself here: http://www.coldfusionjedi.com/demos/cfrss/sim.cfm Have fun. I know I ran it about a hundred times or so. I then used the same code above (and attached to this entry), with a slight modification to the RSS properties:
<cfset meta = { version="rss_2.0", title="FunCo Mall Security Logs", link="http://null", description="Latest log items." }>You can view that XML here: http://www.coldfusionjedi.com/demos/cfrss/zombie.xml (Again, not 'live', just a save from my machine.)
Enjoy. I attached all the files to the blog entry.