Sitemap Generator

Earlier today Yahoo and Google announced their collaboration on Sitemaps.org. Sitemaps provide a way to describe to a search engine what pages make up your web site. I've had sitemap support in BlogCFC for a while, but today I wrote a little UDF you can use to generate sitemap xml. It will take either a list of URLs or a query of URLs. Enjoy. I'll post it to CFLib later in the week.

view plain print about

1<cffunction name="generateSiteMap" output="false" returnType="xml">
2    <cfargument name="data" type="any" required="true">
3    <cfargument name="lastmod" type="date" required="false">
4    <cfargument name="changefreq" type="string" required="false">
5    <cfargument name="priority" type="numeric" required="false">
6
7    <cfset var header = "<?xml version=""1.0"" encoding=""UTF-8""?><urlset xmlns=""http://www.sitemaps.org/schemas/sitemap/0.9"">">
8    <cfset var result = header>
9    <cfset var aurl = "">
10    <cfset var item = "">
11    <cfset var validChangeFreq = "always,hourly,daily,weekly,monthly,yearly,never">
12    <cfset var newDate = "">
13    <cfset var tz = getTimeZoneInfo().utcHourOffset>
14
15    <cfif structKeyExists(arguments, "changefreq") and not listFindNoCase(validChangeFreq, arguments.changefreq)>
16        <cfthrow message="Invalid changefreq (#arguments.changefreq#) passed. Valid values are #validChangeFreq#">
17    </cfif>
18
19    <cfif structKeyExists(arguments, "priority") and (arguments.priority lt 0 or arguments.priority gt 1)>
20        <cfthrow message="Invalid priority (#arguments.priority#) passed. Must be between 0.0 and 1.0">
21    </cfif>
22
23    
24    <cfif structKeyExists(arguments, "lastmod")>
25        <cfset newDate = dateFormat(arguments.lastmod, "YYYY-MM-DD") & "T" & timeFormat(arguments.lastmod, "HH:mm")>
26        <cfif tz gte 0>
27            <cfset newDate = newDate & "-" & tz & ":00">
28        <cfelse>
29            <cfset newDate = newDate & "+" & tz & ":00">
30        </cfif>
31    </cfif>
32
33    
34    <cfif isSimpleValue(arguments.data)>
35        <cfloop index="aurl" list="#arguments.data#">
36            <cfsavecontent variable="item">
37<cfoutput>
38<url>
39    <loc>#xmlFormat(aurl)#</loc>
40    <cfif structKeyExists(arguments,"lastmod")>
41    <lastmod>#newDate#</lastmod>
42    </cfif>
43    <cfif structKeyExists(arguments,"changefreq")>
44    <changefreq>#arguments.changefreq#</changefreq>
45    </cfif>
46    <cfif structKeyExists(arguments,"priority")>
47    <priority>#arguments.priority#</priority>
48    </cfif>
49</url>
50</cfoutput>
51            </cfsavecontent>
52            <cfset item = trim(item)>
53            <cfset result = result & item>
54        </cfloop>
55
56    <cfelseif isQuery(arguments.data)>
57        <cfloop query="arguments.data">
58            <cfsavecontent variable="item">
59<cfoutput>
60<url>
61    <loc>#xmlFormat(url)#</loc>
62    <cfif listFindNoCase(arguments.data.columnlist,"lastmod")>
63        <cfset newDate = dateFormat(lastmod, "YYYY-MM-DD") & "T" & timeFormat(lastmod, "HH:mm")>
64        <cfif tz gte 0>
65            <cfset newDate = newDate & "-" & tz & ":00">
66        <cfelse>
67            <cfset newDate = newDate & "+" & tz & ":00">
68        </cfif>
69        <lastmod>#newDate#</lastmod>
70    </cfif>
71    <cfif listFindNoCase(arguments.data.columnlist,"changefreq")>
72    <changefreq>#changefreq#</changefreq>
73    </cfif>
74    <cfif listFindNoCase(arguments.data.columnlist,"priority")>
75    <priority>#priority#</priority>
76    </cfif>
77</url>
78</cfoutput>
79            </cfsavecontent>
80            <cfset item = trim(item)>
81            <cfset result = result & item>
82
83        </cfloop>
84    </cfif>
85
86    <cfset result = result & "</urlset>">
87
88    <cfreturn result>
89
90</cffunction>

Comments

[Add Comment] [Subscribe to Comments]

aleksandar said on 11-20-2006 at 6:50 AM #
	How actualy it works?

Raymond Camden said on 11-20-2006 at 8:02 AM #
	You pass in either a list of URLs or a query. I added it to CFLib last night and there is a bit more documentation there. http://www.cflib.org/udf.cfm?id=1596

BL said on 11-28-2006 at 10:41 AM #
	do you think it would be hard to build and site crawler and link parser in cf to use with this udf?

Raymond Camden said on 11-28-2006 at 2:31 PM #
	BL: Sure, I'll make it a Friday test. ;)

BL said on 11-28-2006 at 6:21 PM #
	nice. you feelin a little regexy?

dickbob said on 03-03-2007 at 7:07 AM #

Can I offer a couple of amendments in the light of my experience of using this UDF to submit to Google.

Code changes occur after the comment "reformat datetime as w3c datetime / http://www.w3.org/TR/NOTE-datetime";.

1. Change the test of tz to be "gt" rather than "gte". To be honest this is really just a personal style thing, +00:00 looks better than -00:00 to me, and doesn't seem to effect Google.

2. Make the hour number format "00" for the newDate offset eg. numberFormat(tz,"00"). So the lines should read newDate = newDate & "-" & numberFormat(tz,"00") & ":00" and newDate = newDate & "+" & numberFormat(tz,"00") & ":00"

HTH

Raymond Camden said on 03-03-2007 at 4:20 PM #
	I've made both changes. I've also changed the UDF to use stringbuffer, this makes it quicker. Unfortunately - CFLIB is causing me fits now - so it's not hooked up yet. I will refresh it later.

dickbob said on 03-06-2007 at 9:13 AM #
	I know you most likely thought of it but I should have mentioned that the same changes need to be applied when the data is supplied as a query.

Ruth said on 04-19-2007 at 2:45 PM #
	Hi there, I was just tasked to produce an XML sitemap. I noticed that you mention that you would make a Friday test out of the idea of making a site crawler. I searched for that term in your blog and didn't see any results. Did this ever occur?

Raymond Camden said on 04-19-2007 at 5:32 PM #
	Not yet - no.

Ruth said on 04-20-2007 at 3:04 PM #
	I started a solution to create a map on the server side - but it isn't a crawler. I'll post the code after I get it cleaned up enough and some of the kinks worked out. I am having issues with cfdirectory w/recursion at webroot. I get the pesky null pointer error, which I am attributing to archived directories, etc bloating the query. I still need to prove that is the cause.

Raymond Camden said on 04-20-2007 at 3:11 PM #
	Ruth, don't forget that if you confirm a bug, you can report it at: http://www.adobe.com/go/wish

Jeremy Halliwell said on 08-14-2007 at 7:06 AM #
	Ray, the udf on cflib although dated March 9 2007 doesn't have the changes you mentioned you'd added to timezone and the use of stringbuffer. Also the getTimeZoneInfo().utcHourOffset assigned to tz returns (for me at least) an offset value, eg "-1" for UK, so the test which adds the "+" or "-" later is unnecessary and makes the date format invalid (eg 2007-08-04T19:24+-1:00).

Raymond Camden said on 08-14-2007 at 8:00 AM #
	I'm updating it in 10 seconds. Will you please give it a try?

Jeremy Halliwell said on 08-14-2007 at 8:22 AM #
	Yes that works great, thanks Ray.

Snake said on 09-04-2007 at 12:40 PM #
	You say blogcfc has had sitemap support for some time, in what way? Is it supposed to generate a sitemap.xml file ? And if so, how? I can't find any option to do this. Or do I need to update as I am on blogCFC 5.5

Raymond Camden said on 09-04-2007 at 12:53 PM #
	There should be a file named sitemap or googlesitemap.cfm in the root directory.

marco said on 10-12-2007 at 9:17 PM #

Hi I can't figur out how to combine these values in to one and to an xml output:

<cfset siteMapXML = generateSiteMap(data=urls,changefreq="daily",priority="1.0", lastmod=now())>
<cfdump var="#xmlParse(siteMapXML)#">
<cfset siteMapXML = generateSiteMap(qurls)>
<cfdump var="#xmlParse(siteMapXML)#">

I want these combined as a need to put it all to one xml sitemap, the .cfm sitemap takes to long to load, big sitemap.

thanks

Raymond Camden said on 10-23-2007 at 5:03 PM #
	Well I think you can just combine both XML files. You would want to remove the <xml> header from the second one though. Not exactly sure - but it's definitely possible.

m van den oever said on 10-23-2007 at 6:18 PM #
	of course, i was thinking the hard way as usual, thx

Adam said on 01-18-2008 at 1:49 PM #
	Could someone break this down for me? I've read this post over and over and looked at the CFLib documentation. I don't know if I"m missing something or (most likely) I just don't know what I'm doing. Any shove in the right direction is greatly appreciated.

TonyD said on 07-09-2009 at 12:10 AM #

Hi Ray, many retrospective thanks for other tips.

I'm with Adam on this. As indicated at siteMap.org, that the search engine will look for an XML document called sitemap is clear.

That the generateSiteMap functions returns this XML code for the urls provided is also clear. That the xmlparse function turns this into an xml object (and the xml code is visible in codeview on the webpage) is clear.

So instead of using cfdump, you pick whichever of the three options you prefer and then just surround the xmlparse with cfoutput tags and this hands the xml to the search engine?

Its one of those things where you could end up ten years later in a bar discussing XML and girls only to find out that you've been setting up invalid site maps for ten years.

Raymond Camden said on 07-09-2009 at 7:32 AM #
	Yeah, take the result and just output it.

Karan Joshi said on 02-18-2010 at 9:42 PM #
	Used bits n pieces of your code...works like a charm...thanks.

Custard Pie said on 08-23-2010 at 9:56 AM #
	How are using this? I'm able to call it and it seems to output the header and closing mark up, but I haven't made the connection as to how it gets all the site directories to produce a complete site map. I can see I can pass the data argument to it, but what are you using to traverse the site directories so that this component can wrap it in xml?

Raymond Camden said on 08-24-2010 at 7:02 AM #
	I'm not traversing the site. You need to figure that part out yourself. Every site is different. So for example, on this blog, I can make a map by getting all the entries. For a news site, you would get all the news article. Amazon would get all their products. Essentially - look at your site data and set up a query that represents the content.

[Add Comment] [Subscribe to Comments]