Depending on the requirements, you might be able to drop to the OS and just use the OS to merge the files together--which is going to be way faster than using CF.

In Windows, you can do something like the following from a command prompt:

copy /b *.txt big-old-file.txt

Or:

copy /b file1.txt +file2.txt +file3.txt merged1-3.txt

Now obviously if you have to do some parsing to the files before merging them, this may not work for you. But if the goal is to just merge a ton of individual files into a single file, you can do this very easy using command line tools.

As for your solutions, if the outputted file is going to be very large, I'm not a fan of trying to create everything in memory--because it can cause issues w/the JVM. I'm a much bigger fan of appending the data to a file and writing the data in controlled chunks. It's more work, but will work reliably even if you need to merge really large text files (that you don't want to read all in memory at one time.)

You can use java.io.LineNumberReader (which your http://www.cflib.org/udf/Fi... UDF uses) to pull in chunks from an external file, then append those chunks to another file.

This could be overkill for this project, but I thought I'd mention it.

Interesting point Dan, and while "we" ("we" being the collective CF bloggers, speakers, etc) tend to mention not writing OS specific code, this is an example where for practical reasons it may make sense. So question - do you feel like running a speed test?

As to your 3rd idea (lineNumberReader), wouldn't you agree that UDF does not make sense anymore in CF8+? Certainly it is convenient (to go to line N), but since fileReadLine should be using similar code under the hood then it should perform well.

Very interesting Ray. In tests I've run comparing string concatenation to cfsavecontent and the java string buffer, cfsavecontent has been the fastest of all three methods. For string concatenation, I've seen times like 27 seconds vs. about 160ms or less for cfsavecontent, while using the java string buffer was around 420 ms. A caveat, this test was run using cf7.02 on a WinXP box. Let me know if you'd like a copy of the code and I can send it along.

Here are the results of one test (iteration of 50000 in each case):

String Concatenation
string & string: 28297ms
String Length: 650000

CFSaveContent
cfsavecontent: 156ms
String Length: 650001

Using Java String Buffer
java string buffer: 422ms
String Length: 650001

----
Here's the code:
<cfset runtime = CreateObject("java", "java.lang.Runtime").getRuntime()>
<cfset total_memory = runtime.totalMemory()> <cfset runtime.gc()>
<cfset memory_before = (total_memory-runtime.freeMemory()) / 1024 / 1024>
<cfoutput>Memory Before: #round(memory_before)# Megs<br> <cfset string1 = "">
<cftimer label="string & string" type="inline">
<cfloop from="1" to="50000" index="i">
<cfset string1 = string1 & "Hello World. ">
</cfloop>
</cftimer>
<cfoutput>
String Length: #len(string1)#</cfoutput><br>
<cfset memory_after = (total_memory-runtime.freeMemory()) / 1024 / 1024> Memory After: #round(memory_after)# Megs -- Increase of #round(memory_after - memory_before)# Megs<br> <br />
</cfoutput>
<cfset runtime.gc()>
<cfset memory_before = (total_memory-runtime.freeMemory()) / 1024 / 1024>
<cfoutput>Memory Before: #round(memory_before)# Megs<br>
</cfoutput>

<cftimer label="cfsavecontent" type="inline">
<cfsavecontent variable="string2">
<cfloop from="1" to="50000" index="i">
<cfoutput>Hello World.</cfoutput>
</cfloop>
</cfsavecontent>
</cftimer>

<cfoutput>String Length: #len(string2)#</cfoutput><br>
<cfset memory_after = (total_memory-runtime.freeMemory()) / 1024 / 1024>
<cfoutput>Memory After: #round(memory_after)# Megs -- Increase of #round(memory_after - memory_before)# Megs<br></cfoutput>

regards,
larry

Just an addenum to my previous note. I decided to do String Concatenation with CFSavecontent using CF9 on a Windows 7 box. here are the results:

Memory Before: 208 Megs
string & string: 19718ms String Length: 650000
Memory After: 252 Megs -- Increase of 43 Megs

Memory Before: 31 Megs
cfsavecontent: 32ms String Length: 650001
Memory After: 38 Megs -- Increase of 7 Megs

Hello Ray,

I found this entry by Ben Nadel very helpful. Writing Enormous Files Based On Massive Record Sets In ColdFusion @ http://bit.ly/gGqVJJ .

I added some comments at the bottom on how to change his process if you need to specify an encoding type like UTF-8.

Based on your tests above I did this and it took 790ms

<cfscript>
thisTick = getTickCount();
// file to write
fileToWrite = expandPath('./test.txt');
// create the file
theFile = createObject("java","java.io.File").init(javaCast("string",fileToWrite));
// create my java file writer
fileWriter = createObject("java","java.io.BufferedWriter").
init(createObject("java","java.io.OutputStreamWriter").
init(createObject("java","java.io.FileOutputStream").
init(theFile),"UTF8"));

// create string object
stringToWrite = createObject("java","java.lang.StringBuffer").Init();
for(x=1; x<=200000; x++){
stringToWrite.Append(x & chr(13) & chr(10));
}
// write line
fileWriter.write(stringToWrite.toString());
// Flush the buffered output stream to make sure there is no straggling buffer data
fileWriter.flush();
// final count
finalTick = getTickCount() - thisTick;
// write output
writeOutput('<p>Took #finalTick# ms to write.');
</cfscript>

I would do this in my RDBMS. In MySQL, I would just use Navicat to spit the file out.

Ben's post goes down to Java - which would not be necessary in CF8 or higher I'd say. Interesting that yours ran even faster though - almost by half.

Sounds like what's really needed here, is a <cffile action="concatenate" input="#fileList#" file="#outputFile#" />. Especially if you don't need to do any modifications to the data, and are just cat'ing the files together.

One of the fastest ways to do this, would be to use the java nio package (in most cases should outperform using buffered streams), and I imagine that the underlying implementation of such a feature, would look something like this (which you should be able to "createObject"ify to do essentially the same thing, if you don't want to compile and load the java class.):

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.channels.FileChannel;
import java.util.Arrays;
import java.util.List;

public class FileUtils {
public FileUtils() {}

public static void concatenate(String target,String inList) throws IOException {
FileChannel out = new FileOutputStream(target).getChannel();
FileChannel in = null;
try {
List<String> inputs = Arrays.asList(inList.split(","));

for (String inputFile : inputs) {
try {
in = new FileInputStream(inputFile).getChannel();
in.transferTo (0, in.size(), out);
}
finally {
try {
if (in != null)
in.close();
}
catch (Exception e) {}
}
}
}
finally {
try { out.close(); } catch (Exception e) {}
}
}

}

Appreciate your article, was very useful.