Ok, so this is probably me having a "Duh" moment here, but I forgot that imageScaleToFit goes both ways. What do I mean by that? If you are using the code to create a thumbnail, like so:

<cfset imageScaleToFit(myImage,255,255,"highestQuality")>

and the image is less than 255 tall or wide than ColdFusion will increase the size of the image. As I said - duh. Luckily this is simple enough to solve:

<cfif myImage.width gt 255 or myImage.height gt 255> <cfset imageScaleToFit(myImage,255,255,"highestQuality")> </cfif>

You could also make a simple UDF out of this as well but not sure if it is worth the trouble.