Thursday, May 24, 2012

Rancho: The Code Coverage tool for ColdFusion

Code coverage is an essential part of testing, it reflects the completeness of the testing done. Code coverage is instrumental in providing information about how much of the total code is actually tested by the test suite. However code coverage does not guaranty the quality of testing but it reflects the area covered in testing.

Another important point is white box testing and code coverage(functional) are two separate activities. while the former determine the probable inputs and output for a function, the later determine if all lines and branching conditions of the function are executed, in short all paths have been covered or not. There are quiet a bunch of tools available for Java applications. But for ColdFusion applications, till now there is no way to know the code coverage.

This post provides an experimental tool, Rancho, to get code coverage for any ColdFusion application. Rancho can be used as a ColdFusion Builder extension or as an application. Download Rancho as an extension to CFB.

Rancho is the code coverage tool written in CFML for code coverage of CFML application. There are different metrics a code coverage tool can provide(line coverage,functional coverage,decision coverage) but as of now Rancho only provides the line coverage.

How to get code coverage?

There are 3 steps to get the code coverage.
  1. Instrument your CFML application using Rancho.
  2.  Run your test suite.
  3.  Generate a report of code coverage using Rancho.
Steps 2 and 3 can be iterated if you want to tun your testcases in phases.

Using Rancho as an extension to CFB

Download the Rancho and add it as an extension to CFB. View this article to add  an extension to CFB.

 Note: Since this project is hosted on Github and it provides project to download as a zip. This zip can not directly be imported to CFB as extension because extension files are one level down. So either rezip at the level of ide_config.xml OR directly download from RIAForge

You can also run Rancho as an application. Download it from here.

STEP 1 - Instrument:  Go to the Application folder in CFB navigator you need to instrument. Do a right click and select Rancho -> Instrument
Figure 1: Rancho Instrumentation
This will first take backup of the existing folder and start instrumenting all files in that folder.
Figure 2: Instrumentation Done
Note: Instrumentation is a one time activity. Do not instrument more than once.

STEP 2 - Running test suite: Once instrumentation is done you can proceed to running your testcases for the application instrumented.

STEP 3 - Getting Code Coverage Data: Once you are done with your testing or otherwise you can extract a report of code coverage. From Figure 1 click on Rancho -> Code Coverage or you can directly access http://localhost:8500/Rancho/handlers/coveragegrid.cfm
Figure 3: Code Coverage preview

This will show you the preview of your code coverage, indicating total coverage of application. Also it lists code coverage for individual files. You can further drill down to individual files to see executed and un-executed lines.

Figure 4: Code Coverage details
 This shows individual CF statements executed(green) and  un-executed CF statements(red).


Configuring Rancho

While instrumenting Rancho adds a trace before each CF statement. Now there could be some statements that can not be traced like as putting a trace before or after could be an issue. also there are some tags for which trace can not be put before e.g or .
To handle this 2 files doNottrace.txt and putTraceAfter.txt are placed at Rancho\handlers\config. Add any tag/end tag which falls in above category. As of now only some tags/end tags have been added. As you find more such tags please update them to the repository.

Limitations

  • Supports only tags not script, so any CFML code inside will be uncovered.
  • Multiple level of folders is not supported yet. Instrument and generate report for single folder level at a time.
  •  Instrumentation of single file is not supported. Only folders can be instrumented.
  • Only line coverage is provided.

Improvements

Rancho is an experimental code coverage tool so there are some improvements required like:
- Support for .
- Multi folder support
- UI improvements
- Fine tuning tags in doNottrace.txt and putTraceAfter.txt
- Enhance Rancho to extend Function and Decision coverage

I have created a github project(I am new to it). Please contribute to it.

PS:  You can also run Rancho as an application. Download it from here. Place Rancho in webroot of CF and run index.cfm. Also make sure the folder you are instrumenting has write permissions.




16 comments:

Anonymous said...

Screen shots 3 and 4 are missing

Anonymous said...

Screenshots 3 and 4 are missing.

Adam Tuttle said...

Looks like there are several places where you mention specific bits of code. I'm guessing they were tags, because they aren't showing up. Guessing that blogger is blocking anything between < and > (< and >)... Could you try and fix them? It's a little confusing with those bits missing.

Dan said...

This is just what I'm looking for, many thanks.

Now onto my problem. I've installed Rancho and am attempting to Instrument my code directory in the CFB 2 navigator window. I right click and goto Rancho/Instrument. I get a dialog box saying "User finished with the following error message: No response."

Any ideas what's happening here?

Kunal Saini said...

@Dan

Is your directory you are instrumenting has write permissions?

Kunal Saini said...

@Adam

Do you mean to say after downloading Rancho as a zip file on some files < and > are missing. Is there any soecfin such file.

Kunal Saini said...

@Dan
Can you try wih Rancho as an standalone application.
https://github.com/ksaini/Rancho_standalone

Place it at cf_webroot and run index.cfm

Steven Erat said...

I've tried several times to add Rancho as an extension by importing it from the folder I sync'd to Github. This copies Rancho to the CF10 webroot for the CF Project. But each time it reports at the end that an error occurred on import with message "null". The extensions panel shows Rancho, but I never get the menu context for Rancho.

So I then tried to run it as an application by going to http://localhost:8501/Rancho/handlers. There I selected "View Report" and I got an error in Rancho/handlers/coveragegrid.cfm because it uses back slashes "\" (Windows only), and I'm on a Mac. This would be a nice exercise to test commiting to the project by changing all back slashes to forward slashes "/" which will work on both Windows and *nix/Mac.

Finally, I clicked on the link for Instrument Code. There I got another error: "The required parameter IDEEVENTINFO was not provided." at /Rancho/handlers/instrument.cfm:1

Steven Erat said...

I just realized that by "rancho standalone" you meant that you have an entirely different project for that at https://github.com/ksaini/Rancho_standalone.

Ok, I'll try that.

Steven Erat said...

Ok, so the standalone app is installed. However, the same back slash delimiter issue exists. The variable basefile is calculated using "\" not "/".

Variable MYFILE is undefined.

at cfinstrumentprogress2ecfm1936366410$funcREMOVEMULTILINECOMMENT.runFunction(/Users/stevenerat/Rancho_standalone/instrumentprogress.cfm:145) at cfinstrumentprogress2ecfm1936366410$funcMARKFORCODECOVERAGE.runFunction(/Users/stevenerat/Rancho_standalone/instrumentprogress.cfm:67) at cfinstrumentprogress2ecfm1936366410.runPage(/Users/stevenerat/Rancho_standalone/instrumentprogress.cfm:42)

Kunal Saini said...

@Steven
Thanks for pointing "\" issue. I will fix that. Will double check extention.

On http://localhost:8500/Rancho/instrument.cfm have you provided the directory path which you want to instrument. As of now Rancho can not instrument individual files.

Also please make sure that CF has write permission to the directory you are instrumenting.

This exception looks like file is locked.

Kunal Saini said...

@Dan,Steven

Rancho was not working as an extention because Git created a zip from itself and config files are one level down. So either rezip at level of ide_config.xml or download from
http://rancho.riaforge.org/index.cfm?event=action.download

Anonymous said...

I have tried using the standalone version and nothing seems to happen when i click "instrument code". When i click on "view reports". Any chance of some screenshots of the standalone version working?

Anonymous said...

Sorry, when i click on "view reports" i get application error:
Variable FILE_PATH is undefined.

The error occurred in C:/Program Files/Apache Software Foundation/Apache2.2/htdocs/inetpub/wwwroot/rancho/coveragegrid.cfm: line 28

26 : structInsert(baselinestruct,trim(y),trim(z),"yes");
27 : }
28 : Application.baselinecodebase = file_path & '_bak';
29 : WriteOutput(" align=center>LOC executed
" );
30 : WriteOutput("TOTAL LINES : " & totalline & "
");

Had to remove the h2 from the error to post on this blog

Kunal Saini said...

@Anonymous,

When you clicked on instrument code, have you provided any folder path(file path is not supported). Also that folder should have write permissions. I shall provide screenshots in a day or 2 to make it more clear.

Code Fusion, LLC (Kevin Penny) said...

I have an update for your application.
In coverageGrid.cfm I have directories and filenames with multiple 'periods' in them
i.e. /admin.something.com/fusebox.xml.cfm

If you change line 18 'file_path' to this instead, it will work much better

file_path = reverse(listrest(reverse(x),":"));

VS
file_path= left(x,FindNoCase(".",x)+3);

As your code expects the period in the filename to be ONLY ONE, while there are many cases where there could be mulitple in the url -

I'm basically reversing the string, chopping off the first in the list of the colon delimited string from the 'baseline.txt' file url then re-reversing it i.e.
starts out as
C:\websites\admin.jobs2web.com/fusebox.xml.cfm:1
and ends up as
C:\websites\admin.jobs2web.com/fusebox.xml.cfm

Which is I think what you want