The development team I’m a part of at Ohio Mutual Insurance Group is about to develop our own web services for the first time and I want to make sure we test them properly. So, for about two months now I have been experimenting with using SoapUI to aid in developing our services. One very important aspect of my experiment is to determine how difficult it is to run SoapUI functional tests from a Hudson CI job. In this blog are my rough thoughts on how well SoapUI functional tests integrate with the build process we use on Hudson.
Functional testing is not all SoapUI does, but it is one outstanding feature that I’m focusing on for now. Even though SoapUI has a solid interface for running functional tests, it doesn’t make any sense to have compilation, unit tests, and deployment automated but not functional tests. It appears that the SoapUI developers feel the same way and have provided some help in this area. There are two ways to run SoapUI functional tests automatically: integrate with JUnit or use the SoapUI “testrunner” shell script. Unfortunately, the official documentation on both is sparse (see the hyperlinks in the previous sentence) and not helpful beyond a bare minimum of support.
Note: All mention of SoapUI in this entry is related to version 3.5.1.
Integrating with JUnit
The documentation for how to integrate with JUnit is awful. It barely provides the necessary information for using JUnit and only offers part of the story. If you really want to know how to run your SoapUI tests from JUnit you’re mostly stuck with using the SoapUI JavaDocs and your own experimentation (Intellisense is your friend here). However, the one advantage that this approach has over using the testrunner script is that you can have more control over how and when tests are executed. If you need that control but want to automate the process of running the test cases, this is the way to do it. But, if you just need to automate the test case runs without control, the command line test runner is preferable. Here is some help with the JUnit method.
To run SoapUI tests from JUnit tests you have to use the SoapUI API. This means that the SoapUI libraries need to be in your Classpath. The main archive needed is the bin\soapui-3.5.1.jar file. If they’re not already in your Classpath, you may also need some of the JARs in the lib directory. At a minimum you will need the xmlpublic-2.4.0.jar because it contains the XMLException class that is thrown by some of the classes in the SoapUI API.
When you’ve got your Classpath setup you can use the SoapUI API in your JUnit tests. There are several ways to run SoapUI functional tests with the SoapUI API. You can just run all the test cases in all the test suites in a SoapUI project without control over the order (you choose whether to run them synchronously or asynchronously), run a whole test suite, or run individual test cases within a test suite. For the second and third options, you can either blindly iterate over test suites and test cases or you can run test suites or test cases by name. Which way you run the tests is based on how much control you need over the order of test execution. When running a test suite you have control over whether the test cases in the suite are run synchronously or asynchronously. The options that you have available reflect what you can do in SoapUI’s user interface. The problem is digging through the JavaDocs to find out which methods on what objects to call.
Here is an example of a JUnit test that runs all the test cases in a SoapUI project in two different ways. The lessControl method just runs every test case in the project in the simplest way whereas the fullControl method iterates over the test suites and the test cases in the project.
package com.omig.soapUISamples;
import java.util.List;
import org.junit.Test;
import com.eviware.soapui.impl.wsdl.WsdlProject;
import com.eviware.soapui.model.support.PropertiesMap;
import com.eviware.soapui.model.testsuite.TestCase;
import com.eviware.soapui.model.testsuite.TestRunner;
import com.eviware.soapui.model.testsuite.TestSuite;
import com.eviware.soapui.model.testsuite.TestRunner.Status;
import com.eviware.soapui.tools.SoapUITestCaseRunner;
import static org.junit.Assert.*;
public class SoapUITest {
@Test
public void lessControl() throws Exception {
SoapUITestCaseRunner runner = new SoapUITestCaseRunner();
runner.setProjectFile(“SoapUIWorkspace/Sample-soapui-project.xml”);
runner.setPrintReport(true); //Outputs a small table to stdout of test results.
runner.run();
}
@Test
public void fullControl() throws Exception {
WsdlProject project = new WsdlProject(“SoapUIWorkspace/Sample-soapui-project.xml”);
List<TestSuite> testSuites = project.getTestSuiteList();
for( TestSuite suite : testSuites ) {
List<TestCase> testCases = suite.getTestCaseList();
for( TestCase testCase : testCases ) {
System.out.println(“Running SoapUI test [" + testCase.getName() + "]“);
TestRunner runner2 = testCase.run(new PropertiesMap(), false);
assertEquals(Status.FINISHED, runner2.getStatus());
}
}
}
}
With the code above you’ve got a hint at how run SoapUI functional tests from a JUnit test. To actually make this automatic you’ve got (at least) one more step. In my situation we use Ant to script our software builds. We then use Hudson to schedule the builds and make it all run automatically. There are a lot of other ways you could do this but I’ll only show the Ant way here. Any build tool that has support for JUnit will suffice.
Below is an example of an Ant task that runs the JUnit tests that are executing the SoapUI tests. The JUnit output is being generated in XML format and sent to a folder named soapui-test-reports.
<path id="classpath">
<!-- The path to the lib folder in the SoapUI installation.-->
<fileset dir="${env.SOAPUI.LIB}"/>
<!-- The path to the bin folder in the SoapUI installation.-->
<fileset dir="${env.SOAPUI.BIN}"/>
</path>
<path id="soapui-workspace">
<fileset dir="SoapUIWorkspace">
<include name="**/*.xml"/>
</fileset>
</path>
<target name="soapui-tests" depends="compile">
<mkdir dir="../soapui-test-reports" />
<junit printsummary="withOutAndErr"
showoutput="true"
haltonerror="on"
haltonfailure="on"
filtertrace="off"
fork="no"
forkmode="once">
<classpath refid="classpath" />
<classpath refid="soapui-workspace" />
<formatter type="xml" />
<batchtest todir="../soapui-test-reports">
<fileset dir="tests">
<include name="**/*SoapUI*.java"/>
</fileset>
</batchtest>
</junit>
</target>
Testrunner Script
The documentation for the testrunner script is much clearer than for integrating with JUnit. Integrating the testrunner.bat file with your build process is simple and straight-forward. An example of an Ant task to do this is shown below. See SoapUI’s documentation for an explanation of the command line switches.
<target name="soapui-tests-cmdline" depends="compile">
<exec executable="${env.SOAPUI.BIN}/testrunner.bat"
failonerror="yes"
failifexecutionfails="yes">
<arg value="-rjaf"/>
<arg path="../soapui-test-reports"/>
<arg path="SoapUIWorkspace/Insured-Access-soapui-project.xml"/>
</exec>
</target>
Which Method Is Best?
In my experience so far the command line testrunner tool is the better method for several reasons:
- There’s less work because you do not have to write a JUnit test suite/case to run the SoapUI tests.
- It creates a less fragile environment because writing a JUnit test to run the SoapUI tests makes it very difficult not to hard code the path to the SoapUI project file. You don’t have to worry about this with the command line because the input to script is parametrized.
- The command line test runner can output “correct” JUnit output. The JUnit reports generated by the command line test runner split results by test suite then test case. This is ideal when the results are viewed graphically because you get a full count of actual test cases run and you can drill-down to the individual test cases to see its run status*.
- To integrate with JUnit you have to use the SoapUI API, which is not documented well. There are Javadocs, but actual documentation is sparse in them.
* As far as I can tell it is difficult or impossible to do this by integrating with JUnit manually. To do so you would have to somehow generate one JUnit test for each test case in all the test suites. The SoapUI API allows you to easily iterate over the test suites and test cases, but there is no way in Java to dynamically generate methods for each test case.
