Perl & Bash: Mass Substitutions from the Command Line

Today I found myself in a situation where I had to do the same global substitution for every file in a directory. I considered writing a script to handle it, but I wanted to try using a Perl one-liner to get the job done.

Given a directory containing a file with the following contents:

  This is a XML file
  This is a XML file

I can substitute every occurence of XML with text using:

  $ perl -i -pe 's/XML/text/g' file1.xml

Afterwards, the file now reads:

  This is a text file
  This is a text file

If I wanted to do the substitution for every file in the current directory, I can wrap the command in a bash for loop:

  $ for i in `ls`; do perl -i -pe 's/XML/text/g' $i; done

This isn't revolutionary by any means, but its harder for me to forget this technique if I post it here.

Maven & soapUI: Testing a Service

In a previous post, I demonstrated the creation of the business logic for an adding web service, that when given a list of numbers, returns their sum. The goal for me was to create a simple service so I can try out automated testing of the service using soapUI from Maven.

In this post, I'll show how I can run the test suite for the service. In the next post, hopefully I'll demonstrate how we can expose the service in Tomcat using Mule.

The created a test suite, when run, produces the following results as demonstrated by the service's log:

INFO  [http-8080-1][2010-07-07 11:23:34,981] - Sum of: [2, 2] = 4
INFO  [http-8080-1][2010-07-07 11:23:35,109] - Sum of: [1, 2, 3] = 6
INFO  [http-8080-1][2010-07-07 11:23:35,133] - Sum of: [1, 2, 3, 4] = 10
INFO  [http-8080-1][2010-07-07 11:23:35,148] - Sum of: [0, 1] = 1
INFO  [http-8080-1][2010-07-07 11:23:35,167] - Sum of: [0, 0] = 0
INFO  [http-8080-1][2010-07-07 11:23:35,221] - Sum of: [] = 0

In the adder service project, I exported the project containing a test suite to a new integration testing module, adder-it, as:

src/test/resources/adder-soapui-project.xml

The POM for the adder-it module contains the following:

<?xml version="1.0" encoding="utf-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.prystasj.adder</groupId>
    <artifactId>adder</artifactId>
    <version>1.0-SNAPSHOT</version>
  </parent>
  <artifactId>adder-it</artifactId>
  <name>adder Integration Testing</name>
  <description>Integration testing with soapUI.</description>
  <build>
    <plugins>
      <plugin>
        <groupId>eviware</groupId>
        <artifactId>maven-soapui-plugin</artifactId>
        <version>3.5.1</version>
        <configuration>
          <projectFile>src/test/resources/adder-soapui-project.xml</projectFile>
          <host>localhost</host>
          <port>8080</port>
        </configuration>
        <executions>
          <execution>
            <id>soap-integration-test</id>
            <phase>integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

To run the tests, with the service up and running locally on port 8080, invoke Maven with:

$ mvn integration-test

Below is some of the build output statements from a test run:

[INFO] [soapui:test {execution: soap-integration-test}]
soapUI 3.5.1 Maven2 TestCase Runner
19:31:31,085 INFO  [WsdlProject] Loaded project from [file:/home/prystasj/workspace/prystasj/mule/adder/adder-it/src/test/resources/adder-soapui-project.xml]
19:31:31,713 INFO  [SoapUITestCaseRunner] Running soapUI tests in project [adder]
...
19:31:31,734 INFO  [SoapUITestCaseRunner] Running soapUI testcase [Add_2_to_2]
19:31:31,750 INFO  [SoapUITestCaseRunner] running step [Add_2_to_2]
19:31:33,305 INFO  [SoapUITestCaseRunner] Assertion [SOAP Response] has status VALID
19:31:33,306 INFO  [SoapUITestCaseRunner] Assertion [Contains] has status VALID
19:31:33,306 INFO  [SoapUITestCaseRunner] Finished running soapUI testcase [Add_2_to_2], time taken: 1543ms, status: FINISHED
...
19:31:33,307 INFO  [SoapUITestCaseRunner] Running soapUI testcase [Add_1_to_2_to_3]
...
19:31:33,385 INFO  [SoapUITestCaseRunner] Project [adder] finished with status [FINISHED] in 1664ms

To access the soapUI Maven plugin, add the following plugin repository to your settings.xml:

<pluginRepository>
  <id>eviwarePluginRepository</id>
  <url>http://www.eviware.com/repository/maven2/</url>
</pluginRepository>

Groovy & Spock: Adding Numbers Together

Lately, I've been playing around with embedding Mule in Tomcat. To get going, I wanted to start with creating an extermely simple service, and along the way I decided to try out the Spock specification framework to test the logic I'd be using in my service. I liked what I found so much, I thought I'd take a little detour and write about my first use of the framework, comparing it to more a common test class, and save the Tomcat posting for another day.

The simple service that will eventually be developed will take a list of numbers and add them up. Here is the class that will ultimately do the math:

class Adder {
    Integer add(List<Integer> numbers) {
        println "adding: $numbers"
        numbers.inject(0) { sum, item -> sum + item }
    }
}

The add method takes a list of numbers and returns their sum. I'm using Integer as the return type and List on the parameter declaration instead of the def keyword to aid in the generation of the service WSDL down the line. While the println won't make into the final version of the class, it might help us out demonstrating what is being sent to the method during testing.

With Spock we'll be writing a specification that describes the behavior of the Adder class:

• Given a list of numbers, they should be added together to produce a sum.
• Given a list containg no numbers, zero should be returned.

The framework's documentation does a great job explaining how to write a spec, so I will not to try and reproduce much of it here. The spec is pretty much self-explanatory, each of the two requirements manifests itself in a method:

import spock.lang.*

class AdderSpec extends Specification {

    def adder = new Adder()

    def "numbers should be added together to produce a sum"() {
        expect:
            sum == adder.add(numbers)
        where:
            sum | numbers
            4   | [2, 2]
            6   | [1, 2, 3]
            10  | [1, 2, 3, 4]
            1   | [0, 1]
            0   | [0, 0]
    }

    def "summing no numbers at all should return zero"() {
        expect:
            sum == adder.add(numbers)
        where:
            sum | numbers
            0   | []
            0   | null
    }
}  

The where clauses resemble a table, with each entry describing the input to the add() method on the right, and the expected result on the left. The first entry in the first method would have the values for sum and numbers substituted into the expect clause so that it would read:

4 == adder.add([2, 2])

The output from the test run helps illustrate how things are added during a run of the spec:

Running org.prystasj.services.adder.AdderSpec
adding: [2, 2]
adding: [1, 2, 3]
adding: [0, 1]
adding: []
adding: null
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.091 sec

If the value for numbers instead read [2, 3], we would get the following failure when we run the test as 2+3 does not equal 4:

Condition not satisfied:

sum == adder.add(numbers)
|   |  |     |   |
4   |  |     5   [2, 3]
    |  org.prystasj.services.adder.Adder@969c29
        false

While I could of used a helper method to remove the redunduncy between the two expect clauses, I liked this output such much, I decided to keep it, as using a method here would modify the resulting failure description. For example:

check(sum, numbers)
|     |    |
false 4    [2, 3]

Compare the above spec class to a more traditional unit test and see what you think:

import org.junit.Before
import org.junit.Test
import static org.junit.Assert.assertEquals

class AdderTest {

    def underTest

    @Before
    void setUp() {
        underTest = new Adder()
    }

    def verifySumsFor(data) {
        data.each { sum, numbers ->
            assertEquals "$sum <= $numbers", sum, underTest.add(numbers)
        }
    }

    @Test
    void test_numbers_added_together_produce_a_sum() {
        def data = [
            4  : [2, 2],
            6  : [1, 2, 3],
            10 : [1, 2, 3, 4],
            1  : [0, 1],
            0  : [0, 0],
        ]
        verifySumsFor(data)
    }

    @Test
    void test_summing_no_numbers_at_all_should_return_zero() {
        def data = [
            0  : [],
            0  : null,
        ]
        verifySumsFor(data)
    }
}

Which version do you like better? Thanks for reading.

Mule: Transport of Larger Responses in CXF

Upon calling a remote web service via a CXF endpoint in Mule I was presented with the following error:

Caused by: javax.xml.bind.UnmarshalException
 - with linked exception:
[com.ctc.wstx.exc.WstxIOException: Connection reset]
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.handleStreamException(UnmarshallerImpl.java:426)
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:362)
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:339)
        at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:537)
        ... 96 more

At first, I thought either my request or the response from the remote server was malformed XML since the exception class and stack trace seems to suggest a marshalling problem. On the other hand, the exception mentions a connection reset.

To help spare you kind readers the details of my investigation, it turns out the above problem only manifested itself when a large amount of data was being transferred between the services.

The Mule CXF Transport documentation lists an attribute, mtomEnabled, that can be enabled on the definition of the outbound endpoint for the remote service:

    <cxf:endpoint name="remoteService"
                  address="http://constanza.com/service/architectureService"
                  clientClass="com.costanza.ArchitectureService"
                  wsdlPort="ArchitectureServicePort"
                  wsdlLocation="http://constanza.com/service/architectureService?wsdl"
                  operation="getBlueprint"
                  mtomEnabled="true"/>

The attribute turns on the SOAP Message Transmission Optimization Mechanism which encodes the response payload for travel between the services. The Mule documentation refers to this allowing for data to be sent as an attachment. At the very least it tells CXF to be on the look out or to handle a potentially large response.

WSDLs and Message Parts not recognized

I recently ran into an interesting problem consuming a web service where a call to the service produced the following error message:

Message part {http://myservice.com/}getSets was not recognized.  (Does it exist in service WSDL?)

I originally tested the service using soapUI with no problem, but when I tried to invoke the service from elsewhere I was presented with the error.

My service WSDL was auto-generated by CXF. Here I found I had to tweak it a little to help it better conform to the doc/literal style. The message part declaration for the getSets operation originally looked something like this (nothing special):

<wsdl:message name="getSets">
  <wsdl:part element="tns:getSets" name="parameters">
  </wsdl:part>
</wsdl:message>

The definition of the referenced getSets element is where the problem was hiding. The element declaration contained a reference to a complexType instead of having the complexType defined within, or as part of, the element declaration.

The original element defintion:

<xs:element name="getSets" type="tns:getSets"/>
<xs:complexType name="getSets">
  <xs:sequence>
    <xs:element minOccurs="0" name="shipment" type="xs:string"/>
  </xs:sequence>
</xs:complexType>

The new definition that allows calls from both soapUI and another client to work:

<xs:element name="getSets">
  <xs:complexType>
    <xs:sequence>
      <xs:element minOccurs="0" name="shipment" type="xs:string" />
    </xs:sequence>
  </xs:complexType>
</xs:element>

Hopefully this can help someone out who runs into something similar.

Mule: Defining Properties at Runtime

I've had the desire for sometime to investigate telling Mule at run (or start-up) time what set of properties to use for a particular environment as in development, QA, or production, instead of deciding at build or deploy time what the target environment will be. So below I've come up with a rather simplistic service that uses a system property to define which set of properties to use at startup.

We will use the following Mule configuration element to define the environment properties file (explained in more detail here).

  <context:property-placeholder location="classpath:default.properties,classpath:${env}.properties" />

With the above line included in our Mule configuration, the default.properties file will be loaded first, followed by the environment-specific properties file. The latter will have the opportunity to override anything defined by the former.

The service will report the current season. A greeter bean configured in the Mule context will have two properties to be set via injection:

• greetee - who the greeting will be addressed to
• season - the current season (a true property of the environment)

The greeting will take the form of:

  "Hello ${greetee}, it is ${season}"

A default properties file will be used with the hope that we can eliminate having to repeat the property definitions
whose values would be common for every environment. We'll also investigate the ability to override this property in
case a particular environment would need to do so.

The default properties file will hold the 'greetee'.

    # default.properties
    greetee = Friend

We'll have two season-defining properties files, one for spring and one for winter. The winter properties file will simply define the season:

    # winter.properties
    season = winter

The spring properties file will define the season as well as override the greetee property set in default.properties:

    # spring.properties
    greetee = Chap
    season = spring

Since I'm using Maven 2 to build my service, we'll place the properties file in the src/main/resources directory. By convention, Maven will place all files found in this directory in the JAR file produced by the build. To make the files available to Mule on the classpath, we can place the JAR in hte lib/user directory of our Mule deployment so that they are available on the classpath as requested by the property-placeholder element.

The SeasonReporter class will have the two properties injected and will be used as a service component in our Mule model:

class SeasonReporter {
    def greetee
    def season
    def reportSeason() { "Hello $greetee, it is $season" }
}

Our Mule configuration will have an inbound HTTP endpoint that will reply synchronously with the environment-specific greeting. The component that is invoked will be implemented by a SeasonReporter class that takes the two properties used to create the seasonal greeting.

We'll direct Mule to use the reportSeason method with no arguments (by default the HTTP transport will provide a payload of one argument, /season here):

<mule>
    <context:property-placeholder location="classpath:default.properties,classpath:${env}.properties" />

    <model name="seasonModel">
        <service name="seasonService">
            <inbound>
                <http:inbound-endpoint address="http://localhost:8080/season"/>
            </inbound>
            <component>
                <no-arguments-entry-point-resolver>
                    <include-entry-point method="reportSeason"/>
                </no-arguments-entry-point-resolver>
                <spring-object bean="seasonReporter"/>
            </component>
        </service>
    </model>

    <spring:bean name="seasonReporter" class="SeasonReporter" scope="prototype">
        <spring:property name="greetee" value="${greetee}"/>
        <spring:property name="season"  value="${season}"/>
    </spring:bean>
</mule>

To start Mule using the winter properties file, we can pass in the env property when starting Mule on the command line with:

  $ mule/bin/mule start -M-Denv=winter

After starting Mule and hitting URL http://localhost:8080/season in a browser, we are presented with:

  Hello Friend, it is winter

Now we can try starting Mule up for season spring. The greeting should now include the overridden greetee property of 'Chap':

  $ mule/bin/mule start -M-Denv=spring

Giving us:

  Hello Chap, it is spring

Thanks for reading!

Mule & CXF: Multiple Operations and Groovy Components

When I picture a web service in Mule, there is usually a component backing the endpoint. Here I have a case where I would want the service fielding the web service request to delegate its work to another service. To the client, the fact that Mule will pass the actual workload for a request to another service will be hidden.

In this example, we have a mock "warehouse", where a client can store and retrieve a box. A box is uniquely identified by an ID (a String). Each operation will result in the invocation of an additional, but separate, service to do the actual work.

Here's the interface defined for our warehouse:

package prystasj.warehouse;

import javax.jws.WebParam;
import javax.jws.WebResult;
import javax.jws.WebService;

@WebService
public interface Warehouse {

    @WebResult(name="receipt")
    List<String> retrieve(@WebParam(name="boxId") String boxId);

    @WebResult(name="box")
    String store(@WebParam(name="boxId") String boxId);
}

The inteface defines both the operations. To determine which operation the client wishes to invoke, we can inspect the cxf_property of the Mule message created by the CXF Transport.

A request indicating the store operation will be routed to a storageService, while a retrieve operation will be routed to a retrievalService:

<model name="warehouse">
    <service name="warehouseService">
        <inbound>
            <cxf:inbound-endpoint address="http://localhost:8080/warehouse"
                                 serviceClass="prystasj.warehouse.Warehouse"/>
        <inbound>
        <outbound>
            <filtering-router>
                <vm:outbound-endpoint path="retrieval" synchronous="true"/>
                <expression-filter evaluator="groovy" 
                        expression="message.getProperty('cxf_operation').getLocalPart() == 'retrieve'"/>
            <filtering-router>
            <filtering-router>
                <vm:outbound-endpoint path="storage" synchronous="true"/>
                <expression-filter evaluator="groovy" 
                        expression="message.getProperty('cxf_operation').getLocalPart() == 'store'"/>
            <filtering-router>
        <outbound>
    <service>
</model>

To test the processing flow out, we can script a couple of components with Groovy. A couple of println's will help us pick out that the operations were invoked when viewing the Mule log. With a scripting component, we'll have access to a variable payload, which in each operation will contain the web parameters (a boxId in both cases) as defined in the Warehouse interface.

    <service name="retrievalService">
        <inbound>
              <vm:inbound-endpoint path="retrieval"/>
        </inbound>
        <script:component>
            <script:script engine="groovy">
                println "Operation: retrieve; Box ID: $payload"
                new Box(payload)
            <script:script>
        <script:component>
    <service>
    <service name="storageService">
        <inbound>
            <vm:inbound-endpoint path="storage"/>
        </inbound>
        <script:component>
            <script:script engine="groovy">
                println "Operation: store; Box ID: $payload"
                new Receipt(payload)
            <script:script>
        <script:component>
    <service>

The store operation returns an instance of a Box and the retrieve operation returns a Receipt. As both services are invoked synchronously, the result of the component invocations will be returned to the calling Warehouse service, which will pass them through to the client.