Stash's Seriously Named Computing Blog

Xerces Parsing and the Dreaded FWK005 Exception

2012-01-14T11:09:00.000-08:00

One of the Java/Groovy applications I'm currently works with multiple threads. In each thread, an XML service response is marshalled into an object using a method similar to:

    private def messageObjectFrom(String responseXml) {
        def jaxbElement = (JAXBElement) unmarshaller().unmarshal(readerFor(responseXml))
        (MessageType) jaxbElement.value
    }

Under a reasonable amount of load (several threads), the application would report the following exception during an execution of the method:

    javax.xml.bind.UnmarshalException: null
            at javax.xml.bind.helpers.AbstractUnmarshallerImpl.createUnmarshalException(AbstractUnmarshallerImpl.java:315) ~[na:1.6.0_24]
    ...
    Caused by: org.xml.sax.SAXException: FWK005 parse may not be called while parsing.
            at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) ~[com.springsource.org.apache.xerces-2.8.1.jar:na]
            at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) ~[com.springsource.org.apache.xerces-2.8.1.jar:na]
            at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:211) ~[jaxb-impl-2.1.13.jar:2.1.13]
            ... 43 common frames omitted

I've seen the error in the past when parsing XML in a multi-threaded application using Xerces. Several mailing lists have had questions about this error over the years which point to a concurrency issue in the Xerces parser. In my application, the parser is used inside multiple threads.

Most of the postings suggest synchronizing access to the parser. I attempted to do so by adding the synchronized keyword to the method signature:

    synchronized private def messageObjectFrom(String responseXml) {
        def jaxbElement = (JAXBElement) unmarshaller().unmarshal(readerFor(responseXml))
        (MessageType) jaxbElement.value
    }

This indeed did resolve the issue as I haven't run into a FWK005 error since. Its unfortunate the parser itself is not thread-safe, but this solution will do for me for now.

Hope this may help someone else who runs into a similar situation. Thanks!

Maven JAXB2 Plugin and Configuring Package Names

2011-12-06T19:21:00.000-08:00

Several Maven plugins exist for generating classes from a schema definition. One I have yet to try is the Maven JAXB2 Plugin. Below I'll show the minimum setup for using the plugin in your project, as well as a couple of methods for dealing with the package name assigned to the generated classes.

Given the simplistic schema below, placed in src/main/resources of the project:

  <?xml version="1.0" encoding="utf-8"?>
  <xsd:schema xmlns="http://example.org/message/1.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <xsd:element name="message">
      <xsd:complexType>
        <xsd:sequence>
          <xsd:element name="text" type="xsd:string"/>
        </xsd:sequence>
      </xsd:complexType>    
    </xsd:element>
  </xsd:schema>

We can add the plugin definition to our POM:

  <plugin>
    <groupId>org.jvnet.jaxb2.maven2</groupId>
    <artifactId>maven-jaxb2-plugin</artifactId>
    <version>0.8.0</version>
    <executions>
      <execution>
        <goals>
          <goal>generate</goal>
        </goals>
      </execution>
    </executions>
  </plugin>

The classes generated by the plugin can be found in target/generated-sources/xjc. Below is the single class, Message.java, generated by the plugin (with the imports optimized and comments removed for brevity):

  package generated;

  import javax.xml.bind.annotation.*;

  @XmlAccessorType(XmlAccessType.FIELD)
  @XmlType(name = "", propOrder = {"text"})
  @XmlRootElement(name = "message")
  public class Message {

      @XmlElement(required = true)
      protected String text;

      public String getText() {
          return text;
      }

      public void setText(String value) {
          this.text = value;
      }
  }

The first thing that stands out to me is the package name of generated. This might be ok if I wanted to use the generated classes within a single project, but if I wanted to distribute them, a more useful package name might be helpful.

The user guide for the plugin describes a list of configuration options. The one we're looking for might just be generatePackage, which we can add:

  <plugin>
    <groupId>org.jvnet.jaxb2.maven2</groupId>
    <artifactId>maven-jaxb2-plugin</artifactId>
    <version>0.8.0</version>
    <executions>
      <!-- ... -->
    </executions>
    <configuration>
      <generatePackage>org.example.message</generatePackage>
    </configuration>
  </plugin>

The generated class can now be found in target/generated-sources/xjc/org/example/messsage/Message.java, with the new package name:

  package org.example.message;

  public class Message {
     // ...
  }

There are a couple more options for setting the package name, one is to add an annotation to the schema:

  <?xml version="1.0" encoding="utf-8"?>
  <xsd:schema xmlns="http://example.org/message"
      xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
      xmlns:xjc="http://java.sun.com/xml/ns/jaxb/xjc" 
      xmlns:jaxb="http://java.sun.com/xml/ns/jaxb" 
      jaxb:version="2.0">
    <xsd:annotation>
      <xsd:appinfo>
        <jaxb:schemaBindings>
          <jaxb:package name="org.example.test"/>
        </jaxb:schemaBindings>
      </xsd:appinfo>
    </xsd:annotation>
    <xsd:element name="message">
      <!-- ... -->
    </xsd:element>
  </xsd:schema>

If you find annotating a schema to for something like JAXB impurifies the schema, you could influence to derive a namespace by providing a targetNamespace attribute to the root element of the schema:

  <?xml version="1.0" encoding="utf-8"?>
  <xsd:schema xmlns="http://example.org/message"
      targetNamespace="http://example.org/message" ...>
      <!-- ... -->
  </xsd:schema>

If you have any other suggestions or feel like weighing in on the use of xjc annotations in schema definitions, please feel free to add a comment below.

Logging Thread Names with Log4j

2011-11-15T08:11:00.000-08:00

In some applications that start multiple threads to handle some concurrent processing, tracking the activity for a particular thread can be tricky. Take the following log file snippet where a group of JMS messages are sent off all around the same time:


2011-11-15 10:08:25,460 INFO  [JmsMessageSender:?] Sending message to queue: UPDATE.REQ.QUEUE
2011-11-15 10:08:25,460 INFO  [JMSMessageReceiver:47] Using JMS Selector JMSCorrelationID='d6d3278b-7f7d-47f5-9d1f-a67e4100d800'
2011-11-15 10:08:25,461 INFO  [JmsMessageSender:?] Sending message to queue: UPDATE.REQ.QUEUE
2011-11-15 10:08:25,489 INFO  [JMSMessageSender:60] Using replyToQueue UPDATE.RESP.QUEUE
2011-11-15 10:08:25,486 INFO  [JMSMessageReceiver:47] Using JMS Selector JMSCorrelationID='10287b5f-c061-4bba-8d9d-f92dbf6fb239'
2011-11-15 10:08:25,489 INFO  [JMSMessageSender:60] Using replyToQueue UPDATE.RESP.QUEUE
2011-11-15 10:08:25,505 INFO  [JMSMessageReceiver:47] Using JMS Selector JMSCorrelationID='b7892699-8f26-4260-ae36-62fa8b09c9c3'
2011-11-15 10:08:25,517 INFO  [JmsMessageSender:?] Received message from queue UPDATE.RESP.QUEUE
2011-11-15 10:08:25,564 INFO  [JmsMessageSender:?] Received message from queue UPDATE.RESP.QUEUE

With the case above, where many messages are sent and the responses will be eventually dealt with for further processing (which will also be logged), we could filter the log
Message on the thread to help track where something may have gone wrong.

The logging patterns for log4j allow for logging the name of the thread with %t. Here is the pattern used to output the messages above:

log4j.appender.outFile.layout.ConversionPattern=%d{ISO8601} %-5p [%c{1}:%L] %m%n

We can add the %t next to the class (%c):

log4j.appender.outFile.layout.ConversionPattern=%d{ISO8601} %-5p [%t:%c{1}:%L] %m%n

The next run produces similar logging statements with the thread name. As this is a Spring Batch application, the threads are named after the class handling their creation, SimpleAsyncTaskExecutor, with a number appened on as in: SimpleAsyncTaskExecutor-5

2011-11-15 10:57:51,643 INFO  [SimpleAsyncTaskExecutor-5:JMSMessageSender:47] Sending to destination queue UPDATE.REQ.QUEUE
2011-11-15 10:57:51,659 INFO  [SimpleAsyncTaskExecutor-5:JMSMessageSender:60] Using replyToQueue UPDATE.RESP.QUEUE
2011-11-15 10:57:51,665 INFO  [SimpleAsyncTaskExecutor-4:JmsMessageSender:?] Sending message to queue: UPDATE.REQ.QUEUE
2011-11-15 10:57:51,665 INFO  [SimpleAsyncTaskExecutor-4:JMSMessageSender:47] Sending to destination queue UPDATE.REQ.QUEUE
2011-11-15 10:57:51,666 INFO  [SimpleAsyncTaskExecutor-5:JMSMessageReceiver:47] Using JMS Selector JMSCorrelationID='846cc984-84fd-47a5-bfd3-c293da829927'
2011-11-15 10:57:51,687 INFO  [SimpleAsyncTaskExecutor-4:JMSMessageSender:60] Using replyToQueue UPDATE.RESP.QUEUE
2011-11-15 10:57:51,695 INFO  [SimpleAsyncTaskExecutor-4:JMSMessageReceiver:47] Using JMS Selector JMSCorrelationID='d44087e2-bd19-474a-a8d1-71ac0fc618a7'
2011-11-15 10:57:51,827 INFO  [SimpleAsyncTaskExecutor-5:JmsMessageSender:?] Received message from queue UPDATE.RESP.QUEUE
2011-11-15 10:57:51,831 INFO  [SimpleAsyncTaskExecutor-4:JmsMessageSender:?] Received message from queue UPDATE.RESP.QUEUE

Here we can now map request and responses to particular threads if we were to encounter a error (that's logged of course) later in processing.

I'm sure something similar can be done with the newer logging frameworks as well.

Groovy: A Look at Mixins

2011-11-13T12:47:00.000-08:00

Groovy's XML support is often cited as one of the bigger attractions of the language. I often find myself writing classes that parse XML messages of the form:

    class MyXmlParser {
        def parse(String response) {
            def parsedXml = parsed(response)
            doSomethingWith(parsedXml)
        }

        def parsed(String response) {
            new XmlSlurper().parseText(response)
        }
        
        // ...
    }

One of the things that bothers me about the above pattern is the duplication in the parsed method that might spread throughout a group of classes. Using an inheritance scheme here might jump out as a possible solution at first, but that doesn't sit so well with me, as although the classes that use the duplicated method all parse XML, they all may not serve a similar substitutable purpose (one may send parsed results to a database, while another transforms the XML into an object, for example).

A newer feature of Groovy that I've been meaning to look into is the Mixin Transformation. Here, I could mixin the parsed method to all my classes that need to parse some XML. I started with the example linked to the left, but ran into a couple problems doing joint-compilation with Maven. Before we take a look at that, I'll demonstrate the mixin approach I used for this case.

The first step to adding mixins is to ensure the class you want to have make use of the mixin implement a common interface:

    package prystasj.groovy.xml

    interface XmlParser {
        def parse(String xml)
    }

    class MyXmlParser implements XmlParser {
        def parse(String response) {
            def parsedXml = parsed(response)
            doSomethingWith(parsedXml)
        }
        // ...
    }

The next step was to define the mixin class using the @Category annotation:

    package prystasj.groovy.xml

    @Category(XmlParser)
    class XmlParsing {
        def parsed(String response) {
            new XmlSlurper().parseText(response)
        }
    }

Finally, we can apply the @Mixin annotation and remove the parsed method for our parser:

    package prystasj.groovy.xml

    @Mixin(XmlParsing)
    class MyXmlParser implements XmlParser {
        def parse(String response) {
            def parsedXml = parsed(response)
            doSomethingWith(parsedXml)
        }
        // ... method parsed(String response) removed ...
    }

The MyXmlParser class now has the parsed method mixed in and no longer needs to define it. Is this a worthwile use of a mixin however, i.e. is the overhead worth having the duplication removed? In a more concrete situation, we might have many more duplicated methods between a group of classes that we would could add to the mixin, improving its worth.

Now back to the compilation problems I referred to earlier. When I went to compile, the GMaven plugin created the stubs required for joint compilation, and the Maven Compiler Plugin stumbled:


    [ERROR] /home/prystasj/.../target/generated-sources/groovy-stubs/main/prystasj/groovy/xml/MyXmlParser.java:[..] cannot find symbol
    [ERROR] symbol: variable XmlParsing

Everything in the stub looked legitimate, so I was rather stumped. I remembered running into a similar problem before, so I went ahead and tried fully qualifying the references inside the annotations:

    package prystasj.groovy.xml

    @Category(prystasj.groovy.xml.XmlParser)
    class XmlParsing {
        def parsed(String response) {
            new XmlSlurper().parseText(response)
        }
    }

    @Mixin(prystasj.groovy.xml.XmlParsing)
    class MyXmlParser implements XmlParser {
        def parse(String response) {
            def parsedXml = parsed(response)
            doSomethingWith(parsedXml)
        }
        // ... method parsed(String response) removed ...
    }

This resolved the issue and the project compile happily. If I'm not missing something, maybe this might be something worthwile to bring up with the GMaven developers? Thoughts? Thanks for reading.

Spring Batch: Deadlock Inserting Job Instances

2011-11-13T10:42:00.000-08:00

I have been working on my first Spring Batch project and so far I'm a big fan. I think I'm pretty far along and the only real issue I've had is random occurrences of deadlock when the app tries to create instances of the same job simultaneously in the job database (in table BATCH_JOB_INSTANCE):

  2011-11-13 12:48:18,808 ERROR [AbstractStep:212] Encountered an error executing the step
  org.springframework.dao.DeadlockLoserDataAccessException: PreparedStatementCallback;
  SQL [INSERT into BATCH_JOB_INSTANCE(JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION) values (?, ?, ?, ?)]; 
  Deadlock found when trying to get lock; try restarting transaction; 
  nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException:
  Deadlock found when trying to get lock; try restarting transaction
        at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.doTranslate(SQLErrorCodeSQLExceptionTranslator.java:265)
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:602)

I'm using MySQL as the database backing the batch app. The Job Repository, I'm using seemed pretty stock and I never had given it much thought:

     <bean id="jobRepository"
           class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean"
           p:dataSource-ref="dataSource" 
           p:transactionManager-ref="transactionManager" 
           p:lobHandler-ref="lobHandler"/>

One of the properties on the repository that can be set is isolationLevelForCache. I found that setting this value to ISOLATION_READ_UNCOMMITTED helped my deadlock issue.

     <bean id="jobRepository"
           class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean"
           p:dataSource-ref="dataSource" 
           p:transactionManager-ref="transactionManager" 
           p:isolationLevelForCache="ISOLATION_READ_UNCOMMITTED"
           p:lobHandler-ref="lobHandler"/>

Whether or not this is the solution to use come production time is yet to be seen. For now, I surmise this works because the app is starting enough jobs at the same time, that the framework is ready to proceed with one before the insert to the database table is formally committed.

Maven Tomcat Plugin: Selecting an XML Parser

2011-11-09T06:19:00.000-08:00

The Tomcat Maven Plugin is a great tool for quickly working through the cycle of making and verifying changes in a webapp. One such app I was working with included an XML parser from Oracle:

  oracle:oracle_xmlparserv2:jar:10.2.0.3:compile

The inclusion of the parser created a problem as it did not take kindly to a document parsed during startup, causing an error:

  $ mvn tomcat:run 
  ...
  Caused by: oracle.xml.parser.schema.XSDException: Duplicated definition for: 'identifiedType'
          at oracle.xml.parser.schema.XSDBuilder.buildSchema(XSDBuilder.java:731)
          at oracle.xml.parser.schema.XSDBuilder.build(XSDBuilder.java:435)
          at oracle.xml.parser.schema.XSDValidator.processSchemaLocation(XSDValidator.java:879)
          at oracle.xml.parser.schema.XSDValidator.startElement(XSDValidator.java:496)
          at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1224)
          at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:311)
          at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:278)
         ... 52 more

A colleague of mine at work pointed at that you can select another parser through system properties, so I wanted to share his solution here:

  $ mvn tomcat:run \
    -DXMLPARSER=XERCES \
    -DXSLENGINE=XERCES \
    -Djavax.xml.parsers.DocumentBuilderFactory=org.apache.xerces.jaxp.DocumentBuilderFactoryImpl \
    -Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl \
    -Djavax.xml.parsers.SAXParserFactory=org.apache.xerces.jaxp.SAXParserFactoryImpl

The above selects a JAXP parser from Apache, requiring Xerces and Xalan:

  xerces:xercesImpl:${xercesImpl.version}
  xalan:xalan:${xalan.version}

To avoid having to script the above command or have to remember it, you can add the settings to the plugin definition:

    <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>tomcat-maven-plugin</artifactId>
        <version>1.0</version>
        <configuration>
            <port>${tomcatPort}</port>
            <path>/${project.artifactId}-${project.version}</path>
            <systemProperties>
                <env>${env}</env>
                <tomcatPort>${tomcatPort}</tomcatPort>
                <XMLPARSER>XERCES</XMLPARSER>
                <XSLENGINE>XERCES</XSLENGINE>
                <javax.xml.parsers.DocumentBuilderFactory>org.apache.xerces.jaxp.DocumentBuilderFactoryImpl</javax.xml.parsers.DocumentBuilderFactory>
                <javax.xml.transform.TransformerFactory>org.apache.xalan.processor.TransformerFactoryImpl</javax.xml.transform.TransformerFactory>
                <javax.xml.parsers.SAXParserFactory>org.apache.xerces.jaxp.SAXParserFactoryImpl</javax.xml.parsers.SAXParserFactory>
            </systemProperties>
        </configuration>
        <dependencies>
            <dependency>
                <groupId>xerces</groupId>
                <artifactId>xercesImpl</artifactId>
                <version>${xercesImpl.version}</version>
            </dependency>
            <dependency>
                <groupId>xalan</groupId>
                <artifactId>xalan</artifactId>
                <version>${xalan.version}</version>
            </dependency>
        </dependencies>
    </plugin>

Hope this helps anyone else who may run into a similar situation.

Prompted for GNOME Keyring Password During a Subversion Commit

2011-11-04T08:28:00.000-07:00

Out of the blue when trying a Subversion commit, I was presented with a prompt for the password to the GNOME Keyring:

  $ svn ci -m"adding module"
  Password for '(null)' GNOME keyring:
  svn: Commit failed (details follow):
  svn: MKACTIVITY of '/svn/projectr/!svn/act/c5d9f7fb-76ee-4900-9a0b-b55928d418f0': authorization failed:
  Could not authenticate to server: rejected Basic challenge (http://svn.myorginazation.org)

In my case, the password for my login took care of the problem. While I do not understand the reasons for the prompt, the same may work for someone else, as by default the login password is what is being asked for.

If anyone understands why the prompt may have occurred (I haven't done much research yet), please leave a comment. Thanks!

Using Guava's LineReader in a Spring Batch Job

2011-10-29T14:35:00.000-07:00

Reading data from source like flat files and databases are common theme in batch projects. Spring Batch provides a lot of support for reading from these data sources out of the box. The FlatFileItemReader<T> provides a great example. This class extends the abstract AbstractItemCountingItemStreamItemReader<T>, which for one reason or another, I wanted to implement myself for reading a flat file. Extensions of this inteface must implement three methods:

doClose()
doOpen()
doRead()

Since I was implementing the interface myself, I had to come up with a way to actually read the contents of the file myself. In the Google Guava project, I found a class to use, LineReader, that seemed like a good candidate to help alleviate the need for the usual tedious boilerplate code required for reading the contents of the file.

Before describing the use of the LineReader in a Spring Batch project, I wanted to demonstrate an example of using the LineReader outside of the context of a Spring Batch project. The Maven coordinates for the current version of Guava are:

groupId: com.google.guava
artifactId: guava
version: 10.0.1

Below is a sample class that follows the same pattern that must be implemenented when extending the AbstractItemCountingItemStreamItemReader class using a LineReader. The class takes the name of a resource to read from and handles making the resource available to the LineReader in the doOpen() method. Reading from the resource and closing of the resource are handled by the doRead() and doClose() methods respectively.

    package prystasj.sample.guava;

    import com.google.common.io.LineReader;
    import java.io.InputStream;
    import java.io.InputStreamReader;
    import java.io.Reader;

    public class ResourceLineReader {

        private InputStream inputStream;
        private LineReader lineReader;

        public String doRead() throws Exception {
            return lineReader.readLine();
        }

        public void doOpen(String resourceName) throws Exception {
            inputStream = getClass().getClassLoader().getResourceAsStream(resourceName);
            Reader reader = new InputStreamReader(inputStream);
            lineReader = new LineReader(reader);
        }

        public void doClose() throws Exception {
            inputStream.close();
        }

    }

To test the above class, we'll use a file, tests.txt that contains three lines:

  line1
  line2
  line3

As the doRead() method, by way of LineReader.readLine(), will return null when the data from the input resource has been exhausted, we can use that fact to know when to call the doClose() method. At the end of our test we have a check to verify that three lines were read from the file:

package prystasj.sample.guava;

    import org.junit.Before;
    import org.junit.Test;

    import java.util.ArrayList;
    import java.util.List;

    import static org.junit.Assert.assertEquals;
    import static org.junit.Assert.fail;

    public class ResourceLineReaderTest {

        ResourceLineReader resourceLineReader;

        @Before 
        public void setup() {
            resourceLineReader = new ResourceLineReader();
        }

        @Test
        public void reads_lines_from_a_classpath_resource() {
            List<String> itemsRead = new ArrayList<String>();
            String itemRead;

            try {
                resourceLineReader.doOpen("test.txt");
                while((itemRead = resourceLineReader.doRead()) != null) {
                    itemsRead.add(itemRead);
                }
                resourceLineReader.doClose();
            }
            catch(Exception e) {
                fail(e.getMessage());
            }

            assertEquals(3, itemsRead.size());
        }

    }

We can use the same implementation pattern when extending the the AbstractItemCountingItemStreamItemReader<T> abstract class as mentioned earlier in this post:

    package prystasj.sample.batch;

    import com.google.common.io.LineReader;
    import org.springframework.batch.item.file.ResourceAwareItemReaderItemStream;
    import org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader;
    import org.springframework.core.io.Resource;
    import org.springframework.util.ClassUtils;

    import java.io.InputStream;
    import java.io.InputStreamReader;
    import java.io.Reader;

    class BatchFileReader extends AbstractItemCountingItemStreamItemReader<String> implements ResourceAwareItemReaderItemStream<String> {

        private InputStream inputStream;
        private LineReader lineReader;
        private Resource fileResource;

        /** Fulfills execution context requirement that {@code name} is set for the keys prefix. */
        public BibNumbersFileReader() {
            setName(ClassUtils.getShortName(BibNumbersFileReader.class));
        }

        @Override
        protected String doRead() throws Exception {
            return lineReader.readLine();
        }

        @Override
        protected void doOpen() throws Exception {
            inputStream = fileResource.getInputStream();
            Reader reader = new InputStreamReader(inputStream);
            lineReader = new LineReader(reader);
        }

        @Override
        protected void doClose() throws Exception {
            inputStream.close();
            inputStream = null;
        }

        @Override
        public void setResource(Resource resource) {
            this.fileResource = resource;
        }

    }

The above reader can be configured with the below definition to be referenced from within a batch job:

    <bean id="bibNumbersFileReader" class="prystasj.sample.batch.BatchFileReader" scope="step"
          p:resource="file:#{jobParameters['file']}"/>

In the above configuration, the scope of step ensures the bean is reconfigured every time the reader is used within a job. Spring Expression Language is used to inject the the name of the file resource into the reader from the job's parameters.

Groovy 1.8 Playing Nice with JAXB

2011-10-21T07:40:00.000-07:00

Getting Groovy and JAXB to play nice together has been previously discussed many times. A good solution to the problem is presented here: Stack Overflow.

However, I started running into some problems when I upgraded to Groovy 1.8. Take the following class definition:

    import javax.xml.bind.annotation.*

    @XmlRootElement
    @XmlAccessorType(XmlAccessType.NONE)
    public class JobResult {
        @XmlElement String status
        @XmlElement String exitDescription
    }

During compliation, I received the following error:

    symbol: variable XmlAccessType
    @javax.xml.bind.annotation.XmlRootElement() @javax.xml.bind.annotation.XmlAccessorType(value=XmlAccessType.NONE) public class JobResult
    /home/prystasj/workspace/project/target/generated-sources/groovy-stubs/main/org/oclc/controlledheading/admin/JobResult.java:[10,106] an enum annotation value must be an enum constant

Confused, I first tried using a static import:

    import javax.xml.bind.annotation.*
    import static javax.xml.bind.annotation.XmlAccessType.NONE

    @XmlRootElement
    @XmlAccessorType(NONE)
    public class JobResult {
        @XmlElement String status
        @XmlElement String exitDescription
    }

Giving me a different error:

    [ERROR] /home/prystasj/project/target/generated-sources/groovy-stubs/main/prystasj/JobResult.java:[10,93] cannot find symbol
    symbol: variable NONE
    @javax.xml.bind.annotation.XmlRootElement() @javax.xml.bind.annotation.XmlAccessorType(value=NONE) public class JobResult

A solution I used was to fully qualify the path to field inside the XmlAccessorType annotation:

    import javax.xml.bind.annotation.*

    @XmlRootElement
    @XmlAccessorType(javax.xml.bind.annotation.XmlAccessType.NONE)
    public class JobResult {
        @XmlElement String status
        @XmlElement String exitDescription
    }

Groovy: Reuse of Helper Test Methods in a Spec

2011-09-26T01:50:00.000-07:00

There are a lot of message transformers in a lot of the projects I work that work primarily with XML. I want my tests to take a source XML document and verify the output transformation. The example input and output messages live on the classpath, say in directory src/test/resources, and their contents are accessed through the classloader.

Take this example spec written for the Spock framework for Groovy:

    class XmlTransformerSpec extends Specification {

        XmlTransformer transformer = new XmlTransformer()

        def "transforms xml messages for some business goal"() {
            given:
                def inputXml = textOf("input.xml")
                def expectedOutputXml = textOf("expected-output.xml")
            when:
                def actualOutputXml = transformer.transform(inputXml)
            then:
                areIdentical(expectedOutputXml, actualOutputXml)
        }

        def areIdentical(String expected, String actual) {
            XMLUnit.setIgnoreWhitespace true
            new Diff(expected, actual).identical()
        }

        def textOf(resource) {
            getClass().getClassLoader().getResourceAsStream(resource).text
        }

    }

The retrieval of the messages in the form of resources is contained within the textOf() method where it can be resued by additional tests within the spec. What if I wanted to reuse it other test classes? One option I found was to move the method to an abstract class that extends spock.lang.Specification:

    abstract class ResourceAwareSpecification extends Specification {

        def textOf(resource) {
            getClass().getClassLoader().getResourceAsStream(resource).text
        }
    }

Now my test spec, along with any others that need to retrieve the contents of classpath resources, can extend the new specification:

    class XmlTransformerSpec extends ResourceAwareSpecification {
        // method textOf() has been removed...
    }

I wonder if others find themselves loading resources in this way and may make use of an approach like this if it were builtin to a framework like Spock in this or some other form, like a mixin. The helper method comparing the XML using XMLUnit would be another candidate in my mind for such an addition. Thoughts?

Openbox: Defining a Keyboard Shortcut

2011-09-20T16:46:00.000-07:00

The Openbox Window Manager has quickly become my favorite way to use my computer. It use it almost exclusively on all the machines that I use (Ubuntu at work, Mint at home).

Here I'll describe how to add a keyboard shortcut so that we do not have to reach for the mouse to do something common. As I spend most of my time in the terminal, one shortcut I make the most use of is Alt+T to open a terminal program, mrxvt.

The keyboard shortcuts, along with many other things, are defined in ~/.config/openbox/rc.xml. The keyboard element is a child element of the root openbox element. Below is a snippet of an example showing the default shortcut for switching to the desktop to the left. The shortcut C-A-Left describes pressing the Ctrl, Alt, and left arrow keys with the default mappings:

<?xml version="1.0" encoding="UTF-8"?>
<openbox_config xmlns="http://openbox.org/3.4/rc">
  <!-- some config omitted...  -->
  <keyboard>
    <chainQuitKey>C-g</chainQuitKey>
    <!-- Keybindings for desktop switching -->
    <keybind key="C-A-Left">
      <action name="DesktopLeft">
        <dialog>no</dialog>
        <wrap>no</wrap>
      </action>
    </keybind>
    <!-- some keyboard config omitted...  -->
  </keyboard>
  <!-- some config omitted...  -->
</openbox>

To add our terminal shortcut, we add a keybind entry with the Execute action, followed by the command to execute. The key of A-t corresponding to pressing the Alt key along with the t key:

<?xml version="1.0" encoding="UTF-8"?>
<openbox_config xmlns="http://openbox.org/3.4/rc">
  <!-- some config omitted...  -->
  <keyboard>
    <chainQuitKey>C-g</chainQuitKey>
    <!-- my keybindings -->
    <keybind key="A-t">
      <action name="Execute">
        <execute>mrxvt</execute>
      </action>
    </keybind>
    <!-- some keyboard config omitted...  -->
  </keyboard>
  <!-- some config omitted...  -->
</openbox>

We can reload the new configuration from the command line with:

  $ openbox --reconfigure

Or by right-clicking on the desktop and selecting Reconfigure from the Openbox menu.

Note: In order to take the screenshot with the menu shown, I ran gnome-screenshot with a delay of 5 seconds:

  $ gnome-screenshot -d 5

During the delay, you can right click on the desktop to get the menu in the picture.

Installing Clojure 1.2 on Ubuntu

2011-09-20T08:09:00.001-07:00

If you're looking to install the latest stable version of Clojure on Ubuntu or a derivative, below are a few options. Currently, I'm running Ubuntu 10.10, whose package list contains Clojure 1.1.0. We can install this version using apt-get:

  $ sudo apt-get install clojure
  ...
  $ clojure
  Clojure 1.1.0
  user=> (System/exit 0)

The most recent stable release is 1.2.1 (see Clojure Releases). It looks like while there was a request to have the available package updated to 1.2 (Ubuntu Bug #731979), but it looks like it won't be included by default until the 11.10 release of Ubuntu.

If we want to use the most recent stable release, the bug report has a link to the package set to be included in 11.10, clojure1.2, which we can install with dpkg:

  $ sudo dpkg -i clojure1.2_1.2.1+dfsg-3_all.deb

This links the clojure.jar as:

  $ ls -l /usr/share/java/clojure.jar
  lrwxrwxrwx 1 root root 29 2011-09-20 10:39 /usr/share/java/clojure.jar -> /etc/alternatives/clojure.jar

Alternatively, we can download the source from the releases page and build it with ant ourselves: Clojure 1.2.1 Source.

  $ unzip clojure-1.2.1.zip
  $ cd clojure-1.2.1
  $ ant
  ...
  BUILD SUCCESSFUL
  Total time: 1 minute 57 seconds

If the build succeeds, we can just copy the built jar to the install location:

  $ sudo cp clojure.jar /usr/share/java/clojure.jar
  $ clojure
  Clojure 1.2.1
  user=> (System/exit 0)

This approach should also work if we want to run the most recent development versions as well.

Fantom: Constructors, Required Fields, and More

2011-09-15T17:55:00.000-07:00

This post is part of a series of sorts where I write about what I find while exploring the Fantom language. This time I'm going to share what I find out about declaring and using classes as I go. I imagine I will only scratch the surface.

To give you the chance to devote some of your valuable reading time elsewhere, below is a summary of some of the highlights covered below:

Declaring class fields that are required at object creation and throughout the lifetime of the object
Default class field values
Automatic getters and setters

A class can be declared using the class keyword as we're all probably used to. Let's declare a (boring) class Person with a name field and add a main method so we can try and create an instance.

Assignments in Fantom are done via the use of := instead of an equals sign. Here we'll start by simply creating an empty object and printing out to the console that we've succeeded:

    class Person {
        Str name

        static Void main(Str[] args) {
            Person p := Person()
            echo("Person created.")
        }
    }

The variable p is given what looks to be the result of a call to a default constructor. We can run the above with:

   $ fan Person.fan

Our first run gives us the following error:

   Non-nullable field 'name' must be assigned in constructor 'make'

The error hints that the call to Person() results in a call to a constructor named make. Also, the error descrption seems to go against what we may be used to in Java, where class properties can be null. Right of the bat to me, this could demonstrate a potential strength of Fantom as, as given the class definition above, a user cannot create an incomplete Person. If we want to create a Person without assigning a value to the name field, we can append a '?' to the field declaration:

    class Person {
        Str? name

        static Void main(Str[] args) {
            Person p := Person()
            echo("Person created.")
        }
    }

This time we'll get the "success" message indicating we've created a Person. Let's keep the name field required and assign a value to it when the Person is constructed. To do this, we can add a constructor and in the main method create a Person named 'Cosmo':

    class Person {

        Str name

        new make(Str name) {
            this.name = name
        }

        static Void main(Str[] args) {
            Person p := Person("Cosmo")
            echo("Person created with name: " + p.name)
        }
    }

As you can see, the main method now also reports the name of the created Person via a call to the name method. Although it looks like we are accessing the field directly, getters and setters are automatically created for each field which helps us cut down on the amount of boilerplate code our class might contain, not unlike some of the newer JVM langauges. Running things through the interpreter again gives us:

    Person created with name: Cosmo

To demonstrate the generated setter, we can add the following:

    p.name = "George"
    echo("Person renamed to: " + p.name)

The output of our program is now:

   Person created with name: Cosmo
    Person renamed to: George

One question I now have is now that we have declared the name field as non-null at construction time, can we assign the value to null after the object has been created? Let's add the lines below to the main method:

    p.name = null
    echo("Person with null name?: " + p.name)

With the above in place, we now get an error:

   'null' is not assignable to 'sys::Str'

Again, this restriction should in practice help keep our class instances safe and remove the usual null-checking concerns.

Now some might say a class with one field might be pretty boring (if not useless). Lets add another field, age, but not require a user of the class to do a Person's age at construction time by allowing a default value (since age can be a touchy subject anyway). Let's try adding another constructor to be used when both the name and the age of the Person are known:

    class Person {

        Str name
        Int age := 1

        new make(Str name) {
            this.name = name
        }

        new make(Str name, Int age) {
            this.name = name
            this.age = age
        }

        static Void main(Str[] args) {
            Person cosmo := Person("Cosmo", 40)
            Person marshall := Person("Marshall")
            echo("Person " + cosmo.name + ", aged " + cosmo.age)
            echo("Person " + marshall.name + ", aged " + marshall.age)
        }
    }

Here will create two Persons with an declared age and one with the default of 1, and report on both:

    Duplicate slot name 'make'

Ah, we get an error... are we not allowed to have two constructors? Turns out in Fantom, constructors are treated like methods, with the new keyword out in front, and can be either be named make or start with make. Since we will likely prefer for our users to fully populate the class fields, let's name the two parameter constructor make and rename the original:

    new makeNamed(Str name) {
        this.name = name
    }

    new make(Str name, Int age) {
        this.name = name
        this.age = age
    }

Here though, how do we differentiate between which constructor is used for each assignment? The Persons we create are made through a call to Person(...)? Turns out we can call the constructors by name by referring to them through the class definition:

    Person cosmo := Person.make("Cosmo", 40)
    Person marshall := Person.makeNamed("Marshall")

We cannot it appears make calls to Person(...) like we did before and have the compiler infer with constructor to use asin:

    Person cosmo := Person("Cosmo", 40)
    Person marshall := Person("Marshall")

This results in the error:

   Invalid args (sys::Str, sys::Int), not (sys::Str)

Back to the explicit calls, our class now reads:

    class Person {

        Str name
        Int age := 1

        new makeNamed(Str name) {
            this.name = name
        }

        new make(Str name, Int age) {
            this.name = name
            this.age = age
        }

        static Void main(Str[] args) {
            Person cosmo := Person.make("Cosmo", 40)
            Person marshall := Person.makeNamed("Marshall")
            echo("Person " + cosmo.name + ", aged " + cosmo.age)
            echo("Person " + marshall.name + ", aged " + marshall.age)
        }
    }

Giving this attempt a shot gives us...

   Person Cosmo, aged 40
    Person Marshall, aged 1

...output for both Persons, with the second making use of the default age.

This looks like enough to cover for now in one post. Looking at the documentation further, there's looks like there's a lot more to cover at a later date. If I find I misstated any of the terminology or made things more difficult then they need to be, I'll be sure to come back and make corrections.

Creating a Marco in Vim

2011-09-13T09:10:00.000-07:00

I'm a fan of the vim text editor, and try to use it as much as possible. I have hard time though making the effort to learn more than I already know. While I think I can use it pretty effectively, I know I've truly only scratched the surface.

Often I find myself retyping the same commands over and over to repeatedly accomplish the same task. I then make a mental note to learn about creating macros, which would of helped me out. I then proceed to forget about immediately.

Not today though. Today I'm going to create my first macro.

A macro is a series of commands that can be replaced. Marcos can be assigned to buffers so they can be saved and recalled. Buffers have single- character names in the ranges:

To start the definition of a macro in command mode, type 'q' followed by a single character in one of the ranges above.

I'm going to start with the text below and think of some reason to create a macro:

hello my name is george
hello my name is george
hello my name is george

How about we try and replace every occurence with "name" with "first name". We could do a global substitution by typing a colon followed by 's/name/first name/g' and hitting enter, but I want to try out a macro.

A marco could simply:

Search for the next occurrence of name.
Replace "name" with "first name" (or insert the text "first ")

Let's try it out. With my cursor on the first character in the file, if I:

Type 'q' to declare I want to define a macro
Type 'a' to assign the macro to buffer 'a'
Type '/' followed by "name" then Enter to find the first occurence of "name"
Type 'cw' followed by "first name" then Esc to change the word
Type 'q' to complete the macro definition

The text should now have one occurrence of "first name" in place of "name":

hello my first name is george
hello my name is george
hello my name is george

Now we can run the macro typing '@a'. Running it twice will complete the remaning two substitutions, giving us:

hello my first name is george
hello my first name is george
hello my first name is george

We could also run the marco a number of times with one command. Say, given the text above where the first substitution was done, if we wanted to run the marco twice in one shot, instead of invoking it two separate times, we can type '2@a'.

Exceptions for Control Flow: Always Wrong?

2011-09-08T18:37:00.000-07:00

It is generally accepted that exceptions should never be used for control of flow and should only be used for exceptional conditions. On face value, that assertion seems to make sense. For instance, given a method that iterates over an array, the user of the method should not be expected to catch a NoSuchElementException because the method does not make use of a hasNext() method.

If a state checking method is available to help determine if another method can be called, we would all assume it should be used. However, what about cases where a series of methods must be called in sequence to accomplish a larger overall task? In a scenario I'm thinking off, a series of individual methods must be called, where each in the sequence is called only if the previous call succeeded.

Would disobeying the exceptions rule possibly make for cleaner code in such a case?

To work out my thoughts, I thought of a home building example. Let's simplify building a house to three steps: laying a foundation, putting up the framing and walls, and then adding the roof.

Let's also imagine a contractor who's job is to build a house, ensuring its marked as condemned is any of the construction fails. If the contractor attempts to add the framing without the foundation, or the roof without the framing, a catastrophe would surely happen. To avoid the catastrophe, the builder must check the house to ensure the previous step completed before moving on to the next.

Like an Iterator with a hasNext() method, the House class has methods or state variables indicating whether it has the requirements for the next phase of construction and something similarly for its overall state: whether or not its livable or condemned.

    class House {
        boolean hasFoundation
        boolean hasFraming
        boolean hasRoof
        boolean isCondemned
    }

The Contractor classs has one public method, buildHouse(), that returns a House, livable or not, and a private method for completing each step in the construction process:

    class Contractor {

        public House buildHouse() {
            def house = new House()
            
            addFoundationTo(house)
            
            if (house.hasFoundation) {
                addFramingTo(house)
                if (house.hasFraming) {
                    addRoofTo(house) 
                    if (!house.hasRoof()) {
                        house.isCondemend = true
                    }              
                } else {
                    house.isCondemned = true
                }
            } else {
                house.isCondemned = true
            }
            
            house
        }
        
        private def addFoundationTo(house) {
            //...
        }
        
        private def addFramingTo(house) {
            //...
        }
        
        private def addRoofTo(house) {
            //...
        }
    }

This contractor is overly cautious as he wants to condemn the house as soon as possible if one of the constuction steps fails to avoid a catastrophe. Another implementation may wait till the building process is completed before condemning the house:

    class Contractor {

        public House buildHouse() {
            def house = new House()
            
            addFoundationTo(house)
            
            if (house.hasFoundation) {
                addFramingTo(house)
                if (house.hasFraming) {
                    addRoofTo(house)                            
                }
            }
            
            if (!house.hasRoof()) {
                house.isCondemned = true
            }
            
            house
        }
    }

This contractor only looks at the roof and the end of the process to determine if the house should be condemned, which may lead to a lawsuit if the someone where to change the order in which the steps were executed. The contractor's check at the end could include looking at the foundation and framing as well, but he is overly trusting of his workers not to proceeded and the extra checks may be costly.

What if a different contractor required a complaint be raised by a worker if any of the home building steps fails and condemns the house if he receives the complaint before attempting the next step?

    class Contractor {

        public House buildHouse() {
            def house = new House()
            
            try {
                addFoundationTo(house)
                addFramingTo(house)
                addRoofTo(house)                            
            } catch (ConstructionComplaint complaint) {
                house.isCondemned = true
            }
            
            house
        }
        
        private def addFoundationTo(house) throws ConstructionComplaint {
            //...
        }
        
        private def addFramingTo(house) throws ConstructionComplaint {
            //...
        }
        
        private def addRoofTo(house) throws ConstuctionComplaint {
            //...
        }
    }

Here, a specific, but uniform, exception is given to the builder if any of the steps fail, ensuring the next step in the process is not attempted, avoiding a catastrophe.

An argument for this procedure may include that adding another step, like adding a garge, does not increase the complexity of the building process by adding more evaluation (branching), or require the builder to explicitly check that a garge can be added. Conversely, this approach may be overly trusting in that it requires a each worker method properly files a complaint.

I think it's safe to say we've all seen complex code as least as complex as the first implentation of the buildHouse() method. Is it worth asking that the use of exceptions would make the code cleaner? On the other hand, such cases usually present a situation where the user is required to code around what may be a poor design.

This topic and discussion may be nothing new, but I think it should help give thought to reasoning about the rule stated earlier regarding using exceptions for only exceptional conditions. Here were not talking avoiding programmer errors, such as trying to access a non-existent array element, but using exceptions to simplify a process.

While I'm not trying to suggest generally that using exceptions to control flow makes for better programs, I do think it is worthwhlie to consider such rules and understand why they may exist in the first place.

First Steps with Fantom

2011-08-06T14:38:00.000-07:00

I'm now finally getting around to taking my first steps with Fantom, even though I thought I would months ago. Where else would I start then "Hello World", and while I'm here, I'll toss in a comparison to the Java and Groovy versions for good measure:

Below is your classic Java "Hello World" program:

    public class HelloWorld {
        public static void main(String[] args) {
            System.out.println("Hello World!");
        }
    }

Here is a Fantom version of the same program.

    class HelloWorld {
        static Void main(Str[] args) {
            echo("Hello World!")
        }
    }

There are a few differences worth noting:

The public keyword is not required.
The void keyword maps to a Void type.
The equivalent to String in Java is Str in Fantom.
The semicolon ending the singular statement in the Java version is not present in the Fantom version, in fact, including it would cause an error.

Also, the argument to the main method in the Java version is not required in the Fantom version:

 class HelloWorld { static Void main() { echo("Hello World!") } }

Since I spend most of time with Groovy, I thought I'd consider the Groovy version. Like the Fantom version, the public keyword is not required, but the declared argument to the main method is, although we don't have to declare a type:

    class HelloWorld {
        static void main(args) {
            println "Hello World"
        }
    }

Additionally, the parentheses are optional and unlike with the Fantom version, an ending semicolon is allowed.

Now back to Fantom and getting the greeting printed out on the screen. The Fantom class can be placed in a file and run using the fan command:

  $ fan HelloWorld.fan
  Hello World!

With Groovy, the println call can simply be placed in a file and executed as a script using the groovy command:

  $ cat hello.groovy
  println "Hello World!"

  $ groovy hello.groovy
  Hello World!

Knowing this, I thought I'd try doing something similar in Fantom:

  $ cat hello.fan
  echo("Hello World!")

  $ fan echo.fan
  /home/prystasj/workspace/Fantom/echo.fan(2,1): Expected 'class', not 'echo'
  ERROR: cannot compile script

Unfortunately, we don't get the same results, but Fantom does have an interactive shell we can use, fansh:

  $ fansh
  Fantom Shell v1.0.59 ('?' for help)
  fansh> echo("Hello World!")
  Hello World!
  fansh> exit

Next, let's have the Fantom class make use of the array representing the arguments passed into the main method. In Fantom, arrays are also zero-based and accessed in the same way we are all used to:

   class HelloWorld {
        static Void main(Str[] args) {
            echo("Hello and " + args[0])
        }
    }

Running the progam with a single argument will have the argument included in the output:

  $ fan HelloWorld.fan goodbye
  Hello and goodbye

Groovy: Reducing Duplication with Closures

2011-07-23T10:20:00.000-07:00

I've been using Groovy as my primary programming language for a few years now. One of the features that is commonly discussed and I've read about a lot are closures. While I've appreciated what I've read, I've had a difficult time recognizing good places to use them. Like I lot of things, if you "dont use it, you lose it".

One situation where I have been able to find uses for closures recently was using them instead of methods to reduce duplication that I can hopefully demonstrate using the following example scenario. We have a TextAnalyzer class whose sole responsibility is to analyze a string and report the words separated by commas and semicolons and the counts for said delimiting characters.

The TextAnalyzer.analyze() method below travels through a given string, aggregating words and counting commas and semicolons and returns an instance of TextAnalysis that contains the results.

For example, given the string:

    groovy,is,a,language;enjoyable

The result is:

    TextAnalysis(words:[groovy, is, a, language, enjoyable], commas:3, semicolons:1)

Here's the first version of the code. Of note here to me is the duplication that exists when either a comma or semicolon is found. In both clauses, the current word is pushed onto the words list and is reset in preparation for the next word.

    def text = "groovy,is,a,language;enjoyable"

    new TextAnalyzer().analyze(text)

    @groovy.transform.ToString(includeNames=true)
    class TextAnalysis {
        List words
        int commas
        int semicolons
    }

    class TextAnalyzer {

        def analyze(text) {
            def words = []
            def word = ''
            int commas = 0
            int semicolons = 0
            
            text.each { c ->
                if (c == ',') {
                    commas++
                    words << word
                    word = ''
                }
                else if (c == ';') {
                    semicolons++
                    words << word
                    word = ''
                }
                else {
                    word += c
                }
            }
            
            if (word) words << word
            
            new TextAnalysis(words:words, commas:commas, semicolons:semicolons)
        }
    }

One option to remove the duplication is to add another method that would handle the push and reset of the current word, named add below:

    class TextAnalyzer {

        private add(words, word) {
            words << word
            ''
        }

        def analyze(text) {
            def words = []
            def word = ''
            int commas = 0
            int semicolons = 0
            
            text.each { c ->
                if (c == ',') {
                    commas++
                    word = add(words, word)
                }
                else if (c == ';') {
                    semicolons++
                    word = add(words, word)
                }
                else {
                    word += c
                }
            }
            
            if (word) words << word
            
            new TextAnalysis(words:words, commas:commas, semicolons:semicolons)
        }
     
    }

This works and produces the same result, however a couple of things bother me about it. First, the method is doing two things, it's adding the current word to the list, and return a value to reset the word. The user is required to know that it needs to assign the return value to the current word.

We could also used a boolean to indicate a word was complete that could of been evaluated at the end of each character evaluation, but that would add a bit more complexity to the analyze method through additional branching.

Another approach that sits better with me is to use a closure that's internal to the analyze method. Let's call it completeWord:

class TextAnalyzer {

    def analyze(text) {
        def words = []
        def word = ''
        int commas = 0
        int semicolons = 0
        
        def completeWord = { ->
            words << word
            word = ''
        }
        
        text.each { c ->
            if (c == ',') {
                commas++
                completeWord()
            }
            else if (c == ';') {
                semicolons++
                completeWord()
            }
            else {
                word += c
            }
        }
        
        if (word) words << word
        
        new TextAnalysis(words:words, commas:commas, semicolons:semicolons)
    }
    
}

There are a few reasons I like this better besides the reduced duplication. Since the closure is defined within the method, it has access to the variables holding the current word and the list of words, so there's no need to pass the variables around. Also, since there are no other uses for either the previous method or the new closure, the code reads better having the logic to complete a word inside the method itself.

Of course, a really concise version of the analyze method probably would not iterate over the characters in the string at all (I was thinking I could of simply started with this version, but found the iteration helped illustrate things a bit more):

    class TextAnalyzer {
        def analyze(text) {
            new TextAnalysis(
                words: text.split(',|;'),
                commas: text.count(','),
                semicolons: text.count(';')
            )
        }
    }

The above though still has some duplication in the call to String.count(). We remove it using a closure that counts:

    class TextAnalyzer {
        def analyze(text) {

            def count = { c -> text.count(c) }

            new TextAnalysis(
                words: text.split(',|;'),
                commas: count(','),
                semicolons: count(';')
            )
       }
   }

While we have not seen any huge wins here, in a more complex scenario, defining closures within methods could help make our code more readable and easier to maintain, which is always a win.

Spock: Varying Results from a Mocked Method

2011-07-05T18:02:00.000-07:00

In the Groovy world, I'm a big fan of the Spock specification framework and have written about it before. One little challenge I ran into was to setup an interaction for a mock to return a different result depending on how many times its been called. To illustrate what I'm talking about, I'll set things up with the following little scenario.

Let's say we have a class, RecordProcessor whose job is to read records from a source until there are no more records to be read. A collaborator of the class, RecordReader, is responsible for the actual record reading, and is the class whose behavior we want to mock.

The method to be called on the RecordReader takes as a parameter, the number of records to read and reports whether or not there are more records left in the source to be read. For simplicities sake, let's assume the responsibility of the RecordProcessor is to simply invoke the RecordReader until it reports that there are no more records to read.

class RecordProcessor {

    RecordReader recordReader
    int threshold

    def process() {
        boolean isProcessingComplete = true
        boolean recordsRemaining = true

        while(recordsRemaining) {
            recordsRemaining = recordReader.readRecords(threshold)
        }

        // would do more processing evaluation here... 
        processingComplete
    }
}

We want our unit test to determine whether or not the class under test can detect when there are no more records to read and report that processing was completed. Here is the base of the Spock specification where the mocking setup is left to a method:

class RecordProcessorSpec extends Specification {

    def threshold = 5
    RecordProcessor recordProcessor = new RecordProcessor(threshold: threshold)

    def "can process all the records accessible by a record reader"() {
        given:
            recordProcessorHasMockRecordReader()
        when:
            isProcessingComplete = recordProcessor.process()
        then:
            isProcessingComplete
    }
}

The wiki page describing mocking and interactions can be found My first pass at setting up the mock looked like the below, where the first two calls to readRecords method would return true and the third would return false (for more on how mocking is done with Spock, see Interactions).

    def recordProcessorHasMockRecordReader() {
        def mock = Mock(RecordReader)
        2 * mock.readRecods(threshold) >> true
        1 * mock.readRecods(threshold) >> false
        recordProcessor.recordReader = mock
    }

This didn't do the trick however as 3 invocations were detected for the first definition of how the method should behave:

  Too many invocations for:

  2 * mock.readFrom(threshold) >> true   (3 invocations)

I found one answer in using a closure to define the return value of the method invocation that had a bound variable to tracking the number of reads that were asked for:

    def recordProcessorHasMockRecordReader() {
        def mock = Mock(RecordReader)
        def readsAttempted = 0
        3 * mock.readRecods(threshold) >> {
            readsAttempted++
            readsAttepted < 3 ? true : false
        }
        recordProcessor.recordReader = mock
    }

With each invocation of the readRecords method by the RecordProcessor, the readsAttempted variable defined outside of the closure is incremented. With the first two calls, the RecordProcessor is told there are more records to process, and with the third call, it is told that all the records have been read.

Are there any other approaches anyone has found to approach a similar problem? Thanks.

Experimenting with a Groovy Template Engine and PermGen

2011-06-16T15:58:00.000-07:00

I had a Groovy application that was using a SimpleTemplateEngine was running out of PermGen space after running for a long period of time. The app was using version 1.7.10 and was running in a Tomcat 6 container.

Load tests consistently showed a linear increase in the PermGen consumption (monitored by JConsole). Originally, the template engine was used to create the template from the template text with every request. Also, the template was injected as a resource to the class using the engine.

After switching the class to create and cache the created template, we were still experiencing an OutOfMemoryError during the load test. Only after removing the use of the engine in its entirety, where we able to observe the PermGen leveling off.

A run of jmap -permstat PID against the application process showed a large number of entries of the form of:

  0x00000000f96bf800      24      233928  0x00000000f7f19558      dead    groovy/lang/GroovyClassLoader$InnerLoader@0x00000000fd784720

To investigate things, I thought I would try to see if I can reproduce the behavior using only a script run by Maven. As I go, I'll record what I see below. To start, I'll use Groovy 1.8.0.

The script will loop and complete a template with every iteration. The first version of the script will use a singe instance of SimpleTemplateEngine, but will call the createTemplate method with every iteration:

  def runs = 5
  def templateText = 'Run number: ${runNumber}'
  def engine = new groovy.text.SimpleTemplateEngine()

  runs.times { i ->
      def binding = [runNumber:"$i"]
      engine.createTemplate(templateText).make(binding)
  }

We'll start the script with verbose class loading on so we see what classes are loaded into PermGen during execution:

  $ export MAVEN_OPTS="-verbose:class"
  $ mvn -o exec:java -Dexec.mainClass=TemplateEngine | tee /tmp/out
  $ grep SimpleTemplateScript /tmp/out
  [Loaded SimpleTemplateScript1 from file:/groovy/shell]
  [Loaded SimpleTemplateScript2 from file:/groovy/shell]
  [Loaded SimpleTemplateScript3 from file:/groovy/shell]
  [Loaded SimpleTemplateScript4 from file:/groovy/shell]
  [Loaded SimpleTemplateScript5 from file:/groovy/shell]

We might suspect that behavior since not only are reading the template everytime through the loop. A new version of the script below again uses the same SimpleTemplateEngine instance and only creates the template once:

  def runs = 5
  def engine = new groovy.text.SimpleTemplateEngine()
  def createdTemplate = engine.createTemplate(templateText)

  runs.times { i ->
      def binding = [runNumber:"$i"]
      createdTemplate.make(binding)
  }

With this version, we only see one class load:

  $ grep SimpleTemplateScript /tmp/out
  [Loaded SimpleTemplateScript1 from file:/groovy/shell]

This is encouraging, but I want to make sure PermGen truly doesn't increase linerally over time. To do so, I'll add a sleep at the beginning of the script so I can attach JConsole to the Maven process and set the number of runs to a really high number:

  sleep(30000)
  println "Starting..."
  def runs = 50000

The PermGen graph is pretty jagged in the case where the createTemplate method is called each time through the loop. However, it appears that space was recovered as well so we do not end up running out of space:

To test the version where we only create the template once, I'm going to up the number of runs to 5,000,000 as execution is much faster without the repeated calls to the createTemplate method. The PermGen graph rises quickly but levels off:

So far, based on the verbose class loading and graph results, it looks like things are a definite improvement over what I was seeing in the original applcation. Just to be sure, I'm going to same experiment creating the template only once with version 1.7.0. as well.

With the single instance approach, the PermGen increased at a very slow rate. Using the approach where we call the createTemplate method with each iteration, the graph is again jagged:

At the end of this little experiment, I don't see any significant differences between Groovy 1.7 and 1.8. My work application was running in a Tomcat container which may also factor into things, along with a slightly different JVM version. Outside of the container, while the PermGen does rise with every load of a SimpleTemplateScript class, the memory is recovered.

I'll likely have to do some more experimentation to understand the behavior I was seeing in the original applcation.

JMeter Plugin: NonGuiDriver NullPointerException & SOAPException

2011-04-27T11:50:00.000-07:00

When using the Maven JMeter Plugin to try and send web service requests, I ran into the following exception:

  Error in NonGUIDriver java.lang.NullPointerException

My jmeter.log presented the following:

2011/04/27 14:20:46 ERROR - jmeter.save.SaveService: Conversion error com.thoughtworks.xstream.converters.ConversionException: org/apache/soap/SOAPException : org/apache/sp/SOAPException
---- Debugging information ----
message             : org/apache/soap/SOAPException
cause-exception     : java.lang.NoClassDefFoundError
cause-message       : org/apache/soap/SOAPException
first-jmeter-class  : org.apache.jmeter.save.converters.TestElementConverter.unmarshal(TestElementConverter.java:108)
class               : org.apache.jmeter.save.ScriptWrapper
required-type       : org.apache.jmeter.protocol.http.sampler.WebServiceSampler
path                : /jmeterTestPlan/hashTree/hashTree/hashTree[2]/WebServiceSampler
line number         : 741
-------------------------------
2011/04/27 14:20:46 ERROR - jmeter.JMeter:  java.lang.NullPointerException
        at org.apache.jmeter.gui.tree.JMeterTreeModel.addSubTree(JMeterTreeModel.java:91)
        at org.apache.jmeter.JMeter.run(JMeter.java:728)
        at org.apache.jmeter.JMeter.startNonGui(JMeter.java:706)
        ...

The solution was to add a dependency on soap:soap:2.3 to my plugin definition:

      <plugin>
        <groupId>org.apache.jmeter</groupId>
        <artifactId>maven-jmeter-plugin</artifactId>
        <version>1.0</version>
        <configuration>
          <reportDir>${reportDir}</reportDir>
        </configuration>
        <dependencies>          
          <dependency>
            <groupId>soap</groupId>
            <artifactId>soap</artifactId>
            <version>2.3</version>
          </dependency>
        </dependencies>
      </plugin>

After adding the dependency, the tests ran fine:

  Starting the test @ Wed Apr 27 14:42:57 EDT 2011 (1303929777586)
  Tidying up ...    @ Wed Apr 27 14:42:59 EDT 2011 (1303929779839)
  ... end of run

Fantom: Getting FWT to run on Linux 64-bit

2011-04-17T16:25:00.000-07:00

Today I decided to start playing around with the Fantom language. After working through a couple of examples on the Hello World page, I ran into a problem running the example that uses the Fantom Widget Toolkit on my 64-bit machine.

After following the instructions from the Fantom Setup Page, I set my FAN_HOME in my .bash_profile to:

  FAN_HOME=/opt/fantom-1.0.58

I later tried running the Hello FWT example below:

  using fwt
  class FwtHello : Test {
    Void main() {
      Window { Label { text = "Hello world" }, }.open
    }
  }

Which produced the following error:

  hello $ fan hello::FwtHello
  sys::Err: java.lang.UnsatisfiedLinkError: Cannot load 32-bit SWT libraries on 64-bit JVM

I found a swt.jar here:

  /opt/fantom-1.0.58/lib/java/ext/linux-x86/swt.jar

The setup page recommended running fan to get a read on the platform:

 $ fan -version | grep platform
   fan.platform:    linux-x86_64

I created a directory, linux-x86_64, under java/ext, and copied the swt.jar downloaded from the Eclipse site to it.

Afterwhich, the example ran fine.

Linux Mint 10 & Intel Centrino Wireless Card

2011-04-12T17:47:00.000-07:00

Today, I got a new laptop and chose to install Linux Mint 10. While the wired connection worked after the install, I had a little trouble with the wireless card, a Intel Centrino Advanced-N 6200.

For my Google searches, I verified the type of card using:

  $ lspci -v
  ...
  01:00.0 Network controller: Intel Corporation Device 008a (rev 34)
          Subsystem: Intel Corporation Device 5325
          Flags: bus master, fast devsel, latency 0, IRQ 48
          Memory at d1600000 (64-bit, non-prefetchable) [size=8K]
          Capabilities: <access denied>

In short, to fix the problem I had to upgrade to kernel version 2.6.38 from 2.6.35 and install a driver.

To upgrade my kernel, I followed the helpful instructions here: Upgrade Instructions.

After a reboot, the splash screen and dmesg presented the following complaint:

  [   15.131819] iwlagn 0000:01:00.0: request for firmware file 'iwlwifi-6000g2b-5.ucode' failed.
  [   15.131870] iwlagn 0000:01:00.0: no suitable firmware found!

To resolve this issue, I download the driver, iwlwifi-6000g2b-ucode-17.168.5.2 from Intel Wireless WiFi Link drivers for Linux.

After unpacking the tar, I copied the driver to /lib/firmware:

  iwlwifi-6000g2b-ucode-17.168.5.2 $ sudo cp iwlwifi-6000g2b-5.ucode /lib/firmware

After another reboot, I was able to use my wireless card. Hopefully, this post will help anyone else in a similar situation.

Groovy 1.8: Playing with the new @Canonical Transformation

2011-04-06T18:26:00.000-07:00

Groovy 1.8 introduces some new transformations through the use of annotations. One I came across that I wanted to investigate was @Canonical which gives you an implementation of equals(), hashCode(), and toString(), along with tuple constructors.

These transformations can also be applied individually through the @EqualsHashCode, @ToString(), and @TupleConstructors respectively.

As I have no experience with tuple constructors, I hope to take a look at that one later. On the other hand, I have written a fair share of equals() and toString() methods (some better than others) so I'm always looking for a good shortcut for both.

To start, I took a look at @EqualsAndHashCode. I wrote the simplest of classes that has one property and tried to see if comparing two instances with the same value would work. Since the class does not override equals(), this failed as one would expected:

  class Person {
      int age
  }

  def p1 = new Person(age:30)
  def p2 = new Person(age:30)

  assert p1 == p2

The failure presented by running the script:

  Assertion failed: 

  assert p1 == p2
         |  |  |
         |  |  Person@3040c5
         |  false
         Person@1ec459b

Adding a simple equals() method does the trick for now as the assertion passes:

  class Person {
      int age
    
      boolean equals(o) {
          age == o.age
      }
  }

We can use the @EqualsAndHashCode annotation and remove our override of equals(). Here the assertion will stll pass:

  import groovy.transform.EqualsAndHashCode

  @EqualsAndHashCode
  class Person {
      int age    
  }

The next question I had was whether or not there was a way to exclude some properties from taking part in the comparsion. Turns out we set the annotation to ignore certain properties using excludes.

Let's add a second property, ssn, and give both our Person instances different values:

  @EqualsAndHashCode(excludes='ssn')
  class Person {
      int age
      String ssn
  }

  def p1 = new Person(age:30, ssn:'1')
  def p2 = new Person(age:30, ssn:'2')

  assert p1 == p2

No failures here. Turns out the same can be accomplished by declaring the ssn as private:

  @EqualsAndHashCode
  class Person {
      int age
      private String ssn
  }

Now let's add @ToString into the mix:

  import groovy.transform.EqualsAndHashCode
  import groovy.transform.ToString

  @EqualsAndHashCode
  @ToString
  class Person {
      int age
      String ssn
  }

  def p1 = new Person(age:30, ssn:'1')
  def p2 = new Person(age:30, ssn:'1')

  assert p1 == p2
  println p1

The above script prints out:

  Person(30, 1)

This is definitely more helpful than the default we get if we don't override toString():

  Person@1e092

But it could be improved perhaps if we tell the annotation to include the name of the properties in the created String:

  @ToString(includeNames=true)

We now get the more helpful:

  Person(age:30, ssn:1)

Now let's see what happens if we replace both annotations with @Canoncial:

  @Canonical
  class Person {
      int age
      String ssn
  }

  def p1 = new Person(age:30, ssn:'1')
  def p2 = new Person(age:30, ssn:'1')

  assert p1 == p2
  println p1

The assertion passes, but we now are reverted to the String representation that didn't include the field names. An attempt to add the includeNames setting failed with an exception:

  'includeNames'is not part of the annotation groovy.transform.Canonical

I found that if added the @ToString annotation back, we get the display we want.

  @Canonical
  @ToString(includeNames=true)
  class Person {
      int age
      String ssn
  }

So it appears that @Canonical applies both the @EqualsAndHashCode and @ToString annotations, but applies the defaults for each. To be more selective in the behavior of the individual annotations, we need include them individually.

There look's like there are more options for each of the transformations we've used so far. I found the Javadoc for version 1.8-rc-3 in the source release to be very helpful. A link should be made availabe on the Groovy downloads page when 1.8.0 is released.

Installing Groovy 1.8: NoClassDefFoundError: GroovyStarter

2011-04-06T17:07:00.000-07:00

Today, I tried installing Groovy 1.8-rc-3 so I could play with some of the new features of the language. On my Ubuntu machine, the current version I had installed from the repositories was 1.7.6, which in itself is pretty recent:

  $ groovy --version
  Groovy Version: 1.7.6 JVM: 1.6.0_13

Since I installed Groovy through the Ubunutu repositories using apt-get, the groovy executable was already on my path:

  $ which groovy
  /usr/bin/groovy

As I did not want to mess with the default install, to get 1.8 into the picture, I would have to install things manually. After unzipping the install, I expected to be able to run it easily, but I ran into an exception:

  $ /opt/groovy-1.8.0-rc-3/bin/groovy --version
   Exception in thread "main" java.lang.NoClassDefFoundError: org/codehaus/groovy/tools/GroovyStarter
   Caused by: java.lang.ClassNotFoundException: org.codehaus.groovy.tools.GroovyStarter
           at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
           ...
   Could not find the main class: org.codehaus.groovy.tools.GroovyStarter.  Program will exit.

Of course by instinct, I went to Google, but before I went to far, it hit me to check GROOVY_HOME:

  $ echo $GROOVY_HOME
  /usr/share/groovy

After changing the value to my new install, I was able to verify the install:

  $ export $GROOVY_HOME='/opt/groovy-1.8.0-rc-3'
  $ /opt/groovy-1.8.0-rc-3/bin/groovy --version
  Groovy Version: 1.8.0-rc-3 JVM: 1.6.0_13

Nothing groundbreaking here, but hopefully I won't forget about this the next time and jump to a search page.

Spring JMS & Groovy: Sending and Receiving Messages

2011-02-26T09:30:00.001-08:00

I recently wrote some code using Spring JMS to send a JMS text message to a queue and to receive a response. The examples in the Spring documentation use Java of course to send and receive the messages. Here I wanted to demonstrate how the code can be simplified using Groovy .

In order to send and receive the messages, a JmsTemplate object is required. In a typical case, the template would be configured thru Spring and injected. Here, I'm going to omit the creation process here for brevity's sake, especially given any example would be specific to a particular JMS provider, but the documentation does a great job explaining how to accomplish the creation of a template using Spring.

When sending a text message, we'll need a list of parameters:

The text the message should contain.
The name of the queue to place the request on.
The name of the queue to have the response sent to.
A correlation ID so that we can receive the response created for the message being sent.

Since its simple to get out of the way here, a correlation ID can be created by creating a unique String using:

  def createCorrelationId() { UUID.randomUUID().toString() }

We'll also need a method to create a MessageCreator to give to the JmsTemplate when sending the mesage. A Java example of doing so my look like this using an inner class:

    void sendMessage(JmsTemplate jmsTemplate, String text, String requestQueue, String responseQueue, String correlationId) throws JMSException { 
        jmsTemplate.send(requestQueue, new MessageCreator() {
            public Message createMessage(Session session) throws JMSException {
                TextMessage message = session.createTextMessage(text);
                return message;
            }
        });
    }

This example can be made easier on the eyes by using as in Groovy since MessageCreator is an interface (we also get the benefit of not needing the throws declarations):

    def sendMessage(jmsTemplate, text, requestQueue, responseQueue, correlationId) { 
        jmsTemplate.send(requestQueue, { session -> 
            session.createTextMessage(text) 
        } as MessageCreator)
    }

Or we can store the MessageCreator in a variable:

    def sendMessage(jmsTemplate, text, requestQueue, responseQueue, correlationId) { 
        def messageCreator = { session -> 
            session.createTextMessage(text) 
        } as MessageCreator

        jmsTemplate.send(requestQueue, messageCreator)
    }

In order to set the response queue and correlationID though, we'll need access to the text message created by the Session provided by the JmsTemplate:

    def correlationId = createCorrelationId() 

    def sendMessage(jmsTemplate, text, requestQueue, responseQueue, correlationId) { 
        def messageCreator = { session -> 
            session.createTextMessage(text) 
            message.with {
                setText request
                setJMSCorrelationID correlationId
                setJMSReplyTo session.createQueue(responseQueue)
            }

        } as MessageCreator

        jmsTemplate.send(requestQueue, messageCreator)
    }

Since the MessageCreator is defined by a closure, we can extract a method out that the closure will have access to when its executed to create the text message:

    def correlationId = createCorrelationId() 

    def createMessage(session, text, responseQueue, correlationId) {
        def message = session.createTextMessage(text) 
        message.with {
            setText text
            setJMSCorrelationID correlationId
            setJMSReplyTo session.createQueue(responseQueue)
        }
        message
    }

    def sendMessage(jmsTemplate, text, requestQueue, responseQueue, correlationId) { 
        def messageCreator = { session -> 
            createMessage(session, request, responseQueue, correlationId)
        } as MessageCreator

        jmsTemplate.send(requestQueue, messageCreator)
    }

Now that we have a method for sending the request message, we need a way to grab the response. A Java example:

    void receiveResponse(JmsTemplate jmsTemplate, String responseQueue, String correlationId) throws JMSException {
        String selector = "JMSCorrelationID='" + correlationId + "'";
        Message message = (TextMessage) jmsTemplate.receiveSelected(parameters.getRetrieveFromQueue(), selector);
        String response;
        if (message != null) {
            response = message.getText();
        }
        return response;
    }

A Groovier way:

  def receiveResponse(jmsTemplate, responseQueue, correlationId) { 
     def selector = "JMSCorrelationID='$correlationId'" 
     def message = (TextMessage)jmsTemplate.receiveSelected(responseQueue, selector) 
     message?.getText()
  }

In both reception examples, the fact we may not get a response or text in the response. This is something the caller can deal with or we can modify the examples to throw an exception:

   def receiveResponse(jmsTemplate, responseQueue, correlationId) {
        def selector = "JMSCorrelationID='$correlationId'"
        def message = jmsTemplate.receiveSelected(responseQueue, selector) as TextMessage
        def response = message?.getText()

        if (!response) {
            throw new Exception("Got nothing!") // in practice would likely throw a more apt or custom exception
        }

        response
    }

To put it all together, the code examples above can be aggregated to a class:

class JmsMessageSender {

    def createCorrelationId() { UUID.randomUUID().toString() }

    def createMessage(session, text, responseQueue, correlationId) {
        def message = session.createTextMessage(text) 
        message.with {
            setText text
            setJMSCorrelationID correlationId
            setJMSReplyTo session.createQueue(responseQueue)
        }
        message
    }

    def sendMessage(jmsTemplate, text, requestQueue, responseQueue, correlationId) { 
        def messageCreator = { session -> 
            createMessage(session, request, responseQueue, correlationId)
        } as MessageCreator

        jmsTemplate.send(requestQueue, messageCreator)
    }

    def createSelectorFrom(correlationId) {
        "JMSCorrelationID='$correlationId'"
    }

    void receiveResponse(jmsTemplate, responseQueue, correlationId) {
        def selector = createSelectorFrom(correlationId)
        def message  = jmsTemplate.receiveSelected(responseQueue, selector) as TextMessage
        def response = message?.getText()

        if (!response) throw new Exception("Got nothing!")

        response
    }
}

A client code example using the above class through three calls:

    JmsTemplate jmsTemplate // injected
    def correlationId = messageSender.createCorrelationId()
    messageSender.sendMessage(jmsTemplate, text, requestQueue, responseQueue, correlationId)
    def response = messageSender.receiveResponse(jmsTemplate, responseQueue, correlationId)

Thanks for reading!

Perl: Retrieve URLs with LWP and LWP::Simple

2011-01-28T07:59:00.001-08:00

With Perl there are many ways to make requests over the web. One method is to use the LWP module. Below is an example of using it grab the contents of a web page:

use Carp;
use LWP;

my $url = 'http://prystash.blogspot.com';
my $contents = get_contents_from($url);

print $contents;

sub get_contents_from {
    my ($url) = @_;

    my $agent = LWP::UserAgent->new;
    my $request = HTTP::Request->new(GET => $url);
    my $response = $agent->request($request);

    if (!$response->is_success) {
        croak "Could not get URL '$url'";
    }

    return $response->content
}

Another simpler method is to use the LWP::Simple module:

use Carp;
use LWP::Simple;

my $url = 'http://prystash.blogspot.com';
my $contents = get_contents_from($url);

sub get_contents_from {
    my ($url) = @_;
    my $contents = get($url) or croak "Could not get URL '$url'";
    return $contents;
}

Removing Files Older than a Certain Number of Days

2011-01-27T14:06:00.000-08:00

Using the find command, we can remove files that have not been modified in a certain number of days old using the mtime option:

  -mtime n
    File's data was last modified n*24 hours ago.  See the comments for -atime 
    to understand how rounding affects  the  inter‐pretation of file modification times.

To delete files that are older than 6 months, we can use:

  $ find . -type f -mtime +180 | xargs rm

Or alternatively:

  $ find . -type f -mtime +180 -exec rm {} \;

GMaven: A Couple Early Problems Building a Plugin

2011-01-25T16:33:00.000-08:00

My first attempt to build a Maven plugin using GMaven got off to a rough start. I don't want to use this space to complain by any means, but I would to share what I learned in case anyone else runs into something similar.

My first problem had to deal with my use of a newer version of GMaven:

      <plugin>
        <groupId>org.codehaus.gmaven</groupId>
        <artifactId>gmaven-plugin</artifactId>
        <version>1.3</version>
        <configuration>
          <providerSelection>1.7</providerSelection>
        </configuration>
        <extensions>true</extensions>
        <inherited>true</inherited>
        <executions>
          <execution>
            <goals>
              <goal>generateStubs</goal>
              <goal>compile</goal>
              <goal>generateTestStubs</goal>
              <goal>testCompile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>

During the build of the plugin, I was given a deprecation warning stating that no mojo descriptors were found in the project:

[WARNING] Deprecation Alert:
[WARNING] No mojo descriptors were found in this project which has a packaging type of maven-plugin.

I found that reason for the warning was that the stub generation was not retaining the Javadoc annotations used to mark a Mojo. By downgrading to version 1.2 of GMaven and changing the providerSelection to 1.6, the warning went away.

Next when trying to use the plugin in another build, I was present with something ilke:

This realm = plexus.core
urls[0] = file:/opt/apache-maven-2.2.1/lib/maven-2.2.1-uber.jar
Number of imports: 10
import: org.codehaus.classworlds.Entry@a6c57a42
import: org.codehaus.classworlds.Entry@12f43f3b
import: org.codehaus.classworlds.Entry@20025374
import: org.codehaus.classworlds.Entry@f8e44ca4
import: org.codehaus.classworlds.Entry@92758522
import: org.codehaus.classworlds.Entry@ebf2705b
import: org.codehaus.classworlds.Entry@bb25e54
import: org.codehaus.classworlds.Entry@bece5185
import: org.codehaus.classworlds.Entry@3fee8e37
import: org.codehaus.classworlds.Entry@3fee19d8
-----------------------------------------------------
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Internal error in the plugin manager executing goal 'org.prystasj.plugins:jms-testing:1.0-SNAPSHOT:hello': Unable to find the mojo 'hello' (or one of its required components) in the plugin 'org.prystasj.plugins:jms-testing'
org.codehaus.groovy.runtime.GroovyCategorySupport.getCategoryNameUsage(Ljava/lang/String;)Ljava/util/concurrent/atomic/AtomicInteger;

I found the solution to this problem was to exclude the groovy-all-minimal jar, version 1.5.7 with:

   <dependency>
      <groupId>org.codehaus.groovy.maven</groupId>
      <artifactId>gmaven-mojo</artifactId>
      <version>1.0</version>      
      <exclusions>
        <exclusion>
          <groupId>org.codehaus.groovy</groupId>
          <artifactId>groovy-all-minimal</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

I also had another that was similar error related to class CallSiteArray that was alleviated by ensuring I was using Groovy 1.6 everywhere.

Groovy: Sorting a Map by Values

2011-01-10T12:23:00.000-08:00

Here's a real quick Groovy snippet demonstrating one way to sort a Map by the values stored in its entries (mostly so I don't forget how to do it):

def map = ["ghi":6, "abc":4 ,"def":5]
def sortedByValue = map.sort { a,b -> a.value <=> b.value }
println sortedByValue.keySet()

The output from this snippet is:

[abc, def, ghi]

Anyone have any other methods for doing the same thing?

Using Maven to Publish and Verify Schemas

2011-01-08T15:20:00.000-08:00

We use Maven to publish schemas and other documents, like WSDLs, that we would like to share across projects. This makes publishing schema releases, along with schemas that are in-development, easy for consumption by clients. Using Maven also allows to easily to publish the schema together with example documents as a resource bundle in addition to validating the examples messages against the schema during the build so we know the schema and what we would expect a message to look like are in sync.

Here I'm going to demonstrate our base project structure and the minimum POM we use for our schema projects. I'll follow that up by adding in additional plugins to make the project more worthwhile.

To start, we'll use a real simple schema, people.xsd, describing a list of people with each person being described with a first name, last name, and his/her age:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">
  <xs:element name="people">
    <xs:complexType>
      <xs:sequence>
        <xs:element minOccurs="1" maxOccurs="unbounded" ref="person"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  <xs:element name="person">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="firstName" type="xs:string"/>
        <xs:element name="lastName" type="xs:string"/>
        <xs:element name="age" type="xs:string"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

An example document containing two people:

<people>
  <person>
    <firstName>George</firstName>
    <lastName>Costanza</lastName>
    <age>40</age>
  </person>
  <person>
    <firstName>Cosmo</firstName>
    <lastName>Kramer</lastName>
    <age>42</age>
  </person>
</people>

At the root of the project, we of course have the pom.xml. The schema and example documents are kept in the src/main/resources directory:

  pom.xml
  src/main/resources/people.xsd
  src/main/resources/twoPeople.xml
  src/main/resources/onePerson.xml

The POM itself is relatively simple. By default, Maven will look to package a JAR. We have no classes in this project, but the files in the src/main/resources directory will be packaged. Since we want to publish the schema as a separate artifact, we can use the Build Helper plugin to attach the schema to the build for publishing. The location of the schema is defined by the schema property, which we will reuse later:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.prystasj.schemas</groupId>
  <artifactId>people</artifactId>
  <version>1.0-SNAPSHOT</version>
  <name>People Schema</name>
  <properties>
    <schema>src/main/resources/${artifactId}.xsd</schema>
  </properties>
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>build-helper-maven-plugin</artifactId>
        <version>1.5</version>
        <executions>
          <execution>
            <phase>package</phase>
            <id>attach-artifacts</id>
            <goals>
              <goal>attach-artifact</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <artifacts>
            <artifact>
              <file>${schema}</file>
              <type>xsd</type>
            </artifact>
          </artifacts>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

Now we can run the install phase and see that the schema is installed to our local repository:

$ mvn install
[INFO] Scanning for projects...
...
[INFO] [install:install {execution: default-install}]
[INFO] Installing /home/prystasj/workspace/prystasj/writing/maven-xsd/target/people-1.0-SNAPSHOT.jar to /home/prystasj/.m2/repository/org/prystasj/schemas/people/1.0-SNAPSHOT/people-1.0-SNAPSHOT.jar
[INFO] Installing /home/prystasj/workspace/prystasj/writing/maven-xsd/src/main/schemas/people.xsd to /home/prystasj/.m2/repository/org/prystasj/schemas/people/1.0-SNAPSHOT/people-1.0-SNAPSHOT.xsd
...

The schema, along with the JAR, are installed separately to the local repository, and of course we can deploy it for public consumption by running the deploy phase.

Since we have a couple of sample messages, we can add a step to the build to validate them against the schema to help insulate us against any inconsistencies if the schema ever changes. To do just that, we'll add an execution of the XML Maven Plugin:

      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>xml-maven-plugin</artifactId>
        <version>1.0-beta-3</version>
        <executions>
          <execution>
            <goals>
              <goal>validate</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <validationSets>
            <validationSet>
              <systemId>${schema}</systemId>
              <dir>src/main/resources</dir>
              <excludes>
                <exclude>${artifactId}.xsd</exclude>
              </excludes>
            </validationSet>
          </validationSets>
        </configuration>
      </plugin>

The systemId property is used to define the location of the schema we want to validate against relative to the base of the project. Here I'm using the value of the schema property from the original version of the POM. The dir property describes the directory containing the instance documents to validate. I've added an exclude element to ensure the schema itself is not used.

When we run the install phase now, we should see the following build output if the validation succeeds:

[INFO] [surefire:test {execution: default-test}]
[INFO] No tests to run.
[INFO] [xml:validate {execution: default}]
[INFO] [jar:jar {execution: default-jar}]

To make sure things are setup correctly, let's add an instance document that we know should be considered invalid, src/main/resources/invalidPerson.xml, which defines one person without the required age element:

<people>
  <person>
    <firstName>George</firstName>
    <lastName>Costanza</lastName>
  </person>
</people>

Now our build fails, citing:

[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] While parsing /home/prystasj/workspace/prystasj/writing/maven-xsd/src/main/resources/invalidPerson.xml, at file:/home/prystasj/workspace/prystasj/writing/maven-xsd/src/main/resources/invalidPerson.xml, line 5,  column 12: cvc-complex-type.2.4.b: The content of element 'person' is not complete. One of '{age}' is expected.

Finally, we can add the Remote Resources Plugin to create a resource bundle for clients to take advantage of. We do have the JAR that is built that contains both the schema and example documents that client can unpack using the Dependency Plugin, but providing a resource bundle makes things easier, as the contents of the bundle are unpacked as a result of a depending on the bundle. An example use case for a client would be ensuring the creation of messages to be sent are valid.

The resource bundle can be created by adding the plugin:

      <plugin>
        <artifactId>maven-remote-resources-plugin</artifactId>
        <version>1.1</version>
        <executions>
          <execution>
            <goals>
              <goal>bundle</goal>
            </goals>
            <configuration>
              <includes>
                <include>**/*.xml</include>
                <include>**/*.xsd</include>
              </includes>
            </configuration>
          </execution>
        </executions>
      </plugin>

The build will now create a file describing the resources to be included in the bundle:

<?xml version="1.0" encoding="UTF-8"?>
<remoteResourcesBundle xsi:schemaLocation="http://maven.apache.org/plugins/maven-remote-resources-plugin/remote-resources/1.1.0 http://maven.apache.org/plugins/maven-remote-resources-plugin/xsd/remote-resources-1.1.0.xsd"
    xmlns="http://maven.apache.org/plugins/maven-remote-resources-plugin/remote-resources/1.1.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <remoteResources>
    <remoteResource>twoPeople.xml</remoteResource>
    <remoteResource>people.xsd</remoteResource>
    <remoteResource>onePerson.xml</remoteResource>
  </remoteResources>
</remoteResourcesBundle>

A client can now depend on the bundle using the same plugin:

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-remote-resources-plugin</artifactId>
        <executions>
          <execution>
            <goals>
              <goal>process</goal>
            </goals>
            <configuration>
              <resourceBundles>
                <resourceBundle>org.prystasj.schemas:people:1.0-SNAPSHOT</resourceBundle>             
              </resourceBundles>
            </configuration>
          </execution>
        </executions>
      </plugin>

With the addition of the plugin in the client build, the schema and example documents will be available on the classpath:

target/maven-shared-archive-resources/twoPeople.xml
target/maven-shared-archive-resources/people.xsd
target/maven-shared-archive-resources/onePerson.xml
target/classes/twoPeople.xml
target/classes/people.xsd
target/classes/onePerson.xml

Hope this helps demonstrate some potential uses for Maven that might be a little outside-the-box, but useful none the less.

Griffon: Signing an Application

2010-11-08T06:34:00.000-08:00

Last week, I tried using Griffon to help investigate the possibility of creating a Java Web Start application. Everything went smoothly with the exception of some trouble I had signing the application for the "production" environment. To help those in a similar situation, I'll summarize the steps I needed to get things singed below signed below.

Note:I used this thread on Markmail to and the Griffon Quick Start guide to help me out.

I created my application as the guide suggests with:

  $ griffon create-app DemoConsole

Now the guide offers some suggestions for making the resulting application more useful, but here I'm just going to do the bare minimum I know to get the application deployed.

Next, in order to sign the jars in the application for production use, we need to create a keystore:

  $ keytool -genkey -alias GriffonKey

  Enter keystore password:  
  Re-enter new password: 
  What is your first and last name?
    [Unknown]:  John Prystash
  ...
  Enter key password for <GriffonKey>
 (RETURN if same as keystore password):

I used prystasj for both the keystore and key password, this being information we'll need to know later. The resulting keystore file was created at: /home/prystasj/.keystore

The signing and key information is determined from the file griffon-app/conf/BuildConfig.groovy. Below is the relevant file information in its original form:

// key signing information
environments {
    //...
    production {
        signingkey {
            params {
                sigfile = 'GRIFFON'
                keystore = 'CHANGE ME'
                alias = 'CHANGE ME'
                // NOTE: for production keys it is more secure to rely on key prompting
                // no value means we will prompt //storepass = 'BadStorePassword'
                // no value means we will prompt //keypass   = 'BadKeyPassword'
                lazy = false // sign, regardless of existing signatures
            }
        }

        griffon {
            jars {
                sign = true
                pack = true
                destDir = "${basedir}/staging"
            }
            webstart {
                codebase = 'CHANGE ME'
            }
        }
    }
}

We'll need to point the production configuration at the keystore we created earlier and set the alias to GriffonKey. We'll also add the password we gave to the keytool program, which is prystasj in both cases. Finally, we'll be publishing the app to a webserver, whose location we set in the codebase property:

// key signing information
environments {
    //...
    production {
        signingkey {
            params {
                sigfile = 'GRIFFON'
                keystore = '/home/prystasj/.keystore'
                alias = 'GriffonKey'
                storepass = 'prystasj'
                keypass = 'prystasj'
                lazy = false // sign, regardless of existing signatures
            }
        }

        griffon {
            jars {
                sign = true
                pack = true
                destDir = "${basedir}/staging"
            }
            webstart {
                codebase = 'http://myhost.org/prystasj/democonsole/'
            }
        }
    }
}

Now we can build the app with:

  $ griffon prod package webstart

The result is a zip file at dist/webstart. Taking this zip we can unpack on the webserver at the location we set in the codebase property.

Next, we can download the application and run it from: http://myhost.org/prystasj/democonsole/application.jnlp.

Hope this helps anyone else looking to get a Griffon application signed.

Groovy: Retrieving the Value of Multiple XML Elements

2010-10-20T08:18:00.000-07:00

Yesterday, I ran into an interesting case at work where some code was parsing XML using Groovy's XmlSlurper to retrieve the value of an element and treat it as a String. Something along the likes of:

  def xml = "<xml><character>a</character></xml>"
  def node = new XmlSlurper().parseText(xml)
  String result = node.character
  println result

Which simply prints out a. The code was expected to only return value of the first element found, but when a second element is added:

  def xml = "<xml><character>a</character><character>b</character></xml>"
  def node = new XmlSlurper().parseText(xml)
  String result = node.character
  println result

The resulting output is ab, which broke a new test case.

One solution to the issue is to grab the first element with:

  String result = node.character[0]

Another interesting point is that the result of node.character[0] is a NodeChild, not a String. Since the type of the result variable is declared, the right side of the assignment was coerced to a String. If that were not the case, and we had:

  def result = node.character[0]
  println result.getClass().getName()
  println result

The output would be:

  groovy.util.slurpersupport.NodeChildren
  a

Note:We need to use getClass() as result.class would return the class node and not the Class object itself.

Alternatively, we can use the text() method of NodeChildren (which also exists for NodeChild) to ensure we get a String:

  def result = node.character[0].text()
  println result.getClass().getName()
  println result

Giving us:

  java.lang.String
  a

To summarize, putting all the methods discussed so far to use with:

  def xml = "<xml><character>a</character><character>b</character></xml>"
  def node = new XmlSlurper().parseText(xml)

  println node.character
  println node.character.getClass().getName()
  println()

  println node.character.text()
  println  node.character.text().getClass().getName()
  println()

  println node.character[0]
  println node.character[0].getClass().getName()
  println()

  println node.character[0].text()
  println node.character[0].text().getClass().getName()
  println()

We get:

  ab
  groovy.util.slurpersupport.NodeChildren

  ab
  java.lang.String

  a
  groovy.util.slurpersupport.NodeChild

  a
  java.lang.String

While the original gotcha might not be all that hard to resolve, hopefully this gives a little insight to those who might explore things a little bit further.

Bash: Learning about Arrays

2010-10-19T17:28:00.000-07:00

Last year, I wrote about a little Groovy script to help me validate XML documents against a schema. A ended up with a little script to call to help me from having to recall how to run it using Maven. The script takes a schema location and an example instance document as arguments:

  #!/bin/bash
  mvn -o exec:java -Dexec.mainClass=Validator -Dexec.args="$1 $2"

Which I can call with:

  $ ./validate.sh order.xsd order.xml

We store example XML documents along with our internal schemas, and sometimes I find myself running the same script multiple times, once for each document in the project, so I thought having a script to loop through the documents in a directory and report the results would also be helpful.

In order to report both the 'good' instances and the 'bad' instances at the end of the scripts run, I needed to learn a little about bash arrays.

In found that in bash, there are multiple ways to create an array. You can start by simply assigning a value to a yet unused array:

  #!/bin/bash
  good[0]="my.xml"
  echo ${good[0]}

Or you can declare an array using:

  #!/bin/bash
  declare -a good = ("my.xml", "your.xml")
  echo ${good[0]}
  echo ${good[1]}

In my case, I will be iterating through the files and adding them to the appropriate array, so I won't be able to declare the array or its size up front. I used this method to first check if the array is empty in an if statement, and if so declare and initialize it. If the array does in fact exist, I append an element to in the else clause, by using @ to get the length of the array:

  #!/bin/bash
  if [ ${#bad[0]} -eq 0 ]; then
      declare -a bad=("$i")
  else
      bad=("${bad[@]}", $i)
  fi

To roll everything up, my new scripts takes the schema location and a directory as arguments. For every, directory listing in the directory, I run the Groovy code against it. If the validation failed, setting $? to 1, I add the file to the 'bad' list. Otherwise, it goes to the good list:

  #!/bin/bash
  xsd=$1
  dir=$2

  for i in `ls $dir`
  do
      echo -n "$i..."
      mvn -o exec:java -Dexec.mainClass=Validator -Dexec.args="$1 $2/$i"

      if [ $? -eq 1 ]; then
          if [ ${#bad[0]} -eq 0 ]; then
              declare -a bad=("$i")
          else
              bad=("${bad[@]}", $i)
          fi
      else
          if [ ${#good[0]} -eq 0 ]; then
              declare -a good=("$i")
          else
              good=("${good[@]}", $i)
          fi
      fi
  done

  echo
  echo "Good files: "
  echo "${good[@]}"
  echo
  echo "Failed files: "
  echo ${bad[@]}
  echo

At the end of script, I simply echo both lists. An example of the output for the report:

  Good files: 
  good1.xml good2.xml good3.xml

  Failed files: 
  bad1.xml bad2.xml bad3.xml

I'm sure I could remove some of the duplication, but I since I have only two arrays and this is just a helper, I think I'll leave things be for me. If you have any more array advice, please leave a comment!

Trang: Creating Schemas from XML

2010-10-01T09:00:00.000-07:00

Does anyone like writing XML schemas? Sometimes they can be frustrating, and yet always ends up feeling simple when you're done. When given the choice, it always feels good to me to start writing a schema from an example instance document, and of course there are plenty of tools to help.

While a lot of tools are available for a price, Trang however is free, and helps me with the writer's block I tend to get when I'm handed an XML document and asked to make a schema from scratch.

Trang is a Java app that can be downloaded in zip form. Luckily (for me at least), it was available in the Ubuntu package repositores:

$ sudo apt-get install trang

Now that we have Trang installed, let's generate a schema from a simple XML document, languages.xml:

<?xml version='1.0' encoding='UTF-8'?>
<languages>
  <language>
    <name>Groovy</name>
    <platform>JVM</platform>
    <appeared>2003</appeared>
  </language>
  <language>
    <name>Scala</name>
    <platform>JVM</platform>
    <appeared>2003</appeared>
  </language>
  <language>
    <name>Boo</name>
    <platform>CLR</platform>
    <appeared>2003</appeared>
  </language>
</languages>

Now let's tell Trang, gratefully, to make us a schema:

$ trang languages.xml languages.xsd

And a schema is generated for us:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">
  <xs:element name="languages">
    <xs:complexType>
      <xs:sequence>
        <xs:element maxOccurs="unbounded" ref="language"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  <xs:element name="language">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="name"/>
        <xs:element ref="platform"/>
        <xs:element ref="appeared"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  <xs:element name="name" type="xs:NCName"/>
  <xs:element name="platform" type="xs:NCName"/>
  <xs:element name="appeared" type="xs:integer"/>
</xs:schema>

This is a real good start, but I feel like I should make a couple changes. To make things a little easier to understand for consumers, I think I'll change the uses of NCName to string:

<xs:element name="name" type="xs:NCName"/>
<xs:element name="platform" type="xs:NCName"/>

And require at least one language element for the document to be valid:

<xs:element minOccurs="1" maxOccurs="unbounded" ref="language"/>

My edited schema becomes:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">
  <xs:element name="languages">
    <xs:complexType>
      <xs:sequence>
        <xs:element minOccurs="1" maxOccurs="unbounded" ref="language"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  <xs:element name="language">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="name"/>
        <xs:element ref="platform"/>
        <xs:element ref="appeared"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  <xs:element name="name" type="xs:string"/>
  <xs:element name="platform" type="xs:string"/>
  <xs:element name="appeared" type="xs:integer"/>
</xs:schema>

Even with this simple example, Trang has saved me a lot of typing. Trang also has the options to also create RELAX NG and DTD documents if you need them.

Perl & Bash: Mass Substitutions from the Command Line

2010-09-21T14:41:00.001-07:00

Today I found myself in a situation where I had to do the same global substitution for every file in a directory. I considered writing a script to handle it, but I wanted to try using a Perl one-liner to get the job done.

Given a directory containing a file with the following contents:

  This is a XML file
  This is a XML file

I can substitute every occurence of XML with text using:

  $ perl -i -pe 's/XML/text/g' file1.xml

Afterwards, the file now reads:

  This is a text file
  This is a text file

If I wanted to do the substitution for every file in the current directory, I can wrap the command in a bash for loop:

  $ for i in `ls`; do perl -i -pe 's/XML/text/g' $i; done

This isn't revolutionary by any means, but its harder for me to forget this technique if I post it here.

Maven & soapUI: Testing a Service

2010-08-14T16:41:00.000-07:00

In a previous post, I demonstrated the creation of the business logic for an adding web service, that when given a list of numbers, returns their sum. The goal for me was to create a simple service so I can try out automated testing of the service using soapUI from Maven.

In this post, I'll show how I can run the test suite for the service. In the next post, hopefully I'll demonstrate how we can expose the service in Tomcat using Mule.

The created a test suite, when run, produces the following results as demonstrated by the service's log:

INFO  [http-8080-1][2010-07-07 11:23:34,981] - Sum of: [2, 2] = 4
INFO  [http-8080-1][2010-07-07 11:23:35,109] - Sum of: [1, 2, 3] = 6
INFO  [http-8080-1][2010-07-07 11:23:35,133] - Sum of: [1, 2, 3, 4] = 10
INFO  [http-8080-1][2010-07-07 11:23:35,148] - Sum of: [0, 1] = 1
INFO  [http-8080-1][2010-07-07 11:23:35,167] - Sum of: [0, 0] = 0
INFO  [http-8080-1][2010-07-07 11:23:35,221] - Sum of: [] = 0

In the adder service project, I exported the project containing a test suite to a new integration testing module, adder-it, as:

src/test/resources/adder-soapui-project.xml

The POM for the adder-it module contains the following:

<?xml version="1.0" encoding="utf-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>org.prystasj.adder</groupId>
    <artifactId>adder</artifactId>
    <version>1.0-SNAPSHOT</version>
  </parent>
  <artifactId>adder-it</artifactId>
  <name>adder Integration Testing</name>
  <description>Integration testing with soapUI.</description>
  <build>
    <plugins>
      <plugin>
        <groupId>eviware</groupId>
        <artifactId>maven-soapui-plugin</artifactId>
        <version>3.5.1</version>
        <configuration>
          <projectFile>src/test/resources/adder-soapui-project.xml</projectFile>
          <host>localhost</host>
          <port>8080</port>
        </configuration>
        <executions>
          <execution>
            <id>soap-integration-test</id>
            <phase>integration-test</phase>
            <goals>
              <goal>test</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

To run the tests, with the service up and running locally on port 8080, invoke Maven with:

$ mvn integration-test

Below is some of the build output statements from a test run:

[INFO] [soapui:test {execution: soap-integration-test}]
soapUI 3.5.1 Maven2 TestCase Runner
19:31:31,085 INFO  [WsdlProject] Loaded project from [file:/home/prystasj/workspace/prystasj/mule/adder/adder-it/src/test/resources/adder-soapui-project.xml]
19:31:31,713 INFO  [SoapUITestCaseRunner] Running soapUI tests in project [adder]
...
19:31:31,734 INFO  [SoapUITestCaseRunner] Running soapUI testcase [Add_2_to_2]
19:31:31,750 INFO  [SoapUITestCaseRunner] running step [Add_2_to_2]
19:31:33,305 INFO  [SoapUITestCaseRunner] Assertion [SOAP Response] has status VALID
19:31:33,306 INFO  [SoapUITestCaseRunner] Assertion [Contains] has status VALID
19:31:33,306 INFO  [SoapUITestCaseRunner] Finished running soapUI testcase [Add_2_to_2], time taken: 1543ms, status: FINISHED
...
19:31:33,307 INFO  [SoapUITestCaseRunner] Running soapUI testcase [Add_1_to_2_to_3]
...
19:31:33,385 INFO  [SoapUITestCaseRunner] Project [adder] finished with status [FINISHED] in 1664ms

To access the soapUI Maven plugin, add the following plugin repository to your settings.xml:

<pluginRepository>
  <id>eviwarePluginRepository</id>
  <url>http://www.eviware.com/repository/maven2/</url>
</pluginRepository>

Groovy & Spock: Adding Numbers Together

2010-07-06T19:33:00.000-07:00

Lately, I've been playing around with embedding Mule in Tomcat. To get going, I wanted to start with creating an extermely simple service, and along the way I decided to try out the Spock specification framework to test the logic I'd be using in my service. I liked what I found so much, I thought I'd take a little detour and write about my first use of the framework, comparing it to more a common test class, and save the Tomcat posting for another day.

The simple service that will eventually be developed will take a list of numbers and add them up. Here is the class that will ultimately do the math:

class Adder {
    Integer add(List<Integer> numbers) {
        println "adding: $numbers"
        numbers.inject(0) { sum, item -> sum + item }
    }
}

The add method takes a list of numbers and returns their sum. I'm using Integer as the return type and List on the parameter declaration instead of the def keyword to aid in the generation of the service WSDL down the line. While the println won't make into the final version of the class, it might help us out demonstrating what is being sent to the method during testing.

With Spock we'll be writing a specification that describes the behavior of the Adder class:

Given a list of numbers, they should be added together to produce a sum.
Given a list containg no numbers, zero should be returned.

The framework's documentation does a great job explaining how to write a spec, so I will not to try and reproduce much of it here. The spec is pretty much self-explanatory, each of the two requirements manifests itself in a method:

import spock.lang.*

class AdderSpec extends Specification {

    def adder = new Adder()

    def "numbers should be added together to produce a sum"() {
        expect:
            sum == adder.add(numbers)
        where:
            sum | numbers
            4   | [2, 2]
            6   | [1, 2, 3]
            10  | [1, 2, 3, 4]
            1   | [0, 1]
            0   | [0, 0]
    }

    def "summing no numbers at all should return zero"() {
        expect:
            sum == adder.add(numbers)
        where:
            sum | numbers
            0   | []
            0   | null
    }
}

The where clauses resemble a table, with each entry describing the input to the add() method on the right, and the expected result on the left. The first entry in the first method would have the values for sum and numbers substituted into the expect clause so that it would read:

4 == adder.add([2, 2])

The output from the test run helps illustrate how things are added during a run of the spec:

Running org.prystasj.services.adder.AdderSpec
adding: [2, 2]
adding: [1, 2, 3]
adding: [0, 1]
adding: []
adding: null
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.091 sec

If the value for numbers instead read [2, 3], we would get the following failure when we run the test as 2+3 does not equal 4:

Condition not satisfied:

sum == adder.add(numbers)
|   |  |     |   |
4   |  |     5   [2, 3]
    |  org.prystasj.services.adder.Adder@969c29
        false

While I could of used a helper method to remove the redunduncy between the two expect clauses, I liked this output such much, I decided to keep it, as using a method here would modify the resulting failure description. For example:

check(sum, numbers)
|     |    |
false 4    [2, 3]

Compare the above spec class to a more traditional unit test and see what you think:

import org.junit.Before
import org.junit.Test
import static org.junit.Assert.assertEquals

class AdderTest {

    def underTest

    @Before
    void setUp() {
        underTest = new Adder()
    }

    def verifySumsFor(data) {
        data.each { sum, numbers ->
            assertEquals "$sum <= $numbers", sum, underTest.add(numbers)
        }
    }

    @Test
    void test_numbers_added_together_produce_a_sum() {
        def data = [
            4  : [2, 2],
            6  : [1, 2, 3],
            10 : [1, 2, 3, 4],
            1  : [0, 1],
            0  : [0, 0],
        ]
        verifySumsFor(data)
    }

    @Test
    void test_summing_no_numbers_at_all_should_return_zero() {
        def data = [
            0  : [],
            0  : null,
        ]
        verifySumsFor(data)
    }
}

Which version do you like better? Thanks for reading.

Mule: Transport of Larger Responses in CXF

2010-06-26T15:35:00.000-07:00

Upon calling a remote web service via a CXF endpoint in Mule I was presented with the following error:

Caused by: javax.xml.bind.UnmarshalException
 - with linked exception:
[com.ctc.wstx.exc.WstxIOException: Connection reset]
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.handleStreamException(UnmarshallerImpl.java:426)
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:362)
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:339)
        at org.apache.cxf.jaxb.JAXBEncoderDecoder.unmarshall(JAXBEncoderDecoder.java:537)
        ... 96 more

At first, I thought either my request or the response from the remote server was malformed XML since the exception class and stack trace seems to suggest a marshalling problem. On the other hand, the exception mentions a connection reset.

To help spare you kind readers the details of my investigation, it turns out the above problem only manifested itself when a large amount of data was being transferred between the services.

The Mule CXF Transport documentation lists an attribute, mtomEnabled, that can be enabled on the definition of the outbound endpoint for the remote service:

    <cxf:endpoint name="remoteService"
                  address="http://constanza.com/service/architectureService"
                  clientClass="com.costanza.ArchitectureService"
                  wsdlPort="ArchitectureServicePort"
                  wsdlLocation="http://constanza.com/service/architectureService?wsdl"
                  operation="getBlueprint"
                  mtomEnabled="true"/>

The attribute turns on the SOAP Message Transmission Optimization Mechanism which encodes the response payload for travel between the services. The Mule documentation refers to this allowing for data to be sent as an attachment. At the very least it tells CXF to be on the look out or to handle a potentially large response.

WSDLs and Message Parts not recognized

2010-06-22T07:36:00.000-07:00

I recently ran into an interesting problem consuming a web service where a call to the service produced the following error message:

Message part {http://myservice.com/}getSets was not recognized.  (Does it exist in service WSDL?)

I originally tested the service using soapUI with no problem, but when I tried to invoke the service from elsewhere I was presented with the error.

My service WSDL was auto-generated by CXF. Here I found I had to tweak it a little to help it better conform to the doc/literal style. The message part declaration for the getSets operation originally looked something like this (nothing special):

<wsdl:message name="getSets">
  <wsdl:part element="tns:getSets" name="parameters">
  </wsdl:part>
</wsdl:message>

The definition of the referenced getSets element is where the problem was hiding. The element declaration contained a reference to a complexType instead of having the complexType defined within, or as part of, the element declaration.

The original element defintion:

<xs:element name="getSets" type="tns:getSets"/>
<xs:complexType name="getSets">
  <xs:sequence>
    <xs:element minOccurs="0" name="shipment" type="xs:string"/>
  </xs:sequence>
</xs:complexType>

The new definition that allows calls from both soapUI and another client to work:

<xs:element name="getSets">
  <xs:complexType>
    <xs:sequence>
      <xs:element minOccurs="0" name="shipment" type="xs:string" />
    </xs:sequence>
  </xs:complexType>
</xs:element>

Hopefully this can help someone out who runs into something similar.

Mule: Defining Properties at Runtime

2010-05-06T12:49:00.000-07:00

I've had the desire for sometime to investigate telling Mule at run (or start-up) time what set of properties to use for a particular environment as in development, QA, or production, instead of deciding at build or deploy time what the target environment will be. So below I've come up with a rather simplistic service that uses a system property to define which set of properties to use at startup.

We will use the following Mule configuration element to define the environment properties file (explained in more detail here).

  <context:property-placeholder location="classpath:default.properties,classpath:${env}.properties" />

With the above line included in our Mule configuration, the default.properties file will be loaded first, followed by the environment-specific properties file. The latter will have the opportunity to override anything defined by the former.

The service will report the current season. A greeter bean configured in the Mule context will have two properties to be set via injection:

greetee - who the greeting will be addressed to
season - the current season (a true property of the environment)

The greeting will take the form of:

  "Hello ${greetee}, it is ${season}"

A default properties file will be used with the hope that we can eliminate having to repeat the property definitions
whose values would be common for every environment. We'll also investigate the ability to override this property in
case a particular environment would need to do so.

The default properties file will hold the 'greetee'.

    # default.properties
    greetee = Friend

We'll have two season-defining properties files, one for spring and one for winter. The winter properties file will simply define the season:

    # winter.properties
    season = winter

The spring properties file will define the season as well as override the greetee property set in default.properties:

    # spring.properties
    greetee = Chap
    season = spring

Since I'm using Maven 2 to build my service, we'll place the properties file in the src/main/resources directory. By convention, Maven will place all files found in this directory in the JAR file produced by the build. To make the files available to Mule on the classpath, we can place the JAR in hte lib/user directory of our Mule deployment so that they are available on the classpath as requested by the property-placeholder element.

The SeasonReporter class will have the two properties injected and will be used as a service component in our Mule model:

class SeasonReporter {
    def greetee
    def season
    def reportSeason() { "Hello $greetee, it is $season" }
}

Our Mule configuration will have an inbound HTTP endpoint that will reply synchronously with the environment-specific greeting. The component that is invoked will be implemented by a SeasonReporter class that takes the two properties used to create the seasonal greeting.

We'll direct Mule to use the reportSeason method with no arguments (by default the HTTP transport will provide a payload of one argument, /season here):

<mule>
    <context:property-placeholder location="classpath:default.properties,classpath:${env}.properties" />

    <model name="seasonModel">
        <service name="seasonService">
            <inbound>
                <http:inbound-endpoint address="http://localhost:8080/season"/>
            </inbound>
            <component>
                <no-arguments-entry-point-resolver>
                    <include-entry-point method="reportSeason"/>
                </no-arguments-entry-point-resolver>
                <spring-object bean="seasonReporter"/>
            </component>
        </service>
    </model>

    <spring:bean name="seasonReporter" class="SeasonReporter" scope="prototype">
        <spring:property name="greetee" value="${greetee}"/>
        <spring:property name="season"  value="${season}"/>
    </spring:bean>
</mule>

To start Mule using the winter properties file, we can pass in the env property when starting Mule on the command line with:

  $ mule/bin/mule start -M-Denv=winter

After starting Mule and hitting URL http://localhost:8080/season in a browser, we are presented with:

  Hello Friend, it is winter

Now we can try starting Mule up for season spring. The greeting should now include the overridden greetee property of 'Chap':

  $ mule/bin/mule start -M-Denv=spring

Giving us:

  Hello Chap, it is spring

Thanks for reading!

Mule & CXF: Multiple Operations and Groovy Components

2010-04-20T14:32:00.000-07:00

When I picture a web service in Mule, there is usually a component backing the endpoint. Here I have a case where I would want the service fielding the web service request to delegate its work to another service. To the client, the fact that Mule will pass the actual workload for a request to another service will be hidden.

In this example, we have a mock "warehouse", where a client can store and retrieve a box. A box is uniquely identified by an ID (a String). Each operation will result in the invocation of an additional, but separate, service to do the actual work.

Here's the interface defined for our warehouse:

package prystasj.warehouse;

import javax.jws.WebParam;
import javax.jws.WebResult;
import javax.jws.WebService;

@WebService
public interface Warehouse {

    @WebResult(name="receipt")
    List<String> retrieve(@WebParam(name="boxId") String boxId);

    @WebResult(name="box")
    String store(@WebParam(name="boxId") String boxId);
}

The inteface defines both the operations. To determine which operation the client wishes to invoke, we can inspect the cxf_property of the Mule message created by the CXF Transport.

A request indicating the store operation will be routed to a storageService, while a retrieve operation will be routed to a retrievalService:

<model name="warehouse">
    <service name="warehouseService">
        <inbound>
            <cxf:inbound-endpoint address="http://localhost:8080/warehouse"
                                 serviceClass="prystasj.warehouse.Warehouse"/>
        <inbound>
        <outbound>
            <filtering-router>
                <vm:outbound-endpoint path="retrieval" synchronous="true"/>
                <expression-filter evaluator="groovy" 
                        expression="message.getProperty('cxf_operation').getLocalPart() == 'retrieve'"/>
            <filtering-router>
            <filtering-router>
                <vm:outbound-endpoint path="storage" synchronous="true"/>
                <expression-filter evaluator="groovy" 
                        expression="message.getProperty('cxf_operation').getLocalPart() == 'store'"/>
            <filtering-router>
        <outbound>
    <service>
</model>

To test the processing flow out, we can script a couple of components with Groovy. A couple of println's will help us pick out that the operations were invoked when viewing the Mule log. With a scripting component, we'll have access to a variable payload, which in each operation will contain the web parameters (a boxId in both cases) as defined in the Warehouse interface.

    <service name="retrievalService">
        <inbound>
              <vm:inbound-endpoint path="retrieval"/>
        </inbound>
        <script:component>
            <script:script engine="groovy">
                println "Operation: retrieve; Box ID: $payload"
                new Box(payload)
            <script:script>
        <script:component>
    <service>
    <service name="storageService">
        <inbound>
            <vm:inbound-endpoint path="storage"/>
        </inbound>
        <script:component>
            <script:script engine="groovy">
                println "Operation: store; Box ID: $payload"
                new Receipt(payload)
            <script:script>
        <script:component>
    <service>

The store operation returns an instance of a Box and the retrieve operation returns a Receipt. As both services are invoked synchronously, the result of the component invocations will be returned to the calling Warehouse service, which will pass them through to the client.

XMLUnit: Simpler Asserts

2010-03-29T08:29:00.000-07:00

Quite a while ago, I wrote about using XMLUnit in my unit tests to compare XML results.

The method used to create Diffs has been useful, but the project offers a simpler way to compare XML:

import static org.custommonkey.xmlunit.XMLAssert.assertXMLEqual

class AssertXmlEqualTest {
 
    @Test void test_xml_is_similar() {
        def expected = "<hi>bye</hi>"
        def actual = "<hi>bye</hi>"
        assertXMLEqual expected, actual
    }
}

The test passes. If I change the value of actual to something different, say:

    def actual = "<hi>goodbye</hi>"

The test fails as the XML documents are no longer similar:

junit.framework.AssertionFailedError: org.custommonkey.xmlunit.Diff
[different] Expected text value 'bye' but was 'goodbye' - 
  comparing <hi ...>bye<hi> at /hi[1]/text()[1] to <hi ...>goodbye<hi> at /hi[1]/text()[1]

Since I have no reason in this simple case to be inspecting the Diff object created by XMLUnit as described in the previous posting, this method should serve as a nice substitute.

Mule: Service Composition with Dependencies

2010-03-07T12:00:00.001-08:00

Last time around, I did a little work with service composition in hopes to come up with an approach I could use in a real-world project. The approach I came up with last time appeared to fit the bill, until another requirement was added: The response from the first service would be needed to make the request to the second.

The new requirement definitely forced me to rethink my previous approach, as I could no longer simply send the intial request off to be routed to the two services that needed to be consumed, aggregating the responses for a request to the business logic.

For this example, I'll try to come up with a similar problem for a service to solve that follows a similar processing flow: The goal of this new service, given an order number, is to respond with a report about the customer containing their information and order history.

In order to accomplish this task, this new service will need to make use of two remote services:

A customer service: Returns the information for a customer who placed a particular order.
An order history service: Returns the order history for a particular customer.

The information contained in both service responses is used in the business logic contained within this composite service to produce a report, let's say to relate the ordering of certain products to different demographic profiles.

Now at this point, one might say: "Why can't the order history service return the customer information its provided in its request in its response?".

A good question, but some would argue that a service should not respond with data the caller already has or knows. My job here would be a lot of easier if the order history service could be changed to do just that, I could just pass the order history response to my business logic and not need to keep the customer service response around. On the other hand, the order history service should just be charged with returning the information it was requested to produce and not concern itself with how its response will ultimately be used, as long as that response is usable and correct. Not all clients might make use of the customer information returned in the order history response, and we should not request that type of overhead.

Ok, enough talking and more Mule-ing. For now I can start with the two things I know for certain, I will need to make requests to both the customer service and order history service, and they need to happen in order, so let's start there. For the sake of brevity, I'll omit any needed transformations to make the processing paths easier to follow.

<service name="orderService">
    <inbound>
        <jms:inbound-endpoint queue="order.queue" synchronous="true"/>
    </inbound>
    <outbound>
        <chaining-router>
            <vm:outbound-endpoint path="customer.queue" synchronous="true"/>
            <vm:outbound-endpoint path="history.queue" synchronous="true"/>
        </chaining-router>
    </outbound>
</service>

The incoming request will be routed to a VM queue where a listening internal service will route it to the remote customer service:

<service name="customerService">
    <inbound>
        <vm:inbound-endpoint path="customer.queue"/>
    </inbound>
    <outbound>
        <pass-through-router>
            <jms:outbound-endpoint queue="customer.service.request.queue" synchronous="true"/>
        </pass-through-router>
    </outbound>
</service>

The response from the customer service will be provided to the order history service, which is defined similarly.

Now the next problem here is getting both responses to the same place so a request for my business logic can be created and the request carried out. From here, I'll add the internal service containing the business logic:

<service name="reportService">
    <inbound>
        <vm:inbound-endpoint path="responses.queue" synchronous="true"/>
        <custom-inbound-router class="ResponseAggregator"/>
    </inbound>
    <component>
        <spring-object bean="reportingComponent"/>
    </component>
</service>

The inbound router is an extension of Mule's AbstractEventAggregator class that waits for a group of 2 responses (to avoid a context switch I'll omit the implementation details for now). It's job is to take both service responses and create a request for the reportingComponent. Now that we have that defined, how do we get the service responses to its VM queue?

Getting the order history response to the queue is easy, we just add another outbound endpoint to the chaining router:

<service name="orderService">
    <inbound>
        <jms:inbound-endpoint queue="order.queue" synchronous="true"/>
    </inbound>
    <outbound>
        <chaining-router>
            <vm:outbound-endpoint path="customer.queue" synchronous="true"/>
            <vm:outbound-endpoint path="history.queue" synchronous="true"/>
            <vm:outbound-endpoint path="responses.queue" synchronous="true"/>
        </chaining-router>
    </outbound>
</service>

Now we need to get the customer service response to the order history service and to the same response.queue. A "customer response routing service" might work here. This new service would be added as an endpoint in the chaining router between the routings to the customer and history services. This new internal service needs to route its request to the VM queue for the report service and return the request to the chaining router:

<service name="customerRoutingService">
    <inbound>
        <vm:inbound-endpoint path="customer.routing.queue" synchronous="false"/>
    </inbound>
    <outbound>
        <multicasting-router>
            <vm:outbound-endpoint path="responses.queue" synchronous="false"/>
            <vm:outbound-endpoint path="async.queue" synchronous="false"/>
        </multicasting-router>
    </outbound>
    <async-reply>
        <vm:inbound-endpoint path="async.queue"/>
        <single-async-reply-router>
            <!-- we want to return the message that maps to the correlation ID found on the original request -->
            <expression-message-info-mapping messageIdExpression="#[header:JMSCorrelationID]"
                                             correlationIdExpression="#[header:JMSCorrelationID]"/>
        </single-async-reply-router>
    </async-reply>
</service>

Unlike the other internal services that respond synchronously, this service is asynchronous. To the caller, the chaining router, the request will have been handled synchronously, following the Async Request Response style.

The request received (the customer information), will be sent to the desired VM queue and to the async.queue where a reply router will return the customer information response it recevied on its inbound endpoint to the chaining router.

Now we place the routing to the customerRoutingService into the chain:

<service name="orderService">
    <inbound>
        <jms:inbound-endpoint queue="order.queue" synchronous="true"/>
    </inbound>
    <outbound>
        <chaining-router>
            <vm:outbound-endpoint path="customer.queue" synchronous="true"/>
            <vm:outbound-endpoint path="customer.routing.queue" synchronous="true"/>
            <vm:outbound-endpoint path="history.queue" synchronous="true"/>
            <vm:outbound-endpoint path="responses.queue" synchronous="true"/>
        </chaining-router>
    </outbound>
</service>

Since the last endpoint in the chain will be the second message placed on responses.queue, the inbound router for the reportService will be invoked, in turn invoking the component. The response from the component will be returned to the chain, completing it. The chaining router will then return the response to the caller of the orderService.

To sum up the processing flow:

A request is sent to the order service.
The request is forwarded to the remote customer service.
The customer service response is sent to the customer response routing service.
The routing service sends the request to the responses queue (event #1) and to an async reply queue.
A listener on the async reply queue returns the customer service response to the chaining router
The customer service response is forwarded to the remote order history service.
The order history response is sent to the responses queue (event #2).
The aggreagation router listening to the responses queue, having both events, is invoked, creating the reporting request.
The reporting component is invoked creating the service response.
The response is returned to the chaining router, completing the chain.
The response is returned to the caller.

This could one of several approaches to the stated problem, alternatives are welcome!