Saturday, August 30, 2014

Using Maven to Detect Duplicate Classes

Using Maven to Detect Duplicate Classes

As the number of dependencies from some our corporate project grew, we ran into a couple cases where we would be transitively pulling into different versions of the same classes. For a long time, we were lucky, as this did not cause any issues, until someday a developer new to our team introduced trying to run our application using a different operating system. The JVM on his machine loaded up the classes in a different order, and his application instance used an older version of a class that resulted in errors running the application.

This developer thankfully pointed out we can detect when we were pulling in different versions of classes using the Maven Enforcer plugin with some help of some extra rules. We added something like this to our parent POM:

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-enforcer-plugin</artifactId>
        <version>1.3.1</version>
        <executions>
            <execution>
                <id>enforce</id>
                <phase>validate</phase>
                <configuration>
                    <rules>
                        <banDuplicateClasses>
                            <ignoreClasses>
                                <ignoreClass>javax.*</ignoreClass>
                                <ignoreClass>org.junit.*</ignoreClass>
                                <ignoreClass>org.hamcrest.*</ignoreClass>
                                <ignoreClass>org.slf4j.*</ignoreClass>
                            </ignoreClasses>
                            <findAllDuplicates>true</findAllDuplicates>
                        </banDuplicateClasses>
                    </rules>
                    <fail>false</fail>
                </configuration>
                <goals>
                    <goal>enforce</goal>
                </goals>
            </execution>
        </executions>
        <dependencies>
            <dependency>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>extra-enforcer-rules</artifactId>
                <version>1.0-beta-2</version>
            </dependency>
        </dependencies>
    </plugin>

Example output from the validate phase of a build:

    [INFO] --- maven-enforcer-plugin:1.1:enforce (enforce) @ cataloging-api ---
    [INFO] Adding ignore: javax.*
    [INFO] Adding ignore: org.junit.*
    [INFO] Adding ignore: org.hamcrest.*
    [INFO] Adding ignore: org.slf4j.*
    [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
    Duplicate classes found:

      Found in:
        org.company.id.api:id-api:jar:1.28.0:compile
        org.compnay.legacy.id-api:id-common-api:jar:2.8.0:compile
      Duplicate classes:
        org/company/service/id/SAMLEntityId.class
        org/company/service/id/crypto/ServiceKey.class

We were surprised at the amount of duplicate classes we were bringing in. We thought our dependency management was in pretty good shape, but adding the new rules showed we still had some work to do.

It is also worth noting, while our project is large, adding the plugin execution to the build added an insignifcat amount of time to the build overall, so there should be no performance concerns if you were considering adding something similar to your project.

Saturday, August 9, 2014

Spring AOP: Excluding Methods in Pointcuts

We use Spring AOP to make use of aspects in our applications for the purpose of recording statistics about requests they receive. This approach has worked out well because it helps separate the concern of tracking statistics from our business logic. Previous iterations of our applications mixed the two, leaing to code that was generally harder to maintain.

Below is an example of how we might configure an aspect to target all entry points into our business application with the same aspect:

    <aop:config>
        <aop:pointcut id="requestPointcut"
                      expression="execution(public * org.company.service.*.*(..))"/>
        <aop:aspect id="requestAspect" ref="requestStatAspect">
            <aop:around method="around" pointcut-ref="requestPointcut"/>
        </aop:aspect>
    </aop:config>

    <bean id="requestAspect" class="org.company.service.aop.RequestIdentifierAspect" lazy-init="true"/>

With the above, we are saying we want request identifiers managed for every request to our interfaces. With the wildcards in the expression, we are saying target every class in the org.mycompany.myapp.service package and every method in those classes.

This approach worked greatly until we added an interface that provided health reporting for system monitoring purposes. This method lived in a new interface. Let's suppose added the interface method is defined as:

  • public boolean org.company.HealthService.isHealthy()
    

Since the method signature is different (it takes no arguments), running the advice in the aspect would lead to errors as the aspect evaluated the method arguments. We could change the code in the aspect to avoid processing if the arguments contained within the join point do not match what we are expecting. This approach is not that attractive because it one, involves changing code for special cases, and two, could lead to the aspect siltently not invoking its main logic at times we might not expect. If we target a method with the aspect unintentionally, we want to be informed by errors in our log.

An alternative we found was to exclude the new method from the pointcut expression:

  • execution(public * org.company.service.*.*(..)) &amp;&amp; !execution(* org.company.service.HealthService.*(..))
    

Just with adding special case logic to an aspect or code in general to avoid exception cases, we have to be careful that the exceptional cases we avoid targeting with the aspect does not get out of hand as we are tempted to add more in the future.

Saturday, June 28, 2014

Sending data to a server with Groovy

The Groovy language provides a lot of additions to existing Java classes that can make our jobs easier.

A couple of them we were able to take advantage of lately that use closures to make our coding simpler were:

Our use case: We have a service whose responsibility is to send "records" over a TCP/IP connection to a remote server.

Of course we want to include a test for this in our integration tests, but how can we do it? We cannot expect a client to leave a server available open to receive records sent by our tests. We could create our test server to accept and output the sent record.

We'll write a test server using Groovy that can be started at the beginning of the test run and shutdown at the end:

class TestServer {

    static defaultPort = 6666

    static ServerSocket start_on_default_port() {
        start_on defaultPort
    }

    static ServerSocket start_on(port) {

        def server = new ServerSocket(port)
        String receivedRecord ''
        def done = false

        Thread.start { t ->
            while (!done) {
                server.accept { socket ->
                    socket.withStreams { input, output ->
                        done = false

                        while (!done) {
                            def block = new byte[128]
                            int size = input.read block

                            if (size == -1) done = true // there is nothing left to have been received

                            block = trim(block as List)
                            receivedRecord += new String(block)
                        }
                    }
                }
            }
        }
        server
    }

    /** Remove any empty or non-set bytes at the end of the array. */
    private static def trim(List block) {
        block.reverse().findAll { b -> b != 0 }.reverse() as byte[]
    }
}

In our integration tests class, written with Spock, we start and stop the server at the beginning of the run:

@ContextConfiguration
class RecordDeliveryServiceIntegrationSpec extends Specification {

    @Autowired
    RecordDeliveryService service

    static def server

    def setupSpec() {
        server = TestServer.start_on_default_port()
    }

    def cleanupSpec() {
        server.close()
    }

    def 'delivers a record to a server'() {
        expect:
        expectedResponse == service.send 'localhost', TestServier.defaultPort, record
    }

    //...
}

For reference to complete the example, here's what a simplified service implementation could look like, using a methodology similar to that the test server uses to accept the record:

dclass RecordDeliveryService implements DeliveryService {

    public def send(String host, long port, Record record) {

        def socket = new Socket(host, port)

        socket.withStreams { input, output -&gt;
            output &lt;&lt; record.toString()
            output.flush()
            output.close()
        }

        socket.close()
    }
}   //...
}

If you can think of any improvements, or a better approach to the problem, I would be glad to hear from you!

Maven: Failing the Build After Running All Modules

Maven: Failing the Build After Running All Modules

Here's a quick one to help me feel a better about not posting anything for quite a while.

In our large project at work that contains several deployed services, we have several parent integration testing Maven modules that run after our nightly development install. Each has several child modules that each run tests against a particular endpoint in each service.

So naturally, we want our build server to kick off each parent testing module after the install. The problem we ran into is when that one of the earlier child modules failed, the build will stop, without running the rest of the modules (Maven's default behavior).

This would often lead to us making changes to get the tests in the failing module to pass, then re-running the testing build, only to have a later child module to fail. It would be nice to have had all the modules to run, reporting all failures, so that all the required changes and fixes could be made up front, saving us time.

Fortunately, Maven provides an option we learned about do just that:

-fae,--fail-at-end     Only fail the build afterwards;
                       allow all non-impacted builds to continue

There are also a couple more failure related options as well, but we have yet to find any use for them (yet):

-ff,--fail-fast        Stop at first failure in reactorized builds
-fn,--fail-never       NEVER fail the build, regardless of project result

When working on the tests locally, we can pick up the build from within the parent from the point of the failed chilid module after making changes to our running service to fix the test with:

-rf,--resume-from      Resume reactor from specified project

For example:

$ mvn verify -rf :service-integration-testing-records

For more on working with multiple modules, see: Guide to Working with Multiple Modules.

Saturday, May 11, 2013

Spring AOP: Ordering Aspects

Sometimes we may want to apply multiple aspects to the same pointcut when using Spring AOP, keeping each aspect responsible for a single activity or concern. Below we'll discuss a business example of why we may want to order more than one aspects and how to ensure that ordering using Spring. Again, we'll be writing our code in Groovy to keep things simple.

To start our example, let's say we have a service method that takes a Purchase and returns a Shipment:

interface PurchaseProcessor {
    Shipment process(Purchase)
}

Our first use of AOP may be recording some information about the Purchase for statistics purpose. Since statistics keeping is a separate concern from actually processing the Purchase, it serves as a perfect candidate for an aspect. For the purposes of this example, we will use an aspect that provides around advice instead of an alternative like before advice:

class StatRecordingAspect {

    StatRecorder statRecorder

    Object around(ProceedingJoinPoint joinPoint) {
        Purchase purchase = joinPoint.args[0]
        statRecorder.record purchase
        joinPoint.proceed()
    }
}

Later on, we decide we want to inform an internal process via JMS when the processed Shipment is on its way out the door. As the sending of the information could be considered another concern separate from the actual processing of the Purchase we decide it to should be done in an aspect. Additionally, this notification requires information from the statistics from the Purchase:

class NotificationAspect {

    JmsShipmentNotifier notifier
    StatRecorder statRecorder

    Object around(ProceedingJoinPoint joinPoint) {
        Purchase purchase = joinPoint.args[0]
        Shipment shipment = (Shipment)joinPoint.proceed()
        notifier.notifyAbout shipment, statRecord.statsFor(purchase)
        shipment
    }
}

The above business case requires that the stats about the Purchase be recorded before the notification. If we target both aspects at the PurchaseProcessor method via the same pointcut, how do we guarantee that the stats are recorded first?

We could opt to combine the two aspects into one, but it could be argued then the combined aspect would have one more than responsibility, record stats and sending out notifications, and have more than one concern.

A solution provided by Spring AOP is to have each aspect implement the Ordered interface provided by spring-core (Spring also provides an @Order annotation as an alternative to implemented the Ordered interface.):

.
class StatRecordingAspect implements Ordered { //...

    Object around(ProceedingJoinPoint joinPoint) { //... }

    @Override
    int getOrder() {
        0
    }
}

class NotificationAspect implements Ordered { //...

    Object around(ProceedingJoinPoint joinPoint) { //... }

    @Override
    int getOrder() {
        1
    }
}

With the above implementations, the StatRecordingAspect would be executed first as it is the aspect with the lowest order.

If the above makes you a little uneasy because the values returned are not linked to each other in any way, the values seem arbitrary, you could relate the orders to each other with an enumeration:

enum AspectOrder {

    STAT_RECORDING(0),
    NOTIFICATION(1)

    final int value

    AspectOrder(int value) {
        this.value = value
    }

    int value() {
        value
    }
}


class StatRecordingAspect implements Ordered { //...

    Object around(ProceedingJoinPoint joinPoint) { //... }

    @Override
    int getOrder() {
        AspectOrder.STAT_RECORDING.value
    }
}

class NotificationAspect implements Ordered { //...

    Object around(ProceedingJoinPoint joinPoint) { //... }

    @Override
    int getOrder() {
        AspectOrder.NOTIFICATION.value
    }
}

Now in your unit tests you can compare the order value of one aspect to the others when testing the getOrder() method, ensuring they would in practice be ordered as you would expect.

Loading Classpath Resources from a Static Context

Recently, I had to figure out how to load classpath resources from within a static method in a test class. I've been doing it from a non-static method forever, but had to do some searching to figure out how to do it from a static method.

For example to build a scenario where one might want to do this, let's take a generic Record class that is used by many of our classes under test. As JSON is the input method for records from the user of our system, it might make sense to store our records to use during our tests as JSON test resources.

During our tests, we can load the record JSON from a test resource on the classpath with a test helper class like the below. The Groovy test class uses the ObjectMapper class from Jackson:

class RecordBuilder {
    def recordFor(resource) {
        new ObjectMapper().readValue textOf(resource), Record.class
    }
    def textOf(resource) {
        getClass().getClassLoader().getResourceAsStream(resource).text
    }
}

One annoyance with using the above class in our test classes is that each test class would need to create or be given an instance of RecordBuilder to do its work. An alternative is to make the record-building method static, which should be ok since the normal concerns regarding static methods, pro or con, should not apply in a testing scenario as tests are usually executed one at at time.

The problem with making the recordFor method static is the approach to reading the resource from the classpath no longer works, and the solution may not be apparent to those who have not tried to do something similar before. Below includes the answer I was able to find, referring to the RecordBuilder class itself:

class RecordBuilder {
    static def recordFor(resource) {
        new ObjectMapper().readValue textOf(resource), Record.class
    }
    static def textOf(resource) {
        RecordBuilder.class.getClassLoader().getResourceAsStream(resource).text
    }
}

I guess this was a rather long-winded way to show how to load classpath resources from a static context, but I hope it will do. Thanks.

Wednesday, January 23, 2013

Spring & RestTemplate: Dealing with Double Escaped Entities

We've been using Spring's RestTemplate pretty extensively to perform various operations. Several have escaped entities in the query. For example, one query requires its parameters be surrounded by quotes:

http://myhost.com/records?query=no+"1"+owner="John"

We could inject that URL into a bean containing a RestTemplate instance, representing the URL with a String and escaping the quotes with %22 to make the parsing of the bean file easier:

    <bean id="recordUrl" class="java.lang.String">
        <constructor-arg value="http://myhost.com/records?query=no+%221%22+owner=%22John%22"/>
    </bean>

Then executing the query:

  restTemplate.getForObject(recordUrl, String.class, urlVariables)

However, our query would not work as each %22 would be escaped again as %2522. For example:

  DEBUG main org.springframework.web.client.RestTemplate - Created GET request for: "http://myhost.com/records?query=no+%25221%2522+owner=%2522John%2522"

How do we avoid the double escaping?. The RestTemplate Javadoc contains the answer. To avoid the double-escaping, we need to use the version of the RestTemplate method that takes a java.net.URI as the first parameters instead of a String representing the URL:

  restTemplate.getForObject(new URI(recordUrl), String.class, urlVariables)

Or instead of creating a URI, inject an instance of one like what was done with the String originally.