Shameless plug

For all my swedish readers who aren't either working for Citerus or are regular guests in the same IRC channel as I am (should amount to about zero people, I'm afraid) - here's an introductory article on the excellent Grails framework that I've written for PNEHM, Citerus' newsletter on agile development.

I will also host a short (20 minutes) oral presentation on the same subject at the upcoming JFokus conference, in January.

Advisor summary

It's been a while since the last post, I've been busy with my new assignment that begun right after my vacation. Anyway, here a nifty little routine to quickly get an overview of which beans are woven by what advice, and also what advice weaves which beans in a Spring context:

ApplicationContext context = ... ;  // Create your context 

String perBeanSummary = "--- Advisors per bean ---\n";
Map<String,Advised> beanMap = BeanFactoryUtils.beansOfTypeIncludingAncestors(context, Advised.class);
Map<Advisor,Set<String>> advisorMap = new HashMap<Advisor, Set<String>>();

// This gathers advisors per bean and beans per advisors,
// and builds the presentation of advisors per bean 
for (String beanName : beanMap.keySet()) {
  Advised advised = beanMap.get(beanName);
  perBeanSummary += beanName + ":\n\t";
  for (Advisor advisor : advised.getAdvisors()) {
    perBeanSummary += advisor.getAdvice().getClass().getName() + "\n";
    Set<String> beans = advisorMap.get(advisor);
    if (beans == null) {
      beans = new HashSet<String>();
      advisorMap.put(advisor, beans);
    }
    beans.add(beanName);
  }
  perBeanSummary += "\n";
}

// Builds the presentation of beans per advisor
String perAdvisorSummary = "+++ Beans per advisor +++\n";
for (Advisor advisor : advisorMap.keySet()) {
  perAdvisorSummary += advisor.getAdvice().getClass().getName() + "\n";
  for (String beanName : advisorMap.get(advisor)) {
    perAdvisorSummary += "\t" + beanName + "\n";
  }
  perAdvisorSummary += "\n";
}

System.out.println(perBeanSummary);
System.out.println(perAdvisorSummary);

Grooeat work, JetBrains!

The first day after the vacation was fairly productive, after all. In preparation for an upcoming PNEHM article, I've successfully installed a snapshot of IDEA 7.0 and built a Subversion snapshot of the Groovy/Grails plugin. This is the first thing I tried out - it looks very promising:


Dynamic typing and completion!

Update: it turns out IDEA also completes the Groovy additions to the JDK. Not completely unexpected when you've seen the screenshot above, but nevertheless very nice!

Implementing ActiveRecord in Java

The ActiveRecord (AR) design pattern is very popular right now, forming the base of web application frameworks such as Ruby on Rails and the Groovy-based Grails. Martin Fowler defines the pattern as follows:

An object that wraps a row in a database table or view, encapsulates the database access, and adds domain logic on that data.

AR is very closely related to the concept of an Object-Relational Mapper (ORM), and is an alternative to the Data Access Object (DAO) and Repository patterns. In the latter two patterns, persistent data operations are separated from the domain object into one or more dedicated interfaces.

AR is very attractive if you want to build a more powerful domain model, as opposed to using it as a simple data container. It ties the most obvious domain logic into the domain object (namely CRUD), and opens up possibilities to write higher-level properties and operations on your domain objects.

It can also aid in normalizing the three-tier architecture, by which I mean avoiding the all too common situation where you have a request mapped to an object in the MVC layer, which calls the service layer, which initiates a transaction and calls the DAO layer, which calls the ORM, which stores the object in the database. It might be even worse - you may need to convert the bound request data from a form bean to a Data Transfer Object, both of which could be separate from the domain object. Normalizing this operation would mean binding request data to a domain object, which then stores itself.

So, building an ActiveRecord base class in Java means that at least the following methods must be implemented:

public class ActiveRecord<T> {
public static <T> T load(Long id) { .. }
public void store() { .. }
public void delete() { .. }
public static <T> List<T> findAll() { .. }
}

Derived classes will add all sorts of operations, notably a number of specialized finders with similar signatures to findAll().

In order to be able to access non-domain services we need a way to access external services, preferably through Dependency Injection. It's also mandatory that we are able to isolate the domain object for test purposes, and to be able to mock or stub dependencies, all of which is enabled by using DI.

It's also natural to use a full-fledged ORM - Java now has a standard API for that (JPA), and there are a number of different implementations available: Hibernate, TopLink, OpenJPA etc. There's also JDO, iBatis and of course you could also use plain JDBC if you really have to.

I've been using (surprise, surprise!) Hibernate as ORM tool, and Spring for DI and transaction demarcation, all of which are using annotations and build-time weaving using AspectJ and AJDT. Dependency injection of service beans into domain objects is taken care of by the @Configurable annotation, and transaction demarcation by the the @Transactional annotation, so those problems are very cleanly solved.

However: static methods, such as loaders and finders, don't seem to be woven by transactional advice, at least not when using compile-time weaving. I need to investigate this further, but I believe it might be caused by the fact that a "this" joinpoint is being used here. Workarounds include writing transactional advice manually, using TransactionTemplate for example, or following the mixed AR/Repository pattern suggested below. The best solution would be to have an aspect that's able to read transaction attributes from annotations even in a static context.

The first roadblock is how to gain access to the ORM in a static context, the loader and finder methods. We need a reference to the unit-of-work (session) provider - the SessionFactory in this case - in order to create or obtain the current session for performing data operations. Since we don't have an instance of AR or a derived class, we can't access any injected SF reference.

I've thought long and hard about this, and tried a number of different approaches, but the way I see it, you basically have to give up either static loaders/finders or give up dependency injection. Any way you look at it, you will need some sort of static handle to the SF - you might make the SF member of the AR class static, or you could use some variation of SessionFactoryUtil, or maybe use some sort of lightweight holder object that's instantiated, injected and finally discarded as part of the static operation. The problem with static references, and the ServiceLocator pattern, is that you can't isolate objects completely - there can be only one implementation per class loader at a time, and all instances of a class with a static reference must share the same implementation. This makes testing harder and less robust, compared to a pure DI environment: you must make sure to "reset" the service locator reference after each test, even in case of failures, so you'll end up with a number of try - finally blocks everywhere, and you can't run tests in parallell since you can't guarantee what implementation the factory will return. (Crazy) Bob Lee talks a little about that in this Guice presentation.

So, as far as I can tell, you will need to give up DI, or at least mix DI and ServiceLocator in your application, if you want "pure" ActiveRecord. I'll be very interested if anyone can show a way to use AR and DI together, though :-). For now, I prefer a mix between AR and Repository, but more about that in a little while. If you decide to go for pure AR by using a static SF reference, your next problem will be how to tell the ORM which class to load, without adding redundant data in derived classes, such as overriding load() or keeping a static class member pointing to its own class.

When performing a load, you will need to know what class, or sometimes what table, to load the data from, in addition to the supplied identifier property. Optimally, the implementation of load() exists in the top class ActiveRecord only, and should return a correctly typed object. In short, we want to use the API like this:

Customer c = Customer.load(1);
Item item = Item.load(125);

where Customer and Item both inherit ActiveRecord.

Typing is taken care of by generics, as you can see a few paragraphs earlier. But since it's a static method, there's no "this" to ask for the current class, so finding the current class to feed to Session.load(id, clazz) is harder. There is no API access point in Java, and I've fiddled with various reflection hacks against sun.reflect.Reflections for example, to peek at the call stack, but to no avail. I did however find a way by using the "call" joinpoint in AspectJ and the following construct:

/*
* Keep the traditional load() signature as access point.
*/
public static <T> T load(Long id) {
throw new RuntimeException("This body should never be reached, weaving has not been performed");
}

/*
* This method performs an actual load.
*/
protected static <T> T doLoad(Long id, Class clazz) {
return (T) getSession().load(id, clazz);
}

/*
* This inner class aspect intercepts the call to load, and inspects the join point
* to determine what class the static call was made on, and reroutes the call to
* doLoad() with the correct class parameter.
*/
@Aspect
protected static class ClassIdentifier {

@Before("call (* load(Long)) && args(id)")
public T interceptLoad(ProceedingJoinPoint pjp, Long id) {
  Class clazz = pjp.getSignature().getDeclaringType();
  return ActiveRecord.doLoad(id, clazz);
}

}

So, if you are prepared to give up reliable replacement of the ORM interface reference, ActiveRecord is within reach. In fact, you might find that you rarely or never mock or stub something like the SessionFactory, but instead simply use a dedicated data source for testing which is loaded with test data. That's perfectly reasonable imo, if you work directly against the ORM API, but it gets a bit more complicated if you keep your own DAO layer around, tested separately from the domain object, and inject that into your domain objects instead.

If you want to go for pure DI, I would argue that it's reasonable to split CRUD operations between the domain objects and a shared "read only"-repository. Operations that work on an actual instance, such as user.store() or item.delete() (non-static by nature), are placed in the domain objects. But operations that result in one or more instances are retrieved according to various criteria, loaders and finders, are placed on a separate service. Those are the same methods that would be static in ActiveRecord.

This approach is DI-compliant, because each domain object instance has its own non-static reference to the ORM, and the Repository service also is injected with an ORM reference. The repository can be regarded as a third party, where you go to retrieve instances of domain objects. By clever use of generics, the amount of code can be kept low, and a single @Transactional annotation at class level on the implementation marks every method for execution in a read-only transaction.

interface Repository {
// Typing on the methods instead of the interface
// allows us to share this interface across the domain model
<T> T load(Long id, Class clazz);
<T> List<T> findAll(Class clazz);

// Various specific finders are added as they are needed.
Customer findByUsername(String username);
List<Item> findDeliveredItems();                

// Some kind of generic query-object method might be added too
}

This distinction between instance-tied domain logic versus third-party is then extrapolated throughout the application.

Weird but useful generics trick

I stumbled upon the following piece of code while programmatically creating AspectJ proxies with Spring:

// create a factory that can generate a proxy for the given target object
AspectJProxyFactory factory = new AspectJProxyFactory(targetObject);

// add an aspect, the class must be an @AspectJ aspect
// you can call this as many times as you need with different aspects
factory.addAspect(SecurityManager.class);

// you can also add existing aspect instances, the type of the object supplied must be an @AspectJ aspect
factory.addAspect(usageTracker);

// now get the proxy object...
MyInterfaceType proxy = factory.getProxy();

(Pasted from here)

If you look carefully, you'll notice that A) the AspectJProxyFactory class is not parameterized, and B) the proxy creation is assigned to a MyInterfaceType without a cast. I found this to be rather confusing, and took a quick peek at the AspectJProxyFactory source:

public class AspectJProxyFactory extends ProxyCreatorSupport {

[...] // Stuff

public <T> T getProxy() {
  return (T) createAopProxy().getProxy();
}

}

The return type is inferred by the assignment, regardless of the (lack of) type on the owning class, which effectively looks and feels like dynamic and static typing all at the same time!

Kind of weird, but might be useful. I, for one, was not aware of this technique.

Making JDOM run faster

It's time for another cool AspectJ hack here on RMA, once again leveraging Codehaus' excellent Maven plugin. Still working on the same project as the last post, we've decided to move from standard JAXP to JDOM for programmatic XML handling (the JDOM api is so much nicer to work with it's not even funny). As the returning reader may recall, the model in the MVC part of our application basically consists of a number of XML documents, and the view rendering technology is Freemarker. Freemarker is supposed to be able to handle both JAXP, JDOM and dom4j, but a closer look will reveal that it's more or less only JAXP that's up to date, the other wrappers are deprecated and don't work correctly anymore, which is a shame.

Using JDOM, there is a simple workaround however: the DOMOutputter, which is capable of converting an org.jdom.Document to an org.w3c.dom.Document which can then be handed to Freemarker. There is a speed penalty of course, and when profiling we found that almost all the time is spent in this method, in JAXPDOMAdapter:

public Document createDocument() throws JDOMException {

try {
        // We need DOM Level 2 and thus JAXP 1.1.
        // If JAXP 1.0 is all that's available then we error out.
        Class.forName("javax.xml.transform.Transformer");

        // Try JAXP 1.1 calls to build the document
        Class factoryClass =
            Class.forName("javax.xml.parsers.DocumentBuilderFactory");

        // factory = DocumentBuilderFactory.newInstance();
        Method newParserInstance =
            factoryClass.getMethod("newInstance", null);
        Object factory = newParserInstance.invoke(null, null);

        // jaxpParser = factory.newDocumentBuilder();
        Method newDocBuilder =
            factoryClass.getMethod("newDocumentBuilder", null);
        Object jaxpParser  = newDocBuilder.invoke(factory, null);

        // domDoc = jaxpParser.newDocument();
        Class parserClass = jaxpParser.getClass();
        Method newDoc = parserClass.getMethod("newDocument", null);
        org.w3c.dom.Document domDoc =
            (org.w3c.dom.Document) newDoc.invoke(jaxpParser, null);

        return domDoc;
    } catch (Exception e) {
        throw new JDOMException("Reflection failed while creating new JAXP document", e);

You're probably wondering why the hell they're using this much reflection just to create an empty Document (we did, anyway). It avoids compile-time dependencies on certain javax.xml classes, but it sure isn't designed with high performance in mind!

In addition to that, the call trace from DOMOuputter to JAXPDOMAdapter contains private methods, so you can't simply inherit and override with your own implementation. So, what now? AspectJ to the rescue, of course!

What we realy want to do is replace this implementation with one that performs the sam thing but statically, so we wrote this around advice (DocBuilder creation omitted for brevity):

@Aspect
public class FastDOMDocumentCreator {

@Around("execution (* org.jdom.output.DOMOutputter.createDOMDocument(..))")
public Object createDOMDocument(ProceedingJoinPoint pjp) throws Throwable {
    return docBuilder.newDocument();
}

}

This is a lot faster than the default way. But how do we perform weaving of JDOM classes, that are in an external jar? Actually, it's really simple: just place this snippet in the AspectJ plugin section in the pom-xml (details here):

<configuration>
<weaveDependencies>
 <weaveDependency>
 <groupId>jdom</groupId>
 <artifactId>jdom</artifactId>
</weaveDependency>
</configuration>

If you're using AJDT (which I highly recommend), make sure you add the JDOM jar to the inpath in the configuration dialog, that way this weaving runs on the fly just like regular compilation. On a sidenote, AJDT has some really cool features such as contextual information on "weaves" and "weaved by". It supports annotation-style aspects too.

The weaved jar is exploded under the output directory, with the new weaved classes instead of the vanilla ones. This trick pushed the JDOM-to-JAXP document converstion from the top way down on the hotspot list, with a fairly small amount of work.

Update: While examining the situation a bit closer, we found that in fact it's possible to accomplish the same thing without having to weave the JAXPDOMAdapter. Extending AbstractDOMAdapter using the same code as in the aspect, and passing the new class' name to the DOMOutputter constructor will work too. Nevertheless, the technique is still interesting and may be applicable (as the only solution) somwhere else.

Clean solutions are the best

Don't you love it when things just seem to fit together exactly as you want them to? It doesn't happen all too often, but when it does, you just want to blog at the top of your lungs...

I'm working on a project where we communicate with an index server over HTTP, and recieve the query results in XML format. It's a data source, which basically boils down to this interface - the DocumentSource:

Document retrieveDocument( URL url );

The contract of this component is that if the document retrieval succeeds, we get a parsed and ready org.w3c.Document back, and in all other cases it throws some kind of exception. Handling exceptions is done in another part of the application, so don't worry about that right now.

Now, a failure can be either uncontrolled, for example if the index application hits a bug, if there's a network problem, the server might be on fire and so on. It could also be a problem with our implementation of of the afromentioned interface (not likely). In either case, the component will throw an exception to the caller, possibly wrapped in some way.

But a failure can also be controlled, which means that the index server returns a valid XML error message along with HTTP status 200 (OK), if the query is invalid in any way. This is also considered an error, and we handle in like this:

if (isErrorResult(document)) {
  throw new IndexErrorException(method, document);
}

The method variable is the HttpMethod that we tried to execute, containing host, port and query information, and document is the parsed XML response from the index, in this case an error message.

What we want to do now is log this error, as transparently as possible. The snippet above expresses that we're not really interested in handling the error in detail, we just leave it at "ok, there was an error, so let's throw an exception. Here's all the information I have on why and where the error occurred".

That's all very well, but we still have to inspect the XML error message to present the error in a readable way. The best way to do that is of course the message property of the exception.

Here's where another aspect of our application comes in: we use Freemarker to build views (HTML and others) using the XML data we retrieve from the DocumentSource interface shown above. And since the error message is also XML, and we want to build a kind of view - a String - why not use the same approach here? That way we won't have to deal with cumbersome Java DOM apis, and we have maximum power to extract and format the error message the way we want. Sounds like a good idea.

It's also the case that we've abstracted away a lot of the fuss around the template engine, as well as the fact that we're using Freemarker, behind this very simple TemplateService interface:

String mergeTemplateIntoString( String templateName, Map model );

This service is a component, a Spring bean, in our application. It would be great if this service could be used to render the logged string from the document and the HTTP method, but in that case we have to supply the newly instantiated exception with a reference from the context. Sure, this could be accomplished by holding an (otherwise useless) reference in the DocumentSource implementation that is passed along the exception, but I prefer it when things Just Work.

Enter @Configurable and compile-time AspectJ weaving! We slap on an a TemplateService setter and an annotation (with autowire=Autowire.BY_NAME to avoid the need for a boilerplate bean definition) on the exception class. We're using AspectJ aspects for various other tasks already, and weaving is done at compile-time to avoid the proxy problem. The Codehaus Maven plugin works great!

In order to weave the Spring AnnotationBeanConfigurerAspect, we add the following to pom.xml:

<aspectLibraries>
 <aspectLibrary>
  <groupId>org.springframework</groupId>
  <artifactId>spring-aspects</artifactId>
 </aspectLibrary>
</aspectLibraries>

And the aspect must be made aware of the Spring context (in any application context file):

<aop:spring-configured />

So from here on, the exception message may be rendered like this:

@Override
public String getMessage(  ) {
  return templateService.mergeTemplateIntoString( "error", model );
}

where "error" is the name of a template, and the model contains the XML document and some other stuff. The error message is now ready to be read and logged, for example in an @AfterThrowing aspect, but that's another story.

Integrating Struts 1 and Spring

If you want to manage Struts 1 actions with Spring, for example to inject dependencies at runtime, you are going to need a bridging context that essentially duplicates every single action as a bean. (details are here). If you're working on an existing application with lots of actions, writing the bridge context XML will be a pain, so here's a quick Groovy hack that I came up with to generate the Spring context file from the Struts config file:

import groovy.xml.MarkupBuilder

def strutsConfig = new File("path/to/struts-config.xml")

def fromXml = new XmlParser().parse(strutsConfig)

def writer = new StringWriter()
def toXml = new MarkupBuilder(writer)

toXml.beans(
      xmlns:"http://www.springframework.org/schema/beans",
      "xmlns:xsi":"http://www.w3.org/2001/XMLSchema-instance",
   "xsi:schemaLocation":"http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd",
  "default-autowire":"byName") {
 fromXml."action-mappings".action.each {
      bean(name:it["@path"],class:it["@type"])
     }
 }
}

// This simply dumps the XML to stdout, you could of course write to an actual file as well
println writer

This little snippet creates a <bean name="/foo" class="bar.Baz"/> for every <action path="/foo" type="bar.Baz"/> element in the struts-config.xml file, using a few neat Groovy tricks (GPath, closures, GroovyMarkup).

Still, you could do better. You could use a hook such as BeanFactoryPostProcessor that parses the Struts config and dynamically creates beans corresponding to the actions, or even write an XSD that allows Spring to interpret the Struts config file as a context file directly. There's an entry in the SpringIDE blog that shows how you could write a namespace that works in German, so for example <bean/> becomes <bohne/> and so on. That principle might be applicable to this context too.

Bandwidth saver

I never miss an opportunity to push for Freemarker, the most powerful template engine in the world, so here's a quick tip on how to strip all whitespace from your HTML pages:

<#assign out>
<#compress>
!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML>
<HEAD>

  // Your page here

</BODY>
</HTML>
</#compress>
</#assign>
${out?replace('\n','')}

It assigns the compressed body of the outer assign macro to the variable "out", and the finally writes the value of "out" with all newlines stripped.

The entire page is now a singe line of HTML - a developer's nightmare, but a nice a bandwidth saver.

Combined with Sitemesh, another one of my favorite tools, this is easily and transparently applied to all your pages.

A quick peek at SpringIDE 2.0

There are a number of cool new features coming in SpringIDE 2.0, currently at milestone 3. One example is the new SpringExplorer, which is a comprehensive tree view of a Spring context, another is content assist for custom namespaces.

But the by far neatest new feature is the ability to evaluate AspectJ expressions in XML on the fly, and displaying what beans will be advised! Take a close look at this screenshot...an <aop:advisor> element is highlighted on the left, with an AspectJ pointcut expression. On the right, you can see what methods will be advised (the "advises" arrow). A godsend!

One more thing...

I forgot one thing about Test-NG...they switched the order of "expected value"/"actual value" in the assert statements! So after finally having learned the hard way that the "expected" value should be first (JUnit 3), I now have to revert to (the more intuitive but less standard way of) putting the "actual" value first :-)

Today I'm struggling with an issue related to this: http://forum.springframework.org/archive/index.php/t-36086.html

Basically, an @Around expression with "&& args(foo)" works fine when the Eclipse compiler is being used, but not when Maven compiles the sources with Sun Javac 1.6, no matter how much debug info I turn on :-/

A week with Test-NG

On my latest assignment (I work as a consultant), we've decided to let go of the old workhorse that is JUnit 3, and go with one of the new, fancy @Nnotations-driven frameworks JUnit 4 or Test-NG. After a short evaluation period, which consisted of verifying basic Eclipse and Maven support and skimming a few tutorials, the feature sheet of Test-NG looked a bit stronger. The two major selling points of Test-NG over JUnit 4 were the ability to group tests and that Test-NG can run only the tests that failed the previous round.

However, the first week of day-to-day usage of Test-NG has been less than stellar, mostly (but not limited to) the poor IDE integration. My list of pros and cons versus the trusted old JUnit 3, when used inside Eclipse:

Pros:

• The test class is only instantiated once, not once for each method in the test class. As it should be.
• Better setup granularity with both @BeforeClass and @BeforeMethod entry points
• @DataProvider is a nice feature, especially in conjunction with Selenium.

And the cons:

• No built-in macro expansion in Eclipse that creates a test method - in a class that extends the JUnit 3 class TestCase, just type "test" and press Ctrl-Enter, and you have a method skeleton. Removing the need to inherit means more work in this case.
• Keyboard shortcut Ctrl-Alt-X N does not work, so I have to use the mouse every time I want to run a test.
• Can't run more than one test class at once, using multi-select in the file tree browser.
• Can't run an entire package hierarchy of test classes in one shot.
• Can't run a single test method by right-clicking a method name in the "green bar" view.
• Using JMock is a lot more work, either by re-inventing half of MockObjectTestCase manually, or by using plain old inheritance.
• No automatic static imports (a weakness of Eclipse's, to be fair), which forces me to write Assert.assertEquals() instead of just assertEquals() or write the static import manually.

So, a lot of annoyances on the ground level, and we haven't really started to see the benefits of large-scale features such as grouping and run-failed-tests-only. The verdict so far is that Test-NG is as good as or worse than JUnit 3, when counting in IDE integration.

And really, what is so fantastic about not having to inherit TestCase and naming your methods testXXX? Here's the number of times that has been a limiting factor in my work as a Java developer: 0.

Also, annotations are good for a lot of things (@Entity and @Transactional are great), but sometimes you get the feeling that various framework authors are going "OK, we now have annotations in Java, how can we possibly shoe-horn that into JMyFramework4J 2.0?"

TSSJS 2007, full time

TSSJS 2007 is over, or at least it will be in about ten minutes. I left the last session a bit early since it focused a bit too much on SOA(P) and too little on raw XML processing, which was my area of interest. It's a real shame that there's no official after-party with a chance to mingle with the Java in-crowd, everybody just seem to go home. We are staying until Sunday morning, so Friday evening and Saturday will be a mini-vacation.


The second half has been mixed, as was the first one. This morning's keynote was an endless stream of three-letter acronyms and buzzwords from a top Oracle executive, Tomas Kurian. He outlined how Oracle see the future of computing, which was quite painful to sit through. At least until an interesting announcement was made: Oracle buys Tangosol Coherence. Cameron outshone the Oracle guys with his demo, which while it wasn't very substantial, it did actually work. The Demo Devil apparently spares no one, not even senior Oracle engineers.

But the absolute highlight of the entire conference was the "Java performance myths - how do JVMs really work" session with Brian Goetz. He crammed in probably around 150% more words that the average speaker (he talks fast), and it was extremely valuable information. He went through a number of long-loived Java performance myths (object allocation is slow, garbage collection is slow, synchronizarion is slow etc) and explained why they are no longer true. His advice can be summarized into the following sentence:

The JVM is always smarter than you.

Java isn't an interpreted language, it's dynamically compiled, which enables the JVM (or the JIT compiler) to gather statistical data, at runtime, on how code is being executed to aid in optimization of the code. Also, the JVM is tuned to recognize commonly found patterns of execution and optimize them as best it can. This means that the more well-designed, clean code you write, the bigger the chance that the JVM (JIT compiler) will recognize code patterns and be able to do a good job at optimizing the code. Since the compilation into native code happens at runtime, and can even vary over time (code can be re-analyzed and re-compiled dynamically), it's very difficult to try and predict how the source code you write actually will be compiled and executed. This also has the side-effect that writing isolated benchmarks is often useless and/or misleading, since they don't properly reflect real-world usage. Just write as clean, readable and maintainable code as you can, and safely assume that the JVM will do a better job than you at optimization.

One interesting aspect of synchronization improvements in recent JVMs (Sun Java 6, I believe) is that there is no longer any performance difference between using StringBuffer (syncronized) and StringBuilder (not synchronized), since the JVM automatically detects that synchronization isn't needed, and the code executed will be equivalent. I can actually back this up with test data I gathered around the time I wrote about how the Java bytecode compiler automatically converted String concatenation using the + operator to StringBuffer/StringBuilder (depending on which version of javac used). I found no speed difference whatsoever between the two, which profiling a webapp during load-test with several hundred simultaneous threads, which at the time made me quite confused, but now I have an explanation. So the old truth "Concatenating Strings is slow, use StringBuffer" and the not so old truth "StringBuffer is slow, use StringBuilder" are now both false*, and can instead be reduced to "it doesn't matter at all". Since performance is equal, readability wins - use regular concatenation (+).

Of course, in most cases the application bottlenecks aren't in your code at all, but in I/O operations against databases etc.

I wrote a small benchmark (even though I just said that benchmarks are useless) that you can check out and play around with.


* if the JVM comes to the conclusion that StringBuffer synchronization isn't needed, which is often the case.

TSSJS 2007, half time

I'm writing this as I'm waiting for the the presentation "Agile Development Metrics" by Neal Ford to begin. I attended a presentation on Selenium that he did yesterday, and that was one of the better ones so far. The overall quality of the sessions has been mixed, ranging from fairly shallow (Rod on "Spring 2.0 and beyond") or way too basic (Mark Richards on "Advanced (bah!) JPA", Adrian Colyer on "Simplifying Enterprise development with AOP") to very good. Hani biled the JPA session, and my feelings were exactly the same. "Hey, if you add fetch=FetchType.EAGER to your annotation, the relation is no longer lazy!". Duh. Also, showing how to make a POJO transactional using AOP isn't exactly jaw-dropping in 2007.

What is jaw-dropping however is the scale of operations run by investment banks, that John Davis talked about this morning. Java is widely used in the banking world, and outperforms C/C++ a lot of the time, but a 50 ms garbage collection run is unacceptable since it would mean missing business opportunities worth millions of dollars. The solution: hardware garbage collection. He also explained that the speed of light was a limiting factor for performance. Yes, that's right. The speed of light. A theoretical best-case roundtrip for light to travel between New York to Chicago is around 8 ms, which is enough to motivate moving the physical system from NY to Chicago if you need to talk to another system in Chicago, for example. XML isn't widely adopted either, for performance reasons. If it's used at all, you often have a situation with a single element with 4000 attributes, to minimize the number of angle brackets to parse. And when it comes to reliable messaging, no JMS vendor can match the SWIFT system, used by 8000 banks world-wide to transfer money between them. They guarantee 100% reliability, to the point where they will refund any amount submitted to the system that is lost. However, in the 30 years that they've been operating, they haven't yet lost a single message. They average around 12 million messages a day. The particular investment bank that John talked about handled around $500,000,000,000 (five hundred thousand million dollars, or half a trillion) a day, so that's why even milliseconds matter. Quite a few "whoooa!" moments during the speech :-).

The second half of TSSJS contains a few sessions that I hope will be interesting, mainly around clustering and JVM and XML performance and scalability, which will be relevant in my current work assignment right away. There's one presentation held by a Swedish developer, Jonas Bonér of Terracotta fame which I'll be watching. Terracotta has gotten quite a few mentions in other presentations, and it seems really useful in some situations.

Next up is Cameron Purdy from Tangosol. I'll be back with a review of the second half on Saturday, I think.

Viva Las Vegas

Oh how hard it is to come up with headlines for non-technical posts...anyway, I'm going to The ServerSide Java Symposium in Las Vegas tomorrow, which I'm sure will be a fantastic experience. My room at The Venetian is almost as big as my apartement, and both the weather and the schedule look very promising indeed.

So, if you're reading this and are going to TSSJS, say hi if you recognize me (I have a little bit shorter hair than on the photograph right now). My guess is, however, that the only two people in the world that fit both categories are on the same plane as I am...

SimpleFormController demystified - part 2

In the first part we covered the basics about the available controller approaches in Spring MVC, and the simplest possible scenario for using SimpleFormController - creating a new User.

The next step is to adapt out UserController to handle editing existing users as well, which is natural to do in the same view. What we need to do, is of course to load a specified user from our storage and prepopulate the input fields with the current values, and then hand a User object with the new/changed values back to our storage when we submit the form.

SFC uses a command object to bind parameters, just like the ActionForm in Struts. The difference is that in Spring MVC, the command object can be any class, which makes it possible to re-use the domain model classes instead of duplicating all properties in a class that extends ActionForm. As we noted in the first part, SFC places the command object in the model under the default name "command", so that you can access properties using ${command.firstName} etc in the view. If you want to use another name, call setCommandName() in the constructor or set the property in XML.

The default behaviour of SFC is to simply instantiate the command class to get the command object . What we want to do however, is to check if a certain parameter is set (such as "id"), and if it is, load the corresponding instance from storage and use as command object. The proper callback to override in this case is formBackingObject():

@Override
protected Object formBackingObject(HttpServletRequest request) throws Exception {
Long id = ServletRequestUtils.getLongParameter(request, "id");
if (id != null) {
  User user = userDao.load(id);
  if (user != null) {
      return user;
  }
}
return new User();
}

If the "id" parameter is present, and if we can find a persistent entity with that id, we use that as our command object, otherwise simply a new User. The input fields are prepopulated like this:

<input name="firstName" value="${command.firstName}"/>

As far as showing the form, it is all very straightforward. When it's time to submit the form, we need to take a step back and think about when formBackingObject() is called. It is obviously called when we first show the form, that is when we issue a GET request for /user/edit.html?id=1, when the User with id == 1 is loaded and put into to model. But once that requests ends, the User instance is lost, and we need an instance to bind the POST parameters to as well.

There are two ways to solve this: either store the command object, our loaded User, in the HttpSession, or repeat the same process as when showing the form. SimpleFormController has built-in support to store the command object in the session, just call setSessionFrom(true). Default is false, meaning we don't store the command object in the session, but instead call formBackingObject again on post. In order for that to work, you need to supply the id parameter again in the form submit, either as a hidden input:

<input type="hidden" value="${command.id}"/>

or as part of the form action url:

<form action="/user/store.html?id=${command.id}" method="POST">

Otherwise formBackingObject will instantiate a new User and bind parameters to that, most likely resulting in a new User being created by the storage instead of updating the existing one.

Note that the ServletRequestUtils.getLongParameter() method unfortunately handles "?id=" as an invalid Long value instead of null (throwing ServletBindingException), so don't build the id input at all if the User does not have id set:

#if ($command.id)
<input type="hidden" value="${command.id}"/>
#end

Storing the command object in the session has pros and cons: one benefit is that it requires one less call to the storage to fetch the command object, on the other hand putting lots of things in the session consumes memory on the server and increases the load on session replication, if you're running a cluster. As far as simultaneous editing goes, it doesn't really make much difference, since you will overwrite using your input value on submit either way (unless you put a row lock on the database and don't commit the transaction until after posting, but that's beyond the scope of this article).

I will post the source code for a very simple project implementing this article series shortly, watch this space. The next part will be about validation and more sophisticated forms.

SimpleFormController demystified - part 1

Spring MVC is a very flexible powerful Model 2 framework, but as with all flexible frameworks, it comes with a price - confusion. In order to write a controller you have the freedom/burden to choose between implementing one of two single-method interfaces, or inherit from one of the 13 (!) skeleton implementations available. Most of the time you're probably going to want one of the extremes: as much help or as little help as possible. The former scenario is typically things like a LogoutController, where you just invalidate the session and redirect to the default start page or whatever. Good candidates for this is either to implement the Controller or the ThrowawayController interface, or inherit from AbstractController, which adds a few cache header setters and some other stuff. Using the AbstractController approach, it would look like this:

public class LogoutController extends AbstractController {

public ModelAndView handleRequestInternal(HttpServletRequest request, HttpServletResponse response) {
request.getSession().invalidate();
return new ModelAndView("redirect:/start.html");
}

}

Another candidate is when you want to delete a blog post, in which case you're only interested in a single parameter:

public class DeletePostController implements ThrowawayController {

Long id;
BlogDao blogDao;

public Long getId() { return id; }

public void setId(Long id) { this.id = id; }

public ModelAndView execute() {
blogDao.deletePost(getId());
return new ModelAndView("redirect:/posts/list.html");
}

}

The ThrowawayController is instantiated on every request, and binds parameters directly onto the controller class instead of a separate model bean, similar to the WebWork/Struts2 approach.

So in general, if you're only interested in zero or one parameter, and don't care about validation or anything, the as-simple-as-possible approach is probably right.

But this article series is supposed to be about the controller base class that gives the maximum amount of support, SimpleFormController (SFC). Some people think that's an oxymoron, but I'm going to try and show the easy ways to accomplish many common tasks using SFC.

SFC handles two actions: preparing a form view and showing it, and taking care of the submission and showing a resulting view. By default, the request method determines what kind of action to take - GET means prepare and show the form, POST means submission (it is possible to override isFormSubmission(), but it's very rarely necessary and we can disregard that for now). That is why there are two views to configure.

Suppose you have the concept of a User in your domain, and you need a way to create and edit these users in a web form. There is already a domain class named User, that's persisted in some way, and it has two properties: firstName and lastName (we'll add more later on). In order to build a controller that is capable of storing a new User, you need to do the following:

public class UserController extends SimpleFormController {

UserDao userDao;

public UserController(UserDao userDao) {
this.userDao = userDao;
setCommandClass(User.class);
}

protected void doSubmitAction(Object command) {
User user = (User) command;
userDao.store(user);
}

}

And some configuration in yourApp-servlet.xml (view parameters can be set in the constructor too, if you prefer that, and you can move the command class configuration out to XML too):

<bean name="/user/create.html" class="com.example.UserController">
<constructor-arg ref="userDao">
<property name="formView" value="user/editForm">
<property name="successViewView" value="user/created">
</bean>

This is really all there is to it. The form preparation phase is simply an instantiation of the command class, in our case a User, and rendering of the "user/editForm" view. When the form is submitted, all parameters are bound to a User instance and handed to us in the doSubmitAction() callback method where we simply store it. The superclass will then render the success view.

The default handler mapping (what controller that handles what urls) is the BeanNameUrlHandlerMapping, which uses the name attribute for registering controller beans to urls. It accepts wildcards and space-separated lists: name="/foo /bar", name="/user/*" etc.

Don't forget to set the form method to POST in the HTML page, otherwise the controller will think that you want a new form every time you submit the form. Note that the action url will be /user/create.html as well, unless you map both /user/create.html and /user/store.html to the same controller.

The input names should match the properties on the command object:

<input type="text" name="firstName"/>

The command object itself is available in the view under the default name "command", so you can access properties like this: ${command.lastName}. That will be more interesting in part 2, when we'll edit existing users.

StringBuilder vs. StringBuffer and the Java compiler

Since Java 5, a non-synchronized sibling to the popular StringBuffer class exists - StringBuilder. It's API compatible, and migration can be done with search-and-replace. StringBuilder is not thread-safe, but that's rarely a problem. A popular myth among Java developers is that this:

String comma = ", ";
String slow = "Hello" + comma + " world!";

is slower than this:

String comma = ", ";
String fast = new StringBuffer().append("Hello").append(comma).append("world!").toString();

The fact is that the Java compiler automatically converts the first line to the equivalent of the second, resulting in the same bytecode. That is, until Java 5, when the compiler instead selects StringBuilder.

The twist here is that if you're using Java 5 and a library that's very heavy on string handling (contcatenation) which is compatible with Java < 5, it might be a good idea to recompile it with the -target 1.5 parameter to convert all string concatenations to use StringBuilder. Of course, you should also replace any explicit usage of StringBuffer with StringBuilder.

I figured this out while profiling an application using Freemarker, which has a standard distribution that's compatible with Java 1.2.

MySQL mini-FAQ

I've spent some time reading about MySQL and the JDBC driver, and here are a few long-standing questions that I found the answer to:

1. What is the difference between org.gjt.mm.mysql.Driver and com.mysql.jdbc.Driver?
   
   - They are the same, the former extends the latter with nothing but a no-args constructor for backwards compatability.
   
   
2. What is the difference between MySQL Connector/J 3.1.x and 5.0.x?
   
   - The 5.0.x version has support for XA (distributed) transactions.
   
   
3. Where can I find a reference guide for configuring the JDBC driver?
   
   - Here.
   

Quick BeanWrapper tip

If you ever find yourself in the situation where you want to replace a single collaborator property in an otherwise fully wired environment, and all your beans are proxied by AOP, here's a handy tip.

Suppose you're testing your service component Foo, which has a dependency on service component Bar, both interfaces and transactionally proxied. Foo is wired to the actual DAOs and other service collaborators and whatnot, including Bar, but for this test purpose you'd like to replace the Bar dependency with an empty stub that doesn't do anything, or returns a fixed value or whatever. The problem is that you don't have any setBar(Bar bar) property on your Foo interface (you don't, do you? It belongs on the implementation!), so you have to employ some sort of reflection trick. After having wrapped with BeanWrapper, listed PropertyDescriptors, wrapped again, listed more PropertyDescriptors and so on, I finally found a way to modify the property on the actual Foo implementation found behind the transactional proxy:

// Here's my stub 
Bar stubBar = new Bar() {
  public void interactWithComplexStuff() {
    // Just skip this
  }
}
// Wire the new Bar
new BeanWrapperImpl(this.foo).setPropertyValue("targetSource.target.bar", stubBar);

As you can see, the property to set on the proxy is "targetSource.target.<name of actual property>".

Assorted hacks

I've been coding batch services the last couple of days, and found a few useful tricks and tools for testing that I would to share with you, the reader :-).

The task is basically to extract a number of filenames from an XML file (part of a complete processing of the file), and to download those files from a web server and store locally. We're already using XmlBeans to map the XML to JavaBeans, so I can stub the elements that I'm interested in for a particular test. XmlBeans generates interfaces for every node, so I let Eclipse create default implementations of the interfaces I need, which I then override inline like this:

Foo foo = new Foo() {
  @Override
  public Bar getBar() {
    return new Bar() {
      @Override
      public String getStringValue() {
        return "test-bar-value";
      }
    }
  }
  ...
}

to represent (stub out) an XML fragment on this form:

<foo>
  <bar>test-bar-value</bar>
  ...
</foo>

I've decided that I want to write my test against an actual HTTP server, since mocking that part is more work and less return on investment. Now, I don't want to run the test against the production server, and I don't want to rely on the fact that a dedicated Apache has been started either. The solution is to run an HTTP server as part of the test!

There are a number of different Java implementations of HTTP servers, ranging from microscopic to full-blown. I settled for Jetty, which is small, fast, has a nice programmatic API and you can rely on it to handle request-headers correctly, which might not be the case for YetAnotherSuperSmallHttpServer.sourceforge.net.

Anyway, a minimal HTTP file server can be accomplished with the following code:

Server server;

public void setUp() {
  server = new Server(55555);
  ResourceHandler resourceHandler =new ResourceHandler();
  resourceHandler.setResourceBase(
    new ClassPathResource("webroot").getFile().getAbsolutePath());
  server.addHandler(resourceHandler);
  server.start();
}

public void tearDown() {
  if (server != null) {
    server.stop();
  }
}

The resource base is set to the directory "webroot" in the class path (ClassPathResource is a Spring class), which resolves to src/test/resources/webroot when you're using the Maven standard directory layout (like I do). The server runs on localhost on port 55555 for the duration of the test.

Make sure that you have execution rights on all directories from the root down to the actual directory you serve. Many Linux distributions (Fedora, Debian) sets 700 on the home directories. If you want to browse the server, add a simple System.in.read(); after server.start(). At least in Eclipse you can press any key when the console window is active to continue, I'm sure the other two have similar capabilities.

The actual downloading is done using Jakarta HttpClient. I don't have anything particular to say about that, it has a nice API and seems very solid and broadly used.

One important aspect of the task at hand is to not download images unless they are newer than the copy that we already have, so I'm using the "If-Modified-Since" header. Finding the correct date format was harder than I expected (the RFC specs don't count, who reads 'em anyway). Here's a snippet that seems to do the trick:

private static final SimpleDateFormat httpDateFormat = new SimpleDateFormat("EEE, dd-MMM-yy HH:mm:ss zzz", Locale.US);
static {
  httpDateFormat.setTimeZone(TimeZone.getTimeZone("GMT"));
}

Finally, a rant and a warning about java.util.Calendar: if you populate your database with a timestamp "2007-01-02 12:30:00", here's how you do an assertEquals() in a test:

Calendar calendar = Calendar.getInstance();
// Yes, only month numbering starts with 0
calendar.set(2007, 0, 2, 12, 30, 0);
// Yes, there's no set() that includes milliseconds
calendar.set(Calendar.MILLISECOND, 0);
// Yes, we have no bananas
assertEquals(calendar, myLoadedCalendarValue);

This last thing is nothing new, but it deserves a daily beating.

Making transitive ASM dependencies work with Spring, Hibernate and AspectJ

I've recently had the opportunity to work with the new AspectJ integration in Spring 2.0, and it's been, for the most part, a pleasure. A very brief three-step tutorial could look something like this:

1. Create a POJO that holds all your pointcuts, annotate the class with @Aspect and start defining your pointcuts:
   

   
   package se.petrix.stuff.aop;
   
   @Aspect
   public class MyPointcuts {
   
   @Before("execution (* * se.petrix.stuff.dao.*.*(..))")
   public void beforeAnyDaoMethod() {}
   
   }
   

   
   A pointcut answers the question of where the advice should be applied. In this case, the pointcut is defined in the AspectJ expression language and means "before the execution of any method, in any class in the package se.petrix.stuff.dao that takes any argument". It's the annotation that does all the work here, the actual method doesn't do anything, as you can see. It should however be namned in a self-explanatory way, since we will be referring to it later on.
   
   
2. Write another class, which will become the actual advice. This class is also annotated with @Aspect:
   

   
   package se.petrix.stuff.aop;
   
   @Aspect
   public class DaoCallCounterAspect {
   
   private int callCount = 0;
   
   @Pointcut("se.petrix.stuff.aop.MyPointcuts.beforeAnyDaoMethod()")
   public void countDAOCall() {
   logger.info("DAO call count is now: " + (callCount++));
   }
   
   public int getDaoCallCount() { return callCount; }
   
   }
   

   
   This is now a before-advice, and refers to the previously created pointcut. All it does is count the number of calls the the DAO layer, which is yet another pointless example but at least it's not just logging. There's also a way to read the current call count.
   
   
3. XML configuration!
   

   
   <bean class="se.petrix.stuff.aop.DaoCallCounterAspect"/>
   
   <aop:aspectj-autoproxy/>
   

   
   Not too bad, eh? You need the aop namespace configured of course. All weaving is automatic, based on the pointcut expressions.
   

It's completely straightforward the aspect class itself into another bean, for example you might want a controller to be able to read the DAO call count and display to a web page. And vice versa, you can wire other beans into the aspect.

It is of course possible to place pointcut definitions in the same class as the advice, and/or define pointcuts in XML and so on. This is just one way to organize it all. Read this chapter for more information, and take a look at the original AspectJ documentation.

Now to the main subject of this post: the transitive Maven jar dependency conflicts, or rather version mismatches, between Spring, Hibernate, CGLIB and ASM. We're using Spring 2.0 and Hibernate 3.2.0.ga (that's "general availability", not "Gavin Approved" contrary to popular belief), and AspectJ 1.5.3 (weaver and rt jars). This pulls in ASM 1.5.3 and CGLIB 2.1_3 through transitive dependencies defined in the Maven poms for those packages. But this will result in ASM classes not being found (something about node visitor, don't remember exactly), or if you're not careful when overriding dependencies, method signature mismatch (method not found). I don't have the details on which library requires which version, or the exact error messages, at the time of this writing, but I aim to provide a working solution (well, it works for us anyway). Maybe if I have the time, I'll do a more careful dependency analysis.

Anyway, here it goes: first of all, exclude the CGLIB dep from Hibernate, and replace it with the "cglib-nodep" version which includes the ASM files needed:

<dependency>
  <groupId>cglib</groupId>
  <artifactId>cglib-nodep</artifactId>
  <version>2.1_3</version>
</dependency

and

<dependency>
  <groupId>org.hibernate</groupId>
  <artifactId>hibernate</artifactId>
  <version>3.2.0.ga</version>
  <exclusions>
    <exclusion>
      <groupId>cglib</groupId>
      <artifactId>cglib</artifactId>
    </exclusion>
  </exclusions>
</dependency>

Then we specify ASM version 2.2.3 manually:

<dependency>
  <groupId>asm</groupId>
  <artifactId>asm</artifactId>
  <version>2.2.3</version>
</dependency>
<dependency>
  <groupId>asm</groupId>
  <artifactId>asm-all</artifactId>
  <version>2.2.3</version>
</dependency>
<dependency>
  <groupId>asm</groupId>
  <artifactId>asm-attrs</artifactId>
  <version>2.2.3</version>
</dependency>

The reason why we have an overlap with asm, asm-all and asm-attrs is that they are pulled in by another library (not sure which one), so we have to override the version to avoid ending up with different versions of these jars. Another way would probably be to exclude them and only have asm-all 2.2.3. As I said, I don't have the details, but we've struggled quite a bit to get this working and I thought it might useful to provide at working example, even without explaining exactly why it works :-).