Friday, April 17, 2009

Running Spring on Google App Engine

In case you've been living under a rock the last couple of weeks, Google recently announced the addition of Java support to its App Engine. I have written a small sample application that leverages Spring and each of the Google infrastructure services that are exposed either as proprietary APIs or serves as backend to standard APIs.

The application itself, Feeling Lucky Pictures, is very simple: you login using your Google account, and you import images from URLs into your personal gallery. You can also send an email with your pictures attatched.

Integration with the Google infrastructure is as follows:
  • Authentication is of course done with the Google Accounts API.

  • Image import uses a regular java.net.URL input stream, but on the GAE runtime that class is backed by the URL Fetch API.

  • Storage is implemented using JDO, which is backed by a BigTable data store.

  • Reading objects from the data store uses the JSR 107 javax.cache API, which is backed by Memcache.

  • Imported images are enhanced with the I'm feeling lucky filter, part of the Images API.

  • Email is sent with the standard JavaMail API, which also is backed by a custom mail service on GAE.
As I said, the application is built on Spring as I wanted to see what problems, if any, I would run into if I were to run a regular Spring application on GAE. Most things worked out fine, but there were a handful of issues that I discovered and resolved, which is the real value of this application.
  • I decided to make as heavy use of the annotation configuration option as possible, including automatic classpath scanning. However, Spring will then attempt to scan for JPA annotations if certain conditions are met, which in turn will cause the javax.naming.NamingException class to load.

    This class is not on the GAE class whitelist, and working around the missing class by providing a JNDI API jar or a dummy class won't work either. My solution was to roll a JDO-only version of the spring-orm jar. This problem is also discussed in this thread, with alternative solutions.

  • When configuring the LocalPersistenceManagerFactoryBean, don't specify a configuration file property (i.e. "classpath:META-INF/jdoconfig.xml"). Instead, set the persistenceManagerFactoryName property to "transactions-optional", which is the name used in the default configuration file provided by the Eclipse GAE plugin.

  • I wanted to avoid using the various Google API static factories programmatically in my MVC controllers, instead injecting service interfaces like ImageService to improve testability. This worked fine for the most part, simply using the factory-method attribute in the bean definition, with the exception of the Cache interface.

    Cache extends java.util.Map, and for some reason the @Autowire mechanism requires generic key and value types on constructor parameters of Map type. Unfortunately the Cache interface is not possible to parameterize, so my workaround was to embed it in a very simple CacheHolder one-property class, which is produced by a factory bean and injected.

  • I'm caching picture ids per email address, like this: "foo@bar.com":[1,5,7]. In a regular in-memory HashMap cache you can add elements to a collection map value, but on GAE you need to overwrite the old entry with a new collection that contains the old elements plus the new one. See JdoRepository for details.

  • Spring provides a set of abstractions on top of JavaMail, which I wanted to keep. In particular, the JavaMailSender interface and MimeMessageHelper class, for sending emails with attachments. This turned out to be a bit cumbersome, since the GAE mail backend is wired into the implementation of the JavaMail API in an unusual way. There's no SMTP transport, for example.

    What I had to do was to override Spring's implementation of JavaMailSender and align creation and usage of javax.mail.Session and javax.mail.Transport exactly with the howto instructions for the GAE mail service. Basically you can't let the Transport connect and then send the message on the established connection, you need to do it all in one pass using the static Transport.send() method.

    Oh, and don't leave the body of an email empty, or you'll get a really poor error message.

Other than that it's business as usual. The complete source code is available on Google Code, and it's MIT licensed so you can do whatever you want with it. If it helped you out, or if you've found a better or simpler solution to any of the problems above, please drop a comment on this article.

Friday, March 27, 2009

DDDSample 1.1.0 released

What the title says :-)

It's been 6 months since 1.0, and quite a lot has happened. If you're interested in domain-driven design, take a look at it and let us know what you think, either on the international Yahoo DDD group or the Swedish Google group.

If you're interested in learning more about DDD, Citerus has a variety of offers for you and your team.

Monday, March 23, 2009

Five tips for successfully deploying Maven

Maven is one of those things that people seem to hate rather intensely, but nevertheless adoption is steadily rising in the Java community. I've worked with Maven almost daily since the 1.0 betas, and here are five things that I think could help your team working more efficiently with Maven.

  1. Use a repository manager

    A repository manager is basically an on-demand mirroring repository cache that you set up inside your IT infrastructure and use as primary repository for your builds. They basically work like this: if you build a project that depends on, for example, commons-lang-2.4.jar, the repository manager will download the artifact from the main Maven repository on the web, cache it locally and return it to the build client that asked for it. All subsequent builds that use the same managed repository will get the commons-lang jar delivered from the cache, not from the web.

    This has many advantages. First of all, it's fast. All project members, except the first one, will download any given dependency at LAN speed, which is especially nice when you're setting up a build environment from scratch (new project member, staging a clean build, etc). And of course it saves external bandwidth for other purposes and to lower costs.

    Second, it's safer. It allows you to run centralized and incremental backups on all external dependencies that you projects use, and you reduce your dependency on the availability of public repositories.

    Third, it's convenient. From time to time you will need a library that's not (yet) available in any public repository, so you have to publish it somewhere. A repository manager makes that really easy. And if you're sharing internal libraries or interfaces between projects, it's extremely handy to deploy to the managed repository. You can even set up your continuous integration build to automatically deploy snapshots.

    I've had a pleasant experience working with Nexus, but there are others. A repository manager should be as natural a part of you infrastructure as SCM and CI if you're using Maven.

  2. Specify plugin versions

    By default, Maven will automatically download a new version of any given plugin whenever there is one available. Given that Maven is 99% made up of plugins (there's even a plugin plugin!), this is a potential point of breakage over time and in my opinion a design mistake.

    As of version 2.0.9, the default behaviour is improved by locking down the versions of the core plugins (where "core" is defined by this list). However, you still need to explicitly define versions for all non-core plugins, and that can be done at the top level pom.xml in a hierarchial project using the pluginManagement section.

    <pluginManagement>
    <plugins>
    <plugin>
    <artifactid>maven-assembly-plugin</artifactid>
    <version>2.2-beta-2</version>
    </plugin>
    <plugin>
    <artifactid>maven-antrun-plugin</artifactid>
    <version>1.2</version>
    </plugin>
    </plugins>
    </pluginManagement>

    Do this for the plugins that you actually use. Note that for plugins with group id org.apache.maven.plugin, you can omit the groupId element.

    This will make your builds more stable and eliminate a fairly rare but very annoying and confusing set of problems.

  3. Learn how to use the dependency plugin

    Maven introduced the concept of transitive depedencies to the Java community, and has been a source of confusion ever since. The dependency plugin is an invaluable tool for analyzing the results of the dependency algorithm, and to handle dependencies in various ways. Here are a couple of things you can do with it:

    • dependency:tree
      shows (you guessed it) the dependency tree for the project, what dependencies are being pulled in and why. It's a nice overview and can help you tweak the dependency structure by excluding artifacts or override versions and so on. Example output:

      [INFO] +- org.apache.activemq:activemq-core:jar:5.2.0:compile
      [INFO] | +- org.apache.camel:camel-core:jar:1.5.0:compile
      [INFO] | +- org.apache.geronimo.specs:geronimo-jms_1.1_spec:jar:1.1.1:compile
      [INFO] | +- org.apache.activemq:activeio-core:jar:3.1.0:compile
      [INFO] | | \- backport-util-concurrent:backport-util-concurrent:jar:2.1:compile
      [INFO] | \- org.apache.geronimo.specs:geronimo-j2ee-management_1.0_spec:jar:1.0:compile
    • dependency:go-offline
      Downloads all project dependencies and plugins, transitively. It's a good command to run both if you want to work offline for a while and if you want to get as many of the external dependencies in place in a single shot with no manual intervention while you go grab a cup of coffee and/or read another item in Effective Java ;-)

    • dependency:copy
      dependency:copy-dependencies
      If you ever need to handle artifacts as files, copying all or some of them to a custom location for whatever reason, this is a good approach.

    There are many more things you can do with it, and mastering it will help you get on top of the transitive dependency situation.

  4. Use the documentation

    Well, duh. But a weak point of Maven in the eyes of many people is the lack of documentation and the sometimes poorly organized information. There are a few good points of reference though, that you can spread around you team by setting up links on the Wiki for example:

    • The Definitive Guide to Maven: a free book from Sonatype, available both as HTML and PDF. Good for the beginner, and sometimes as a reference. If you don't know where to start, start here.

    • The plugin list: a comprehensive list to the official plugins, with links to each project page and JIRA subsection. Most of the core functionality is actually performed by one of these plugins, and you can learn a lot by studying things like the resources plugin documentation.

    • The POM reference: for the slightly more advanced user. Every element in the POM is explained. Don't forget to specify the XSD information in your POM file to get the most help from your XML editor.

  5. Understand the conventions

    Maven is a conventions-based tool, relieving you from scripting common task like compiling source code, running tests or packaging a web application into a war file. Learning the conventions - directory structure, build phases - and working along them will make your life easier a lot of the time.

    There are definitely situations even in moderately sized projects to customize the build however, and Maven can sometimes be quite cumbersome to work with when you need to break the conventions. But by understanding the conventions and having the mindset that there is a good chance what you're trying to do can be accomplished within the realms of the conventions, you might be able to find a different approach than you otherwise might have.

    Perhaps that ugly jar-splitting, file-copying, token-replacing antrun hack that you spent an agonizing week writing could be replaced by extracting part of the project into a separate module and included as a dependency instead? It's a lot easier to swim downstream than upstream.

Maven is not perfect by any means, but it has brought standardization and conventions to the world of Java development. Project structure, directory structure, public metadata and artifact repository publishing to name a few. There are lots of plugins available, both central and third-party ones, and most IDEs and continuous integration servers support Maven very well.

A lot of the standardization may even outlive the Maven tool itself, as demonstrated by two newer build system for Java: Buildr and Gradle. They both use many of the same conventions, and could challenge Maven by perhaps being able to scale down in complexity more easily and have a lower threshold for newcomers. Progress on Maven slowed down a bit after 2.0, but recently the 2.1.0 version was released with a number of important improvments, for example parallel resolution of depenencies.

Friday, January 16, 2009

New PNEHM article

I'm about to publish my next PNEHM article, a step-by-step port of a short algorithm written in Java to Groovy. For my Swedish readers, here's a sneak peak (the article is in Swedish).

UPDATE: the article is now published. Enjoy!

Thursday, December 18, 2008

DDDSample tutorial at QCon London

I'm very proud to announce that I will be giving a tutorial on the DDD sample application at QCon London in March next year, together with my colleague Patrik Fredriksson. The presentation will be basically the same as the one we're giving at the Swedish conference JFokus in January, but in English (obviously :-)).

Hope to see you there! The tutorial will cover the upcoming second release of the application.

Sunday, November 23, 2008

On layering in DDDSample

There has been considerable interest in the DDDSample application on the Swedish DDD user group mailing list - people are scrutinizing the code, asking questions and raising concerns. This last week has been mostly about layers and packages, and I think this blog is a good forum to provide a little background and explain the rationale behind how the sample application is structured.

When we first started working on the application, we used a fairly standard layering with a web user interface layer, a service layer with interfaces, implementations, transaction demarcation and so on, and a repository layer for persistence, implemented in Hibernate (roughly matching the DAO layer found in most applications). These layers all resided in their own package hirearchy. In addition to that, the domain model had its own package, although it wasn't a layer in the usual sense - it was used by several other layers, but calls did not pass through the domain model on to some other layer. This suited us well for the time being, since there were many other aspects of the application that required our immediate attention.

As the application grew more mature, we began looking at how the structure of the application could illustrate important DDD concepts such as aggregates and isolating the domain. In his book, Eric Evans uses this diagram when talking about layered archicture:


By far the most important layer is the domain layer, so we decided to take a closer look at the contents of our domain layer and the domain package. A pretty obvious decision was to place each aggregate in its own subpackage below domain, so we had domain.cargo, domain.handling and so on. Deciding which services were domain services and which were application services was harder, but we settled for a separation where domain services performed tasks that you could talk about with a domain expert, using the ubiquitous language. Signatures consisted completely of domain model types. In some cases, it was natural to place the interface of a domain service in the domain layer, but the implementation elsewhere.

But the decision that would turn out to be the most controversial was placing the repository definitions (i.e. the interfaces) in the domain layer, alongside the aggregate root for which it was used to retrieve, store and search. The concept of an aggregate root is closely linked to that of a repository: all access to an aggregate is through the root, so consequently the repository works with aggregate roots, and there is one repository per aggregate root (and thus per aggregate). Also, repositories are expressed in the ubiquitous language.

At this point our domain layer consisted of the domain model, separated into aggregates, domain services and one repository per aggregate, which we felt pretty good about (and still do). It expressed many important DDD concepts in a clear way, and it was slightly unorthodox compared to many mainstream designs. All this was located under the domain package.

The question of what to do with the rest of the application remained. The other three layers are not as interesting from a DDD perspective, so we decided not to pursue the effort of organizing the rest of the code into per-layer-package hierarchies, but instead separate it from the domain package and organize it by a combination of technology and use cases.

We did however consciously include two very different approaches to user interface exposure. The tracking web interface, which runs in the same JVM as the main application, is the low-overhead-thight-coupling way of doing it (in terms of layers and lines of code), where the MVC controller acts as application layer and calls the domain layer repository directly. The tracked cargo is thinly wrapped in the view rendering phase to make it easier to work with in a JSP EL environment.

The booking web interface on the other hand can run in a different JVM and works against an RMI facade on top of the domain layer, passing custom DTOs back and forth. The point is that we don't generally recommend a mandatory, slavishly delegating application layer between the presentation and the domain layers. Sometimes the controller in the MVC layer can play the part of application just as well. Another important point here is that you should always shield the domain model objects from presentation requirements, and DTOs or a thin presentation wrapper and so on are good options for doing that.

The top level package for all non-domain-layer code was named application, which turned out to be a bad idea (mine) since the name coincided with one the the layers in the picture above. For the record, the rationale behind it was "application" as in "computer program", i.e. everything about the code that wasn't part of the domain. A better name would have been something like nondomain or even other.

This problem immediately became apparent when I presented the application to the New York DDD User Group, so a separate ui package was extracted for the web MVC controllers and supporting code. This actually turned out to make matters slightly worse, since we now had top-level packages with names matching three out of the four layers in the DDD layer model, immediatly leading people to ask for the missing infrastructure package. The discussions on the mailing list and other forums helped us realize that there is a need for an explicit infrastructure package, so the next version of DDDSample will include such a package containing the parts that we consider to be part of the infrastructure layer.

It appears from the discussions on the Swedish user group mailing list that many people think of the infrastructure as being identical to the persistence aspect (the database and the O/R mapper), but we have a wider definition of infrastructure which also includes messaging, scheduling, thread pools, the Spring container, the servlet container and external services such as mail senders and in our case the routing team's graph path finding service.

Looking at the picture above, the arrows between the layers actually illustrate the point quite well: the presentation, service and domain layers all work with the infrastructure layer, but it's important to realize that it doesn't mean that you should execute SQL statements in your JSP pages, but rather that each layer interacts with some part of the infrastructure. Also, the infrastructure layer can be used for passing asynchronous messages between layers.

In general, we consider code and configuration files that we write in order to hook into the infrastructure to be part of the infrastructure layer - Hibernate repository implementations, HBM mapping files, Spring context definition files, the RoutingService implementation and so on. Sometimes the distinction is harder to make, such as having an application service implement the JMS MessageListener interface to act message-driven.

As a rule of thumb, never state rules of thumb when it comes to software development. But if I were to do that anyway, I'd say that the infrastructure layer should be completely separable from the rest of the application by stubbing out external services, implementing persistence in-memory, and using synchronous calls or simple threads for messaging.

Sunday, November 02, 2008

DDD updates

For about a year now, I've been part of a team at Citerus that has worked with Eric Evans to create a sample application showcasing domain-driven design, and we have recently released the first version to the public.

Al least two porting efforts have already been initiated: Qi4J and Sculptor. Some people are blogging and twittering about it, too :-)

The Swedish DDD user group held its first meeting at the Omegapoint office in Stockholm, where Patrik Fredriksson and I presented the application. In August I presented the application to the New York DDD user group, and in January 2009 we will give a three-hour tutorial on basic DDD and the sample application during the JFokus conference.

On tuesday, November 4, Eric Evans and Patrik Fredriksson will host a one-day seminar on DDD in Stockholm.

Friday, April 11, 2008

DSL for time and money

(Well, for time anyway)

A couple of weeks ago, I did some work together with Eric Evans when he came to Uppsala to give his excellent course in domain driven design, which was co-hosted by Citerus and Patrik Fredriksson.

Eric is the project leader of the Time and Money Java library, which makes working with dates, time intervals, currencies and so on a breeze. However, inspired by this article by Guillame Laforge, I wanted to see if I could create something similar by leveraging Groovy and the Time and Money library. These are a few simple examples that I came up with in an hour:


println 1.minute

=> '1 minute'


println 5.minutes + 1.minutes

=> '6 minutes'


println "2003-05-16" + 3.weeks - 50.years

=> 'Sat Jun 06 01:00:00 CET 1953'

Looking at these statements from top to bottom, we first have

1.minute

The number 1 is of course an instance of java.lang.Integer, a full-blown object. On that instance, we access something called minute, which kind of looks like a field, but is actually a JavaBean property thanks to Groovy's built-in support for those. So what we really have is an invocation of Integer.getMinute(), a method that doesn't exist. But don't worry - here's how we can use metaprogramming to add that method to the Integer class:

Integer.metaClass.getProperty = {symbol ->
switch (symbol) {
case ["minute"]:
return Duration.minutes(delegate)
default:
return null
}
}

In Groovy, every class has a corresponding open metaclass, the ExpandoMetaClass, that may be used to dynamically add methods on classes. The method getProperty is invoked when a JavaBean property is accessed, and here we assign a closure to be evaluated on invocation. The closure recieves one argument, symbol, which is a String containing the name of the property accessed, in this case "minute". This particular case is chosen to be converted to a Time and Money datatype, Duration, by passing the delegate (that's the instance we're invoking the getter on, i.e. 1) to the appropriate factory method. The result is that a Duration instance is returned, representing one minute.

Moving on to the next one, we have

5.minutes + 1.minute

The terms being added are familiar by now, although we need to expand the previous closure to this:

Integer.metaClass.getProperty = {symbol ->
switch (symbol) {
case ["minute"]:
return Duration.minutes(delegate)
case ["years", "quarters", "months", "weeks", "days", "hours", "minutes", "seconds"]:
return Duration.getMethod(symbol, int).invoke(null, delegate)
default:
return null
}
}

Fortunately, the Time and Money library has nicely named methods that correspond exactly to how we want to express durations in this DSL, so we can be very efficient and use reflection invocation of factory methods. Oh, and it's very nice to be able to switch on lists, isn't it? :-)

But there's one more thing to it: the overloading of the + operator. We've already established that both 1.minute and 5.minutes are instances of Duration, and luckily the Duration class already has a plus(Duration) method on it, that Groovy will automatically evalute. It's not always the case that there is such a method available though, as we find out when we move on to the third case:


"2003-05-16" + 3.weeks - 50.years

Evaluating from left to right, we're initially adding a String and a Duration, but String does not have any plus(Duration) method, so we're going to have to add that:


String.metaClass.plus = {Duration duration ->
return duration.addedTo(TimePoint.parseGMTFrom(delegate, "yyyy-MM-dd"))
}

Here, we're using the Time and Money API to represent the String as a TimePoint, to which a Duration may be added, producing another TimePoint. Continuing our evaluation, we now have to subtract a Duration from a TimePoint, which should be quite familiar by now:

TimePoint.metaClass.minus = {Duration duration ->
return duration.subtractedFrom(delegate)
}

These are just a few examples, and the possibilities are vast.

Thursday, April 10, 2008

Grails Pet Store 0.2 released

I finally managed to wrap up a semi-stable milestone of Grails Pet Store, and the roadmap is now available in the form of tagged issues. Hopefully there will be a live instance available Real Soon - watch this spot for updates.

Sunday, March 02, 2008

Podcast from JFokus available

The JFokus presentations are finally available online. Both my presentation on Grails and my collegue Patrik Fredriksson's presentation on the specification pattern are available here. Type your name, press login, then press Play on the next page. Don't ignore the yellow information box if you're running Mac :-)

Tuesday, February 26, 2008

Damn you DBUnit!

I simply can't get over how powerful the Groovy XML and SQL support is, especially when you combine the two. Did you ever find yourself in the position where you wanted to convert a Hypersonic database to a DBUnit dataset? I did, and I told my co-worker, somewhat disgruntled, that "I bet this could be done with 30 lines of Groovy". Well, it could:


import groovy.sql.Sql
import groovy.xml.MarkupBuilder

def sql = Sql.newInstance("jdbc:hsqldb:my_db", "sa", "", "org.hsqldb.jdbcDriver")

def sw = new StringWriter()
def xml = new MarkupBuilder(sw)

xml.dataset {
sql.eachRow "select * from system_tables where table_type != 'SYSTEM TABLE'", {
table(name:it.TABLE_NAME.toLowerCase()) {
sql.rows("select * from ${t}", { md ->
md.columnCount.times {
column md.getColumnName(it + 1).toLowerCase() ?: ""
}
}).each { r ->
row {
r.size().times {
value r[it]
}
}
}
}
}
}

println sw

This is why I like Groovy - it's powerful, yet elegant.

Sunday, February 24, 2008

Groovy power

The concurrent API that was added to Java 5 is very powerful for sumbitting tasks to a worker thread pool, but when you combine it with the Groovy ability to implement single-method interfaces with closures you have a real winner.


import java.util.concurrent.Callable
import java.util.concurrent.Executors

def executorService = Executors.newFixedThreadPool(4)

def x = {
20.times {
println "X"
}
} as Callable

def y = {
20.times {
println " Y"
}
} as Callable

executorService.invokeAll([x, y])

executorService.shutdown()

which of course has an output similar to this:

X
Y
Y
Y
X
Y
X
Y
X
Y
Y
X
X
Y
X
Y
Y
X
X
X

Pretty neat, huh?

Friday, November 30, 2007

Shameless plug

For all my swedish readers who aren't either working for Citerus or are regular guests in the same IRC channel as I am (should amount to about zero people, I'm afraid) - here's an introductory article on the excellent Grails framework that I've written for PNEHM, Citerus' newsletter on agile development.

I will also host a short (20 minutes) oral presentation on the same subject at the upcoming JFokus conference, in January.

Tuesday, September 18, 2007

Advisor summary

It's been a while since the last post, I've been busy with my new assignment that begun right after my vacation. Anyway, here a nifty little routine to quickly get an overview of which beans are woven by what advice, and also what advice weaves which beans in a Spring context:


ApplicationContext context = ... ; // Create your context

String perBeanSummary = "--- Advisors per bean ---\n";
Map<String,Advised> beanMap = BeanFactoryUtils.beansOfTypeIncludingAncestors(context, Advised.class);
Map<Advisor,Set<String>> advisorMap = new HashMap<Advisor, Set<String>>();

// This gathers advisors per bean and beans per advisors,
// and builds the presentation of advisors per bean
for (String beanName : beanMap.keySet()) {
Advised advised = beanMap.get(beanName);
perBeanSummary += beanName + ":\n\t";
for (Advisor advisor : advised.getAdvisors()) {
perBeanSummary += advisor.getAdvice().getClass().getName() + "\n";
Set<String> beans = advisorMap.get(advisor);
if (beans == null) {
beans = new HashSet<String>();
advisorMap.put(advisor, beans);
}
beans.add(beanName);
}
perBeanSummary += "\n";
}

// Builds the presentation of beans per advisor
String perAdvisorSummary = "+++ Beans per advisor +++\n";
for (Advisor advisor : advisorMap.keySet()) {
perAdvisorSummary += advisor.getAdvice().getClass().getName() + "\n";
for (String beanName : advisorMap.get(advisor)) {
perAdvisorSummary += "\t" + beanName + "\n";
}
perAdvisorSummary += "\n";
}

System.out.println(perBeanSummary);
System.out.println(perAdvisorSummary);

Monday, August 13, 2007

Grooeat work, JetBrains!

The first day after the vacation was fairly productive, after all. In preparation for an upcoming PNEHM article, I've successfully installed a snapshot of IDEA 7.0 and built a Subversion snapshot of the Groovy/Grails plugin. This is the first thing I tried out - it looks very promising:


Dynamic typing and completion!

Update: it turns out IDEA also completes the Groovy additions to the JDK. Not completely unexpected when you've seen the screenshot above, but nevertheless very nice!

Thursday, July 12, 2007

Implementing ActiveRecord in Java

The ActiveRecord (AR) design pattern is very popular right now, forming the base of web application frameworks such as Ruby on Rails and the Groovy-based Grails. Martin Fowler defines the pattern as follows:

An object that wraps a row in a database table or view, encapsulates the database access, and adds domain logic on that data.

AR is very closely related to the concept of an Object-Relational Mapper (ORM), and is an alternative to the Data Access Object (DAO) and Repository patterns. In the latter two patterns, persistent data operations are separated from the domain object into one or more dedicated interfaces.

AR is very attractive if you want to build a more powerful domain model, as opposed to using it as a simple data container. It ties the most obvious domain logic into the domain object (namely CRUD), and opens up possibilities to write higher-level properties and operations on your domain objects.

It can also aid in normalizing the three-tier architecture, by which I mean avoiding the all too common situation where you have a request mapped to an object in the MVC layer, which calls the service layer, which initiates a transaction and calls the DAO layer, which calls the ORM, which stores the object in the database. It might be even worse - you may need to convert the bound request data from a form bean to a Data Transfer Object, both of which could be separate from the domain object. Normalizing this operation would mean binding request data to a domain object, which then stores itself.

So, building an ActiveRecord base class in Java means that at least the following methods must be implemented:

public class ActiveRecord<T> {
public static <T> T load(Long id) { .. }
public void store() { .. }
public void delete() { .. }
public static <T> List<T> findAll() { .. }
}

Derived classes will add all sorts of operations, notably a number of specialized finders with similar signatures to findAll().

In order to be able to access non-domain services we need a way to access external services, preferably through Dependency Injection. It's also mandatory that we are able to isolate the domain object for test purposes, and to be able to mock or stub dependencies, all of which is enabled by using DI.

It's also natural to use a full-fledged ORM - Java now has a standard API for that (JPA), and there are a number of different implementations available: Hibernate, TopLink, OpenJPA etc. There's also JDO, iBatis and of course you could also use plain JDBC if you really have to.

I've been using (surprise, surprise!) Hibernate as ORM tool, and Spring for DI and transaction demarcation, all of which are using annotations and build-time weaving using AspectJ and AJDT. Dependency injection of service beans into domain objects is taken care of by the @Configurable annotation, and transaction demarcation by the the @Transactional annotation, so those problems are very cleanly solved.

However
: static methods, such as loaders and finders, don't seem to be woven by transactional advice, at least not when using compile-time weaving. I need to investigate this further, but I believe it might be caused by the fact that a "this" joinpoint is being used here. Workarounds include writing transactional advice manually, using TransactionTemplate for example, or following the mixed AR/Repository pattern suggested below. The best solution would be to have an aspect that's able to read transaction attributes from annotations even in a static context.

The first roadblock is how to gain access to the ORM in a static context, the loader and finder methods. We need a reference to the unit-of-work (session) provider - the SessionFactory in this case - in order to create or obtain the current session for performing data operations. Since we don't have an instance of AR or a derived class, we can't access any injected SF reference.

I've thought long and hard about this, and tried a number of different approaches, but the way I see it, you basically have to give up either static loaders/finders or give up dependency injection. Any way you look at it, you will need some sort of static handle to the SF - you might make the SF member of the AR class static, or you could use some variation of SessionFactoryUtil, or maybe use some sort of lightweight holder object that's instantiated, injected and finally discarded as part of the static operation. The problem with static references, and the ServiceLocator pattern, is that you can't isolate objects completely - there can be only one implementation per class loader at a time, and all instances of a class with a static reference must share the same implementation. This makes testing harder and less robust, compared to a pure DI environment: you must make sure to "reset" the service locator reference after each test, even in case of failures, so you'll end up with a number of try - finally blocks everywhere, and you can't run tests in parallell since you can't guarantee what implementation the factory will return. (Crazy) Bob Lee talks a little about that in this Guice presentation.

So, as far as I can tell, you will need to give up DI, or at least mix DI and ServiceLocator in your application, if you want "pure" ActiveRecord. I'll be very interested if anyone can show a way to use AR and DI together, though :-). For now, I prefer a mix between AR and Repository, but more about that in a little while. If you decide to go for pure AR by using a static SF reference, your next problem will be how to tell the ORM which class to load, without adding redundant data in derived classes, such as overriding load() or keeping a static class member pointing to its own class.

When performing a load, you will need to know what class, or sometimes what table, to load the data from, in addition to the supplied identifier property. Optimally, the implementation of load() exists in the top class ActiveRecord only, and should return a correctly typed object. In short, we want to use the API like this:

Customer c = Customer.load(1);
Item item = Item.load(125);

where Customer and Item both inherit ActiveRecord.

Typing is taken care of by generics, as you can see a few paragraphs earlier. But since it's a static method, there's no "this" to ask for the current class, so finding the current class to feed to Session.load(id, clazz) is harder. There is no API access point in Java, and I've fiddled with various reflection hacks against sun.reflect.Reflections for example, to peek at the call stack, but to no avail. I did however find a way by using the "call" joinpoint in AspectJ and the following construct:

/*
* Keep the traditional load() signature as access point.
*/
public static <T> T load(Long id) {
throw new RuntimeException("This body should never be reached, weaving has not been performed");
}

/*
* This method performs an actual load.
*/
protected static <T> T doLoad(Long id, Class clazz) {
return (T) getSession().load(id, clazz);
}

/*
* This inner class aspect intercepts the call to load, and inspects the join point
* to determine what class the static call was made on, and reroutes the call to
* doLoad() with the correct class parameter.
*/
@Aspect
protected static class ClassIdentifier {

@Before("call (* load(Long)) && args(id)")
public T interceptLoad(ProceedingJoinPoint pjp, Long id) {
Class clazz = pjp.getSignature().getDeclaringType();
return ActiveRecord.doLoad(id, clazz);
}

}

So, if you are prepared to give up reliable replacement of the ORM interface reference, ActiveRecord is within reach. In fact, you might find that you rarely or never mock or stub something like the SessionFactory, but instead simply use a dedicated data source for testing which is loaded with test data. That's perfectly reasonable imo, if you work directly against the ORM API, but it gets a bit more complicated if you keep your own DAO layer around, tested separately from the domain object, and inject that into your domain objects instead.

If you want to go for pure DI, I would argue that it's reasonable to split CRUD operations between the domain objects and a shared "read only"-repository. Operations that work on an actual instance, such as user.store() or item.delete() (non-static by nature), are placed in the domain objects. But operations that result in one or more instances are retrieved according to various criteria, loaders and finders, are placed on a separate service. Those are the same methods that would be static in ActiveRecord.

This approach is DI-compliant, because each domain object instance has its own non-static reference to the ORM, and the Repository service also is injected with an ORM reference. The repository can be regarded as a third party, where you go to retrieve instances of domain objects. By clever use of generics, the amount of code can be kept low, and a single @Transactional annotation at class level on the implementation marks every method for execution in a read-only transaction.

interface Repository {
// Typing on the methods instead of the interface
// allows us to share this interface across the domain model
<T> T load(Long id, Class clazz);
<T> List<T> findAll(Class clazz);

// Various specific finders are added as they are needed.
Customer findByUsername(String username);
List<Item> findDeliveredItems();

// Some kind of generic query-object method might be added too
}

This distinction between instance-tied domain logic versus third-party is then extrapolated throughout the application.

Wednesday, June 27, 2007

Weird but useful generics trick

I stumbled upon the following piece of code while programmatically creating AspectJ proxies with Spring:

// create a factory that can generate a proxy for the given target object
AspectJProxyFactory factory = new AspectJProxyFactory(targetObject);

// add an aspect, the class must be an @AspectJ aspect
// you can call this as many times as you need with different aspects
factory.addAspect(SecurityManager.class);

// you can also add existing aspect instances, the type of the object supplied must be an @AspectJ aspect
factory.addAspect(usageTracker);

// now get the proxy object...
MyInterfaceType proxy = factory.getProxy();

(Pasted from here)

If you look carefully, you'll notice that A) the AspectJProxyFactory class is not parameterized, and B) the proxy creation is assigned to a MyInterfaceType without a cast. I found this to be rather confusing, and took a quick peek at the AspectJProxyFactory source:

public class AspectJProxyFactory extends ProxyCreatorSupport {

[...] // Stuff

public <T> T getProxy() {
return (T) createAopProxy().getProxy();
}

}
The return type is inferred by the assignment, regardless of the (lack of) type on the owning class, which effectively looks and feels like dynamic and static typing all at the same time!

Kind of weird, but might be useful. I, for one, was not aware of this technique.

Wednesday, May 30, 2007

Making JDOM run faster

It's time for another cool AspectJ hack here on RMA, once again leveraging Codehaus' excellent Maven plugin. Still working on the same project as the last post, we've decided to move from standard JAXP to JDOM for programmatic XML handling (the JDOM api is so much nicer to work with it's not even funny). As the returning reader may recall, the model in the MVC part of our application basically consists of a number of XML documents, and the view rendering technology is Freemarker. Freemarker is supposed to be able to handle both JAXP, JDOM and dom4j, but a closer look will reveal that it's more or less only JAXP that's up to date, the other wrappers are deprecated and don't work correctly anymore, which is a shame.

Using JDOM, there is a simple workaround however: the DOMOutputter, which is capable of converting an org.jdom.Document to an org.w3c.dom.Document which can then be handed to Freemarker. There is a speed penalty of course, and when profiling we found that almost all the time is spent in this method, in JAXPDOMAdapter:

public Document createDocument() throws JDOMException {

try {
// We need DOM Level 2 and thus JAXP 1.1.
// If JAXP 1.0 is all that's available then we error out.
Class.forName("javax.xml.transform.Transformer");

// Try JAXP 1.1 calls to build the document
Class factoryClass =
Class.forName("javax.xml.parsers.DocumentBuilderFactory");

// factory = DocumentBuilderFactory.newInstance();
Method newParserInstance =
factoryClass.getMethod("newInstance", null);
Object factory = newParserInstance.invoke(null, null);

// jaxpParser = factory.newDocumentBuilder();
Method newDocBuilder =
factoryClass.getMethod("newDocumentBuilder", null);
Object jaxpParser = newDocBuilder.invoke(factory, null);

// domDoc = jaxpParser.newDocument();
Class parserClass = jaxpParser.getClass();
Method newDoc = parserClass.getMethod("newDocument", null);
org.w3c.dom.Document domDoc =
(org.w3c.dom.Document) newDoc.invoke(jaxpParser, null);

return domDoc;
} catch (Exception e) {
throw new JDOMException("Reflection failed while creating new JAXP document", e);

You're probably wondering why the hell they're using this much reflection just to create an empty Document (we did, anyway). It avoids compile-time dependencies on certain javax.xml classes, but it sure isn't designed with high performance in mind!

In addition to that, the call trace from DOMOuputter to JAXPDOMAdapter contains private methods, so you can't simply inherit and override with your own implementation. So, what now? AspectJ to the rescue, of course!

What we realy want to do is replace this implementation with one that performs the sam thing but statically, so we wrote this around advice (DocBuilder creation omitted for brevity):

@Aspect
public class FastDOMDocumentCreator {

@Around("execution (* org.jdom.output.DOMOutputter.createDOMDocument(..))")
public Object createDOMDocument(ProceedingJoinPoint pjp) throws Throwable {
return docBuilder.newDocument();
}

}

This is a lot faster than the default way. But how do we perform weaving of JDOM classes, that are in an external jar? Actually, it's really simple: just place this snippet in the AspectJ plugin section in the pom-xml (details here):

<configuration>
<weaveDependencies>
<weaveDependency>
<groupId>jdom</groupId>
<artifactId>jdom</artifactId>
</weaveDependency>
</configuration>

If you're using AJDT (which I highly recommend), make sure you add the JDOM jar to the inpath in the configuration dialog, that way this weaving runs on the fly just like regular compilation. On a sidenote, AJDT has some really cool features such as contextual information on "weaves" and "weaved by". It supports annotation-style aspects too.

The weaved jar is exploded under the output directory, with the new weaved classes instead of the vanilla ones. This trick pushed the JDOM-to-JAXP document converstion from the top way down on the hotspot list, with a fairly small amount of work.

Update: While examining the situation a bit closer, we found that in fact it's possible to accomplish the same thing without having to weave the JAXPDOMAdapter. Extending AbstractDOMAdapter using the same code as in the aspect, and passing the new class' name to the DOMOutputter constructor will work too. Nevertheless, the technique is still interesting and may be applicable (as the only solution) somwhere else.

Wednesday, May 16, 2007

Clean solutions are the best

Don't you love it when things just seem to fit together exactly as you want them to? It doesn't happen all too often, but when it does, you just want to blog at the top of your lungs...

I'm working on a project where we communicate with an index server over HTTP, and recieve the query results in XML format. It's a data source, which basically boils down to this interface - the DocumentSource:

Document retrieveDocument( URL url );

The contract of this component is that if the document retrieval succeeds, we get a parsed and ready org.w3c.Document back, and in all other cases it throws some kind of exception. Handling exceptions is done in another part of the application, so don't worry about that right now.

Now, a failure can be either uncontrolled, for example if the index application hits a bug, if there's a network problem, the server might be on fire and so on. It could also be a problem with our implementation of of the afromentioned interface (not likely). In either case, the component will throw an exception to the caller, possibly wrapped in some way.

But a failure can also be controlled, which means that the index server returns a valid XML error message along with HTTP status 200 (OK), if the query is invalid in any way. This is also considered an error, and we handle in like this:

if (isErrorResult(document)) {
throw new IndexErrorException(method, document);
}

The method variable is the HttpMethod that we tried to execute, containing host, port and query information, and document is the parsed XML response from the index, in this case an error message.

What we want to do now is log this error, as transparently as possible. The snippet above expresses that we're not really interested in handling the error in detail, we just leave it at "ok, there was an error, so let's throw an exception. Here's all the information I have on why and where the error occurred".

That's all very well, but we still have to inspect the XML error message to present the error in a readable way. The best way to do that is of course the message property of the exception.

Here's where another aspect of our application comes in: we use Freemarker to build views (HTML and others) using the XML data we retrieve from the DocumentSource interface shown above. And since the error message is also XML, and we want to build a kind of view - a String - why not use the same approach here? That way we won't have to deal with cumbersome Java DOM apis, and we have maximum power to extract and format the error message the way we want. Sounds like a good idea.

It's also the case that we've abstracted away a lot of the fuss around the template engine, as well as the fact that we're using Freemarker, behind this very simple TemplateService interface:

String mergeTemplateIntoString( String templateName, Map model );

This service is a component, a Spring bean, in our application. It would be great if this service could be used to render the logged string from the document and the HTTP method, but in that case we have to supply the newly instantiated exception with a reference from the context. Sure, this could be accomplished by holding an (otherwise useless) reference in the DocumentSource implementation that is passed along the exception, but I prefer it when things Just Work.

Enter @Configurable and compile-time AspectJ weaving! We slap on an a TemplateService setter and an annotation (with autowire=Autowire.BY_NAME to avoid the need for a boilerplate bean definition) on the exception class. We're using AspectJ aspects for various other tasks already, and weaving is done at compile-time to avoid the proxy problem. The Codehaus Maven plugin works great!

In order to weave the Spring AnnotationBeanConfigurerAspect, we add the following to pom.xml:

<aspectLibraries>
<aspectLibrary>
<groupId>org.springframework</groupId>
<artifactId>spring-aspects</artifactId>
</aspectLibrary>
</aspectLibraries>

And the aspect must be made aware of the Spring context (in any application context file):

<aop:spring-configured />
So from here on, the exception message may be rendered like this:

@Override
public String getMessage( ) {
return templateService.mergeTemplateIntoString( "error", model );
}

where "error" is the name of a template, and the model contains the XML document and some other stuff. The error message is now ready to be read and logged, for example in an @AfterThrowing aspect, but that's another story.

Sunday, May 06, 2007

Integrating Struts 1 and Spring

If you want to manage Struts 1 actions with Spring, for example to inject dependencies at runtime, you are going to need a bridging context that essentially duplicates every single action as a bean. (details are here). If you're working on an existing application with lots of actions, writing the bridge context XML will be a pain, so here's a quick Groovy hack that I came up with to generate the Spring context file from the Struts config file:

import groovy.xml.MarkupBuilder

def strutsConfig = new File("path/to/struts-config.xml")

def fromXml = new XmlParser().parse(strutsConfig)

def writer = new StringWriter()
def toXml = new MarkupBuilder(writer)

toXml.beans(
xmlns:"http://www.springframework.org/schema/beans",
"xmlns:xsi":"http://www.w3.org/2001/XMLSchema-instance",
"xsi:schemaLocation":"http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd",
"default-autowire":"byName") {
fromXml."action-mappings".action.each {
bean(name:it["@path"],class:it["@type"])
}
}
}

// This simply dumps the XML to stdout, you could of course write to an actual file as well
println writer

This little snippet creates a <bean name="/foo" class="bar.Baz"/> for every <action path="/foo" type="bar.Baz"/> element in the struts-config.xml file, using a few neat Groovy tricks (GPath, closures, GroovyMarkup).

Still, you could do better. You could use a hook such as BeanFactoryPostProcessor that parses the Struts config and dynamically creates beans corresponding to the actions, or even write an XSD that allows Spring to interpret the Struts config file as a context file directly. There's an entry in the SpringIDE blog that shows how you could write a namespace that works in German, so for example <bean/> becomes <bohne/> and so on. That principle might be applicable to this context too.