A week with Test-NG

On my latest assignment (I work as a consultant), we've decided to let go of the old workhorse that is JUnit 3, and go with one of the new, fancy @Nnotations-driven frameworks JUnit 4 or Test-NG. After a short evaluation period, which consisted of verifying basic Eclipse and Maven support and skimming a few tutorials, the feature sheet of Test-NG looked a bit stronger. The two major selling points of Test-NG over JUnit 4 were the ability to group tests and that Test-NG can run only the tests that failed the previous round.

However, the first week of day-to-day usage of Test-NG has been less than stellar, mostly (but not limited to) the poor IDE integration. My list of pros and cons versus the trusted old JUnit 3, when used inside Eclipse:

Pros:

• The test class is only instantiated once, not once for each method in the test class. As it should be.
• Better setup granularity with both @BeforeClass and @BeforeMethod entry points
• @DataProvider is a nice feature, especially in conjunction with Selenium.

And the cons:

• No built-in macro expansion in Eclipse that creates a test method - in a class that extends the JUnit 3 class TestCase, just type "test" and press Ctrl-Enter, and you have a method skeleton. Removing the need to inherit means more work in this case.
• Keyboard shortcut Ctrl-Alt-X N does not work, so I have to use the mouse every time I want to run a test.
• Can't run more than one test class at once, using multi-select in the file tree browser.
• Can't run an entire package hierarchy of test classes in one shot.
• Can't run a single test method by right-clicking a method name in the "green bar" view.
• Using JMock is a lot more work, either by re-inventing half of MockObjectTestCase manually, or by using plain old inheritance.
• No automatic static imports (a weakness of Eclipse's, to be fair), which forces me to write Assert.assertEquals() instead of just assertEquals() or write the static import manually.

So, a lot of annoyances on the ground level, and we haven't really started to see the benefits of large-scale features such as grouping and run-failed-tests-only. The verdict so far is that Test-NG is as good as or worse than JUnit 3, when counting in IDE integration.

And really, what is so fantastic about not having to inherit TestCase and naming your methods testXXX? Here's the number of times that has been a limiting factor in my work as a Java developer: 0.

Also, annotations are good for a lot of things (@Entity and @Transactional are great), but sometimes you get the feeling that various framework authors are going "OK, we now have annotations in Java, how can we possibly shoe-horn that into JMyFramework4J 2.0?"