JUnit 5

Next Generation Testing
on the JVM

Nicolai Parlog

codefx.org / @nipafx

JVM Group Stockholm

Heads Up

JUnit 5 is work in progress!

This is based on Milestone 2
(released 23rd of July 2016).

Give feedback!

JUnit 5 Links:

Also:

Basics

Basics
Dynamic Tests
Extensions
Architecture
Tools & Setup

What’s New?

class JUnit5Test {

	@Test
	void someTest() {
		assertTrue(true);
	}

}

⇝ Package visibility suffices!

What’s New?

@BeforeAll
static void beforeAll() { ... }

@BeforeEach
void beforeEach() { ... }

@AfterEach
void afterEach() { ... }

@AfterAll
static void afterAll() { ... }

⇝ Lifecycle annotations have new names.

What’s New?

@Test
@Disabled("Y U No Pass?!")
void failingTest() {
	assertTrue(false);
}

⇝ @Ignored is now @Disabled.

What’s New?

@Test
@DisabledOnFriday
void failingTest() {
	assertTrue(false);
}

⇝ Convenient Extensibility.

But how?

What’s New?

@Test
void someTest() {
	...
	assertEquals(
		expected,
		actual,
		"Should be equal.");
}

⇝ Failure message comes last.

What’s New?

@Test
void someTest() {
	...
	assertEquals(
		expected,
		actual,
		() -> "Should " + "be " + "equal.");
}

⇝ Failure message can be created lazily.

What’s New?

@Test
void assertAllProperties() {
	Address ad = new Address(
	  "City", "Street", "42");

	assertAll("address",
	  () -> assertEquals("C", ad.city),
	  () -> assertEquals("Str", ad.street),
	  () -> assertEquals("63", ad.number)
	);
}

⇝ assertAll gathers results from multiple assertions

What’s New?

Output if assertAll fails:

org.opentest4j.MultipleFailuresError:
	address (3 failures)
	expected: <C> but was: <City>
	expected: <Str> but was: <Street>
	expected: <63> but was: (42)

What’s New?

void methodUnderTest() {
	throw new IllegalStateException();
}

@Test
void assertExceptions() {
	Exception ex = assertThrows(
		Exception.class,
		this::methodUnderTest);
	assertEquals("Msg", ex.getMessage());
}

⇝ assertThrows to assert
exception type and other properties

What’s New?

class CountTest {
	// lifecycle and tests
	@Nested
	class CountGreaterZero {
		// lifecycle and tests
		@Nested
		class CountMuchGreaterZero {
			// lifecycle and tests
		}
	}
}

⇝ @Nested to organize tests in inner classes

What’s New?

@DisplayName("A count")
class CountTest {
	@Nested
	@DisplayName("when greater zero")
	class CountGreaterZero {
		@Test
		@DisplayName("is positive")
		void isPositive() { ... }
	}
}

⇝ @DisplayName to show a nice name

What’s new?

The effects of @Nested and @DisplayName:

Nested and DisplayName

What’s new?

@Test
void someTest(MyServer server) {
	// do something with `server`
}

⇝ Test has parameters!

But where do they come from?

What’s New?

Summary

  • lifecycle works much like before

  • many details were improved

  • @Nested and @DisplayName
    make a nice couple

  • parameter injection

  • no lambdas (so far)

That's all very nice but how is it
Next Generation Testing?

Dynamic Tests

Basics
Dynamic Tests
Extensions
Architecture
Tools & Setup

Defining Tests

Up to now tests were identified

  • by names
    (test…​ in JUnit 3 and before)

  • by annotations
    (@Test in JUnit 4 and 5)

⇝ Tests had to be known at compile time.

Defining Tests

So what?!

What if we want to create tests

  • for a set of parameters

  • based on non-source files

  • with lambdas

⇝ We need to define tests at run time.

Dynamic Tests To The Rescue!

Allow creation of tests at run time.

  • tests are wrapped into DynamicTest-s

  • methods that create them
    are annotated with @TestFactory

Creating Tests

@TestFactory
List<DynamicTest> createPointTests() {
	return Arrays.asList(
		dynamicTest(
			"A Great Test For Point",
			() -> { /* test code */ } ),
		dynamicTest(
			"Another Great Test For Point",
			() -> { /* test code */ } )
	);
}

Implementation

The rest is straight-forward:

  1. JUnit detects @TestFactory methods

  2. calls them to generate tests

  3. adds tests to the test tree

  4. eventually runs them

Parameterized Tests

void pointTest(Point p) { /*...*/ }

@TestFactory
Stream<DynamicText> testingPoints() {
	return Stream.of(
			/* create points */)
		.map(p ->
			dynamicTest(
				"Testing " + p,
				() -> pointTest(p)));
}

Parameterized Tests

void pointTest(Point p) { /*...*/ }

@TestFactory
Stream<DynamicText> testingPoints() {
	List<Point> points = asList(
		/* create points */);
	return DynamicTest.stream(
		points,
		p -> "Testing " + p,
		p -> pointTest(p));
}

File-Based Tests

void pointTest(Point p) { /*...*/ }

@TestFactory
Stream<DynamicText> testingPoints() {
	return Files
		.lines(pathToPointFile)
		.map(Point::parse)
		.map(p -> dynamicTest(
			"Testing " + p,
			() -> pointTest(p)));
}

Lambda Tests

So what about lambdas?

This would be great:

class PointTest {

	"A great test for point" -> {
		/* test code */
	}

}

But how?

Lambda Tests

public class LambdaTest {

	private List<DynamicTest> tests;

	protected void λ(
			String name, Executable test) {
		tests.add(dynamicTest(name, test));
	}

	@TestFactory List<DynamicTest> tests() {
		return tests;
	}
}

Lambda Tests

class PointTest extends LambdaTest {{

	λ("A Great Test For Point", () -> {
		/* test code goes here */
	});

}}

  • the inner braces create an initialization block

  • code therein is run during construction

  • JUnit will pick up the tests by calling tests()

Lambda Tests

While we’re hacking…​ what about this?

class PointTest extends LambdaTest {{

	λ(a_great_test_for_point -> {
		/* test code goes here */
	});

}}

Learn how to access
a lambda’s parameter name
with this one weird trick.

Dynamic Tests

Summary

  • with @TestFactory and DynamicTest
    we can create tests at run time:

    • for sets of parameters

    • based on external input

    • as lambdas (yay!)

  • not fully integrated in lifecycle (yet? #378)

That's all very nice but is this already
Next Generation Testing?

Extensions

Basics
Dynamic Tests
Extensions
Architecture
Tools & Setup

Extensions in JUnit 4

Runners

Manage a test’s full lifecycle.

@RunWith(MockitoJUnitRunner.class)
public class MyTest { ... }

  • very flexible

  • heavyweight

  • exclusive

Extensions in JUnit 4

Rules

Execute code before and after statements.

public class MyTest {
	@Rule
	public MockitoRule rule =
		MockitoJUnit.rule();
}

  • added in 4.7

  • lightweight

  • limited to before/after behavior

Extensions in JUnit 4

Extension model is not optimal:

  • two competing mechanisms

    • each with limitations

    • but with considerable overlap

  • composition can cause problems

Approach in JUnit 5

From JUnit 5’s Core Principles:

Prefer extension points over features

Quite literally,
JUnit 5 has Extension Points

Extension Points

  • Test Instance Post Processor

  • BeforeAll Callback

  • Test and Container Execution Condition

  • BeforeEach Callback

  • Parameter Resolution

  • Before Test Execution

  • After Test Execution

  • Exception Handling

  • AfterEach Callback

  • AfterAll Callback

Implementing Extensions

  • one interface for each extension point

  • method arguments capture context

public interface BeforeEachCallback
		extends Extension {

	void beforeEach(
		TestExtensionContext context);
}

  • an extension can use multiple points
    to implement its feature

Benchmark Extension

We want to benchmark our tests!

  • for each test method

  • write the elapsed time to console

How?

  • before test execution: store test launch time

  • after test execution: print elapsed time

Benchmark Extension

public class BenchmarkExtension implements
		BeforeTestExecutionCallback,
		AfterTestExecutionCallback {

	private long launchTime;

	// ...
}

Benchmark Extension

@Override
public void beforeTestExecution(
		TestExtensionContext context) {
	launchTime = currentTimeMillis();
}

@Override
public void afterTestExecution(
		TestExtensionContext context) {
	printf("Test '%s' took %d ms.%n",
		context.getDisplayName(),
		currentTimeMillis() - launchTime);
}

Other Examples

Remember This?

@Test
@DisabledOnFriday
void failingTest() {
	assertTrue(false);
}

Let’s see how it works!

Disabled Extension

public class DisabledOnFridayCondition
		implements TestExecutionCondition {
	@Override
	public ConditionEval.Result evaluate(
			TestExtensionCtx. context) {
		if (isFriday())
			return disabled("Weekend!");
		else
			return enabled("Fix it!");
	}
}

Other Examples

What about parameter injection?

@Test
void someTest(MyServer server) {
	// do something with `server`
}

Parameter Injection

public class MyServerParameterResolver
		implements ParameterResolver {
	@Override
	public boolean supports(
			ParameterContext p, ...) {
		return MyServer.class
			== p.getParameter().getType();
	}
	@Override
	public Object resolve( ... ) {
		return new MyServer();
	}
}

Applying Extensions

How do we apply extensions?

@ExtendWith(DisabledOnFridayCondition.class)
class JUnit5Test {
	...
}

That’s technical and verbose…​ :(

Applying Extensions

Meta-annotations to the rescue!

  • JUnit 5’s annotations are meta-annotations

  • JUnit 5 checks recursively for annotations

⇝ We can create our own annotations!

Creating Annotations

@ExtendWith(DisabledOnFridayCondition.class)
public @interface DisabledOnFriday { }

@Test
@Tag("integration")
@ExtendWith(BenchmarkExtension.class)
@ExtendWith(MyServerParameterResolver.class)
public @interface IntegrationTest { }

@IntegrationTest
@DisabledOnFriday
void testLogin(MyServer server) { ... }

Extensions

Summary

  • flexibility because of many extension points

  • extensions compose well

  • customizable due to meta-annotations

(We left out some details.)

That's all very nice but how is it
Next Generation Testing?

Architecture

Basics
Dynamic Tests
Extensions
Architecture
Tools & Setup

JUnit 4 Architecture

  • a single JAR (ignoring Hamcrest)

  • used by

    • developers

    • extensions

    • IDEs, build-tools

  • no separation of concerns

JUnit 4 Architecture

  • tools provide us with awesome features!

  • but API is not powerful enough

I know, I’ll use reflection!

  • nothing was safe!

  • bound tools to implementation details

  • made maintenance and evolution very hard

Dead End

Part of JUnit’s success is its great tool support!

But the same tools locked development in.

The success of JUnit as a platform prevents the development of JUnit as a tool.
(Johannes Link)

Approach in JUnit 5

Separation of concerns:

  1. an API to write tests against

  2. a mechanism to discover and run tests

  3. an API for tools to run tests

Approach in JUnit 5

Separation of concerns V 2.0:

  1. an API to write tests against

  2. a mechanism to discover and run tests

    1. specific engine per variant of tests
      (e.g. JUnit 4 or JUnit 5)

    2. orchestration of engines

    3. API between them

  3. an API for tools to run tests

Subprojects & Modules

JUnit Jupiter 5.0.0-M2

  • junit-jupiter-api

  • junit-jupiter-engine

JUnit Vintage 4.12.0-M2

  • junit-vintage-engine

JUnit Platform 1.0.0-M2

  • junit-platform-engine

  • junit-platform-runner

  • a lot more

JUnit 5 Modules

architecture limited lean

Architecture

Summary

  • clear separation of concerns

  • API for developers

  • API for tools

That's all very nice but how is it
Next Generation Testing?

Because it opens up the platform!

Moar Engines!

  • want to run JUnit 4 tests?
    ⇝ create an engine for it

  • want TestNG to have support like JUnit?
    ⇝ create an engine for it

  • want to write tests in natural language?
    ⇝ create an engine for it

Moar Engines!

architecture lean

Open Platform

Once JUnit 5 adoption sets in:

  • tools are decoupled from implementation details

  • tools can support all frameworks (almost) equally well

  • new frameworks start with full tool support

  • developers can try out new things

A new generation of test frameworks might arise!

Open Platform

JUnit’s success as a platform
becomes available to everybody.

This heralds the
next generation of testing on the JVM!

Architecture

Summary

  • clear separation of concerns:
    APIs for developers, tools,
    and new frameworks

  • opens up the platform

  • tool support for everybody!

(There’s even more to the story.)

Tools & Setup

Basics
Dynamic Tests
Extensions
Architecture
Tools & Setup

Writing Tests

As Easy As Pie!

Add this:

org.junit.jupiter
junit-jupiter-api
5.0.0-M2

Have fun!

Running Tests

Not much native support, yet

  • Maven: discussions (#31, #1206, mails)

  • Gradle: not even an issue

  • IntelliJ: EA support in 2016.2

  • Eclipse: slow progress (#217, #488566)

  • NetBeans: not even an issue

Running Tests

As Part Of JUnit 4

  • individual classes:

    @RunWith(JUnitPlatform.class)
public class JUnit5Test { ... }

  • all classes:

    @RunWith(JUnitPlatform.class)
@SelectPackages({ "my.test.package" })
public class JUnit5TestSuite { }

Running Tests

With Build Tools

JUnit 5 team provides rudimentary
Gradle plugin and Maven Surefire provider
(see user guide for details)

Running Tests

From Console

There is a console launcher:

# run all tests
junit-platform-console
	-p ${path_to_compiled_test_classes}
	-a
# run a specific test
junit-platform-console
	-p ${path_to_compiled_test_classes}
	org.codefx.demo.junit5.HelloWorldTest

Tools & Setup

Summary

  • you can start writing tests right away

  • only IntelliJ has native support

  • running with JUnit 4 is a good compromise

(Read about the setup details.)

Next Generation Testing On The JVM

  • new API is an incremental improvement
    full of thoughtful details

  • dynamic tests are very useful

  • extension model looks very promising

  • architecture opens up the platform

  • tool support is not there yet

(Read more about JUnit 5.)

Questions?

Find Me

codefx.org / @nipafx / +NicolaiParlog

Me

you can hire me

since 2016: editor of sitepoint.com/java

2014-2016: Java developer at Disy

2011-2014: Java developer at Fraunhofer ISI

until 2010: CS and Math at TU Dortmund

Image Credits