JUnit 5

Next Generation Testing
on the JVM

Heads Up

JUnit 5 is work in progress!

This is based on Milestone 3
(released 30th of November 2016).

Milestone 4 should come out any day now!

Give feedback!

Basics

Basics
Dynamic Tests
Extensions
Architecture
Tools & Setup

What’s New?

class JUnit5Test {

	@Test
	void someTest() {
		assertTrue(true);
	}

}

⇝ Package visibility suffices!

What’s New?

@BeforeAll
static void beforeAll() { ... }

@BeforeEach
void beforeEach() { ... }

@AfterEach
void afterEach() { ... }

@AfterAll
static void afterAll() { ... }

⇝ Lifecycle annotations have new names.

What’s New?

@Test
@Disabled("Y U No Pass?!")
void failingTest() {
	assertTrue(false);
}

@Ignored is now @Disabled.

What’s New?

@Test
@DisabledOnFriday
void failingTest() {
	assertTrue(false);
}

⇝ Convenient Extensibility.

But how?

What’s New?

@Test
void assertAllProperties() {
	Address ad = new Address(
	  "City", "Street", "42");

	assertAll("address",
	  () -> assertEquals("C", ad.city),
	  () -> assertEquals("Str", ad.street),
	  () -> assertEquals("63", ad.number)
	);
}

assertAll gathers results from multiple assertions

What’s New?

Output if assertAll fails:

org.opentest4j.MultipleFailuresError:
	address (3 failures)
	expected: <C> but was: <City>
	expected: <Str> but was: <Street>
	expected: <63> but was: (42)

What’s New?

class CountTest {
	// lifecycle and tests
	@Nested
	class CountGreaterZero {
		// lifecycle and tests
		@Nested
		class CountMuchGreaterZero {
			// lifecycle and tests
		}
	}
}

@Nested to organize tests in inner classes

What’s New?

@DisplayName("A count")
class CountTest {
	@Nested
	@DisplayName("when greater zero")
	class CountGreaterZero {
		@Test
		@DisplayName("is positive")
		void isPositive() { ... }
	}
}

@DisplayName to show a nice name

What’s new?

The effects of @Nested and @DisplayName:

Nested and DisplayName

What’s new?

@Test
void someTest(MyServer server) {
	// do something with `server`
}

⇝ Test has parameters!

But where do they come from?

What’s New?

Summary

  • lifecycle works much like before

  • many details were improved

  • @Nested and @DisplayName
    make a nice couple

  • parameter injection

That's all very nice but how is it
Next Generation Testing?

Dynamic Tests

Basics
Dynamic Tests
Extensions
Architecture
Tools & Setup

Defining Tests

Up to now tests were identified

  • by names
    (test…​ in JUnit 3 and before)

  • by annotations
    (@Test in JUnit 4 and 5)

⇝ Tests had to be known at compile time.

Defining Tests

So what?!

What if we want to create tests

  • for a set of parameters

  • based on non-source files

  • with lambdas

⇝ We need to define tests at run time.

Dynamic Tests To The Rescue!

Allow creation of tests at run time.

  • tests are wrapped into DynamicTest-s

  • methods that create them
    are annotated with @TestFactory

Creating Tests

@TestFactory
List<DynamicTest> createPointTests() {
	return Arrays.asList(
		dynamicTest(
			"A Great Test For Point",
			() -> { /* test code */ } ),
		dynamicTest(
			"Another Great Test For Point",
			() -> { /* test code */ } )
	);
}

Implementation

The rest is straight-forward:

  1. JUnit detects @TestFactory methods

  2. calls them to generate tests

  3. adds tests to the test tree

  4. eventually runs them

Parameterized Tests

void pointTest(Point p) { /*...*/ }

@TestFactory
Stream<DynamicText> testingPoints() {
	return Stream.of(
			/* create points */)
		.map(p ->
			dynamicTest(
				"Testing " + p,
				() -> pointTest(p)));
}

Parameterized Tests

void pointTest(Point p) { /*...*/ }

@TestFactory
Stream<DynamicText> testingPoints() {
	List<Point> points = asList(
		/* create points */);
	return DynamicTest.stream(
		points,
		p -> "Testing " + p,
		p -> pointTest(p));
}

File-Based Tests

void pointTest(Point p) { /*...*/ }

@TestFactory
Stream<DynamicText> testingPoints() {
	return Files
		.lines(pathToPointFile)
		.map(Point::parse)
		.map(p -> dynamicTest(
			"Testing " + p,
			() -> pointTest(p)));
}

Dynamic Tests

Summary

  • with @TestFactory and DynamicTest
    we can create tests at run time:

    • for sets of parameters

    • based on external input

    • as lambdas (yay!)

  • not fully integrated in lifecycle (yet? #378)

That's all very nice but is this already
Next Generation Testing?

Extensions

Basics
Dynamic Tests
Extensions
Architecture
Tools & Setup

Approach To Extension

From JUnit 5’s Core Principles:

Prefer extension points over features

Quite literally,
JUnit 5 has Extension Points

Extension Points

  • Test Instance Post Processor

  • BeforeAll Callback

  • Test and Container Execution Condition

  • BeforeEach Callback

  • Parameter Resolution

  • Before Test Execution

  • After Test Execution

  • Exception Handling

  • AfterEach Callback

  • AfterAll Callback

Implementing Extensions

  • one interface for each extension point

  • method arguments capture context

public interface BeforeEachCallback
		extends Extension {

	void beforeEach(
		TestExtensionContext context);
}
  • an extension can use multiple points
    to implement its feature

Benchmark Extension

We want to benchmark our tests!

  • for each test method

  • write the elapsed time to console

How?

  • before test execution: store test launch time

  • after test execution: print elapsed time

Benchmark Extension

public class BenchmarkExtension implements
		BeforeTestExecutionCallback,
		AfterTestExecutionCallback {

	private long launchTime;

	// ...
}

Benchmark Extension

@Override
public void beforeTestExecution(
		TestExtensionContext context) {
	launchTime = currentTimeMillis();
}

@Override
public void afterTestExecution(
		TestExtensionContext context) {
	printf("Test '%s' took %d ms.%n",
		context.getDisplayName(),
		currentTimeMillis() - launchTime);
}

Other Examples

Remember This?

@Test
@DisabledOnFriday
void failingTest() {
	assertTrue(false);
}

Let’s see how it works!

Disabled Extension

public class DisabledOnFridayCondition
		implements TestExecutionCondition {

	@Override
	public ConditionEvaluationResult evaluate(
			TestExtensionContext context) {
		if (isFriday())
			return disabled("Weekend!");
		else
			return enabled("Fix it!");
	}

}

Other Examples

What about parameter injection?

@Test
void someTest(MyServer server) {
	// do something with `server`
}

Parameter Injection

public class MyServerParameterResolver
		implements ParameterResolver {

	@Override
	public boolean supports(ParameterContext p, ...) {
		return MyServer.class
			== p.getParameter().getType();
	}

	@Override
	public Object resolve(ParameterContext p, ...) {
		return new MyServer();
	}

}

Applying Extensions

How do we apply extensions?

@ExtendWith(DisabledOnFridayCondition.class)
class JUnit5Test {
	...
}

That’s technical and verbose…​ :(

Applying Extensions

Meta-annotations to the rescue!

  • JUnit 5’s annotations are meta-annotations

  • JUnit 5 checks recursively for annotations

⇝ We can create our own annotations!

Creating Annotations

@ExtendWith(DisabledOnFridayCondition.class)
public @interface DisabledOnFriday { }

@Test
@Tag("integration")
@ExtendWith(BenchmarkExtension.class)
@ExtendWith(MyServerParameterResolver.class)
public @interface IntegrationTest { }

@IntegrationTest
@DisabledOnFriday
void testLogin(MyServer server) { ... }

Extensions

Summary

  • flexibility because of many extension points

  • extensions compose well

  • customizable due to meta-annotations

(We left out some details.)

That's all very nice but how is it
Next Generation Testing?

Architecture

Basics
Dynamic Tests
Extensions
Architecture
Tools & Setup

JUnit 4 Architecture

  • a single JAR (ignoring Hamcrest)

  • used by

    • developers

    • extensions

    • IDEs, build-tools

  • no separation of concerns

JUnit 4 Architecture

  • tools provide us with awesome features!

  • but API is not powerful enough

I know, I’ll use reflection!

  • nothing was safe!

  • bound tools to implementation details

  • made maintenance and evolution very hard

Dead End

Part of JUnit’s success is its great tool support!

But the same tools locked development in.

The success of JUnit as a platform prevents the development of JUnit as a tool.
(Johannes Link)

Approach in JUnit 5

Separation of concerns:

  1. an API to write tests against

  2. a mechanism to discover and run tests

  3. an API for tools to run tests

Approach in JUnit 5

Separation of concerns V 2.0:

  1. an API to write tests against

  2. a mechanism to discover and run tests

    1. specific engine per variant of tests
      (e.g. JUnit 4 or JUnit 5)

    2. orchestration of engines

    3. API between them

  3. an API for tools to run tests

Subprojects & Modules

JUnit Jupiter 5.0.0-M3
  • junit-jupiter-api

  • junit-jupiter-engine

JUnit Vintage 4.12.0-M3
  • junit-vintage-engine

JUnit Platform 1.0.0-M3
  • junit-platform-engine

  • junit-platform-runner

  • a lot more

JUnit 5 Modules

architecture limited lean

Architecture

Summary

  • clear separation of concerns

  • API for developers

  • API for tools

That's all very nice but how is it
Next Generation Testing?

Because it opens up the platform!

Moar Engines!

  • want to run JUnit 4 tests?
    ⇝ create an engine for it

  • want TestNG to have support like JUnit?
    ⇝ create an engine for it

  • want to write tests in natural language?
    ⇝ create an engine for it

Moar Engines!

architecture lean

Open Platform

Once JUnit 5 adoption sets in:

  • tools are decoupled from implementation details

  • tools can support all frameworks (almost) equally well

  • new frameworks start with full tool support

  • developers can try out new things

A new generation of test frameworks might arise!

Open Platform

JUnit’s success as a platform
becomes available to everybody.

This heralds the
next generation of testing on the JVM!

It Already Shows

Some community engines:

  • jqwik: "a simpler JUnit test engine"

  • Specsy: "a BDD-style unit-level testing framework"

  • Spek: "a Kotlin specification framework for the JVM"

Other projects:

Architecture

Summary

  • clear separation of concerns:
    APIs for developers, tools,
    and new frameworks

  • opens up the platform

  • tool support for everybody!

(There’s even more to the story.)

Tools & Setup

Basics
Dynamic Tests
Extensions
Architecture
Tools & Setup

Writing Tests

As Easy As Pie!

Add this:

org.junit.jupiter
junit-jupiter-api
5.0.0-M3

Have fun!

Running Tests

Native support is w.i.p.

  • Maven: work started #1206

  • Gradle: work started #1037

  • IntelliJ: since 2016.2; 2016.3.1 required for M3

  • Eclipse: EA support with 4.7 M4
    with some extra work #488566

  • NetBeans: not even an issue

Running Tests

As Part Of JUnit 4

  • individual classes:

    @RunWith(JUnitPlatform.class)
    public class JUnit5Test { ... }
  • all classes:

    @RunWith(JUnitPlatform.class)
    @SelectPackages({ "my.test.package" })
    public class JUnit5TestSuite { }

Running Tests

With Build Tools

JUnit 5 team provides rudimentary
Gradle plugin and Maven Surefire provider
(see user guide for details)

Running Tests

From Console

There is a console launcher:

# run all tests
junit-platform-console \
	-cp ${path_to_compiled_test_classes} \
	--scan-class-path
# run a specific test
junit-platform-console \
	-cp ${path_to_compiled_test_classes} \
	--select-class \
	${fully_qualified_test_class_name}

Tools & Setup

Summary

  • you can start writing tests right away

  • only IntelliJ has native support

  • running with JUnit 4 is a good compromise

(Read about the setup details.)

Next Generation Testing On The JVM

  • new API is an incremental improvement
    full of thoughtful details

  • dynamic tests are very useful

  • extension model looks very promising

  • architecture opens up the platform

  • tool support is not there yet

About Me

Want More?

I’m writing a book!
EA soon ⇝ tiny.cc/fx-weekly

two-day Java 9 course
Zürich, 20.-21.04.
⇝ register at 42talents.com

Image Credits