JUnit 5

Next Generation Testing
on the JVM

Heads Up

JUnit 5 is work in progress!

This is based on Milestone 4
(released 1st of April 2017).

GA release is tentatively scheduled for September!

Give feedback!

Basics

Basics
Dynamic Tests
Parameterized Tests
Extensions
Architecture
Setup & Tools

What’s New?

class JUnit5Test {

	@Test
	void someTest() {
		assertTrue(true);
	}

}

⇝ Package visibility suffices!

What’s New?

@BeforeAll
static void beforeAll() { ... }

@BeforeEach
void beforeEach() { ... }

@AfterEach
void afterEach() { ... }

@AfterAll
static void afterAll() { ... }

⇝ Lifecycle annotations have new names.

What’s New?

@Test
@Disabled("Y U No Pass?!")
void failingTest() {
	assertTrue(false);
}

@Ignore is now @Disabled.

What’s New?

@Test
@DisabledOnFriday
void failingTest() {
	assertTrue(false);
}

⇝ Convenient Extensibility.

But how?

What’s New?

@Test
void assertAllProperties() {
	Address ad = new Address(
	  "City", "Street", "42");

	assertAll("address",
	  () -> assertEquals("C", ad.city),
	  () -> assertEquals("Str", ad.street),
	  () -> assertEquals("63", ad.number)
	);
}

assertAll gathers results from multiple assertions

What’s New?

Output if assertAll fails:

org.opentest4j.MultipleFailuresError:
	address (3 failures)
	expected: <C> but was: <City>
	expected: <Str> but was: <Street>
	expected: <63> but was: (42)

What’s New?

class CountTest {
	// lifecycle and tests
	@Nested
	class CountGreaterZero {
		// lifecycle and tests
		@Nested
		class CountMuchGreaterZero {
			// lifecycle and tests
		}
	}
}

@Nested to organize tests in inner classes

What’s New?

@DisplayName("A count")
class CountTest {
	@Nested
	@DisplayName("when greater zero")
	class CountGreaterZero {
		@Test
		@DisplayName("is positive")
		void isPositive() { ... }
	}
}

@DisplayName to show a nice name

What’s new?

The effects of @Nested and @DisplayName:

Nested and DisplayName

What’s new?

@Test
void someTest(MyServer server) {
	// do something with `server`
}

⇝ Test has parameters!

But where do they come from?

What’s New?

Summary

  • lifecycle works much like before

  • many details were improved

  • @Nested and @DisplayName
    make a nice couple

  • parameter injection

That's all very nice but how is it
Next Generation Testing?

Dynamic Tests

Basics
Dynamic Tests
Parameterized Tests
Extensions
Architecture
Setup & Tools

Defining Tests

Up to now tests were identified

  • by names
    (test…​ in JUnit 3 and before)

  • by annotations
    (@Test in JUnit 4 and 5)

⇝ Tests had to be known at compile time.

Defining Tests

So what?!

What if we want to create tests

  • for a set of parameters

  • based on non-source files

  • with lambdas

⇝ We need to define tests at run time.

Dynamic Tests To The Rescue!

Allow creation of tests at run time.

  • tests are wrapped into DynamicTest-s

  • methods that create them
    are annotated with @TestFactory

These are fully-fledged tests:

  • integrated into lifecycle (soon)

  • viewed by tools as separate tests

For more details, check this post.

Parameterized Tests

Basics
Dynamic Tests
Parameterized Tests
Extensions
Architecture
Setup & Tools

Parameterizing Tests

Dominant use case for parameterizing tests:

  • have a test method

  • define input data

  • run the method once per input

JUnit 5 has native support for that!

Parameterizing Tests

@ParameterizedTest
@ValueSource(strings = { " Hello", "JUnit" })
void testValues(String word) { ... }
  • declare test method with @ParameterizedTest

  • define input data with different sources

  • JUnit does the rest:

Parameterized tests

Argument Sources

Parameterized tests need sources for arguments.

These are included:

  • @ValueSource is very simple
    but only works for single arguments

  • @EnumSource injects some or all enum values

  • @MethodSource calls a method to create args

  • @CsvSource allows defining args as CSV strings

  • @CsvFileSource loads args from CSV file

More than one source can be used per method!

Enum Source

Calling a test with all values of an enum:

@ParameterizedTest
@EnumSource(TimeUnit.class)
void testAllEnumValues(TimeUnit unit) { ... }

Calling a test with a few values of an enum:

@ParameterizedTest
@EnumSource(
	value = TimeUnit.class,
	names = {"NANOSECONDS", "MICROSECONDS"})
void testSomeEnumValues(TimeUnit unit) { ... }

Method Source

Call a method to provide arguments:

@ParameterizedTest
@MethodSource(names = "createWords")
void testWords(String word) { ... }

static Stream<String> createWords() {
	return Stream.of("Hello", "JUnit");
}

Method Source

For multiple arguments, return Arguments:

@ParameterizedTest
@MethodSource(names = "createWordsWithLength")
void testLength(String word, int length) { ... }

static Stream<Arguments> createWordsWithLength() {
	return Stream.of(
			create("Hello", 5),
			create("JUnit 5", 7));
}

CSV Sources

Define values as CSV strings:

@ParameterizedTest
@CsvSource({ "(0/0), 0", "(0/1), 1", "(1/1), 1.41" })
void testPointNorm(String point, double norm) { ... }
  • each string provides args for one execution

  • strings are split by comma to get individual args

Argument Converters

How do strings get converted?

  • JUnit handles primitives, enums, java.time.*

  • you can implement ArgumentConverter
    and apply with @ConvertWith:

@ParameterizedTest
@CsvSource({ "(0/0), 0", "(0/1), 1", "(1/1), 1.41" })
void testPointNorm(
	@ConvertWith(PointConverter.class) Point point,
	double norm) { ... }

CSV File Sources

This also applies to CSV files:

@ParameterizedTest
@CsvFileSource(resources = "/pointNorms.csv")
void testPointNorm(
	@ConvertWith(PointConverter.class) Point point,
	double norm) { ... }

More Sources

You can create your own sources:

  1. implement ArgumentsProvider

static class RandomIntegerProvider
		implements ArgumentsProvider {

	@Override
	public Stream<Arguments> arguments(...) {
		return new Random().ints(0, 10)
			.mapToObj(ObjectArrayArguments::create)
			.limit(3);
	}

}

More Sources

You can create your own sources:

  1. apply with @ArgumentsSource

@ParameterizedTest
@ArgumentsSource(RandomIntegerProvider.class)
void testRandomIntegers(Integer argument) { ... }

Parameterized Tests

Summary

JUnit 5 has native support for
one variant of parameterized tests:

  • declare test method with @ParameterizedTest

  • define input data with different sources, e.g.
    @ValueSource, @MethodSource, @CsvSource,
    or self-made sources

Other variants can be implemented via extensions!

That's all very nice but is this already
Next Generation Testing?

Extensions

Basics
Dynamic Tests
Parameterized Tests
Extensions
Architecture
Setup & Tools

Approach To Extension

From JUnit 5’s Core Principles:

Prefer extension points over features

Quite literally,
JUnit 5 has Extension Points

Extension Points

  • Instance Post Processor

  • Template Invocation

  • Container and Test Execution Condition

  • BeforeAll and BeforeEach Callbacks

  • Parameter Resolution

  • Before Test Execution

  • After Test Execution

  • Exception Handling

  • AfterEach and AfterAll Callbacks

Implementing Extensions

  • one interface for each extension point

  • method arguments capture context

public interface BeforeEachCallback
		extends Extension {

	void beforeEach(
		TestExtensionContext context);
}
  • an extension can use multiple points
    to implement its feature

Benchmark Extension

We want to benchmark our tests!

  • for each test method

  • write the elapsed time to console

How?

  • before test execution: store test launch time

  • after test execution: print elapsed time

Benchmark Extension

public class BenchmarkExtension implements
		BeforeTestExecutionCallback,
		AfterTestExecutionCallback {

	private long launchTime;

	// ...
}

Benchmark Extension

@Override
public void beforeTestExecution(
		TestExtensionContext context) {
	launchTime = currentTimeMillis();
}

@Override
public void afterTestExecution(
		TestExtensionContext context) {
	printf("Test '%s' took %d ms.%n",
		context.getDisplayName(),
		currentTimeMillis() - launchTime);
}

Other Examples

Remember This?

@Test
@DisabledOnFriday
void failingTest() {
	assertTrue(false);
}

Let’s see how it works!

Disabled Extension

public class DisabledOnFridayCondition
		implements TestExecutionCondition {

	@Override
	public ConditionEvaluationResult evaluate(
			TestExtensionContext context) {
		if (isFriday())
			return disabled("Weekend!");
		else
			return enabled("Fix it!");
	}

}

Other Examples

What about parameter injection?

@Test
void someTest(MyServer server) {
	// do something with `server`
}

Parameter Injection

public class MyServerParameterResolver
		implements ParameterResolver {

	@Override
	public boolean supports(ParameterContext p, ...) {
		return MyServer.class
			== p.getParameter().getType();
	}

	@Override
	public Object resolve(ParameterContext p, ...) {
		return new MyServer();
	}

}

Applying Extensions

How do we apply extensions?

@ExtendWith(DisabledOnFridayCondition.class)
class JUnit5Test {
	...
}

That’s technical and verbose…​ :(

Applying Extensions

Meta-annotations to the rescue!

  • JUnit 5’s annotations are meta-annotations

  • JUnit 5 checks recursively for annotations

⇝ We can create our own annotations!

Creating Annotations

@ExtendWith(DisabledOnFridayCondition.class)
public @interface DisabledOnFriday { }

@Test
@Tag("integration")
@ExtendWith(BenchmarkExtension.class)
@ExtendWith(MyServerParameterResolver.class)
public @interface IntegrationTest { }

@IntegrationTest
@DisabledOnFriday
void testLogin(MyServer server) { ... }

Extensions

Summary

  • flexibility because of many extension points

  • extensions compose well

  • customizable due to meta-annotations

(We left out some details.)

That's all very nice but how is it
Next Generation Testing?

Architecture

Basics
Dynamic Tests
Parameterized Tests
Extensions
Architecture
Setup & Tools

JUnit 4 Architecture

  • a single JAR (ignoring Hamcrest)

  • used by

    • developers

    • extensions

    • IDEs, build-tools

  • no separation of concerns

JUnit 4 Architecture

  • tools provide us with awesome features!

  • but API is not powerful enough

I know, I’ll use reflection!

  • nothing was safe!

  • bound tools to implementation details

  • made maintenance and evolution very hard

Dead End

Part of JUnit’s success is its great tool support!

But the same tools locked development in.

The success of JUnit as a platform prevents the development of JUnit as a tool.
(Johannes Link)

Approach in JUnit 5

Separation of concerns:

  1. an API to write tests against

  2. a mechanism to discover and run tests

  3. an API for tools to run tests

Approach in JUnit 5

Separation of concerns V 2.0:

  1. an API to write tests against

  2. a mechanism to discover and run tests

    1. specific engine per variant of tests
      (e.g. JUnit 4 or JUnit 5)

    2. orchestration of engines

    3. API between them

  3. an API for tools to run tests

Subprojects & Modules

JUnit Jupiter 5.0.0-M4
  • junit-jupiter-api

  • junit-jupiter-engine

JUnit Vintage 4.12.0-M4
  • junit-vintage-engine

JUnit Platform 1.0.0-M4
  • junit-platform-engine

  • junit-platform-runner

  • a lot more

JUnit 5 Modules

architecture limited lean

Architecture

Summary

  • clear separation of concerns

  • API for developers

  • API for tools

That's all very nice but how is it
Next Generation Testing?

Because it opens up the platform!

Moar Engines!

  • want to run JUnit 4 tests?
    ⇝ create an engine for it!

  • want TestNG to have support like JUnit?
    ⇝ create an engine for it!

  • want to write tests in natural language?
    ⇝ create an engine for it!

Moar Engines!

architecture lean

Open Platform

Once JUnit 5 adoption sets in:

  • tools are decoupled from implementation details

  • tools can support all frameworks (almost) equally well

  • new frameworks start with full tool support

  • developers can try out new things

A new generation of test frameworks might arise!

Open Platform

JUnit’s success as a platform
becomes available to everybody.

This heralds the
next generation of testing on the JVM!

It Already Shows

Some community engines:

  • jqwik: "a simpler JUnit test engine"

  • Specsy: "a BDD-style unit-level testing framework"

  • Spek: "a Kotlin specification framework for the JVM"

Other projects:

Architecture

Summary

  • clear separation of concerns:
    APIs for developers, tools,
    and new frameworks

  • opens up the platform

  • tool support for everybody!

(There’s even more to the story.)

Setup & Tools

Basics
Dynamic Tests
Parameterized Tests
Extensions
Architecture
Setup & Tools

Writing Tests

As Easy As Pie!

Add this:

org.junit.jupiter
junit-jupiter-api
5.0.0-M4

Have fun!

Running Tests

Native support is w.i.p.

  • Maven: work started #1206

  • Gradle: work started #1037

  • IntelliJ: since 2016.2; 2017.1.2 required for M4

  • Eclipse: EA support with 4.7 M4
    with some extra work #488566

  • NetBeans: not even an issue

Running Tests

With Build Tools

JUnit 5 team provides rudimentary
Gradle plugin and Maven Surefire provider
(see user guide for details)

Running Tests

As Part Of JUnit 4

  • individual classes:

    @RunWith(JUnitPlatform.class)
    public class JUnit5Test { ... }
  • all classes:

    @RunWith(JUnitPlatform.class)
    @SelectPackages({ "my.test.package" })
    public class JUnit5TestSuite { }

Running Tests

From Console

There is a console launcher:

# run all tests
java -jar junit-platform-console-standalone.jar
	--class-path ${path_to_compiled_test_classes}
	--scan-class-path
# run a specific test
java -jar junit-platform-console-standalone.jar
	--class-path ${path_to_compiled_test_classes}
	--select-class ${fully_qualified_test_class_name}

Tools & Setup

Summary

  • you can start writing tests right away

  • only IntelliJ has native support

  • provisional Maven/Gradle integration works

  • running with JUnit 4 is a good compromise

(Read about the setup details.)

Next Generation Testing On The JVM

  • new API is an incremental improvement
    full of thoughtful details

  • dynamic tests are an important concept

  • wohoo, native parameterized tests!

  • extension model looks very promising

  • architecture opens up the platform

  • tool support is not there yet

About Nicolai Parlog

37% off with
code fccparlog

tiny.cc/j9ms

Want More?

⇜ Get my book!

I write a mean newsletter,
currently mostly about Java 9.

You can hire me.

Image Credits