Evaluating JDK Flight Recorder Events with JfrUnit

Gunnar Morling, open source software developer at Red Hat, presented JfrUnit, a new testing utility that can be used to detect performance degradation with JUnit or Spock Framework… Interpreting performance test results such as response times can be difficult because there may be regressions caused by factors other than the application itself, such as other processes or the network. JfrUnit can be used to test the performance of an application by measuring memory allocation, I / O operations, database queries, or other application-specific items.

JDK Flight Recorder (JFR) collects events from a running application that can be used to diagnose or profile the application. These events can be almost anything from memory allocation to garbage collection.

The tool can be used directly from the command line, but it is often used in conjunction with JDK Mission Control, providing a graphical interface and various plugins that can be used with JFR. JfrUnit allows you to create assertions that validate JFR events from within your application.

JfrUnit supports OpenJDK 16, and the dependency is available in Maven Central:

<dependency>
  <groupId>org.moditect.jfrunit</groupId>
  <artifactId>jfrunit</artifactId>
  <version>1.0.0.Alpha1</version>
  <scope>test</scope>
</dependency>

JUnit test implementation starts by adding annotation @JfrEventTest to the unit class of the test if the test is not annotated @QuarkusTestbecause the Quarkus testing framework interacts automatically with the JFR record. Tests use annotation @EnableEvent to collect specific events, such as garbage collection events. After executing the program logic, the method jfrEvents.awaitEvents() waits for any JFR events from the JVM or application before assertions are used to verify that the event occurred:

@JfrEventTest
public class GarbageCollectionTest {
    public JfrEvents jfrEvents = new JfrEvents();

    @Test
    @EnableEvent("jdk.GarbageCollection")
    public void testGarbageCollectionEvent() throws Exception {
        System.gc();

        jfrEvents.awaitEvents();

        assertThat(jfrEvents).contains(event("jdk.GarbageCollection"));
    }
}

Alternatively, you can use the Spock framework to write the same test:

class GarbageCollectionSpec extends Specification {
JfrEvents jfrEvents <span style="margin: 0px; box-sizing: border-box; display: inline; -webkit-box-orient: horizontal; -webkit-box-direction: normal; flex-flow: row wrap; vertical-align: top; color: rgb(166, 127, 89); background: rgba(255, 255, 255, 0.5);" class="token operator">=</span> <span style="margin: 0px; box-sizing: border-box; display: inline; -webkit-box-orient: horizontal; -webkit-box-direction: normal; flex-flow: row wrap; vertical-align: top; color: rgb(0, 119, 170);" class="token keyword">new</span> <span style="margin: 0px; box-sizing: border-box; display: inline; -webkit-box-orient: horizontal; -webkit-box-direction: normal; flex-flow: row wrap; vertical-align: top;" class="token class-name">JfrEvents</span><span style="margin: 0px; box-sizing: border-box; display: inline; -webkit-box-orient: horizontal; -webkit-box-direction: normal; flex-flow: row wrap; vertical-align: top; color: rgb(153, 153, 153);" class="token punctuation">(</span><span style="margin: 0px; box-sizing: border-box; display: inline; -webkit-box-orient: horizontal; -webkit-box-direction: normal; flex-flow: row wrap; vertical-align: top; color: rgb(153, 153, 153);" class="token punctuation">)</span><font style="margin: 0px; box-sizing: border-box;"></font>


    @EnableEvent('jdk.GarbageCollection')
    def 'Contains a garbage collection Jfr event'() {
        when:
        System.gc()

        then:
        jfrEvents['jdk.GarbageCollection']
    }
}

In addition to checking if the event has occurred, you can also check the details of the event, such as the duration of the method execution Thread.sleep():

@Test
@EnableEvent("jdk.ThreadSleep")
public void testThreadSleepEvent() throws Exception {
    Thread.sleep(42);

    jfrEvents.awaitEvents();

    assertThat(jfrEvents)
            .contains(event("jdk.ThreadSleep")
            .with("time", Duration.ofMillis(42)));
}

JfrUnit allows you to create more complex scripts as well. Consider the following example, which collects memory allocation events and summarizes their data before claiming that the memory allocation is between certain values:

@Test
@EnableEvent("jdk.ObjectAllocationInNewTLAB")
@EnableEvent("jdk.ObjectAllocationOutsideTLAB")
public void testAllocationEvent() throws Exception {
    String threadName = Thread.currentThread().getName();

    // Application logic which creates objects

    jfrEvents.awaitEvents();
    long sum = jfrEvents.filter(this::isObjectAllocationEvent)
            .filter(event -> event.getThread().getJavaName().equals(threadName))
            .mapToLong(this::getAllocationSize)
            .sum();

    assertThat(sum).isLessThan(43_000_000);
    assertThat(sum).isGreaterThan(42_000_000);
}

private boolean isObjectAllocationEvent(RecordedEvent re) {
    String name = re.getEventType().getName();
    return name.equals("jdk.ObjectAllocationInNewTLAB") ||
            name.equals("jdk.ObjectAllocationOutsideTLAB");
}

private long getAllocationSize(RecordedEvent recordedEvent) {
    return recordedEvent.getEventType().getName()
            .equals("jdk.ObjectAllocationInNewTLAB") ?
            recordedEvent.getLong("tlabSize") :
            recordedEvent.getLong("allocationSize");
}

Including multiple events is also possible using the wildcard ““, for example, @EnableEvent("jdk.ObjectAllocation") can be used to fire all ObjectAllocation events.

To reset the collected events, you can use the method jfrEvents.reset()to ensure that only events are collected after the method is executed reset()… For example, when running multiple iterations and validating the results at each iteration:

for (int i = 0; i < ITERATIONS; i++) {
    // Application logic

    jfrEvents.awaitEvents();


    // Assertions

    jfrEvents.reset();
}

Frameworks like Hibernate do not generate events by themselves, but in these cases JMC agent can be used to create events. The JMC agent can generate SQL query events, which can then be used to estimate the (number) of SQL queries entering the database. This is demonstrated in the DevNation Tech Talk. Continuous performance regression testing with JfrUnit, and an example is available on GitHub: Examples for JfrUnit…