Thinking about high bandwidth? Meet Virtual Flows

The purpose of this article is to explore known facts about the upcoming expansion of the Java threading model.

No, don’t worry, Java’s current multithreading model remains, but behind the scenes, something good is already knocking on the virtual door.

Yes, we are talking about JEP-425: Virtual Threads.

Overcoming current concurrency limitations

Let’s first look at Java’s current multithreading model. It provides an implementation of the Thread class. A thread can be thought of as a unit of Java concurrency that can perform so-called runnable tasks. An instance of the Thread class is also an object, but there’s more going on behind the scenes.

For example: each newly created thread gets its own stack dedicated to store local variables, method calls, references, which are placed on top during execution inside the thread’s current scope.

The thread will terminate after the task has completed by calling the join() method. It can be mapped 1:1 to the platform thread, controlled by the underlying system, which manages the scheduling of instructions to be executed.

Since the underlying platform cannot create an unlimited number of platform threads, it becomes obvious that the current Java threading model must be used wisely and the order in which new threads are created needs to be monitored.

The limiting factor is mainly caused by available resources (CPU, memory, etc.), although Java itself may feel different.

try(ExecutorService executor = Executors.newSingleThreadExecutor(THREAD_FACTORY)){
   executor.execute(() -> … );
}

Example 1: Single threaded executor executing a task runnable

Over the past decades, a parallel program written in Java has been able to execute runnable tasks in parallel, i.e. simultaneously. Currently, Java already provides the concepts Executors (performers – example 1.) or Thread Pools (thread pools – example 2.) that help developers manage available platform resources and avoid unwanted use of system resources, such as new calls Thread() and start().

try( ExecutorService executor = Executors.newFixedThreadPool(10, THREAD_FACTORY)){
   executor.submit(() -> … );   
}

Example 2: A pool of fixed initiated threads that run callable task

Starting with Java SE 8, Java also contains the concept ComputableFeature (example 3.), which helps to perform asynchronous tasks in isolated threads and runs a common thread pool by default. To be more precise, it uses the common ForkJoinPool (picture 1).

The ForkJoin framework was another big improvement in the release of Java SE 7. Its purpose was to make it easier to use all available processor cores correctly, but it could lead to some downsides caused, for example, by unintentional use of executors (Example 3.)

record ComputableTask(AtomicInteger counter, int failedCycle) implements Runnable {
   @Override
   public void run() {
      // May thrown an exception
       readFileContent(counter, failedCycle);
       System.out.printf("""
               DONE: thread: '%s', cycle: '%d', failedCycle:'%d'
               """, Thread.currentThread().getName(), counter.get(), failedCycle);
   }
}
...
completableFuture.thenRun(new ComputableTask(counter, failedCycle));
... 
Example output:
DONE: thread: 'main', cycle: '1', failedCycle:'2'
DONE: thread: 'ForkJoinPool.commonPool-worker-1', cycle: '2', failedCycle:'2'
FINISHED: cycles:'100'

PExample 3 Usage ComputableFuture may have disadvantages, such as the inability to interrupt execution, debugging, or meaningful StackTrace.

To be honest, since the Java platform is very multi-threaded, so when talking about concurrency, we must also consider garbage collectors, debuggers, profilers, or other technologies that are affected by the “threading game”.

Meet virtual flows

Okay, this is what we have at the moment.

Something exciting is about to happen. The next major extension of the concurrency model is coming. These are virtual threads.

I’ll explain what virtual threads are, where they come from and why we need them! The motivation may be obvious. Let’s brush up on the basics.

Idea thread-sharing (thread sharing), represented by a pool of threads (ForJoinPoolpool, etc.) between tasks can help improve throughput, but compared to style thread-per-request (stream per request) it can have significant drawbacks. Idea thread-per-request allows you to make the code maintainable, understandable and debuggable. This style allows you to perform and observe a task from start to finish (the root cause is easy to identify). Thread sharing complicates all of this.

...
var threadFactory = new ThreadFactory() {
  ... 
  @Override
  public Thread newThread(Runnable r) {
    var t = new Thread(threadGroup, r, "t-" + counter.getAndIncrement());
    t.setDaemon(true);
    return t;
  }
};
...
var executor = Executors.newFixedThreadPool(THREADS_NUMBER, threadFactory);
for (int i = 0; i < EXECUTION_CYCLES; i++) {
  executor.submit(new ThreadWorker(i, MAX_CYCLES, ALLOCATION_SIZE));
}
...

Example 4 Current Approach flow per request with fixed pool size and factory

But here’s the good news. Virtual threads tend to maintain a thread-per-request style to bring clarity to code execution and keep the thread structure clear. An approach Virtual Threads looks promising as it attempts to use operating system resources (carried by the platform thread, Figure 1) and maintain understandable code (compare examples 4 and 5).

There are two ways to create a virtual thread:

Both create a new virtual thread for each task.

Figure 1. The ForkJoin shared pool of threads is shared with virtual threads and even a custom factory, and each virtual thread belongs to the
Figure 1. The ForkJoin shared pool of threads is shared with virtual threads and even a custom factory, and each virtual thread belongs to the “VirtualThread” group.

The virtual thread is shared (not CPU-bound, Figure 1) and passed through the platform thread (CPU-bound). Therefore, the user should not make any assumptions about its assignment to a platform thread. These virtual threads are cheap and should be created for a short term task and should never be pooled due to design (Figure 3.).

var threadFactory = Thread.ofVirtual()
               .name("ForkJoin-custom-factory-", 0)
               .factory();
var counter = new AtomicInteger(0);
var failedCycle = new Random().nextInt(CYCLE_MAX - 1) + 1;
try (var executor = Executors.newThreadPerTaskExecutor(threadFactory)) {
  for (int i = 0; i < EXECUTION_CYCLES; i++) {
    executor.submit(new ComputableTask(counter, failedCycle));
  }
}

Example 5: Java SE 19 proposes a method newThreadPerTaskExecutorwhich starts a thread for each task that runs, and a thread factory that serves the virtual thread.

Virtual threads allow you to execute hundreds of tasks at the same time(!), which could otherwise cause JVM crashes or out-of-memory exceptions, using a common threading model (example 4. For example, with THREAD_NUMBER = 10_000).

A few things to remember

virtual flow always running like a thread demon with NORM_PRIORITYwhich means that using the installer has no effect. Because virtual threads are contained within active threads, they cannot be part of any thread group. Using Thread.getThreadGroup returns VirtualThreads.

The virtual stream is not has permission when working with Security Manager, which is already obsolete anyway (JEP-411, Java SE 17, ref. 4).

As mentioned, a virtual thread behaves pretty much the same as normal threads, which means they can use thread-local and thread-local inherited variables (careful as a virtual thread should never be pooled)

Remember one more thing

Java SE 19 also has another very important improvement. ExecutorService now extends interface AutoCloseable and it is recommended to use the design try-with-resource. This plays well with the goal of removing finalization (JEP-421, ref 3).

An additional extension related to the upcoming Virtual Thread implementation is the Java Flight Recorder events.

Figure 2: Upcoming Java Flight Recorder Events for Virtual Flows

Almost no Darksiders

There may be some disadvantages. One is that VirtualThread plans to use a shared thread pool that is also shared by other processes running in the JVM, such as the ForkJoin framework (Figure 1). This could hypothetically cause an out-of-memory exception when trying to allocate the thread’s stack, or cause the application to fail to create a thread.

Another problem is the potential incompatibility with the existing implementation of concurrency, since, for example, a ThreadGroup always returns a value VirtualThreads, but the fact is that it cannot be destroyed, resumed or stopped. These methods always throw an exception. The ThreadMXBean is intended to be used only for platform threads and some others…

Conclusion

Figure 3: ComputableTaskEvent created by the task. It shows the use of virtual threads. Virtual threads are served by a factory (example 5).

The concept of virtual threads looks very promising. Not only does it help increase application throughput by running many more concurrent tasks at the same time (Figure 3), but it also provides a framework for “theoretically” easy refactoring of already existing code (Example 5. thread-per-request style), see section “ There are almost no Darkseiders.”

After all, JEP-425 is still under active development and we have to wait for the upcoming results in Java SE 19.

To test the examples with the current state, you can go to the GitHub project (link 5).

Figure 4.: Traditional “request per stream” approach showing its limitations compared to Figure 3.

We recommend reading

  1. Project Loom Early-Access Build 19-loom+5-429 (2022/4/4)

  2. JEP-425: Virtual Threads (Preview)

  3. JEP-421: Deprecate Finalization for Removal

  4. JEP-411: Deprecate the Security Manager for Removal

  5. GitHub Java 19 Examples

Similar Posts

Leave a Reply Cancel reply