Mpsc scheduler

[franz1981]

This is a first PoC for an alternative scheduler targeting low number of cores.
It has no work-stealing implemented (so it would be better to have VT non-voluntary preemption, but can still suffer from unbalanced work-loads (obviously!) - but to keep latencies and cpu usage tight, it collapse the subpollers as carrier-confined in the carrier loop.
I've introduced a pollerMode 4 for this.

In term of peak tps it is in between the #226 performance and the custom scheduler with native transport i.e. the number of hops from I/O polling is the same.

Architecture

• Single MPSC queue per carrier, no work stealing
• Carrier affinity via affinityHint, set once at onStart, onContinue always routes home
• roundRobinAffinity() for initial VT distribution across carriers
• Poller Mode 4: each carrier owns its own epoll fd, VT fds register directly: no sub-pollers, no master poller thread. Plain HashMap<Integer, Thread> for fd→VT mapping (carrier-local, single-thread access)
• Event loop interleaves task drain (configurable 50µs budget) with non-blocking epoll_wait(0), blocks only when genuinely idle
• @Contended on queue producer/consumer fields to avoid false sharing
• Activated with -Djdk.virtualThreadScheduler.useMpsc=true -Djdk.pollerMode=4

If pwait2 would be available it could collapse into it a timer wheel structure (or just a priority q, since there's no sharing) for timers.
I have not performed any specific optimization on wakeups; eventFd is knew to be terrible and in Netty indeed we shield concurrent eventFd writes with a slightly more complex state machine. Here I made it simple as the most of the interaction(s) are local.

Sustained Load Results

Benchmark: same as #226

MPSC (Mode 4) : this new scheduler
FJP (sticky): the FJP scheduler

Both are running Netty event loops as "sticky" VirtualThreads, assigned (if the scheduler allow it) round robin to each carrier.

CPU-bound (50K req/s, 2 server CPUs, 1ms mock delay, 100 connections)

Metric	MPSC (Mode 4)	FJP (sticky)
p50	1.15ms	1.20ms
p75	1.23ms	1.61ms
p90	4.98ms	2.95ms
p99	33.82ms	44.56ms
CPU utilization	1.71 CPUs	1.82 CPUs
Context switches/sec	8.7K	36.3K
Page faults	5.4K	218

MPSC matches or beats FJP at p50-p75 and p99 with 6% less CPU. Context switches are 4x lower. p90 is wider.

I/O-bound (120K req/s, 12 server CPUs, 30ms mock delay, 7000 connections)

Metric	MPSC (Mode 4)	FJP
p50	30.28ms	30.28ms
p90	32.24ms	30.54ms
p99	182.45ms	56.10ms
CPU utilization	8.61 CPUs	10.0 CPUs
Context switches/sec	54.6K	106K

Same throughput. MPSC uses 14% less CPU with half the context switches. Tail latency (p99) is worse under heavy I/O.

---

• I confirm that I make this contribution in accordance with the OpenJDK Interim AI Policy.

---

Progress

• Change must not contain extraneous whitespace

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/loom.git pull/227/head:pull/227
$ git checkout pull/227

Update a local copy of the PR:
$ git checkout pull/227
$ git pull https://git.openjdk.org/loom.git pull/227/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 227

View PR using the GUI difftool:
$ git pr show -t 227

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/loom/pull/227.diff

[bridgekeeper]

👋 Welcome back franz1981! A progress list of the required criteria for merging this PR into fibers will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[franz1981]

As a design point, given the low core assumption, this scheduler could use a mpmc shared queue where to spill work which exceed a threshold for each carrier (I would make the mpsc mailboxes soft bounded) - in case the VT to spill has not sticky affinity.
If sticky, will end up in the mpsc queue.
The VT started or continued from a sticky one instead, could spill into the mpmc shared queue.
Than, each carrier, can use a prime-based interval while draining it, no matter what.
This should keep a locality- first design with some form of redistribution under load.
BTW this is similar to what go does, although it uses a more "reactive" approach using nmspinner to search for work.

[He-Pin]

this will be faster for the eventloop think, in pekko stream, the mailbox is a mpsc queue now and faster than it was

[franz1981]

Some more benchmarking data from the CPU bound test:

30K tps:

40K tps:

50K tps:

60K tps:

and p50/p99 vs load:

And at 30K tps load:

  ┌──────────────────────┬────────────┬────────────┬────────────┐
  │        Metric        │    MPSC    │    FJP     │   Ratio    │
  ├──────────────────────┼────────────┼────────────┼────────────┤
  │ Context switches/sec │ 17,412     │ 32,942     │ 1.9x fewer │
  ├──────────────────────┼────────────┼────────────┼────────────┤
  │ CPU migrations/sec   │ 50         │ 1,881      │ 37x fewer  │
  ├──────────────────────┼────────────┼────────────┼────────────┤
  │ Page faults/sec      │ 730        │ 1,093      │ 1.5x fewer │
  ├──────────────────────┼────────────┼────────────┼────────────┤
  │ CPU utilization      │ 1.34 cores │ 1.57 cores │ 15% less   │
  ├──────────────────────┼────────────┼────────────┼────────────┤
  │ Instructions         │ 70.0B      │ 72.7B      │ 4% fewer   │
  ├──────────────────────┼────────────┼────────────┼────────────┤
  │ IPC                  │ 1.30       │ 1.22       │ 7% higher  │
  └──────────────────────┴────────────┴────────────┴────────────┘

[franz1981]

@AlanBateman By moving the poller to be per carrier (as this pr), un-shared, would allow me to more easily use edge-triggered poll, further reducing the difference between Netty event loops/async I/O frameworks - and what Loom I/O does, saving one syscall per blocking read.
That said, is not guaranteed to be a win as with edge-triggered poll if a socket is used by different carrier in its life-cycle, its fd will end up awaking N different carriers pollers, with only one of them which have a parked VT, so probably ONESHOT is the best possible (and cleanest) choice, for this case.
Netty event loops physically "event loop confined" a socket so this won't EVER happen.
The different ownership model w Loom is what would drive this specific optimization to be appliable.

At this point a better name for this scheduler would be "event loop scheduler"

[franz1981]

@AlanBateman IIRC the way eventFd is setup to not accumulate writes and a single poll would drain them all: so I've pushed be4dd8f as an additional optimization, although rare, in case on a cycle of VT draining, none register for some I/O events; if that happen, the per carrier local poller just skip the non blocking poll.

[franz1981]

As a side note for Netty uring integration: in modern Linux kernel versions the uring ring buffer can provide its own (pollable) fd which can be registered for POLLIN ONE SHOT against the per carrier poller/sub-poller allowing Netty VT to safely park/unpack.
Clearly this need OpenJDK to expose "something" to allow registering such fd, somehow.

[He-Pin]

Can we have an IntMap here?

[franz1981]

Yeah it was my first choice tbh but thanks to:

• Netty selectors to be very few (the poller register the selector FD and not each connection)
• FD reuse

The numbers were low enough that boxing was using the int cache.
But yes, in a final version would be better to use a primitive map

[franz1981]

I've noticed that I forgot to re-add roundRobin affinity to the subpollers: which means they are randomly assigned!
That's not going to work with this scheduler as it doesn't implement work stealing.

In addition to that I've performed few perf tests in a I/O bound scenario and the custom scheduler I built, which have a different prioritization for the Netty event loop, still beat MPSC with poller mode 4.
The difference is relevant enough (172K vs 166K) and a first analysis shows that the custom scheduler is more aggressively non blocking polling for Netty's I/O causing smaller batches to be execute, and better interleaving the other VTs, while MPSC pm 4 is too batchy.
It's not clear to me yet how this is happening, but in my brain I would have expected the same exact performance.
Interestingly it seems that custom scheduler w EPOLL is slightly better than with NIO (which still use a per carrier poller mode 3 for Netty's selector too), but NIO is not as bad as MPSC w NIO.