HotSpot ZGC under FEX corrupts object fields or crashes in ZGC stack/root processing

[mcfi]

Symptom

Running x86-64 HotSpot with explicit -XX:+UseZGC under FEX on an AArch64 host fails quickly on a small Java allocation/GC stress test.

The most common failure is an impossible Java exception:

java.lang.NullPointerException: Cannot load from byte/boolean array because "this.bytes" is null
        at FEXZGCBarrierRepro$Payload.sample(FEXZGCBarrierRepro.java:28)
        at FEXZGCBarrierRepro.worker(FEXZGCBarrierRepro.java:73)

The bytes field is final and is initialized immediately before the failing load. In other runs, the same test aborts the JVM with a ZGC-related fatal SIGSEGV, for example:

# SIGSEGV (0xb) at pc=0x00007ffff75888ca
# Problematic frame:
# V  [libjvm.so+0x13888ca]  ZUncoloredRoot::process(zaddress_unsafe*, unsigned long)+0xba

Environment

Host: CentOS Stream 10 aarch64 on Apple Silicon
Kernel: 6.12.0-254.el10.aarch64
FEX commit: c171c0619207fe9316d107c3e6b839fb3db8554f
Guest JDK: Temurin OpenJDK 26.0.2+10, linux x86-64

The guest java binary is x86-64:

ELF 64-bit LSB pie executable, x86-64, interpreter /lib64/ld-linux-x86-64.so.2

Reproducer

Compile with an x86-64 JDK and run through FEX:

/path/to/FEX jdk-26.0.2+10/bin/javac FEXZGCBarrierRepro.java

/path/to/FEX jdk-26.0.2+10/bin/java \
  -Xms128m -Xmx128m -XX:+UseZGC \
  FEXZGCBarrierRepro

FEXZGCBarrierRepro.java:

import java.util.concurrent.CountDownLatch;

public final class FEXZGCBarrierRepro {
    private static final int WORKERS = 4;
    private static final int ITERATIONS = 450_000;
    private static final int ROOT_SET_SIZE = 4_096;
    private static final int GC_REQUESTS = 160;

    private static final Object[][] roots = new Object[WORKERS][];
    private static volatile Object gcRoot;
    private static long checksum;

    private FEXZGCBarrierRepro() {
    }

    private static final class Payload {
        private final byte[] bytes;
        private final int seed;

        private Payload(int seed) {
            this.seed = seed;
            bytes = new byte[128 + (seed & 255)];
            bytes[0] = (byte) seed;
            bytes[bytes.length - 1] = (byte) (seed >>> 8);
        }

        private int sample() {
            return mix(seed, bytes[0], bytes[bytes.length - 1]);
        }
    }

    private static int mix(int seed, int first, int last) {
        return seed ^ first ^ last;
    }

    private static long expectedChecksum() {
        long total = 0;
        for (int worker = 0; worker < WORKERS; worker++) {
            for (int i = 0; i < ITERATIONS; i++) {
                int seed = (worker << 24) ^ i;
                total += mix(seed, (byte) seed, (byte) (seed >>> 8));
            }
        }
        return total;
    }

    private static void await(CountDownLatch start) {
        try {
            start.await();
        } catch (InterruptedException exception) {
            Thread.currentThread().interrupt();
            throw new RuntimeException(exception);
        }
    }

    private static void sleepBriefly() {
        try {
            Thread.sleep(1);
        } catch (InterruptedException exception) {
            Thread.currentThread().interrupt();
            throw new RuntimeException(exception);
        }
    }

    private static void worker(int id, CountDownLatch start) {
        await(start);

        Object[] localRoots = new Object[ROOT_SET_SIZE];
        long localChecksum = 0;
        for (int i = 0; i < ITERATIONS; i++) {
            Payload payload = new Payload((id << 24) ^ i);
            localRoots[i & (ROOT_SET_SIZE - 1)] = payload;
            localChecksum += payload.sample();
        }

        roots[id] = localRoots;
        synchronized (FEXZGCBarrierRepro.class) {
            checksum += localChecksum;
        }
    }

    private static void requestGc(CountDownLatch start) {
        await(start);

        for (int i = 0; i < GC_REQUESTS; i++) {
            byte[][] batch = new byte[32][];
            for (int j = 0; j < batch.length; j++) {
                batch[j] = new byte[32 * 1_024 + ((i + j) & 1_023)];
                batch[j][0] = (byte) (i ^ j);
            }

            gcRoot = batch;
            System.gc();
            sleepBriefly();
        }
    }

    public static void main(String[] args) throws Exception {
        Thread.setDefaultUncaughtExceptionHandler((thread, exception) -> {
            System.err.println("uncaught exception in " + thread.getName());
            exception.printStackTrace();
            Runtime.getRuntime().halt(100);
        });

        CountDownLatch start = new CountDownLatch(1);
        Thread[] threads = new Thread[WORKERS];
        for (int i = 0; i < threads.length; i++) {
            final int id = i;
            threads[i] = new Thread(() -> worker(id, start), "worker-" + id);
            threads[i].start();
        }

        Thread gcThread = new Thread(() -> requestGc(start), "gc-requester");
        gcThread.start();
        start.countDown();

        for (Thread thread : threads) {
            thread.join();
        }
        gcThread.join();

        long expected = expectedChecksum();
        if (checksum == expected) {
            System.out.println("ok checksum=" + checksum);
        } else {
            throw new AssertionError("checksum=" + checksum + ", expected=" + expected);
        }
    }
}

Observed results

On FEX with -XX:+UseZGC, the test failed 3/3:

trial 1: exit 134, fatal SIGSEGV in ZUncoloredRoot::process(...)
trial 2: exit 100, NullPointerException because this.bytes is null
trial 3: exit 100, NullPointerException because this.bytes is null

Controls

The same class passes under FEX with ZGC when compiled code is disabled:

/path/to/FEX jdk-26.0.2+10/bin/java \
  -Xint -Xms128m -Xmx128m -XX:+UseZGC \
  FEXZGCBarrierRepro

Output:

ok checksum=-4923005376

The same class also passes under FEX with the default-style G1 collector:

/path/to/FEX jdk-26.0.2+10/bin/java \
  -Xms64m -Xmx64m -XX:+UseG1GC \
  FEXZGCBarrierRepro

Output:

ok checksum=-4923005376

As a comparator, the same x86-64 JDK and same rootfs pass under QEMU user-mode with ZGC and compiled code enabled:

qemu-x86_64-static -L /path/to/rootfs \
  jdk-26.0.2+10/bin/java \
  -Xms128m -Xmx128m -XX:+UseZGC \
  FEXZGCBarrierRepro

Output:

ok checksum=-4923005376