Async signal ucontext RIP rewrite corrupts guest state

[mcfi]

Symptom

amd64 Go under FEX crashes nondeterministically when Go async preemption is enabled. Disabling async preemption avoids the issue:

FEX ~/opt/go/bin/go test fmt
# crashes / corrupts runtime state

GODEBUG=asyncpreemptoff=1 FEX ~/opt/go/bin/go test fmt
# ok fmt

Minimal non-Go repro

This mimics the key part of Go async preemption: an async SIGURG handler rewrites the interrupted ucontext RIP to a trampoline, then resumes the original PC.

#define _GNU_SOURCE
#include <pthread.h>
#include <signal.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ucontext.h>

extern uint64_t worker(uint64_t n);
extern char worker_lo[], worker_hi[];

asm(".text\n"
    ".globl worker,worker_lo,worker_hi\n"
    "worker:\n"
    "worker_lo:\n"
    "  push %rbx\n"
    "  mov %rdi, %rcx\n"
    "  xor %rax, %rax\n"
    "  xor %rbx, %rbx\n"
    "  mov $0x9E3779B97F4A7C15, %rsi\n"
    "1:\n"
    "  add $1, %rax\n"
    "  add %rax, %rbx\n"
    "  mov %rbx, %rdx\n"
    "  imul %rsi, %rdx\n"
    "  xor %rdx, %rax\n"
    "  rol $7, %rax\n"
    "  add %rbx, %rax\n"
    "  sub $1, %rcx\n"
    "  jnz 1b\n"
    "  pop %rbx\n"
    "  ret\n"
    "worker_hi:\n");

uintptr_t resume_slot;
extern void trampoline_global(void);
asm(".text\n"
    ".globl trampoline_global\n"
    "trampoline_global:\n"
    "  jmp *resume_slot(%rip)\n");

static volatile int stop_flag;
static volatile unsigned long preemptions;
static pthread_t worker_tid;

static void handler(int sig, siginfo_t *si, void *uc) {
  (void)sig;
  (void)si;

  ucontext_t *u = (ucontext_t *)uc;
  greg_t *g = u->uc_mcontext.gregs;
  uintptr_t rip = (uintptr_t)g[REG_RIP];

  if (rip < (uintptr_t)worker_lo || rip >= (uintptr_t)worker_hi)
    return;

  preemptions++;
  resume_slot = rip;
  g[REG_RIP] = (greg_t)(uintptr_t)trampoline_global;
}

static void *signaller(void *arg) {
  (void)arg;

  while (!stop_flag) {
    pthread_kill(worker_tid, SIGURG);
    for (volatile int i = 0; i < 200; i++) {}
  }

  return NULL;
}

int main(void) {
  const uint64_t n = 200000;
  const int rounds = 10;

  uint64_t expected = worker(n);
  printf("expected=0x%016llx\n", (unsigned long long)expected);

  struct sigaction sa;
  memset(&sa, 0, sizeof(sa));
  sa.sa_sigaction = handler;
  sa.sa_flags = SA_SIGINFO | SA_RESTART;
  sigaction(SIGURG, &sa, NULL);

  worker_tid = pthread_self();

  pthread_t t;
  pthread_create(&t, NULL, signaller, NULL);

  int bad = 0;
  for (int i = 0; i < rounds; i++) {
    uint64_t got = worker(n);
    if (got != expected) {
      bad++;
      printf("mismatch round %d: got=0x%016llx want=0x%016llx\n",
             i, (unsigned long long)got, (unsigned long long)expected);
    }
  }

  stop_flag = 1;
  pthread_join(t, NULL);

  printf("preemptions=%lu mismatches=%d/%d\n", preemptions, bad, rounds);
  return bad ? 1 : 0;
}

Build/run inside an amd64 guest/rootfs:

gcc -O2 -pthread repro.c -o repro
FEX ./repro

Expected: nonzero preemptions, zero mismatches.

Observed under FEX: checksum mismatches, hangs, or crashes. This suggests FEX is exposing an imprecise async guest context when the signal handler rewrites RIP, which matches Go's SIGURG async preemption behavior.

[Sonicadvance1]

Yea, known issue atm that FEX isn't async signal safe. I have some plans for working around it but haven't gotten there yet. Nice to have the small repro test.