Summary
When a guest executable is invoked through two or more levels of symlink,
FEX expands $ORIGIN in its DT_RPATH/DT_RUNPATH to the directory of the
intermediate symlink rather than the directory of the fully-resolved binary.
Libraries referenced relative to $ORIGIN then fail to load.
One level of symlink works. The real path works. Only chains of two or more
break.
Minimal reproducer
mkdir -p t/bin t/lib alt
cat > lib.c <<'EOF'
#include <stdio.h>
void hello(void) { printf("shared library loaded\n"); }
EOF
cat > main.c <<'EOF'
extern void hello(void);
int main(void) { hello(); return 0; }
EOF
gcc -shared -fPIC -o t/lib/libhello.so lib.c
gcc main.c -Wl,-rpath,'$ORIGIN/../lib' -Lt/lib -lhello -o t/bin/prog
# one level of symlink
ln -sf "$PWD/t/bin/prog" /usr/local/bin/prog-1level
# two levels of symlink
ln -sf "$PWD/t/bin/prog" "$PWD/alt/prog"
ln -sf "$PWD/alt/prog" /usr/local/bin/prog-2level
./t/bin/prog # real path
prog-1level # one symlink
prog-2level # two symlinks
Actual behaviour (under FEX)
real path OK (shared library loaded)
1-level symlink OK (shared library loaded)
2-level symlink FAIL (prog-2level: error while loading shared libraries:
libhello.so: cannot open shared object file: No such file or directory)
Expected behaviour
All three print shared library loaded. They do on native x86-64: glibc derives
$ORIGIN from the fully-resolved executable path, so the number of symlink hops
is irrelevant.
Real-world impact: the JDK
This is how it shows up in practice on an Ubuntu 22.04 x86-64 rootfs:
$ java -version
java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
$ ls -l /usr/bin/java
lrwxrwxrwx 1 root root 22 /usr/bin/java -> /etc/alternatives/java
$ ls -l /etc/alternatives/java
lrwxrwxrwx 1 root root 43 /etc/alternatives/java -> /usr/lib/jvm/java-17-openjdk-amd64/bin/java
$ /usr/lib/jvm/java-17-openjdk-amd64/bin/java -version # same binary, real path
openjdk version "17.0.19" 2026-04-21
OpenJDK Runtime Environment (build 17.0.19+10-1-22.04.2-Ubuntu)
java's RPATH is [$ORIGIN:$ORIGIN/../lib] and libjli.so is present at
/usr/lib/jvm/java-17-openjdk-amd64/lib/libjli.so.
LD_DEBUG=libs on the failing invocation shows the wrong expansion:
search path=/etc/alternatives/glibc-hwcaps/x86-64-v3:...:/etc/alternatives:
/etc/alternatives/../lib/...:/etc/alternatives/../lib (RPATH from file java)
$ORIGIN became /etc/alternatives, so $ORIGIN/../lib resolved to /etc/lib.
LD_DEBUG=libs on the working direct invocation expands it correctly:
search path=/usr/lib/jvm/java-17-openjdk-amd64/bin:
/usr/lib/jvm/java-17-openjdk-amd64/bin/../lib (RPATH from file /usr/lib/jvm/java-17-openjdk-amd64/bin/java)
trying file=/usr/lib/jvm/java-17-openjdk-amd64/bin/../lib/libjli.so
calling init: /usr/lib/jvm/java-17-openjdk-amd64/bin/../lib/libjli.so
Note also the differing provenance string: (RPATH from file java) in the
failing case versus the full path in the working case, suggesting the guest
loader is handed a partially-resolved path.
Environment
- FEX tag
FEX-2608 (commit e869aa644a16e4332cdc15c1ea0b4d13d482385d), built
from source, RelWithDebInfo with -O3, TUNE_ARCH=generic, TUNE_CPU=none,
vixl simulator/disassembler, Zydis, jemalloc-glibc and LTO all disabled.
- Guest userspace: Ubuntu 22.04.5 x86-64 rootfs, executed inside a Podman
container with FEX registered through binfmt_misc.
Summary
When a guest executable is invoked through two or more levels of symlink,
FEX expands
$ORIGINin itsDT_RPATH/DT_RUNPATHto the directory of theintermediate symlink rather than the directory of the fully-resolved binary.
Libraries referenced relative to
$ORIGINthen fail to load.One level of symlink works. The real path works. Only chains of two or more
break.
Minimal reproducer
Actual behaviour (under FEX)
Expected behaviour
All three print
shared library loaded. They do on native x86-64: glibc derives$ORIGINfrom the fully-resolved executable path, so the number of symlink hopsis irrelevant.
Real-world impact: the JDK
This is how it shows up in practice on an Ubuntu 22.04 x86-64 rootfs:
java's RPATH is[$ORIGIN:$ORIGIN/../lib]andlibjli.sois present at/usr/lib/jvm/java-17-openjdk-amd64/lib/libjli.so.LD_DEBUG=libson the failing invocation shows the wrong expansion:$ORIGINbecame/etc/alternatives, so$ORIGIN/../libresolved to/etc/lib.LD_DEBUG=libson the working direct invocation expands it correctly:Note also the differing provenance string:
(RPATH from file java)in thefailing case versus the full path in the working case, suggesting the guest
loader is handed a partially-resolved path.
Environment
FEX-2608(commite869aa644a16e4332cdc15c1ea0b4d13d482385d), builtfrom source,
RelWithDebInfowith-O3,TUNE_ARCH=generic,TUNE_CPU=none,vixl simulator/disassembler, Zydis, jemalloc-glibc and LTO all disabled.
container with FEX registered through
binfmt_misc.