feat: add FreeBSD support by jaeyoon-choi · Pull Request #2 · xnvme/devbind

jaeyoon-choi · 2026-07-30T07:08:43Z

Summary

Add FreeBSD support to devbind. A platform implementation picked at runtime
maps the same CLI onto each platform's native tooling, so one command
prepares an NVMe device for DPDK/SPDK and xNVMe on both Linux and FreeBSD.
Verified end-to-end on FreeBSD 15.1 and 14.4 VMs with an emulated NVMe
device, including a real nic_uio bind with the dpdk kmod installed, and
re-verified on Linux the same way. The branch is structured for per-commit
review; every commit passes the test suite on its own.

DPDK/SPDK and xNVMe support FreeBSD, but preparing a device there — loading
nic_uio, running devctl detach, enabling bus-mastering by hand — was
manual. devbind covered only Linux, with the sysfs paths hardcoded in the
tool.

The Linux code now sits behind a small Platform interface, and a FreeBSD
implementation joins it:

operation	Linux	FreeBSD
enumerate / inspect	`lspci`, sysfs	`pciconf -l`, `camcontrol`
unbind	sysfs `driver/unbind`	`devctl detach`
bind (kernel driver)	sysfs `drivers/<drv>/bind`	`devctl set driver`
bind (user space framework)	sysfs `driver_override` + `bind`	`hw.nic_uio.bdfs` + `kldload`
in-use check	`lsof`	`fstat` + `camcontrol` (nvmeX → ndaY)
user space driver	`vfio-pci`, `uio_pci_generic`	`nic_uio`
bus-mastering	`setpci`	`pciconf -w`

FreeBSD has no VFIO, so nic_uio (the uio_pci_generic analogue) is the
user space path there, and iommugroup reports None. Naming a driver the
running platform does not have is rejected before the device is touched.

The five commits are meant to be read one at a time. A fix for the 0.3.10
--device lookup comes first: it read an unmapped key and matched nothing.
Two refactors follow. One decouples scanning and binding from argparse. The
other moves the Linux code onto a Linux class behind the Platform
interface — a pure move, easiest to read with --color-moved, and output on
Linux is unchanged. A small fix then turns platform errors into one-line
messages instead of tracebacks. The feat commit at the top adds the FreeBSD
platform, unit tests for both platforms, and the docs.

Two notes for review:

pciconf -l output differs by release. 14.3 and earlier print packed
chip=/card= fields. 15.x and 14.4 print split vendor=/device=
fields. Both are parsed, and the unit fixtures cover both.
nic_uio cannot be attached with devctl set driver. It only probes
devices listed in hw.nic_uio.bdfs, so binding registers the device
there, drops any pinned driver, and reloads the module, which claims the
device by itself. The attach is verified through pciconf -l before the
command register is written.

karlowich · 2026-08-04T11:02:59Z

This brings us halfway to drop xnvme-driver (SPDK's setup.py) on FreeBSD.
We also need to handle the contigmem part (DPDK's solution to the missing hugepages on FreeBSD):

From SPDK setup.py:

function _configure_freebsd() {
	local freebsd_bufsz=${FREEBSD_BUFSZ:-256}

	if ! check_for_driver_freebsd; then
		echo "DPDK drivers (contigmem and/or nic_uio) are missing, aborting" >&2
		return 1
	fi
	configure_freebsd_pci "$@"
	if [[ $SKIP_HUGE == yes ]]; then
		# Do nothing as requested
		return 0
	fi
	# If contigmem is already loaded but the HUGEMEM specified doesn't match the
	#  previous value, unload contigmem so that we can reload with the new value.
	if kldstat -q -m contigmem; then
		# contigmem may be loaded, but the kernel environment doesn't have to
		# be necessarily set at this point. If it isn't, kenv will fail to
		# pick up the hw. options. Handle it.
		if ! contigmem_num_buffers=$(kenv hw.contigmem.num_buffers); then
			contigmem_num_buffers=-1
		fi 2> /dev/null
		if ((contigmem_num_buffers != HUGEMEM / freebsd_bufsz)); then
			kldunload contigmem.ko
		fi
	fi
	if ! kldstat -q -m contigmem; then
		kenv hw.contigmem.num_buffers=$((HUGEMEM / freebsd_bufsz))
		kenv hw.contigmem.buffer_size=$((freebsd_bufsz * 1024 * 1024))
		kldload contigmem.ko
	fi
}

@safl Do you think we should put the contigmem part in devbind too? Or perhaps it needs a separate tool and repo.

karlowich

Great.
I propose adding the test_linux.py in a later commit, to avoid changing the test in 3 consecutive commits.

Also see the few inline comments.

karlowich · 2026-08-04T11:10:07Z

+    handles: list = field(default_factory=list)
+
+
+class Backend(abc.ABC):


I think this should be called platform instead. That puts it more in line with the doc-string: "Platform-specific PCI device-driver binding operations"

Renamed to Platform, and get_backend() became get_platform() to match.

karlowich · 2026-08-04T11:11:05Z

+    DRIVERS: set = set()
+
+    def driver_names(self) -> set:
+        """Return the set of driver-names this backend can bind to"""


Let's rename backend to platform here too.

Done. backend is now platform throughout the docstrings and comments too.

karlowich · 2026-08-04T11:11:59Z


-class System:
-    DRIVERS = {"nvme", "vfio-pci", "vfio-noiommu", "uio_pci_generic"}
+class LinuxBackend(Backend):


if we rename from backend to platform, then this class could just be named Linux

Renamed to Linux.

karlowich · 2026-08-04T11:13:17Z


-def unbind(device: Device):
-    log.info(f"Unbinding({device.bdf}) from '{device.driver}'")
+def get_backend() -> Backend:


This should be renamed

Renamed to get_platform().

safl · 2026-08-04T16:02:35Z

This brings us halfway to drop xnvme-driver (SPDK's setup.py) on FreeBSD. We also need to handle the contigmem part (DPDK's solution to the missing hugepages on FreeBSD):

From SPDK setup.py:

function _configure_freebsd() {
	local freebsd_bufsz=${FREEBSD_BUFSZ:-256}

	if ! check_for_driver_freebsd; then
		echo "DPDK drivers (contigmem and/or nic_uio) are missing, aborting" >&2
		return 1
	fi
	configure_freebsd_pci "$@"
	if [[ $SKIP_HUGE == yes ]]; then
		# Do nothing as requested
		return 0
	fi
	# If contigmem is already loaded but the HUGEMEM specified doesn't match the
	#  previous value, unload contigmem so that we can reload with the new value.
	if kldstat -q -m contigmem; then
		# contigmem may be loaded, but the kernel environment doesn't have to
		# be necessarily set at this point. If it isn't, kenv will fail to
		# pick up the hw. options. Handle it.
		if ! contigmem_num_buffers=$(kenv hw.contigmem.num_buffers); then
			contigmem_num_buffers=-1
		fi 2> /dev/null
		if ((contigmem_num_buffers != HUGEMEM / freebsd_bufsz)); then
			kldunload contigmem.ko
		fi
	fi
	if ! kldstat -q -m contigmem; then
		kenv hw.contigmem.num_buffers=$((HUGEMEM / freebsd_bufsz))
		kenv hw.contigmem.buffer_size=$((freebsd_bufsz * 1024 * 1024))
		kldload contigmem.ko
	fi
}

@safl Do you think we should put the contigmem part in devbind too? Or perhaps it needs a separate tool and repo.

Yeah, separate tool. Since, the main thing about the suite of tools: devbind, hugepages, iommu, etc. are that instead of a single monolithic setup.sh or xnvme-driver then the tools manage one specific task. If it were to be squeezed in anywhere, then it would be in the hugepages tool. The challenge of course is that currently then these tools are inherently Linux specific, however, the devbind tool should have equivalent functionality of scanning the pcie, changing driver association etc. but yeah... a dedicated "configmem" tool is preferable.

0.3.10 taught --device to bypass the class filter. The check reads props.get("bdf"), but the scan loop stores the address under "slot", the key lspci prints. The rename to bdf happens later, in Device.from_dict. The lookup always came back empty. --device matched nothing at all, not even NVMe devices that used to match. The command still exited 0, so the failure was silent. Read the slot key. A regression test for the bypass lands with the Linux test module in the platform split. Signed-off-by: Jaeyoon Choi <j_yoon.choi@samsung.com>

device_scan, bind, and unbind read the argparse namespace directly. That coupled them to the CLI. unbind never used args at all. bind took it only to pass it along. Tests could not call the helpers without building a fake args object. Take classcode, bdf, and driver_name as plain parameters instead. main() passes the parsed values at the call sites. This keeps the upcoming platform interface free of argparse. Behavior is unchanged. Signed-off-by: Jaeyoon Choi <j_yoon.choi@samsung.com>

jaeyoon-choi · 2026-08-10T08:37:55Z

Great. I propose adding the test_linux.py in a later commit, to avoid changing the test in 3 consecutive commits.

Also see the few inline comments.

Thanks for the review! Both points are addressed:

Renamed throughout: Backend → Platform, LinuxBackend → Linux,
FreeBsdBackend → FreeBSD, and get_backend() → get_platform().
tests/test_linux.py now lands once, in its final form, in the
platform-split commit, so it is no longer touched by three
consecutive commits.

jaeyoon-choi · 2026-08-10T08:41:33Z

This brings us halfway to drop xnvme-driver (SPDK's setup.py) on FreeBSD. We also need to handle the contigmem part (DPDK's solution to the missing hugepages on FreeBSD):
From SPDK setup.py:
function _configure_freebsd() {
	local freebsd_bufsz=${FREEBSD_BUFSZ:-256}

	if ! check_for_driver_freebsd; then
		echo "DPDK drivers (contigmem and/or nic_uio) are missing, aborting" >&2
		return 1
	fi
	configure_freebsd_pci "$@"
	if [[ $SKIP_HUGE == yes ]]; then
		# Do nothing as requested
		return 0
	fi
	# If contigmem is already loaded but the HUGEMEM specified doesn't match the
	#  previous value, unload contigmem so that we can reload with the new value.
	if kldstat -q -m contigmem; then
		# contigmem may be loaded, but the kernel environment doesn't have to
		# be necessarily set at this point. If it isn't, kenv will fail to
		# pick up the hw. options. Handle it.
		if ! contigmem_num_buffers=$(kenv hw.contigmem.num_buffers); then
			contigmem_num_buffers=-1
		fi 2> /dev/null
		if ((contigmem_num_buffers != HUGEMEM / freebsd_bufsz)); then
			kldunload contigmem.ko
		fi
	fi
	if ! kldstat -q -m contigmem; then
		kenv hw.contigmem.num_buffers=$((HUGEMEM / freebsd_bufsz))
		kenv hw.contigmem.buffer_size=$((freebsd_bufsz * 1024 * 1024))
		kldload contigmem.ko
	fi
}
@safl Do you think we should put the contigmem part in devbind too? Or perhaps it needs a separate tool and repo.
Yeah, separate tool. Since, the main thing about the suite of tools: devbind, hugepages, iommu, etc. are that instead of a single monolithic setup.sh or xnvme-driver then the tools manage one specific task. If it were to be squeezed in anywhere, then it would be in the hugepages tool. The challenge of course is that currently then these tools are inherently Linux specific, however, the devbind tool should have equivalent functionality of scanning the pcie, changing driver association etc. but yeah... a dedicated "configmem" tool is preferable.

I will integrate contigmem into hugepages.

jaeyoon-choi · 2026-08-10T08:43:18Z

Karl, Could you please add Simon to reviewer?

karlowich · 2026-08-10T08:55:45Z

@jaeyoon-choi I don't have the rights on this repo to do that, but I will let him know.

karlowich

Great!

karlowich · 2026-08-11T13:14:22Z

@jaeyoon-choi Looks like the tests are failing on formatting. You need to run make format to fix the issues.

Prepare for adding other platforms. The module-level scan, bind, unbind, and probe helpers move onto a Linux class behind a small abstract Platform interface. get_platform() picks the implementation for the running platform. System.drivers/limits become instance state filled in from the platform. The memlock remediation text becomes a platform hint, assembled into the same warning. The parse-time driver vocabulary splits off as KNOWN_DRIVERS, and Device.MANDATORY_KEYS becomes _LSPCI_KEYS next to the lspci parser. The tool and sysfs notes from the file header move onto the Linux class as its docstring. Output and behavior on Linux are unchanged. scan_devices() keeps the --device bypass of 0.3.10. Add the Linux unit tests: lspci parsing and filtering, the --device bypass, the driver_override -> bind -> setpci sequence, and platform selection. Signed-off-by: Jaeyoon Choi <j_yoon.choi@samsung.com>

safl

Reviewed the whole branch at 351486b against main. The nic_uio bind reads correctly now: registering the device in hw.nic_uio.bdfs, dropping any pinned driver, reloading the module, and verifying the attach through pciconf -l before writing the command register is the right shape, and the new tests cover it. On Linux here, --list output matches main and the --device fix in 3db4842 is confirmed: main prints no props for a live BDF, this branch prints one. ruff format --check and ruff check pass with the pinned 0.11.4, and the suite passes on 3.13.

Two things worth a look, both inline. Separately, the description above still describes the design as Backend/LinuxBackend, which the code no longer uses, and its table has the same staleness as the README one.

I could not exercise any FreeBSD path. The kenv plus kldunload/kldload sequence, devctl clear driver -f, and the CAM mapping rest on the VM testing you describe.

safl · 2026-08-13T08:26:24Z

    system = System()
-    system.probe_drivers()
-    system.probe_limits()
+    system.drivers = platform.probe_drivers()
+    system.probe_limits(platform.memlock_remediation_hint())

    if args.list:
        system.pp()

-    devices = list(device_scan(args))
+    devices = list(platform.scan_devices(args.classcode, args.device))

    try:


probe_drivers() and list(scan_devices(...)) sit outside the try below, so the two calls most likely to fail on a machine without PCI sysfs still end in a traceback rather than the one-line error 01c31f9 introduces.

Reproduced on Linux by making Path.resolve raise FileNotFoundError for /sys/bus/pci/drivers:

File "src/devbind/devbind.py", line 165, in probe_drivers (path.name for path in Path("/sys/bus/pci/drivers").resolve(strict=True).glob("*")) FileNotFoundError: [Errno 2] No such file or directory: '/sys/bus/pci/drivers'

Moving both calls inside the existing try covers it.

safl · 2026-08-13T08:26:24Z

+| enumerate / inspect | `lspci`, sysfs | `pciconf -l`, `camcontrol` |
+| user space framework | `vfio-pci`, `uio_pci_generic` | `nic_uio` |
+| unbind | sysfs `driver/unbind` | `devctl detach` |
+| bind | sysfs `drivers/<drv>/bind` | `devctl set driver` |


This is now wrong for the case the change is actually about. nic_uio binds through hw.nic_uio.bdfs plus kldload; only kernel drivers go through devctl set driver. The commit message for 351486b explains it correctly, so it is just the table that is stale.

An unsupported platform, a --bind name the platform does not have, and a failing probe, scan, bind, or unbind all ended in a Python traceback. The driver-name case gets worse once more platforms exist. A name owned by another platform would reach the device before failing. Catch the platform errors in main() and exit with a one-line error. The probe and the scan run inside that same block, so a machine without PCI sysfs gets the one-line error too. Validate --bind against the active platform's driver_names() before touching the device. Signed-off-by: Jaeyoon Choi <j_yoon.choi@samsung.com>

Implement the Platform interface with FreeBSD's native tools: - pciconf -l enumerates devices. Both output formats are parsed: the packed chip=/card= fields of 14.3 and earlier, and the split vendor=/device= fields of 15.x and 14.4. - devctl detach / devctl set driver implement unbind and bind for kernel drivers. nic_uio cannot be attached that way: it only probes devices listed in the hw.nic_uio.bdfs tunable. Binding to nic_uio registers the device there, drops any driver pinned by an earlier set driver, and (re)loads the module, which detaches the previous driver and claims the device by itself. Binding back to a kernel driver removes the bdfs entry again. - camcontrol devlist maps an nvmeX controller to its CAM disk (ndaY), so the fstat in-use check also sees users of the disk device. - kldstat reports whether nic_uio is loaded. - scan_devices() honors the --device class-filter bypass, matching the Linux platform. - pciconf -w enables bus-mastering once the nic_uio attach is verified through pciconf -l. A failed write is reported instead of ignored. Helpers convert between the pciX:B:S:F selector and the domain:bus:device.function bdf form. nic_uio is added to KNOWN_DRIVERS, completion, and the docs. User-facing text drops its Linux assumptions: the --bind help says driver file instead of .ko file, and the memlock warning says DMA mapping instead of VFIO_IOMMU_MAP_DMA. Unit tests cover both pciconf formats, the fstat heuristic, kldstat probing, camcontrol parsing, the nic_uio bdfs flow, and the devctl bind for kernel drivers. Signed-off-by: Jaeyoon Choi <j_yoon.choi@samsung.com>

jaeyoon-choi · 2026-08-14T01:34:13Z

Reviewed the whole branch at 351486b against main. The nic_uio bind reads correctly now: registering the device in hw.nic_uio.bdfs, dropping any pinned driver, reloading the module, and verifying the attach through pciconf -l before writing the command register is the right shape, and the new tests cover it. On Linux here, --list output matches main and the --device fix in 3db4842 is confirmed: main prints no props for a live BDF, this branch prints one. ruff format --check and ruff check pass with the pinned 0.11.4, and the suite passes on 3.13.

Two things worth a look, both inline. Separately, the description above still describes the design as Backend/LinuxBackend, which the code no longer uses, and its table has the same staleness as the README one.

I could not exercise any FreeBSD path. The kenv plus kldunload/kldload sequence, devctl clear driver -f, and the CAM mapping rest on the VM testing you describe.

Thank you for the review!

karlowich mentioned this pull request Aug 4, 2026

fix: read PCI address from 'slot' field in device_scan #1

Closed

karlowich suggested changes Aug 4, 2026

View reviewed changes

jaeyoon-choi added 2 commits August 10, 2026 15:55

jaeyoon-choi force-pushed the freebsd-support branch 2 times, most recently from 24c1e7e to 5e8f0dd Compare August 10, 2026 06:59

karlowich approved these changes Aug 10, 2026

View reviewed changes

jaeyoon-choi force-pushed the freebsd-support branch 3 times, most recently from 510b10b to 351486b Compare August 13, 2026 06:03

safl reviewed Aug 13, 2026

View reviewed changes

jaeyoon-choi added 2 commits August 14, 2026 10:13

jaeyoon-choi force-pushed the freebsd-support branch from 351486b to 3179a33 Compare August 14, 2026 01:15

		handles: list = field(default_factory=list)


		class Backend(abc.ABC):

Uh oh!

Conversation

jaeyoon-choi commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

karlowich commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

karlowich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

safl commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jaeyoon-choi commented Aug 10, 2026

Uh oh!

jaeyoon-choi commented Aug 10, 2026

Uh oh!

jaeyoon-choi commented Aug 10, 2026

Uh oh!

karlowich commented Aug 10, 2026

Uh oh!

karlowich left a comment

Choose a reason for hiding this comment

Uh oh!

karlowich commented Aug 11, 2026

Uh oh!

safl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaeyoon-choi commented Aug 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

jaeyoon-choi commented Jul 30, 2026 •

edited

Loading

karlowich commented Aug 4, 2026 •

edited

Loading

safl commented Aug 4, 2026 •

edited

Loading