R1 DMA drill: reproduce the root-node DMA panic, fix the /devices snapshot race #3

Merged

forge merged 5 commits from r1-dma-drill into main 2026-09-28 18:11:50 +00:00

Conversation 0 Commits 5 Files changed 16 +421 -56

forge commented 2026-09-28 13:52:54 +00:00

Contributor

Copy link

Stacked on #2 (r1-gem). This adds the DMA drill, a test aid that reproduces on purpose the kernel panic from R1 trial 2 so that a crash dump could be taken, and fixes a race in run-r1-test.sh that the runs exposed.

Changes

• drm::Driver::gem_dma lets a driver choose at attach time how GEM object memory is bound for DMA. The default is DMA_ATTR against the driver's own node. GemDma::against names another node; it is unsafe and documented as drill-only.
• rdrmnull reads the string property dma-mode from its .conf:
  • none: the default and the current behaviour
  • self: bind against its own node; needs a parent nexus that does DMA (tools/test/rdrmnull-dma-self.conf, parent="/")
  • root-node: allocate the handles against ddi_root_node() from the pseudo node (tools/test/rdrmnull-dma-root-node.conf)
• run-r1-test.sh installs the .conf named by RDRMNULL_CONF. The R1 test expects bound objects when the configuration asks for DMA.
• run-r1-test.sh now takes the /devices snapshot after devfsadm -C. devfs drops the clone@0 nodes of idle drivers (bridge, e1000g, llc1, logindmux, ptm) about 30 s after boot, and the devfsadm -C at the end of the script recreates them. The first run on a freshly booted VM therefore failed the /devices comparison with five entries unrelated to rdrmnull.

Runs

The runs used disposable KVM guests: OpenIndiana Hipster 2025.10 updated to osnet 23197, which is illumos-cda5b9a49f, the same kernel as oibldofgate. Each guest had a 2 GiB dump zvol and a 2 GiB swap zvol. xtask kernel-check passed, with mdb answers identical to oibldofgate.

Run	R0 PASS	R1 PASS	32-bit probes	Script checks	FAIL
r1-gem c4a8512, first run in a fresh VM	105	153	5 + 4	4 of 5	1: the /devices race above
c4a8512, repeat in the same VM	105	153	5 + 4	5	0
c4a8512, second fresh VM	105	153	5 + 4	5	0
drill c80b75c, none	105	153	5 + 4	5	0 on repeat (first run: the same race)
drill b2430d0 (this head), none, first run in a fresh VM	105	153	5 + 4	5	0
drill self (root nexus parent)	105	153	5 + 4	4 of 5	1: the known /etc/path_to_inst residue

In the self run, the 1,228,800-byte dumb buffer had 300 cookies, one per page, summing exactly. Four 16 MiB objects had 14,639 cookies. All kstats and vmem arenas returned to zero. The instance entry "/rdrmnull@0,0" 0 "rdrmnull" stays in /etc/path_to_inst after rem_drv -C, after devfsadm -C and across a reboot.

With self on a pseudo node, every CREATE_DUMB that needs memory fails, because the pseudo nexus returns DDI_DMA_BADATTR. There was no leak and no panic.

The panic

The root-node run reproduced the panic on the first attempt, at the first CREATE_DUMB that allocates memory (100x1, bpp 12):

BAD TRAP: type=e (#pf Page fault) rp=fffffe000861c890 addr=0 occurred in module "<unknown>" due to a NULL pointer dereference
  0 ()   <- ddi_dma_addr_bind_handle+0xcc (frame fffffe000861ca40)
  rdrmnull: drm::gem::SysMem::new+0x158
  rdrmnull: ddi::module::glue::ioctl
  genunix: cdev_ioctl, spec_ioctl, fop_ioctl, ioctl

mdb on the dump established the mechanism:

• ddi_dma_alloc_handle(root) uses the root's own bus_ops, because the root node is rootnex's node, and rootnex_dma_allochdl(root, root) hands out a handle with dmai_rdip set to the root.
• ddi_dma_addr_bind_handle special-cases the root for the nexus hop, but it takes the function from DEVI(rdip)->devi_bus_dma_bindfunc.
• ddi_optimize_dtree() fills that field only for nodes with a parent, so for the root node it is 0 and the call goes to address 0.
• ddi_dma_buf_bind_handle and ddi_dma_unbind_handle have the same gap. ddi_dma_bindhdl() handles the root correctly.

Judgements:

• (a) This is a driver bug. ddi_dma_alloc_handle(9F) requires "the device's dev_info structure".
• (b) The kernel could have returned an error instead of calling through NULL. This is worth a low-priority illumos issue.
• (c) illumos-ddi already needs unsafe DevInfo::from_raw to reach the root node. The proposal is a runtime guard in DmaHandle::alloc (refuse a node without a parent) and a non-Copy own-node token for DMA.

The full analysis, with the mdb transcript and the reproduction recipe, is in the workspace note notes/illumos-drm-dma-panic-20260928.md.

The drill hook (GemDma::against) exists only to reproduce this. If this PR is merged, consider putting it behind a test-only feature.

Update 2026-09-28: the own-node rule in illumos-ddi and the panic-drill feature

illumos-ddi now makes this misuse impossible in safe Rust (illumos/illumos-ddi#2, branch dma-own-node, stacked on #1). This branch pins it at b24c307298. The later commit on that branch, dba0697, changes docs only.

API

• ddi::Driver::attach receives a ddi::OwnNode, the non-Copy, non-Clone capability for the driver's own node.
• DmaHandle::new and try_new take &OwnNode.
• illumos-ddi refuses the root node and other drivers' nodes with DmaError::NotOwnNode (ENXIO) before calling the kernel.

This PR (3a1d1f2, 62710d9, 51939da)

• drm::Device::attach(node: OwnNode, driver, ctx) hands the node to GemCore, which binds object memory against it. SysMem::new takes &OwnNode. DMA errors map through From<DmaError> for Errno.
• GemDma::against now exists only with the test-only cargo feature panic-drill of drm and rdrmnull, which is off by default. It allocates with raw ddi_dma_alloc_handle(9F), because illumos-ddi's guard would refuse the root node.
  • A default rdrmnull refuses dma-mode="root-node" at attach with WARNING: rdrmnull0: dma-mode "root-node" panics the system on purpose and needs rdrmnull built with the panic-drill feature; this build refuses it.
  • The default module has no reference to ddi_root_node.
• The drill module is built with cargo xtask build rdrmnull --features panic-drill. xtask build gained --features.
• CI runs clippy with --features rdrmnull/panic-drill so that the drill keeps compiling.
• The README's DMA section describes all of this. This answers the note at the end of this description: the hook is now behind a test-only feature.

Tests. Disposable Hephaistos guests from the golden overlay, illumos cda5b9a49f, 2 vCPUs, 3 GiB. Logs are in the workspace under logs/hephaistos/illumos-drm/20260928-ddi-own-node-*.

Run	Build	Result
dma-mode none	default	R0 105, probe32 5, R1 153, probe32 4 PASS; no FAIL; fail=0
dma-mode="self", root nexus parent	default	the same counts; 300 cookies for the 1,228,800-byte dumb buffer; kstats back to 0; the only FAIL is the known /etc/path_to_inst residue
root-node	default	refused at attach with the warning above; no node, no panic
root-node	panic-drill	R0 105 PASS, then the same panic as in the note at the first binding CREATE_DUMB; savecore wrote vmdump.0 (118 MB, FMA SUNOS-8000-KL)

In the panic-drill run, the panic string is BAD TRAP: type=e (#pf Page fault) ... addr=0 occurred in module "<unknown>" due to a NULL pointer dereference, with pc=0 and rdi=rsi=fffffe04e7db3c28, the root node. The caller frame is SysMem::alloc, the drill path. The dump was deleted with the guest, because the note's dump already covers it.

On the host (macOS, Rust 1.97.1), check and clippy -D warnings pass with and without the feature, and so do fmt, doc -D warnings, cargo test -p drm -p drm-uapi (42 + 1 passed) and uapi-check.

Stacked on #2 (`r1-gem`). This adds the DMA drill, a test aid that reproduces on purpose the kernel panic from R1 trial 2 so that a crash dump could be taken, and fixes a race in `run-r1-test.sh` that the runs exposed. ## Changes - `drm::Driver::gem_dma` lets a driver choose at attach time how GEM object memory is bound for DMA. The default is `DMA_ATTR` against the driver's own node. `GemDma::against` names another node; it is `unsafe` and documented as drill-only. - `rdrmnull` reads the string property `dma-mode` from its `.conf`: - `none`: the default and the current behaviour - `self`: bind against its own node; needs a parent nexus that does DMA (`tools/test/rdrmnull-dma-self.conf`, `parent="/"`) - `root-node`: allocate the handles against `ddi_root_node()` from the pseudo node (`tools/test/rdrmnull-dma-root-node.conf`) - `run-r1-test.sh` installs the `.conf` named by `RDRMNULL_CONF`. The R1 test expects bound objects when the configuration asks for DMA. - `run-r1-test.sh` now takes the `/devices` snapshot after `devfsadm -C`. devfs drops the `clone@0` nodes of idle drivers (bridge, e1000g, llc1, logindmux, ptm) about 30 s after boot, and the `devfsadm -C` at the end of the script recreates them. The first run on a freshly booted VM therefore failed the `/devices` comparison with five entries unrelated to rdrmnull. ## Runs The runs used disposable KVM guests: OpenIndiana Hipster 2025.10 updated to osnet 23197, which is `illumos-cda5b9a49f`, the same kernel as oibldofgate. Each guest had a 2 GiB dump zvol and a 2 GiB swap zvol. `xtask kernel-check` passed, with mdb answers identical to oibldofgate. | Run | R0 PASS | R1 PASS | 32-bit probes | Script checks | FAIL | | --- | --- | --- | --- | --- | --- | | `r1-gem` c4a8512, first run in a fresh VM | 105 | 153 | 5 + 4 | 4 of 5 | 1: the `/devices` race above | | c4a8512, repeat in the same VM | 105 | 153 | 5 + 4 | 5 | 0 | | c4a8512, second fresh VM | 105 | 153 | 5 + 4 | 5 | 0 | | drill c80b75c, `none` | 105 | 153 | 5 + 4 | 5 | 0 on repeat (first run: the same race) | | drill b2430d0 (this head), `none`, first run in a fresh VM | 105 | 153 | 5 + 4 | 5 | 0 | | drill `self` (root nexus parent) | 105 | 153 | 5 + 4 | 4 of 5 | 1: the known `/etc/path_to_inst` residue | In the `self` run, the 1,228,800-byte dumb buffer had 300 cookies, one per page, summing exactly. Four 16 MiB objects had 14,639 cookies. All kstats and vmem arenas returned to zero. The instance entry `"/rdrmnull@0,0" 0 "rdrmnull"` stays in `/etc/path_to_inst` after `rem_drv -C`, after `devfsadm -C` and across a reboot. With `self` on a pseudo node, every `CREATE_DUMB` that needs memory fails, because the pseudo nexus returns `DDI_DMA_BADATTR`. There was no leak and no panic. ## The panic The `root-node` run reproduced the panic on the first attempt, at the first `CREATE_DUMB` that allocates memory (`100x1`, bpp 12): ``` BAD TRAP: type=e (#pf Page fault) rp=fffffe000861c890 addr=0 occurred in module "<unknown>" due to a NULL pointer dereference 0 () <- ddi_dma_addr_bind_handle+0xcc (frame fffffe000861ca40) rdrmnull: drm::gem::SysMem::new+0x158 rdrmnull: ddi::module::glue::ioctl genunix: cdev_ioctl, spec_ioctl, fop_ioctl, ioctl ``` mdb on the dump established the mechanism: - `ddi_dma_alloc_handle(root)` uses the root's own bus_ops, because the root node is rootnex's node, and `rootnex_dma_allochdl(root, root)` hands out a handle with `dmai_rdip` set to the root. - `ddi_dma_addr_bind_handle` special-cases the root for the nexus hop, but it takes the function from `DEVI(rdip)->devi_bus_dma_bindfunc`. - `ddi_optimize_dtree()` fills that field only for nodes with a parent, so for the root node it is 0 and the call goes to address 0. - `ddi_dma_buf_bind_handle` and `ddi_dma_unbind_handle` have the same gap. `ddi_dma_bindhdl()` handles the root correctly. Judgements: - (a) This is a driver bug. `ddi_dma_alloc_handle(9F)` requires "the device's dev_info structure". - (b) The kernel could have returned an error instead of calling through NULL. This is worth a low-priority illumos issue. - (c) illumos-ddi already needs `unsafe DevInfo::from_raw` to reach the root node. The proposal is a runtime guard in `DmaHandle::alloc` (refuse a node without a parent) and a non-`Copy` own-node token for DMA. The full analysis, with the mdb transcript and the reproduction recipe, is in the workspace note `notes/illumos-drm-dma-panic-20260928.md`. The drill hook (`GemDma::against`) exists only to reproduce this. If this PR is merged, consider putting it behind a test-only feature. ## Update 2026-09-28: the own-node rule in illumos-ddi and the `panic-drill` feature illumos-ddi now makes this misuse impossible in safe Rust (illumos/illumos-ddi#2, branch `dma-own-node`, stacked on #1). This branch pins it at `b24c307298`. The later commit on that branch, `dba0697`, changes docs only. **API** - `ddi::Driver::attach` receives a `ddi::OwnNode`, the non-`Copy`, non-`Clone` capability for the driver's own node. - `DmaHandle::new` and `try_new` take `&OwnNode`. - illumos-ddi refuses the root node and other drivers' nodes with `DmaError::NotOwnNode` (`ENXIO`) before calling the kernel. **This PR** (`3a1d1f2`, `62710d9`, `51939da`) - `drm::Device::attach(node: OwnNode, driver, ctx)` hands the node to `GemCore`, which binds object memory against it. `SysMem::new` takes `&OwnNode`. DMA errors map through `From<DmaError> for Errno`. - `GemDma::against` now exists only with the test-only cargo feature `panic-drill` of `drm` and `rdrmnull`, which is off by default. It allocates with raw `ddi_dma_alloc_handle(9F)`, because illumos-ddi's guard would refuse the root node. - A default `rdrmnull` refuses `dma-mode="root-node"` at attach with `WARNING: rdrmnull0: dma-mode "root-node" panics the system on purpose and needs rdrmnull built with the panic-drill feature; this build refuses it`. - The default module has no reference to `ddi_root_node`. - The drill module is built with `cargo xtask build rdrmnull --features panic-drill`. `xtask build` gained `--features`. - CI runs clippy with `--features rdrmnull/panic-drill` so that the drill keeps compiling. - The README's DMA section describes all of this. This answers the note at the end of this description: the hook is now behind a test-only feature. **Tests.** Disposable Hephaistos guests from the golden overlay, illumos `cda5b9a49f`, 2 vCPUs, 3 GiB. Logs are in the workspace under `logs/hephaistos/illumos-drm/20260928-ddi-own-node-*`. | Run | Build | Result | | --- | --- | --- | | `dma-mode` none | default | R0 105, probe32 5, R1 153, probe32 4 PASS; no FAIL; `fail=0` | | `dma-mode="self"`, root nexus parent | default | the same counts; 300 cookies for the 1,228,800-byte dumb buffer; kstats back to 0; the only FAIL is the known `/etc/path_to_inst` residue | | `root-node` | default | refused at attach with the warning above; no node, no panic | | `root-node` | `panic-drill` | R0 105 PASS, then the same panic as in the note at the first binding `CREATE_DUMB`; savecore wrote `vmdump.0` (118 MB, FMA `SUNOS-8000-KL`) | In the `panic-drill` run, the panic string is `BAD TRAP: type=e (#pf Page fault) ... addr=0 occurred in module "<unknown>" due to a NULL pointer dereference`, with `pc=0` and `rdi=rsi=fffffe04e7db3c28`, the root node. The caller frame is `SysMem::alloc`, the drill path. The dump was deleted with the guest, because the note's dump already covers it. On the host (macOS, Rust 1.97.1), `check` and `clippy -D warnings` pass with and without the feature, and so do `fmt`, `doc -D warnings`, `cargo test -p drm -p drm-uapi` (42 + 1 passed) and `uapi-check`.

forge added 2 commits 2026-09-28 13:52:54 +00:00

Add a DMA drill to rdrmnull: bind against its own node or the root node ... c80b75cdc9

Driver::gem_dma lets a driver choose at attach time how GEM object memory
is bound for DMA: DMA_ATTR against its own node by default. GemDma::against
names another node; it is unsafe and exists only for the drill.

rdrmnull reads the string property dma-mode from its .conf: none (the
default, no DMA as before), self (bind against its own node, which needs a
DMA-capable parent: tools/test/rdrmnull-dma-self.conf attaches it under the
root nexus) and root-node (allocate the handles against ddi_root_node(9F)
from the pseudo node, which panicked an R1 trial;
tools/test/rdrmnull-dma-root-node.conf). run-r1-test.sh installs the file
named by RDRMNULL_CONF, and the R1 test expects bound objects when the
configuration asks for DMA.

Take the /devices snapshot of run-r1-test.sh after devfsadm -C ... b2430d00a1

devfs drops the clone@0 nodes of idle drivers (bridge, e1000g, llc1,
logindmux, ptm) some seconds after boot, and the devfsadm -C at the end of
the script recreates them, so the first run on a freshly booted VM failed
the /devices comparison with five entries unrelated to rdrmnull. Run
devfsadm -C before the first snapshot as well.

forge added 3 commits 2026-09-28 17:23:42 +00:00

Allocate GEM DMA handles from the attach capability ... 3a1d1f2d84

illumos-ddi now allocates DMA handles only from ddi::OwnNode, the
capability for the driver's own node that ddi::Driver::attach receives
(illumos-ddi dma-own-node, b24c307298; its docs/DESIGN.md section 5.12),
and refuses the root node and other drivers' nodes with
DmaError::NotOwnNode. Pin that revision and pass the capability through:

- drm::Device::attach takes the OwnNode and hands it to GemCore, which
  keeps it and binds object memory against it; SysMem::new takes
  &OwnNode. DMA errors map through illumos-ddi's From<DmaError> for Errno.
- GemDma::against, the drill hook that names another node, now exists
  only with the new test-only cargo feature panic-drill (drm and
  rdrmnull), off by default. It uses raw ddi_dma_alloc_handle(9F) calls,
  because illumos-ddi's guard would refuse the node.
- rdrmnull without the feature refuses dma-mode "root-node" at attach
  with a warning, and its module has no reference to ddi_root_node.

Tested on a Hephaistos guest (illumos cda5b9a49f): dma-mode none and
self give R0 105, R1 153 and probes 5 and 4 PASS with no test FAIL (self
leaves the known /etc/path_to_inst residue); the panic-drill build with
root-node still panics as on 2026-09-28 and savecore keeps the dump; the
default build refuses root-node at attach without a panic.

Pass --features through cargo xtask build ... 62710d9272

cargo xtask build rdrmnull --features panic-drill builds the DMA drill module; without the option the build is unchanged.

Describe the own-node rule and the panic-drill feature ... 51939da4a2

The README's DMA section explains that DMA handles come from the attach capability and that the root-node drill needs the test-only panic-drill feature. CI runs clippy with that feature so that the drill keeps compiling.

forge referenced this pull request from illumos/illumos-ddi 2026-09-28 17:24:37 +00:00

Allocate device resources only from the driver's own node (OwnNode) #2

forge changed target branch from r1-gem to main 2026-09-28 18:11:41 +00:00

forge merged commit 6b21460a4b into main 2026-09-28 18:11:50 +00:00

forge referenced this pull request from a commit 2026-09-28 18:11:51 +00:00

Merge pull request #3 from r1-dma-drill: DMA drill, OwnNode adaptation and the panic-drill feature

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

illumos/illumos-drm!3

No description provided.