illumos/illumos-ddi
Watch
2
0
Fork
You've already forked illumos-ddi
 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

Private tier (file descriptors for R2), segmap check, pollhead drop rule #5

Merged
toasterson merged 13 commits from r2-private-fd into main 2026-10-05 16:34:04 +00:00
Conversation 0 Commits 13 Files changed 22 +1338 -36
forge commented 2026-09-29 19:27:36 +00:00
Contributor
Copy link

The private tier planned in docs/DESIGN.md section 4 adds what milestone R2 of the Rust DRM core needs to hand kernel objects to user space as file descriptors (illumos-drm WI-454). This PR also adds two public-tier pieces and the fixes for two findings of that work.

Stacked on PR 4 (r3-kms-glue, R3's chpoll(9E) glue); the base of this PR is that branch. Merge PR 4 first. The illumos-drm R2 pull request pins this branch's head.

Reconciliation with R3

R2 had its own chpoll(9E) API. After the rebase onto r3-kms-glue it uses R3's (a !Unpin PollHead, Pin registrations, PollHeads, the glue deciding when to register) and drops its own. The two changes R2's review required are separate commits on top of R3's API:

  • POLLNVAL now passes through like POLLERR and POLLHUP (PollEvents::NVAL, PollEvents::ALWAYS).
  • A pollhead can only be dropped in thread context (panic-to-type rule):
    • A bare PollHead is now !Send. R3 had unsafe impl Send.
    • Heads that cross threads live in the new PollBox (a pinned KBox head that is Send/Sync, with new, try_new, head and release(&Sleepable)) or in a PollHeads array.
    • With the private tier, dropping either in interrupt context leaks the heads and logs a warning instead of calling pollhead_clean(9F) there; poll::leaked_heads counts them.
    • PollHead::registration_unchecked (unsafe) registers a head the instance does not own, such as a sync file's head.

What was built

  • crates/illumos-kernel-private-sys: pinned to gate cda5b9a49f. The crate is empty unless the feature unstable-gate-cda5b9a49f is enabled. It binds:

    • file_t, falloc, unfalloc, setf, getf, releasef, f_setfd_or (os/fio.c, sys/file.h)
    • the public prefix of vnode_t, caller_context_t, fop_open, fop_close, vn_copypath, vn_rele (sys/vnode.h)
    • makespecvp, specfind (sys/fs/snode.h, module fs/specfs)
    • servicing_interrupt (sys/disp.h)

    ufalloc is left unbound on purpose: falloc calls it, and nothing needs a descriptor without a file.

  • Gate citations instead of manual pages: the first doc line of every item is `name`. Private, `path:line`.. xtask lint-man --gate checks that the cited line exists and names the item, and keeps the crate out of the manual-page checks. Result: 19 citations, 0 errors against a checkout at 0fb1d5d8d3, which has not changed any cited file since cda5b9a49f.

  • ddi::private::fd (feature unstable-gate-cda5b9a49f):

    • NewFd::open gives the calling process a reserved descriptor for one of the calling driver's own minor numbers: makespecvp, the caller's path through specfind + vn_copypath, VOP_OPEN, falloc.
    • install uses setf and f_setfd_or; Drop undoes everything.
    • FileRef holds a descriptor (getf/releasef) and names the driver's minor if the descriptor is one of them.
    • Both need the IoctlArg of the ioctl being served, and both refuse LDI callers and kernel threads.

  • segmap(9E) check: ops: [char, devmap] installs glue that runs CharDevice::segmap_check(MapRequest) before ddi_devmap_segmap(9F). A mapping that devmap_map(9E) refuses would otherwise stay in the process until it exits, because devmap_setup does not unmap it.

  • IoctlArg records the device number of the ioctl; only the glue can set it.

  • rselftest gains the feature private-gate and the ioctl RSELFTEST_IOC_ANONFD, with 6 descriptor checks and 2 poll-head guard checks. CI builds it with -N fs/specfs and runs clippy with the feature.

Panics made impossible (docs/DESIGN.md section 5.12)

  1. 2026-09-29: NULL dereference at segdev_create+0xe4 on the first mmap(2) of a dma-buf descriptor.
    • Cause: the makespecvp vnode had no path. segdev_create calls specfind, which only returns vnodes whose path is visible in the caller's zone, and does not check the result.
    • Type rule: only the glue sets the device number in IoctlArg (compile_fail,E0624 doctest), and NewFd::open copies the caller's file path.
    • Guards: EINVAL for an argument that does not come from the glue. The devmap glue returns ENXIO when specfind finds nothing. This also covers descriptors passed into another zone, which panic any devmap driver today.
    • Test: an rselftest check.
    • A local illumos-gate fix (fail the mapping with ENODEV) was prepared but is not filed.
  2. Review finding, no panic yet: a pollhead freed in interrupt context.
    • Type rule: PollHead is !Send, and PollBox/PollHeads carry heads across threads (compile_fail,E0277 doctest: a Pin<KBox<PollHead>> in a soft interrupt handler's state, with a PollBox twin that compiles).
    • Guard: the leak described above.
    • Tests: rselftest checks that a released PollBox is not leaked, and that a PollBox plus a two-head PollHeads dropped in a soft interrupt are leaked (3 heads), with the two warnings in /var/adm/messages.

Verification

Kernel runs used a fresh disposable guest (Hephaistos, golden overlay, illumos cda5b9a49f, 2 vCPU, 3 GiB) and this branch's head faec57598b.

  • Host checks:
    • cargo clippy --workspace, and again with the feature and with rselftest's private-gate
    • cargo fmt
    • cargo doc
    • cargo test -p ddi --lib: 9 tests
    • doctests: 19, on stable and on nightly-2026-04-16 (error codes checked)
    • every commit of the branch builds in all three configurations
  • xtask kernel-check: illumos-kernel-private-sys: sizes 2/2, offsets 21/21, symbols 13/13 (makespecvp and specfind come from specfs, the rest from genunix). ddi-sys gives the same answers as before: ddi_ncpus_expected is not in this kernel.
  • rselftest with private-gate: all 57 checks passed (42 at attach, 7 LDI, 8 private).
  • The illumos-drm R2 kernel suites exercise every wrapper; see that PR.

Logs (workspace): logs/hephaistos/illumos-drm/20260929-r2-rebased-rselftest-private.log, 20260929-r2-rebased-kernel-check.log, 20260929-r2-panic-*.log. Notes: notes/illumos-drm-segdev-panic-20260929.md, notes/illumos-gate-segdev-specfind-proposal.md. Dump on Hephaistos: ~/illumos-drm-test/20260928/dumps/r2-segdev-20260929/.

Open issues

  • Without the private feature the interrupt-context drop guard is absent, because servicing_interrupt is private. The type rule (!Send) still applies. This is documented.
  • devmap_setup leaving a refused mapping in place is an upstream issue. It is only noted in the local gate proposal.
  • ddi_ncpus_expected is missing from the cda5b9a49f kernel (existing ddi-sys binding, unrelated to this PR).
The private tier planned in `docs/DESIGN.md` section 4 adds what milestone R2 of the Rust DRM core needs to hand kernel objects to user space as file descriptors (illumos-drm WI-454). This PR also adds two public-tier pieces and the fixes for two findings of that work. **Stacked on PR 4** (`r3-kms-glue`, R3's `chpoll(9E)` glue); the base of this PR is that branch. Merge PR 4 first. The illumos-drm R2 pull request pins this branch's head. ## Reconciliation with R3 R2 had its own `chpoll(9E)` API. After the rebase onto `r3-kms-glue` it uses R3's (a `!Unpin` `PollHead`, `Pin` registrations, `PollHeads`, the glue deciding when to register) and drops its own. The two changes R2's review required are separate commits on top of R3's API: - `POLLNVAL` now passes through like `POLLERR` and `POLLHUP` (`PollEvents::NVAL`, `PollEvents::ALWAYS`). - **A pollhead can only be dropped in thread context** (panic-to-type rule): - A bare `PollHead` is now `!Send`. R3 had `unsafe impl Send`. - Heads that cross threads live in the new `PollBox` (a pinned `KBox` head that is `Send`/`Sync`, with `new`, `try_new`, `head` and `release(&Sleepable)`) or in a `PollHeads` array. - With the private tier, dropping either in interrupt context leaks the heads and logs a warning instead of calling `pollhead_clean(9F)` there; `poll::leaked_heads` counts them. - `PollHead::registration_unchecked` (`unsafe`) registers a head the instance does not own, such as a sync file's head. ## What was built - **`crates/illumos-kernel-private-sys`**: pinned to gate `cda5b9a49f`. The crate is empty unless the feature `unstable-gate-cda5b9a49f` is enabled. It binds: - `file_t`, `falloc`, `unfalloc`, `setf`, `getf`, `releasef`, `f_setfd_or` (`os/fio.c`, `sys/file.h`) - the public prefix of `vnode_t`, `caller_context_t`, `fop_open`, `fop_close`, `vn_copypath`, `vn_rele` (`sys/vnode.h`) - `makespecvp`, `specfind` (`sys/fs/snode.h`, module `fs/specfs`) - `servicing_interrupt` (`sys/disp.h`) `ufalloc` is left unbound on purpose: `falloc` calls it, and nothing needs a descriptor without a file. - **Gate citations instead of manual pages**: the first doc line of every item is `` `name`. Private, `path:line`. ``. `xtask lint-man --gate` checks that the cited line exists and names the item, and keeps the crate out of the manual-page checks. Result: 19 citations, 0 errors against a checkout at `0fb1d5d8d3`, which has not changed any cited file since `cda5b9a49f`. - **`ddi::private::fd`** (feature `unstable-gate-cda5b9a49f`): - `NewFd::open` gives the calling process a reserved descriptor for one of the calling driver's own minor numbers: `makespecvp`, the caller's path through `specfind` + `vn_copypath`, `VOP_OPEN`, `falloc`. - `install` uses `setf` and `f_setfd_or`; `Drop` undoes everything. - `FileRef` holds a descriptor (`getf`/`releasef`) and names the driver's minor if the descriptor is one of them. - Both need the `IoctlArg` of the ioctl being served, and both refuse LDI callers and kernel threads. - **`segmap(9E)` check**: `ops: [char, devmap]` installs glue that runs `CharDevice::segmap_check(MapRequest)` before `ddi_devmap_segmap(9F)`. A mapping that `devmap_map(9E)` refuses would otherwise stay in the process until it exits, because `devmap_setup` does not unmap it. - **`IoctlArg` records the device number** of the ioctl; only the glue can set it. - `rselftest` gains the feature `private-gate` and the ioctl `RSELFTEST_IOC_ANONFD`, with 6 descriptor checks and 2 poll-head guard checks. CI builds it with `-N fs/specfs` and runs clippy with the feature. ## Panics made impossible (docs/DESIGN.md section 5.12) 1. **2026-09-29: NULL dereference at `segdev_create+0xe4`** on the first `mmap(2)` of a dma-buf descriptor. - Cause: the `makespecvp` vnode had no path. `segdev_create` calls `specfind`, which only returns vnodes whose path is visible in the caller's zone, and does not check the result. - Type rule: only the glue sets the device number in `IoctlArg` (`compile_fail,E0624` doctest), and `NewFd::open` copies the caller's file path. - Guards: `EINVAL` for an argument that does not come from the glue. The devmap glue returns `ENXIO` when `specfind` finds nothing. This also covers descriptors passed into another zone, which panic any devmap driver today. - Test: an rselftest check. - A local illumos-gate fix (fail the mapping with `ENODEV`) was prepared but is not filed. 2. **Review finding, no panic yet: a pollhead freed in interrupt context.** - Type rule: `PollHead` is `!Send`, and `PollBox`/`PollHeads` carry heads across threads (`compile_fail,E0277` doctest: a `Pin<KBox<PollHead>>` in a soft interrupt handler's state, with a `PollBox` twin that compiles). - Guard: the leak described above. - Tests: rselftest checks that a released `PollBox` is not leaked, and that a `PollBox` plus a two-head `PollHeads` dropped in a soft interrupt are leaked (3 heads), with the two warnings in `/var/adm/messages`. ## Verification Kernel runs used a fresh disposable guest (Hephaistos, golden overlay, illumos `cda5b9a49f`, 2 vCPU, 3 GiB) and this branch's head `faec57598b`. - Host checks: - `cargo clippy --workspace`, and again with the feature and with rselftest's `private-gate` - `cargo fmt` - `cargo doc` - `cargo test -p ddi --lib`: 9 tests - doctests: 19, on stable and on `nightly-2026-04-16` (error codes checked) - every commit of the branch builds in all three configurations - `xtask kernel-check`: `illumos-kernel-private-sys: sizes 2/2, offsets 21/21, symbols 13/13` (`makespecvp` and `specfind` come from `specfs`, the rest from `genunix`). `ddi-sys` gives the same answers as before: `ddi_ncpus_expected` is not in this kernel. - rselftest with `private-gate`: all 57 checks passed (42 at attach, 7 LDI, 8 private). - The illumos-drm R2 kernel suites exercise every wrapper; see that PR. Logs (workspace): `logs/hephaistos/illumos-drm/20260929-r2-rebased-rselftest-private.log`, `20260929-r2-rebased-kernel-check.log`, `20260929-r2-panic-*.log`. Notes: `notes/illumos-drm-segdev-panic-20260929.md`, `notes/illumos-gate-segdev-specfind-proposal.md`. Dump on Hephaistos: `~/illumos-drm-test/20260928/dumps/r2-segdev-20260929/`. ## Open issues - Without the private feature the interrupt-context drop guard is absent, because `servicing_interrupt` is private. The type rule (`!Send`) still applies. This is documented. - `devmap_setup` leaving a refused mapping in place is an upstream issue. It is only noted in the local gate proposal. - `ddi_ncpus_expected` is missing from the `cda5b9a49f` kernel (existing `ddi-sys` binding, unrelated to this PR). 
The first private-tier crate of docs/DESIGN.md section 4: file_t, the
public prefix of vnode_t, falloc, unfalloc, setf, getf, releasef,
f_setfd_or, fop_open, fop_close, vn_rele and makespecvp, everything an
ioctl needs to hand a kernel object to user space as a file descriptor
the way pipe(2) does. The crate is empty unless the feature
unstable-gate-cda5b9a49f is enabled, and it declares fs/specfs as its
module dependency.

Items cite the gate file and line instead of a manual page, and
xtask lint-man now checks those citations against the gate checkout
(the cited line must exist and name the item) and keeps the private
crate out of the manual-page checks. xtask kernel-check covers the
crate like every other -sys crate.
Behind the feature unstable-gate-cda5b9a49f, which pulls in
illumos-kernel-private-sys. NewFd::open gives an ioctl a new descriptor
of the calling process for one of the calling driver's own minor
numbers: makespecvp, VOP_OPEN (the driver's open(9E) runs for the
minor), falloc; the descriptor stays reserved and invisible until
NewFd::install (setf, then f_setfd_or for close-on-exec, as pipe2(2)
does), and dropping the NewFd undoes everything, so the number can be
copied out first. FileRef::get holds a descriptor with getf/releasef and
reports the minor if the file is a character device of the calling
driver. Both take the IoctlArg of the ioctl being served and refuse LDI
callers and kernel threads (ddi_get_pid 0), since descriptors belong to
the calling process; device numbers come from ddi_driver_major of the
driver's own node. CI runs clippy with the feature.
A dma-buf mmap on the first R2 kernel run panicked the test guest
(29 September 2026, illumos cda5b9a49f): BAD TRAP, NULL dereference at
segdev_create+0xe4. The devmap framework looks the mapped device's vnode
up with specfind, which only returns vnodes whose cached path is visible
in the caller's zone, and uses the result without a check; the vnode
NewFd::open made with makespecvp has no path.

The type rule: IoctlArg now records the device number of the ioctl, and
only the glue can set it (IoctlArg::from_entry is crate-private; a
compile_fail doctest checks it, E0624). NewFd::open finds the caller's
own file with specfind for that number and copies its path to the new
vnode with vn_copypath (both bound in illumos-kernel-private-sys); an
argument made with IoctlArg::new, or a caller whose file has no visible
vnode, is refused with EINVAL. The guard: with the private tier, the
devmap(9E) glue refuses (ENXIO) a mapping of a device specfind cannot
find, which also covers a descriptor passed into another zone and
mapped there, a panic of every devmap driver today.

rselftest gains the private-gate feature and RSELFTEST_IOC_ANONFD: a
forged argument must be refused, a real one must make a close-on-exec
descriptor of the driver's own minor that FileRef recognises, and
another device's descriptor must not be recognised.
When a devmap_map(9E) callback fails, devmap_setup() in vm/seg_dev.c
frees the mapping handles and returns ENXIO but leaves the new segment
in the address space, so the device stays mapped, and its file open,
until the process exits. The first R2 run of illumos-drm showed it: a
refused MAP_PRIVATE mapping of a dma-buf kept the dma-buf alive after
its last close.

ops: [char, devmap] now installs glue as segmap(9E) that calls the new
CharDevice::segmap_check with the request (offset, length, protections,
flags; MapRequest::is_shared) before ddi_devmap_segmap(9F), so that a
driver can refuse before any segment exists. The default allows
everything, so existing drivers are unchanged.
DESIGN.md: the private tier exists now, with its rules (feature named
after the gate commit, gate citations checked by lint-man, kernel-check
coverage, wrappers only under ddi::private); segmap_check in module!;
the device number IoctlArg records; a Panics prevented row for the
segdev_create panic of 29 September 2026 with its type rule, guards and
tests; the private-gate checks of rselftest and their run; the
kernel-check results of the private crate; the repository layout.
For the poll-head guard of the ddi crate: a head must not be cleaned in
interrupt context (pollhead_clean(9F) is for user and kernel context).
poll(2) reports POLLERR, POLLHUP and POLLNVAL whether they were asked for
or not; the glue masked POLLNVAL away. PollEvents gains NVAL and ALWAYS.
pollhead_clean(9F) is for user and kernel context. A bare PollHead is now
!Send, so it cannot be moved into a timeout or an interrupt handler and
dropped there. Heads that cross threads live in the new PollBox (a pinned
allocation of its own, released with the Sleepable token) or a PollHeads
array; with the private tier, an implicit drop of either in interrupt
context is detected with servicing_interrupt and leaks the heads with a
warning, counted by poll::leaked_heads.

PollHead::registration_unchecked registers a head that a file or object
owns rather than the instance, under the contract that it lives until
the polled file's close(9E).

compile_fail doctest (a pinned boxed head in a soft interrupt handler,
E0277) with a PollBox twin; rselftest private-gate checks that a released
PollBox is not leaked and that heads dropped in a soft interrupt are.
DESIGN.md: PollBox and the thread-context drop rule, private-tier results
Some checks failed
ci / check-hosted (pull_request) Failing after 1s
Details
ci / kmod-illumos (pull_request) Failing after 1s
Details
faec57598b 
Section 5.2 describes registration_unchecked, PollBox and the !Send head;
section 5.12 gains the row for heads freed in interrupt context. The
private tier lists disp.h, kernel-check covers 13 symbols and rselftest
57 checks. Evidence rows no longer point into the workspace.
toasterson changed target branch from r3-kms-glue to main 2026-10-05 16:33:34 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
illumos/illumos-ddi!5
No description provided.