R2: fences, dma_resv, syncobj, sync_file and PRIME with anonymous fds #6

Merged

toasterson merged 25 commits from r2-sync into main 2026-10-05 16:36:18 +00:00

Conversation 0 Commits 25 Files changed 34 +8187 -133

forge commented 2026-09-29 19:27:40 +00:00

Contributor

Copy link

Anima work item: WI-454 01a0e387-6ec8-7d63-a751-7a2de07be520

Milestone R2 of the Rust DRM core: fences, reservations, sync objects with timelines, sync files, and PRIME, with file descriptors handed out from ioctls.

Stacked on PR 5 (r3-kms, R3's KMS); the base of this PR is that branch. The workspace Cargo.toml pins illumos-ddi r2-private-fd at faec57598b (illumos-ddi 5 (illumos/illumos-ddi#5), stacked on illumos-ddi PR 4). Cargo.lock was regenerated from that git dependency without a patch, and cargo check --locked passes. Merge order: illumos-ddi PR 4, then the illumos-ddi R2 PR, then PR 5, then this one. After that, move the pin to the illumos-ddi merge commit.

Reconciliation with R3

• This branch was rebased onto r3-kms. The shared files keep both sides:
  • device.rs: the kms and sync fields; clone_open sets up sync, then KMS. close releases KMS, then GEM, then sync, and frees the clone id last. detach tears down KMS, clears sync, then quiesces GEM and sync.
  • caps.rs, README.md, the R0 test's capability table, and rdrmnull.
• chpoll(9E) uses R3's API from illumos-ddi (CharDevice::chpoll -> (PollEvents, PollRegistration<'a>)):
  • DRM files go to R3's KMS event queue. R2's own always-writable DRM-file poll is dropped.
  • Anonymous minors go to SyncCore::anon_chpoll. A sync file keeps its head in a ddi::PollBox and registers it with the unsafe PollHead::registration_unchecked, because the anonymous-file entry, not the instance, owns it.
  • Sync object files, which are always ready, register a device-owned idle head through the safe API.
• rdrmnull's test ioctls moved to FENCE = DRM_COMMAND_BASE + 4 and BO_WAIT = + 5, because R3's SCANOUT takes + 3. run-r3-test.sh links with -N fs/specfs like the other scripts.
• R3's note on IN_FENCE_FD/OUT_FENCE_PTR now says that connecting them to R2's sync files is open.

What was built

• drm::fence:
  • Fence (Linux dma_fence) with a context, a wraparound-safe seqno, an error and a timestamp.
  • Callbacks are pre-allocated and run outside the fence lock, in the signaller's context. A callback may return a fence to signal next, so cascades run iteratively. remove_callback waits for a running callback.
  • Waits take absolute monotonic deadlines and are interruptible (ETIME, EINTR).
  • Fence::array (all or any, first error) and Fence::chain (timeline points, collected as they signal), with chain_find_seqno and chain_query.
  • A per-device FenceDomain: it holds its locks at the driver's interrupt priority and refuses high-level priorities, owns the sync kstat, and parks interrupt-context drops for a task queue, because mutex_destroy(9F) is thread-context only.
• drm::wwmutex implements wait-die, as Linux's reservation class does, with EDEADLK, lock_slow, lock_all, EALREADY and EINTR.
• drm::resv has the kernel, write, read and bookkeep usages, adds fences under the ww guard, iterates while locked, waits, and returns a singleton fence. Every GEM object has one.
• drm::syncobj and drm::syncobj_ioctl:
  • all eleven SYNCOBJ_* ioctls, with Linux's checks and errnos
  • WAIT_ALL, WAIT_FOR_SUBMIT, WAIT_AVAILABLE, timelines and TRANSFER
  • export and import as sync object files, and as sync files (EXPORT_SYNC_FILE, IMPORT_SYNC_FILE)
• drm::sync_file: poll(2) (pollwakeup(9F) from the signaller's context), SYNC_IOC_MERGE and SYNC_IOC_FILE_INFO.
• drm::anon_fd and drm::prime:
  • dma-buf, sync object and sync file descriptors are specfs files of anonymous minors of the driver (docs/anon-fd.md)
  • PRIME_HANDLE_TO_FD (DRM_CLOEXEC, DRM_RDWR, one dma-buf per object)
  • PRIME_FD_TO_HANDLE (returns the existing handle or a new one)
  • dma-buf mmap(2), and DMA_BUF_IOCTL_SYNC, which waits for the reservation at the start of CPU access
• GET_CAP reports PRIME = import|export for GEM drivers, and SYNCOBJ/SYNCOBJ_TIMELINE according to the new Features::SYNCOBJ and Features::SYNCOBJ_TIMELINE.
• rdrmnull has a test timeline that a soft interrupt signals; timeout(9F) triggers it (Driver::fence_priority). It adds the test-only ioctls FENCE and BO_WAIT in the driver range, and signals pending fences with ECANCELED at detach.
• Tests: host unit tests, several of them running eight threads (R2's 71 of the 108 drm tests); tools/test/drm_r2_test.c; tools/test/rdrmnull-two.conf. run-r1-test.sh runs them.

Decisions

• File descriptors: a descriptor is a specfs vnode for an anonymous minor of the DRM driver itself, opened inside the ioctl as pipe(2) and LDI do (makespecvp, VOP_OPEN, falloc, setf). The alternatives were a clone-open pseudo driver or a private vnode type. Close, poll, mmap, fstat and detach safety come from specfs. See docs/anon-fd.md.
• Fences: signalling never allocates, and callbacks run outside the lock. Signal and drop take Ctx tokens, waits take Sleepable, and locks sit at the driver's declared interrupt priority.
• Wound/wait: wait-die, like Linux's reservation_ww_class, with a real two-thread deadlock-avoidance test.
• Cross-device: sync objects and sync files from any device of the driver can be imported (the code is shared), except between devices whose fences signal at different interrupt priorities (ENOTSUP). A dma-buf of another device gives ENOTSUP, and another driver's file gives EINVAL, because each module links its own core until R5.
• Mappings are refused in segmap(9E) with Linux's errnos (EINVAL, EACCES) instead of the ENXIO of devmap_setup. This also fixes a refused mapping lingering until the process exits; the R1 test expects the new errnos.

Deviations

• The work item asks for "one fd object type with a private vnode". The implementation uses specfs vnodes of the driver's own minors instead (accepted; the reasons are in docs/anon-fd.md).
• ufalloc is not bound: falloc calls it, and nothing needs a descriptor without a file.
• Differences from Linux: TRANSFER refuses flags other than WAIT_FOR_SUBMIT, PRIME_FD_TO_HANDLE refuses a non-zero flags, and arrays of handles are limited to 65536 entries (EINVAL). EXPORT_SYNC_FILE of an unknown handle gives ENOENT (README).
• Deferred:
  • poll(2) on a dma-buf (implicit sync through its reservation) answers ENOTSUP.
  • Cross-device dma-buf import answers ENOTSUP.

Verification (after the rebase onto R3)

All runs used a fresh disposable guest r2-run4 on Hephaistos (golden overlay, illumos cda5b9a49f, 2 vCPU, 3 GiB). The module was built from fc1e54ff2a against illumos-ddi faec57598b.

• run-r1-test.sh, one device: R0 105, R0 probe 5, R1 153, R1 probe 4, R2 206 (including 18 child checks), R2 probe 3, pending-fence cancel PASS. The export-vs-last-close race: 1600 exports of three buffers from four threads, 0 errors. /etc, /devices and /dev/dri unchanged, fail=0.
• run-r1-test.sh, two devices (rdrmnull-two.conf): R0 105, R1 153, R2 216. A dma-buf is refused across devices with ENOTSUP; a sync object and a sync file are imported and waited on across devices. Race 0 errors, cancel PASS, state unchanged, fail=0.
• run-r3-test.sh (R3's suite, with this module): R0 105/5, R1 153/4, R3 184, R3 probe 3, 9 modetest/unload/state PASS lines, vblank 60.07 Hz, fail=0.
• After the last review nit (the sync file head allocated with KM_SLEEP, 247bb8b), run-r1-test.sh ran again with one device: fail=0, race 0 errors, cancel PASS, state unchanged. That log has R2 at 205 PASS lines because one child line is missing: the child's WAIT_FOR_SUBMIT check printed at the same moment as the parent's next line, and the two share the log file (writes without O_APPEND). The child exited 0, which the parent checks ("the sync object child passed all its checks").
• illumos-ddi rselftest with private-gate: 57/57. kernel-check of the private tier: 13/13 symbols.
• Before the rebase: DTrace showed 490 sync-file pollwakeup(9F) calls in an interrupt thread at PIL 1, from the timeline soft interrupt. The first run panicked in segdev_create; that is fixed in illumos-ddi by the panic-to-type rule (see that PR).
• Host checks with the pin, --locked, no patch:
  • cargo check
  • cargo clippy, also with panic-drill
  • cargo fmt
  • cargo doc
  • cargo test -p drm -p drm-uapi: 108 + 1
  • uapi-check: 0 differences
  • the kernel library builds with nightly-2026-04-16
  • with the illumos-ddi patch, every commit of the branch builds and passes fmt

Logs (workspace): logs/hephaistos/illumos-drm/20260929-r2-rebased-one.log, 20260929-r2-rebased-two.log, 20260929-r2-rebased-r3.log, 20260929-r2-rebased-final-one.log, 20260929-r2-rebased-rselftest-private.log, 20260929-r2-rebased-kernel-check.log. Earlier runs: 20260929-r2-final-*-device.log, 20260929-r2-round2-*.log, 20260929-r2-run3-full.log, 20260929-r2-panic-console.log, 20260929-r2-panic-mdb.log, 20260929-r2-softint-pollwakeup-dtrace.log. Notes: notes/illumos-drm-segdev-panic-20260929.md, notes/illumos-gate-segdev-specfind-proposal.md. Dump on Hephaistos: ~/illumos-drm-test/20260928/dumps/r2-segdev-20260929/.

Review: a separate reviewer applied the Anima rules.

• Round 1 rejected the branch with ten findings, all fixed: the pin and lock, a race on anonymous-minor reuse, a use-after-free between defer and quiesce, the poll registration rule, aliasing of callback links, the semantics of DMA_BUF_IOCTL_SYNC, the pass-through of chpoll events, two errno or documentation points, and per-instance timers.
• Round 2, on this rebased branch, approved it. Its preconditions (repin with a locked build, a kernel run of the rebased module) are met above, and its nit (PollBox::new) is fixed.

Open issues

• dma-buf poll, and dma-buf import across devices or drivers (R4/R5).
• KMS in-fences and out-fences (IN_FENCE_FD, OUT_FENCE_PTR) need R2 and R3 together; they are follow-up work.
• For R3 to decide: detach tears down KMS before the GEM and sync checks. A detach refused because a fence is still pending (new with R2), or because GEM objects remain (already in R3), leaves the device attached with its KMS state gone.
• The local illumos-gate proposals (the segdev_create NULL check; devmap_setup not unmapping refused mappings) are not filed.

Update: restacked on the detach fix

Rebased onto PR 5's detach fix; R2's pending-fence refusal now runs among the checks before any teardown. The pending-fence test confirms that after the refused detach the display (modetest) and the sync paths still work and that the detach cancelled the pending fence. Kernel runs on illumos cda5b9a49f: R0-R2 502 PASS / 0 FAIL, R3 476 PASS / 0 FAIL. The "detach order" open issue is resolved.

Anima work item: WI-454 `01a0e387-6ec8-7d63-a751-7a2de07be520` Milestone R2 of the Rust DRM core: fences, reservations, sync objects with timelines, sync files, and PRIME, with file descriptors handed out from ioctls. **Stacked on PR 5** (`r3-kms`, R3's KMS); the base of this PR is that branch. The workspace `Cargo.toml` pins illumos-ddi `r2-private-fd` at `faec57598b` (illumos-ddi 5 (https://code.aopc.cloud/illumos/illumos-ddi/pulls/5), stacked on illumos-ddi PR 4). `Cargo.lock` was regenerated from that git dependency without a patch, and `cargo check --locked` passes. Merge order: illumos-ddi PR 4, then the illumos-ddi R2 PR, then PR 5, then this one. After that, move the pin to the illumos-ddi merge commit. ## Reconciliation with R3 - This branch was rebased onto `r3-kms`. The shared files keep both sides: - `device.rs`: the `kms` and `sync` fields; `clone_open` sets up sync, then KMS. `close` releases KMS, then GEM, then sync, and frees the clone id last. `detach` tears down KMS, clears sync, then quiesces GEM and sync. - `caps.rs`, `README.md`, the R0 test's capability table, and `rdrmnull`. - `chpoll(9E)` uses R3's API from illumos-ddi (`CharDevice::chpoll -> (PollEvents, PollRegistration<'a>)`): - DRM files go to R3's KMS event queue. R2's own always-writable DRM-file poll is dropped. - Anonymous minors go to `SyncCore::anon_chpoll`. A sync file keeps its head in a `ddi::PollBox` and registers it with the `unsafe` `PollHead::registration_unchecked`, because the anonymous-file entry, not the instance, owns it. - Sync object files, which are always ready, register a device-owned `idle` head through the safe API. - rdrmnull's test ioctls moved to `FENCE` = `DRM_COMMAND_BASE + 4` and `BO_WAIT` = `+ 5`, because R3's `SCANOUT` takes `+ 3`. `run-r3-test.sh` links with `-N fs/specfs` like the other scripts. - R3's note on `IN_FENCE_FD`/`OUT_FENCE_PTR` now says that connecting them to R2's sync files is open. ## What was built - `drm::fence`: - `Fence` (Linux `dma_fence`) with a context, a wraparound-safe seqno, an error and a timestamp. - Callbacks are pre-allocated and run outside the fence lock, in the signaller's context. A callback may return a fence to signal next, so cascades run iteratively. `remove_callback` waits for a running callback. - Waits take absolute monotonic deadlines and are interruptible (`ETIME`, `EINTR`). - `Fence::array` (all or any, first error) and `Fence::chain` (timeline points, collected as they signal), with `chain_find_seqno` and `chain_query`. - A per-device `FenceDomain`: it holds its locks at the driver's interrupt priority and refuses high-level priorities, owns the `sync` kstat, and parks interrupt-context drops for a task queue, because `mutex_destroy(9F)` is thread-context only. - `drm::wwmutex` implements wait-die, as Linux's reservation class does, with `EDEADLK`, `lock_slow`, `lock_all`, `EALREADY` and `EINTR`. - `drm::resv` has the kernel, write, read and bookkeep usages, adds fences under the ww guard, iterates while locked, waits, and returns a singleton fence. Every GEM object has one. - `drm::syncobj` and `drm::syncobj_ioctl`: - all eleven `SYNCOBJ_*` ioctls, with Linux's checks and errnos - `WAIT_ALL`, `WAIT_FOR_SUBMIT`, `WAIT_AVAILABLE`, timelines and `TRANSFER` - export and import as sync object files, and as sync files (`EXPORT_SYNC_FILE`, `IMPORT_SYNC_FILE`) - `drm::sync_file`: `poll(2)` (`pollwakeup(9F)` from the signaller's context), `SYNC_IOC_MERGE` and `SYNC_IOC_FILE_INFO`. - `drm::anon_fd` and `drm::prime`: - dma-buf, sync object and sync file descriptors are specfs files of anonymous minors of the driver (`docs/anon-fd.md`) - `PRIME_HANDLE_TO_FD` (`DRM_CLOEXEC`, `DRM_RDWR`, one dma-buf per object) - `PRIME_FD_TO_HANDLE` (returns the existing handle or a new one) - dma-buf `mmap(2)`, and `DMA_BUF_IOCTL_SYNC`, which waits for the reservation at the start of CPU access - `GET_CAP` reports `PRIME` = import|export for GEM drivers, and `SYNCOBJ`/`SYNCOBJ_TIMELINE` according to the new `Features::SYNCOBJ` and `Features::SYNCOBJ_TIMELINE`. - `rdrmnull` has a test timeline that a soft interrupt signals; `timeout(9F)` triggers it (`Driver::fence_priority`). It adds the test-only ioctls `FENCE` and `BO_WAIT` in the driver range, and signals pending fences with `ECANCELED` at detach. - Tests: host unit tests, several of them running eight threads (R2's 71 of the 108 drm tests); `tools/test/drm_r2_test.c`; `tools/test/rdrmnull-two.conf`. `run-r1-test.sh` runs them. ## Decisions - **File descriptors**: a descriptor is a specfs vnode for an *anonymous minor* of the DRM driver itself, opened inside the ioctl as `pipe(2)` and LDI do (`makespecvp`, `VOP_OPEN`, `falloc`, `setf`). The alternatives were a clone-open pseudo driver or a private vnode type. Close, poll, mmap, `fstat` and detach safety come from specfs. See `docs/anon-fd.md`. - **Fences**: signalling never allocates, and callbacks run outside the lock. Signal and drop take `Ctx` tokens, waits take `Sleepable`, and locks sit at the driver's declared interrupt priority. - **Wound/wait**: wait-die, like Linux's `reservation_ww_class`, with a real two-thread deadlock-avoidance test. - **Cross-device**: sync objects and sync files from any device of the driver can be imported (the code is shared), except between devices whose fences signal at different interrupt priorities (`ENOTSUP`). A dma-buf of another device gives `ENOTSUP`, and another driver's file gives `EINVAL`, because each module links its own core until R5. - Mappings are refused in `segmap(9E)` with Linux's errnos (`EINVAL`, `EACCES`) instead of the `ENXIO` of `devmap_setup`. This also fixes a refused mapping lingering until the process exits; the R1 test expects the new errnos. ## Deviations - The work item asks for "one fd object type with a private vnode". The implementation uses specfs vnodes of the driver's own minors instead (accepted; the reasons are in `docs/anon-fd.md`). - `ufalloc` is not bound: `falloc` calls it, and nothing needs a descriptor without a file. - Differences from Linux: `TRANSFER` refuses flags other than `WAIT_FOR_SUBMIT`, `PRIME_FD_TO_HANDLE` refuses a non-zero `flags`, and arrays of handles are limited to 65536 entries (`EINVAL`). `EXPORT_SYNC_FILE` of an unknown handle gives `ENOENT` (README). - Deferred: - `poll(2)` on a dma-buf (implicit sync through its reservation) answers `ENOTSUP`. - Cross-device dma-buf import answers `ENOTSUP`. ## Verification (after the rebase onto R3) All runs used a fresh disposable guest `r2-run4` on Hephaistos (golden overlay, illumos `cda5b9a49f`, 2 vCPU, 3 GiB). The module was built from `fc1e54ff2a` against illumos-ddi `faec57598b`. - `run-r1-test.sh`, one device: R0 105, R0 probe 5, R1 153, R1 probe 4, R2 206 (including 18 child checks), R2 probe 3, pending-fence cancel PASS. The export-vs-last-close race: 1600 exports of three buffers from four threads, 0 errors. `/etc`, `/devices` and `/dev/dri` unchanged, fail=0. - `run-r1-test.sh`, two devices (`rdrmnull-two.conf`): R0 105, R1 153, R2 216. A dma-buf is refused across devices with `ENOTSUP`; a sync object and a sync file are imported and waited on across devices. Race 0 errors, cancel PASS, state unchanged, fail=0. - `run-r3-test.sh` (R3's suite, with this module): R0 105/5, R1 153/4, R3 184, R3 probe 3, 9 modetest/unload/state PASS lines, vblank 60.07 Hz, fail=0. - After the last review nit (the sync file head allocated with `KM_SLEEP`, `247bb8b`), `run-r1-test.sh` ran again with one device: fail=0, race 0 errors, cancel PASS, state unchanged. That log has R2 at 205 PASS lines because one child line is missing: the child's `WAIT_FOR_SUBMIT` check printed at the same moment as the parent's next line, and the two share the log file (writes without `O_APPEND`). The child exited 0, which the parent checks ("the sync object child passed all its checks"). - illumos-ddi `rselftest` with `private-gate`: 57/57. `kernel-check` of the private tier: 13/13 symbols. - Before the rebase: DTrace showed 490 sync-file `pollwakeup(9F)` calls in an interrupt thread at PIL 1, from the timeline soft interrupt. The first run panicked in `segdev_create`; that is fixed in illumos-ddi by the panic-to-type rule (see that PR). - Host checks with the pin, `--locked`, no patch: - `cargo check` - `cargo clippy`, also with `panic-drill` - `cargo fmt` - `cargo doc` - `cargo test -p drm -p drm-uapi`: 108 + 1 - `uapi-check`: 0 differences - the kernel library builds with `nightly-2026-04-16` - with the illumos-ddi patch, every commit of the branch builds and passes fmt Logs (workspace): `logs/hephaistos/illumos-drm/20260929-r2-rebased-one.log`, `20260929-r2-rebased-two.log`, `20260929-r2-rebased-r3.log`, `20260929-r2-rebased-final-one.log`, `20260929-r2-rebased-rselftest-private.log`, `20260929-r2-rebased-kernel-check.log`. Earlier runs: `20260929-r2-final-*-device.log`, `20260929-r2-round2-*.log`, `20260929-r2-run3-full.log`, `20260929-r2-panic-console.log`, `20260929-r2-panic-mdb.log`, `20260929-r2-softint-pollwakeup-dtrace.log`. Notes: `notes/illumos-drm-segdev-panic-20260929.md`, `notes/illumos-gate-segdev-specfind-proposal.md`. Dump on Hephaistos: `~/illumos-drm-test/20260928/dumps/r2-segdev-20260929/`. Review: a separate reviewer applied the Anima rules. - Round 1 rejected the branch with ten findings, all fixed: the pin and lock, a race on anonymous-minor reuse, a use-after-free between defer and quiesce, the poll registration rule, aliasing of callback links, the semantics of `DMA_BUF_IOCTL_SYNC`, the pass-through of chpoll events, two errno or documentation points, and per-instance timers. - Round 2, on this rebased branch, approved it. Its preconditions (repin with a locked build, a kernel run of the rebased module) are met above, and its nit (`PollBox::new`) is fixed. ## Open issues - dma-buf poll, and dma-buf import across devices or drivers (R4/R5). - KMS in-fences and out-fences (`IN_FENCE_FD`, `OUT_FENCE_PTR`) need R2 and R3 together; they are follow-up work. - For R3 to decide: `detach` tears down KMS before the GEM and sync checks. A detach refused because a fence is still pending (new with R2), or because GEM objects remain (already in R3), leaves the device attached with its KMS state gone. - The local illumos-gate proposals (the `segdev_create` NULL check; `devmap_setup` not unmapping refused mappings) are not filed. ## Update: restacked on the detach fix Rebased onto PR 5's detach fix; R2's pending-fence refusal now runs among the checks before any teardown. The pending-fence test confirms that after the refused detach the display (modetest) and the sync paths still work and that the detach cancelled the pending fence. Kernel runs on illumos cda5b9a49f: R0-R2 502 PASS / 0 FAIL, R3 476 PASS / 0 FAIL. The "detach order" open issue is resolved.

forge added 22 commits 2026-09-29 19:27:41 +00:00

Add fences, fence arrays and fence chains ... d4aa2c6601

drm::fence is the core of R2: a Fence (Linux dma_fence) on a context
with a wraparound-safe sequence number, signalled once with an optional
error and a timestamp, waited for with a monotonic deadline and an
optional interruptible wait (ETIME, EINTR), and callbacks that are
allocated ahead of time and run outside the fence lock in the
signaller's context. Signalling never allocates; a callback may return a
fence to signal next, and the signalling loop runs such cascades
iteratively. remove_callback either unlinks a callback that has not
started or waits until it has finished. Arrays (all or any, first error)
and chain points (Linux dma_fence_chain, with chain_find_seqno and
chain_query for timeline sync objects) are fences whose members complete
them through such callbacks.

Fences belong to a device's FenceDomain: its interrupt priority is the
one their locks are created with (high-level priorities are refused), it
counts everything for the kstat, and it parks fences whose last
reference goes in interrupt context for its task queue, because
mutex_destroy(9F) is not allowed there. Destruction releases held fences
iteratively. drm::ksync maps the locks, condition variables and the
monotonic clock to ddi in the kernel and to std in host tests, which
drive signalling against concurrent add and remove from eight threads,
waits, arrays, out-of-order timelines and 100000-point chains on a 64 KiB
stack.

Add wound/wait mutexes and reservation objects ... 246cbebafe

drm::wwmutex is Linux's ww_mutex with the wait-die rule its reservation
class uses: a transaction (WwAcquireCtx) with a stamp waits for younger
holders and gets EDEADLK from older ones while it holds other mutexes of
the class, then backs off with lock_slow; EALREADY for a mutex it holds,
EINTR for interruptible waits, and lock_all runs the back-off loop. Host
tests lock two mutexes in opposite orders from two threads at the same
moment and random subsets from eight threads.

drm::resv is Linux's dma_resv with the kernel, write, read and bookkeep
usages: adding needs the reservation's ww guard, replaces signalled and
superseded fences of the same context, and readers iterate under a short
internal mutex (no RCU). Waits, snapshots and singleton fences (an array
when several remain) are tested from eight threads acting as jobs.

Add sync objects with timelines and the multi-object wait ... e5e3a9e79f

drm::syncobj is Linux's drm_syncobj as plain logic over drm::fence: a
fence slot that is binary (SIGNAL, RESET, replace) or a timeline of
chain points (add_point, signal at a point, find_fence with Linux's
point lookup, query of the last reached or last submitted point).
wait_many implements SYNCOBJ_WAIT and SYNCOBJ_TIMELINE_WAIT: all or any,
WAIT_FOR_SUBMIT and WAIT_AVAILABLE, an absolute monotonic deadline (a
past one polls), ETIME and EINTR, first_signaled. Fence callbacks and
submission waiters point at one Waiter on the waiting thread's stack,
and the cleanup removes all of them (waiting for running callbacks)
before it goes. Host tests cover binary and timeline semantics,
out-of-order points, submission from another thread, and eight waiters
against four signallers.

Add PRIME, the sync object ioctls and sync files over anonymous minors ... d98b9f0c24

Objects handed to user space as file descriptors get an anonymous minor
of the driver (bit 30, the device index, a 24-bit number) and a specfs
file for it, created inside the ioctl with ddi::private::fd; close(9E)
of the minor drops the object, chpoll(9E) polls a sync file, devmap(9E)
maps a dma-buf, ioctl(9E) serves SYNC_IOC_MERGE, SYNC_IOC_FILE_INFO and
DMA_BUF_IOCTL_SYNC. open(9E) accepts an anonymous minor only while the
export path opens it. Imports look the descriptor up with getf and find
the object by minor (drm::anon_fd).

PRIME_HANDLE_TO_FD exports a GEM object as a dma-buf (DRM_CLOEXEC,
DRM_RDWR; one dma-buf per object while it is open), PRIME_FD_TO_HANDLE
returns the file's existing handle or creates one (same device;
ENOTSUP across devices). The eleven SYNCOBJ ioctls follow Linux's
checks and errnos (drm::syncobj_ioctl); HANDLE_TO_FD and FD_TO_HANDLE
export and import sync objects and, with the sync-file flags, fences as
sync files (drm::sync_file). GET_CAP reports PRIME import and export for
GEM drivers and SYNCOBJ and SYNCOBJ_TIMELINE per the new driver
features. GEM objects gain a reservation, and mappings a base offset so
that dma-buf mappings start at 0. The device's R2 state (fence domain,
per-file sync object tables, anonymous files, the sync kstat) is
drm::fence_dev; Driver::fence_priority names the interrupt priority
fences are signalled at.

rdrmnull gains a test timeline signalled from a soft interrupt that
timeout(9F) triggers, and the test-only ioctls FENCE (into a sync
object point, a reservation and/or a sync file) and BO_WAIT; detach
signals what is left with ECANCELED. It declares chpoll and the sync
object features.

Add the R2 kernel test and run it with the R0 and R1 tests ... 38cacb2d9b

tools/test/drm_r2_test.c checks, as root against rdrmnull: the PRIME and
sync object capabilities; a dumb buffer exported as a dma-buf, sent over
SCM_RIGHTS to an exec'd child that imports it on its card and render
files, maps it through the handle and the dma-buf and sees and changes
the parent's contents; one dma-buf per object, read-only exports,
private and out-of-range mappings, DMA_BUF_IOCTL_SYNC and the PRIME
error paths; binary and timeline sync object semantics with fences that
rdrmnull's soft interrupt signals (points out of order, WAIT_FOR_SUBMIT,
WAIT_AVAILABLE, TRANSFER, QUERY); sync files woken in poll(2) by the
soft interrupt, SYNC_IOC_MERGE and FILE_INFO, import and export through
sync objects; a sync object and a sync file passed to another process;
reservation waits for writers and readers; EINTR; eight threads doing
all of it at once; and the sync kstat back at zero. A 32-bit probe
checks ENOTSUP. run-r1-test.sh builds and runs it when it is present and
prints the sync kstat next to the gem one; the R0 test expects the R2
capabilities.

Link rdrmnull with -N fs/specfs ... 23f969abfe

makespecvp, which the anonymous file descriptors use, lives in the
specfs module, and krtld resolves a module's undefined symbols only
against its dependencies and the primary modules. The test scripts, the
CI link step and the README link command now name the dependency, as
cargo xtask link does from the private crate's kmod metadata.

rdrmnull: reconnect the timeline after a refused detach ... 043d63e2a8

before_detach clears the soft interrupt the fence timers trigger; if the
detach is then refused and the device used again, the fence ioctl
restores it, so that its fences still signal.

drm_r2_test: ask for the XPG4.2 socket API for SCM_RIGHTS 0e2a0f34c8

Refuse mappings in segmap(9E); free the clone id last; timer margin ... 448d53f678

The first run without a panic showed three problems:

* An mmap(2) that devmap_map(9E) refused (a private mapping of a
  dma-buf) kept the dma-buf open until the process exited, because
  devmap_setup leaves the segment behind. The core now refuses in
  segmap(9E) (illumos-ddi's CharDevice::segmap_check), before a segment
  exists: shared only, an object's exact offset and a file holding a
  handle for DRM files, the range for dma-bufs, ENODEV for sync files
  and sync object files. The errnos are Linux's (EINVAL, EACCES) instead
  of the ENXIO devmap_setup reports; the R1 test expects them now.
* Concurrent opens failed with EBUSY: close(9E) freed the clone id
  before the file's GEM and sync object tables were gone, so an open
  that reused the id found the table slot taken. The id is freed last.
  (The race existed since R1; the longer close of R2 made it visible to
  the R0 stress test.)
* timeout(9F) counts from the current tick and can fire up to a tick
  before the delay has passed on the monotonic clock, so the soft
  interrupt found the fence not yet due and nothing signalled it.
  rdrmnull's fence timers wait one tick more.

The R2 test's merge checks now expect one fence for a merge of two
fences of rdrmnull's single timeline (the later one, as Linux merges
per context) and two for a timeline point merged with such a fence.

Publish the sync kstat from the fence domain, also after deferred drains ... bcf18992c3

The counters were copied to the kstat only on ioctl paths, so a fence
that the release task destroyed stayed visible in the kstat until the
next ioctl (the R2 test's PROCESSES check saw a timeline point that was
already gone). The kstat now belongs to the FenceDomain, which publishes
after every drain; SyncCore::publish forwards to it. The R2 test prints
the deferred and drain counters with the live ones.

Document R2: fences, reservations, sync objects, file descriptors ... bd6fd7b917

docs/anon-fd.md records the file descriptor design (a specfs file of an
anonymous minor of the driver, made inside the ioctl as pipe(2) makes
its descriptors), the alternatives (a clone-open pseudo driver, a
private vnode with its own operations) and why, and its details: minor
numbers, open rules, one dma-buf per object, the vnode path, mappings,
poll, imports and context. The README reports the R2 run, describes the
new modules and rdrmnull's timeline, and adds the decisions on fences,
reservations and wound/wait mutexes, sync objects, file descriptors and
PRIME, the private tier dependency, the segmap(9E) refusal of mappings
with Linux's errnos, the R2 ioctl rows and the R2 locks.

Fence callback links are atomic; a parking release keeps its domain ... 6311e928f9

Review round 1 of R2:

* CbHdr.next was a plain pointer written through a raw pointer while
  shared references to the header existed (aliasing UB). It is an
  AtomicPtr now, used with the fence locked or by the signaller that
  owns the detached list, and remove_callback unlinks through a
  predecessor instead of a pointer to a link.
* Fence::defer_put took the domain as a raw pointer from the fence it
  parked; once parked, a drain on another processor could destroy the
  fence and let quiesce free the domain while defer still used it. It
  now holds a counted reference across the call, which quiesce sees
  (it refuses while any reference but the device's exists), as GemRef
  does for the GEM state. The comments say so.

Keep an anonymous minor's entry while an export has booked an open ... 7a4c142a8f

Review round 1 found an id-reuse race: close(9E) of a dma-buf's last
file removed the entry and freed its number even while a second export
of the same object had booked an open of it; a third export could then
take the number for another object, and the second export's open, which
specfs runs after the close, landed on that object.

The table is now plain logic (drm::anon_fd::AnonTable) with the rule:
close(9E) keeps an entry, and its number, while an open is booked, and
settle removes it if that open then fails. Host tests check the rules
and run eight threads exporting three objects against last closes with
specfs's per-number serialisation; with the old close rule the stress
test fails (the file lands on the wrong object).

Import sync objects and sync files across devices; DMA_BUF_IOCTL_SYNC waits ... b8d6a4242e

Review round 1 and the orchestrator's decisions:

* Sync object files and sync files of any device of the driver can be
  imported (SYNCOBJ_FD_TO_HANDLE, IMPORT_SYNC_FILE, SYNC_IOC_MERGE), since
  the devices share this code: SyncCore keeps a per-module registry of
  the devices, and SyncCore::lookup finds the object in the table of the
  device the minor names, which the held file keeps attached. A dma-buf
  of another device stays ENOTSUP; files of other drivers are EINVAL.
* DMA_BUF_IOCTL_SYNC at the start of CPU access waits for the object's
  reservation, interruptibly: the writers before reading, all fences
  before writing, as dma_buf_begin_cpu_access does.
* EXPORT_SYNC_FILE of an unknown handle is ENOENT, as in Linux.
* The syncobj_ioctl rules say which pad fields are checked (as in Linux,
  not those of WAIT and TIMELINE_WAIT) and which unknown handles are
  ENOENT; the README lists every R2 deviation from Linux.

rdrmnull timers wake only their device; test races, two devices, cancel ... 8ddabe45a3

* rdrmnull's fence timers are boxed closures that trigger the soft
  interrupt of their own instance only (review round 1).
* The detach reports how many pending test fences it signalled with
  ECANCELED; run-r1-test.sh leaves a 10 s fence pending with no file
  open (drm_r2_test pending) and checks that the unload cancels it.
* drm_r2_test races exports of one buffer against the last close of its
  dma-buf from four threads, and with a second device
  (tools/test/rdrmnull-two.conf) checks that a dma-buf does not cross
  devices (ENOTSUP) while a sync object and a sync file do.

run-r1-test.sh: remove the /dev/dri links of every rdrmnull device 003006f9ae

run-r1-test.sh: find rdrmnull links with ls, illumos has no readlink(1) b883b246e5

Refuse cross-device imports across fence priorities; race three buffers ... 916de71ebd

Review round 2 (non-blocking notes):

* A sync object or sync file of another device is imported only if both
  devices signal their fences at the same interrupt priority: the locks
  that wait on the imported fences (waiters, chain points, arrays) are
  created with the importer's priority. ENOTSUP otherwise; README and
  docs/anon-fd.md say so.
* The export's retry path, dead since the booking rule, is gone.
* The kernel race test exports three buffers from four threads, so that
  minor numbers are recycled between objects, and checks every import
  returns its own buffer.

run-r3-test.sh: link rdrmnull with -N fs/specfs ... a36b8c8b7f

Since R2 the module calls makespecvp and specfind, which live in the
specfs module; run-r0-test.sh and run-r1-test.sh already link with it.

Document the sync file pollhead on illumos-ddi's chpoll API ... fc1e54ff2a

Sync files keep their head in a PollBox and register it with
registration_unchecked; sync object files register a device head that is
never woken.

sync_file: allocate the pollhead with KM_SLEEP ... 247bb8b5bc

SyncFile::new runs in thread context with the Sleepable token, so the
head does not need a non-sleeping allocation that can fail with ENOMEM
(review round 2).

Pin illumos-ddi at its r2-private-fd branch for the private tier ... 11074d036b

r2-private-fd (faec57598b) is r3-kms-glue plus the private tier the R2
file descriptors need (drm enables its unstable-gate-cda5b9a49f feature),
the segmap(9E) check, PollBox and the thread-context drop rule for
pollheads. Cargo.lock regenerated from the git dependency, without any
path patch.

toasterson force-pushed r2-sync from 11074d036b to e59b6a8677 2026-09-29 20:27:30 +00:00 Compare

toasterson referenced this pull request 2026-09-29 22:38:33 +00:00

R4: virtio-gpu driver (virgl, scanout, cursor), guest verification pending #7

toasterson referenced this pull request 2026-09-30 22:36:32 +00:00

G6: one DRM device numbering across drivers (misc/drm_minor) and Linux's ioctl size rule #8

toasterson changed target branch from r3-kms to main 2026-10-05 16:36:14 +00:00

toasterson merged commit e2cb8f8b39 into main 2026-10-05 16:36:18 +00:00

toasterson referenced this pull request from a commit 2026-10-05 16:36:18 +00:00

Merge pull request 'R2: fences, dma_resv, syncobj, sync_file and PRIME with anonymous fds' (#6) from r2-sync into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

illumos/illumos-drm!6

No description provided.