G7: both DRM stacks on one machine, drmadm selects the one that binds the GPU #10

Merged
toasterson merged 11 commits from g7-coexist into main 2026-10-05 16:46:56 +00:00
Owner

Anima work item: G7 01a1028d-7fdb-7411-9ddb-af9b0f9d4444

Stacked on #9 (r5a-facade); pins illumos-ddi 6e65077 (illumos-ddi #9, g7-module-busy, stacked on its #8). The gfx-drm half is the branch drm-coexist of OpenIndiana/gfx-drm (c8967ec, two commits on master d109414; no PR there yet) and must be merged, and a driver/graphics/drm built from it installed, before a machine loads both stacks.

Both DRM stacks install on one machine, are never loaded together, and drmadm(8) selects the one that binds the GPU and loads at boot.

Guards (both sides)

  • gfx-drm drm_sunmod.c _init: EBUSY + cmn_err warning ("drm: the DRM core misc/drm_core is loaded; ...") while drm_core is loaded. The existing misc/drm_core guard covers the other direction.
  • Both walk modules under mod_lock as mod_name_to_modid() does and compare mod_modname (same for every load path); C reaches modules/mod_lock (sys/modctl.h, private) directly, Rust through ddi::private::module_loaded. No hold: mod_hold_by_name waits for a busy module, and a module is busy in its _init, so two _inits holding each other could deadlock; the walk waits only for mod_lock, which the framework does not hold while calling _init or waiting (modinstall asserts it; mod_hold_by_modctl drops it in cv_wait).
  • Rule (changed in review round 1 on both sides): the other core counts when installed, or in memory and busy (during _init/_fini). A core only in memory (a dependency whose _init failed: mod_install_requisites does not unload it, modunload cannot) does not count; counting it locked the other core out (found in guest run 1). Each core is loaded and busy before its own check, both checks hold mod_lock, so of two concurrent loads at least one refuses.

drmadm (tools/drmadm/drmadm, drmadm.8, package driver/graphics/drmadm, required by drm-core)

  • status: selection (since the boot / from the next boot), packages of both stacks, loaded modules of both, Intel aliases on i915/i915n/unbound, /dev/dri link count, every exclude: naming a DRM module in /etc/system and /etc/system.d.
  • select legacy|new: refuses while a module of either stack is loaded (lists them and the modunload steps) and while /etc/system.d/.drmadm.lock is held; then (1) writes /etc/system.d/driver:graphics:drmadm first (selection, time, the managed alias set, exclude: for the other stack's modules as class/name and bare name — the kernel checks both mod_filename and mod_modname, and only the bare name catches modload /kernel/misc/amd64/drm); (2) moves the Intel aliases with update_drv -n (files only; next boot): -d one alias per call, read back (with a list it matches nothing and exits 0), -a with the list; rollback (aliases back, previous fragment) on failure or HUP/INT/TERM; (3) bootadm update-archive; prints every change. -R root for a mounted BE.
  • Alias set: what the installed driver/graphics/drm declares for i915 (pkg contents -m) ∪ bound to i915/i915n ∪ recorded in the fragment. i915n is reserved for the G5 Intel driver; until it exists select new leaves the aliases unbound (GPU falls back to vgatext via pciclass,030000) and says so. Its package must ship its driver action without aliases (IPS refuses a driver action whose alias another driver action declares); no package here binds an Intel GPU at install.
  • Decision: update_drv over editing driver_aliases (lock, conflict checks, atomic rewrite); -n because the selection is for boot and -a/-d without it would attach/unconfigure now. /etc/system.d exists since 2018 (bootadm assembles .self-assembly; dot files skipped).
  • drm-devlink's SUNW_drm_link serves both stacks. Its removal rule ^dri/[a-z]D?[0-9]+$ matched no card0/controlD64/renderD128; gfx-drm drm-coexist fixes it ([a-z]+), so after a switch devfsadm -C cleans /dev/dri (drmadm prints the step; man page, README, x230 doc include it). libdrm 2.4.134 skips stale links (stat + libdevinfo).

Evidence

Guest g7-run3 on hephaistos (fresh overlay of golden-20260930, illumos 7e802e88a3, virtio-gpu-gl, 2 vCPU/3 GiB); gfx-drm built on oibldofgate as libdrm-update + the two drm-coexist commits (b614900, p5p sha256 8d42d281…59cc9), plus the plain drm-coexist tree (c8967ec); logs logs/oibldofgate/20261003-g7-{coexist,combo}-build.log. run-g7-test.sh phases, all PASS, 81 checks, 0 FAIL (logs/hephaistos/illumos-drm/20261003-g7-run3-*.log):

  • baseline (OI's own drm 2025.0.0.95): drm_core refuses beside legacy; legacy loads beside drm_core (no reverse guard); old link rule leaves dangling /dev/dri/card0.
  • install: both stacks into BE g7 (drm with guard, drm-devlink, libdrm 2.4.134, drm-core, drmadm, virtio-gpu, cdrmnull).
  • guards (22): each core refuses with its warning while the other is loaded, incl. /kernel/misc/amd64/drm by path and dependent i915, virtio_gpu, cdrmnull; stuck dependency (::modctl bits l) does not block; 100 concurrent loads: legacy 51, new 49, neither 0, both 0.
  • overlap (6): DTrace chill in modinstall holds one core loaded+busy before _init; the other refuses, both directions.
  • select-new (13): refused while loaded and while locked (nothing changed); 110 aliases off i915, unbound; fragment with 6 excludes and 110 recorded aliases; .self-assembly has it; second select changes nothing; pkg verify driver/graphics/drm reports the aliases.
  • after-new (7, after reboot): misc/drm, path load and drv/i915 excluded (no guard ran); packaged drivers attach; libdrm lists 2 devices.
  • select-legacy (4) / after-legacy (12, after reboot): 110 aliases back, pkg verify clean; new modules excluded; legacy drm and i915 modules load (no Intel GPU in the guest, so i915 cannot attach); libdrm 0 devices with 4 stale links; devfsadm -C removed all 4.
  • i915n (2): aliases to a stand-in i915n (add_drv -n) and back. select-root (3): -R on a mounted clone only. rollback (5): failing update_drv -d on a scratch root → exit 1, aliases and previous fragment as before. recover-remove/after-recover (5): hand removal + reboot → nothing excluded, either core loads; then select legacy restored the 110 aliases.
  • R0–R5a on the selected new stack: run-r5a-test.sh with LEGACY_GUARD=skip PASS, 75 checks, 0 FAIL (R2 200×2, R4 63, R5a 81×4, run-g6-test PASS, run-r1-test cdrmnull fail=0) — 20261003-g7-run3-r5a-suite2.log. The first attempt (…-r5a-suite.log) failed 16 checks: unprivileged opens of cdrmnull's nodes left 0600 by the uninstalled package, and a /dev/dri state check broken by the fixed link rule (fixed in e0e43e0).
  • No panic in any run (consoles g7-console/). Earlier attempts: run1 found the stuck-dependency lock-out, run2 that r() is a ksh93 alias.
  • Host: fmt, clippy (illumos target), cargo test -p drm-core -p drm-uapi -p drm-facade (124+2+1), shellcheck of drmadm, mandoc -T lint.

Review

Opus, anima-review bar. Round 1 rejected: guards counted stuck dependencies (B1), update_drv -d list no-op (B2), alias loss on failure (B3), stale links undocumented (B4) — all fixed. Round 2 approved; its fix-before-merge items (IPS wording, status version with non-first publisher, rollback ordering, untested rollback, signal during moves, vgatext alias, pkg update note, deterministic overlap test) are done in 3ba24f1/98ea76c.

Open issues

  • gfx-drm drm-coexist must merge (and be packaged) before the X230 runs both stacks.
  • Packaged virtio_gpu/cdrmnull nodes attached at boot are 0600 root root despite minor_perm 0666 (20261003-g7-run3-pkgperm.log); persists across reload, rem_drv -C clears. Packaging question beyond G7.
  • select needs every DRM module unloaded: on the X230 stop the display manager and unload i915/drm first, or use -R.
  • pkg fix or a pkg update of driver/graphics/drm that adds i915 aliases undoes part of select new; documented (rerun select).
  • A pkg operation that creates a BE copies the running one; a later select in the running BE doesn't reach it (documented).
  • The guest test's later phases used the fixed drmadm from the run directory rather than the package installed at 89ea95e.
  • boot -a recovery verified against source (mod_askparams), not in a guest.
Anima work item: G7 `01a1028d-7fdb-7411-9ddb-af9b0f9d4444` Stacked on #9 (`r5a-facade`); pins illumos-ddi `6e65077` (illumos-ddi #9, `g7-module-busy`, stacked on its #8). The gfx-drm half is the branch `drm-coexist` of OpenIndiana/gfx-drm (`c8967ec`, two commits on `master` `d109414`; no PR there yet) and must be merged, and a `driver/graphics/drm` built from it installed, before a machine loads both stacks. Both DRM stacks install on one machine, are never loaded together, and `drmadm(8)` selects the one that binds the GPU and loads at boot. ## Guards (both sides) - gfx-drm `drm_sunmod.c` `_init`: `EBUSY` + `cmn_err` warning ("drm: the DRM core misc/drm_core is loaded; ...") while `drm_core` is loaded. The existing `misc/drm_core` guard covers the other direction. - Both walk `modules` under `mod_lock` as `mod_name_to_modid()` does and compare `mod_modname` (same for every load path); C reaches `modules`/`mod_lock` (`sys/modctl.h`, private) directly, Rust through `ddi::private::module_loaded`. No hold: `mod_hold_by_name` waits for a busy module, and a module is busy in its `_init`, so two `_init`s holding each other could deadlock; the walk waits only for `mod_lock`, which the framework does not hold while calling `_init` or waiting (`modinstall` asserts it; `mod_hold_by_modctl` drops it in `cv_wait`). - Rule (changed in review round 1 on both sides): the other core counts when installed, or in memory and busy (during `_init`/`_fini`). A core only in memory (a dependency whose `_init` failed: `mod_install_requisites` does not unload it, `modunload` cannot) does not count; counting it locked the other core out (found in guest run 1). Each core is loaded and busy before its own check, both checks hold `mod_lock`, so of two concurrent loads at least one refuses. ## drmadm (`tools/drmadm/drmadm`, `drmadm.8`, package `driver/graphics/drmadm`, required by `drm-core`) - `status`: selection (since the boot / from the next boot), packages of both stacks, loaded modules of both, Intel aliases on `i915`/`i915n`/unbound, `/dev/dri` link count, every `exclude:` naming a DRM module in `/etc/system` and `/etc/system.d`. - `select legacy|new`: refuses while a module of either stack is loaded (lists them and the `modunload` steps) and while `/etc/system.d/.drmadm.lock` is held; then (1) writes `/etc/system.d/driver:graphics:drmadm` first (selection, time, the managed alias set, `exclude:` for the other stack's modules as `class/name` and bare name — the kernel checks both `mod_filename` and `mod_modname`, and only the bare name catches `modload /kernel/misc/amd64/drm`); (2) moves the Intel aliases with `update_drv -n` (files only; next boot): `-d` one alias per call, read back (with a list it matches nothing and exits 0), `-a` with the list; rollback (aliases back, previous fragment) on failure or HUP/INT/TERM; (3) `bootadm update-archive`; prints every change. `-R root` for a mounted BE. - Alias set: what the installed `driver/graphics/drm` declares for `i915` (`pkg contents -m`) ∪ bound to `i915`/`i915n` ∪ recorded in the fragment. `i915n` is reserved for the G5 Intel driver; until it exists `select new` leaves the aliases unbound (GPU falls back to `vgatext` via `pciclass,030000`) and says so. Its package must ship its driver action without aliases (IPS refuses a driver action whose alias another driver action declares); no package here binds an Intel GPU at install. - Decision: `update_drv` over editing `driver_aliases` (lock, conflict checks, atomic rewrite); `-n` because the selection is for boot and `-a`/`-d` without it would attach/unconfigure now. `/etc/system.d` exists since 2018 (`bootadm` assembles `.self-assembly`; dot files skipped). ## Links and libdrm - `drm-devlink`'s `SUNW_drm_link` serves both stacks. Its removal rule `^dri/[a-z]D?[0-9]+$` matched no `card0`/`controlD64`/`renderD128`; gfx-drm `drm-coexist` fixes it (`[a-z]+`), so after a switch `devfsadm -C` cleans `/dev/dri` (drmadm prints the step; man page, README, x230 doc include it). libdrm 2.4.134 skips stale links (stat + libdevinfo). ## Evidence Guest `g7-run3` on hephaistos (fresh overlay of `golden-20260930`, illumos `7e802e88a3`, virtio-gpu-gl, 2 vCPU/3 GiB); gfx-drm built on oibldofgate as `libdrm-update` + the two `drm-coexist` commits (`b614900`, p5p sha256 `8d42d281…59cc9`), plus the plain `drm-coexist` tree (`c8967ec`); logs `logs/oibldofgate/20261003-g7-{coexist,combo}-build.log`. `run-g7-test.sh` phases, all PASS, 81 checks, 0 FAIL (`logs/hephaistos/illumos-drm/20261003-g7-run3-*.log`): - baseline (OI's own drm 2025.0.0.95): drm_core refuses beside legacy; legacy **loads** beside drm_core (no reverse guard); old link rule leaves dangling `/dev/dri/card0`. - install: both stacks into BE `g7` (drm with guard, drm-devlink, libdrm 2.4.134, drm-core, drmadm, virtio-gpu, cdrmnull). - guards (22): each core refuses with its warning while the other is loaded, incl. `/kernel/misc/amd64/drm` by path and dependent `i915`, `virtio_gpu`, `cdrmnull`; stuck dependency (`::modctl` bits `l`) does not block; 100 concurrent loads: legacy 51, new 49, neither 0, both 0. - overlap (6): DTrace chill in `modinstall` holds one core loaded+busy before `_init`; the other refuses, both directions. - select-new (13): refused while loaded and while locked (nothing changed); 110 aliases off `i915`, unbound; fragment with 6 excludes and 110 recorded aliases; `.self-assembly` has it; second select changes nothing; `pkg verify driver/graphics/drm` reports the aliases. - after-new (7, after reboot): `misc/drm`, path load and `drv/i915` excluded (no guard ran); packaged drivers attach; libdrm lists 2 devices. - select-legacy (4) / after-legacy (12, after reboot): 110 aliases back, `pkg verify` clean; new modules excluded; legacy `drm` and `i915` modules load (no Intel GPU in the guest, so `i915` cannot attach); libdrm 0 devices with 4 stale links; `devfsadm -C` removed all 4. - i915n (2): aliases to a stand-in `i915n` (`add_drv -n`) and back. select-root (3): `-R` on a mounted clone only. rollback (5): failing `update_drv -d` on a scratch root → exit 1, aliases and previous fragment as before. recover-remove/after-recover (5): hand removal + reboot → nothing excluded, either core loads; then `select legacy` restored the 110 aliases. - R0–R5a on the selected new stack: `run-r5a-test.sh` with `LEGACY_GUARD=skip` PASS, 75 checks, 0 FAIL (R2 200×2, R4 63, R5a 81×4, run-g6-test PASS, run-r1-test cdrmnull fail=0) — `20261003-g7-run3-r5a-suite2.log`. The first attempt (`…-r5a-suite.log`) failed 16 checks: unprivileged opens of `cdrmnull`'s nodes left 0600 by the uninstalled package, and a `/dev/dri` state check broken by the fixed link rule (fixed in `e0e43e0`). - No panic in any run (consoles `g7-console/`). Earlier attempts: run1 found the stuck-dependency lock-out, run2 that `r()` is a ksh93 alias. - Host: fmt, clippy (illumos target), `cargo test -p drm-core -p drm-uapi -p drm-facade` (124+2+1), shellcheck of drmadm, `mandoc -T lint`. ## Review Opus, anima-review bar. Round 1 rejected: guards counted stuck dependencies (B1), `update_drv -d` list no-op (B2), alias loss on failure (B3), stale links undocumented (B4) — all fixed. Round 2 approved; its fix-before-merge items (IPS wording, `status` version with non-first publisher, rollback ordering, untested rollback, signal during moves, vgatext alias, pkg update note, deterministic overlap test) are done in `3ba24f1`/`98ea76c`. ## Open issues - gfx-drm `drm-coexist` must merge (and be packaged) before the X230 runs both stacks. - Packaged `virtio_gpu`/`cdrmnull` nodes attached at boot are `0600 root root` despite `minor_perm` `0666` (`20261003-g7-run3-pkgperm.log`); persists across reload, `rem_drv -C` clears. Packaging question beyond G7. - `select` needs every DRM module unloaded: on the X230 stop the display manager and unload `i915`/`drm` first, or use `-R`. - `pkg fix` or a `pkg update` of `driver/graphics/drm` that adds `i915` aliases undoes part of `select new`; documented (rerun select). - A `pkg` operation that creates a BE copies the running one; a later select in the running BE doesn't reach it (documented). - The guest test's later phases used the fixed `drmadm` from the run directory rather than the package installed at `89ea95e`. - `boot -a` recovery verified against source (`mod_askparams`), not in a guest.
gfx-drm's legacy stack (misc/drm, i915) and this repository's
(misc/drm_core and its drivers) install together; each core refuses to
load while the other is loaded. drmadm(8) chooses which one the system
loads at boot:

- drmadm status: the selection (in effect since the boot or from the next
  one), the installed packages of both stacks, their loaded modules, the
  Intel GPU aliases bound to i915 and to i915n (the name reserved for the
  Intel driver of this stack) and every exclude: line naming a module of
  either stack.
- drmadm select legacy|new: refuses while a module of either stack is
  loaded or another drmadm holds its lock; moves the Intel GPU aliases
  (those driver/graphics/drm declares for i915, those bound to i915 or
  i915n, and those recorded before) with update_drv -n, which edits
  /etc/driver_aliases without attaching anything, leaving them unbound
  when the target driver is not installed; writes
  /etc/system.d/driver:graphics:drmadm with the selection, the alias set
  and exclude: lines for the other stack's modules (class/name and bare
  name, so that an explicit-path modload is excluded too); and runs
  bootadm update-archive. -R root works on a mounted boot environment.

The package driver/graphics/drmadm ships it with its manual page;
driver/graphics/drm-core requires it, and it stays when the new stack is
removed, so that its fragment can still be changed.
The guest test of both DRM stacks on one machine, in phases across
reboots: the baseline on OpenIndiana's driver/graphics/drm, the
installation of both stacks into a new boot environment, both guards
(explicit paths, dependent drivers, concurrent loads of both cores),
drmadm select new and legacy with their refusals, the excludes in effect
after each boot, the stale /dev/dri links after a switch, libdrm's view of
the devices and drmadm -R on a mounted boot environment.

run-r5a-test.sh skips its legacy guard check with LEGACY_GUARD=skip, for a
system where the new stack is selected and the legacy core is excluded.
The section describes both guards and why they cannot deadlock, drmadm's
alias and exclude mechanism (update_drv -n, the /etc/system.d fragment,
the boot archive), the IPS consequences, the device links and recovery.
docs/x230.md is the operator note for the ThinkPad X230.
The first G7 guest run found that misc/drm_core's guard counted gfx-drm's
misc/drm as loaded after a refused load of drv/i915: misc/drm, loaded as
i915's dependency, failed its _init and stayed in memory, not installed,
invisible to modinfo and not removable with modunload, so misc/drm_core
refused until misc/drm was loaded again. illumos-ddi 983751f counts a
module as loaded when it is installed, or in memory and busy (during its
_init or _fini). gfx-drm's guard uses the same rule. The README section
describes it.
syslogd folds repeated warnings into "last message repeated", so the
guard checks count messages in ::msgbuf. The guards phase checks that a
core left in memory by a refused dependency (drv/i915 under
misc/drm_core, drv/virtio_gpu under misc/drm) does not keep the other
core out, and the races start the second load after a delay that sweeps
0 to 98 ms, alternating which core starts first, and must be won by both
cores at least once.
Round-1 review findings:

- update_drv -d -i with a list of aliases matches no line of
  driver_aliases (delete_entry compares the whole argument) and still
  exits 0; drmadm now removes one alias per call and reads the file back.
- The fragment, which records the alias set, is written before the
  aliases move and rewritten (with a new time) whenever they move; if a
  move fails, drmadm gives the aliases back and restores the previous
  fragment.
- select says that devfsadm -C after the next boot removes the other
  stack's /dev/dri links; status counts them; the manual page and
  docs/x230.md include the step.
- Only exclude and EXCLUDE count as exclude lines, as in the kernel; -R
  needs an absolute path and is passed to update_drv quoted.
- The manual page refers to i915(7D) and names vgatext without a manual
  page; the README and drmadm.p5m say what happens to a driver action
  with a taken alias (add_drv refuses it, IPS leaves the driver
  unregistered); the guard's exclusion argument rests on mod_lock.

run-g7-test.sh gains the phase i915n (aliases to a stand-in i915n and
back), and illumos-ddi is pinned at 6e65077.
/bin/sh on illumos is ksh93, which predefines the alias r (hist -s), so
$(r etc/driver_aliases) ran the history command and every awk over a root
file read standard input. Found by the G7 guest run.
With gfx-drm's corrected SUNW_drm_link removal rule, devfsadm -C removes
the /dev/dri links and then the empty directory, so the state check after
the test compares the links only and treats a missing directory as an
empty one. Found by the G7 run of run-r5a-test.sh.
- status shows the version for packages of a publisher that is not
  ranked first (pkg list prints NAME (PUBLISHER) VERSION for them).
- rollback first checks whether the aliases went back: only then is the
  previous fragment restored (or the new one removed); otherwise the new
  fragment, which records them, stays and drmadm says so. A HUP, INT or
  TERM while the aliases move rolls back too.
- README and drmadm.p5m: IPS refuses to install a driver action whose
  alias another driver action in the image declares; vgatext binds
  through pciclass,030000; select legacy gives i915 at least its
  package's aliases; after pkg fix or a pkg update of driver/graphics/drm
  that adds i915 aliases, select new again (also in drmadm(8) and
  docs/x230.md).
- run-g7-test.sh: overlap holds one core in modinstall with a DTrace chill
  (after mod_loaded and mod_busy are set, before _init) and loads the
  other, which must refuse; rollback makes every update_drv -d fail on a
  scratch root and checks the exit status, the aliases and the fragment;
  recover-remove and after-recover exercise the hand recovery.
README: G7 results and open issues; run-g7-test.sh: start the overlap's second load once the chill holds
Some checks failed
ci / check-hosted (pull_request) Failing after 19m38s
ci / kmod-illumos (pull_request) Failing after 58m39s
98ea76c50f
The overlap phase waits for DTrace's report that the chill holds the first
core (switch rate 1000 Hz) before it loads the second; with a fixed 150 ms
delay the second load came before the first reached modinstall. docs/x230.md
notes that a selection does not reach a boot environment pkg created
before it.
toasterson changed target branch from r5a-facade to main 2026-10-05 16:46:52 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
illumos/illumos-drm!10
No description provided.