Repository navigation
test(onnx): magika_idle_threads fails on macOS with ORT 1.31 (Neural Engine threads, not an intra-op pool) #4191
Description
Description
crates/headroom-core/tests/magika_idle_threads.rs (added in #4189 for #4187) fails on macOS when ORT_DYLIB_PATH points at ONNX Runtime 1.31.0. It reports 8 to 9 extra OS threads against an allowance of 2. The same test passes against ORT 1.30.0 on the same machine and commit.
None of the extra threads is an intra-op pool, and none spins. Five are Apple Neural Engine service threads, started by Apple's frameworks once ORT 1.31.0 has loaded. So the #4187 fix still holds. The test's thread count just can't tell these threads from a pool on macOS with this ORT version. CI only runs this test on Linux, which is why it passes there.
To Reproduce
On an Apple Silicon Mac, at main (f1bc2609):
pip install 'onnxruntime==1.31.0' # any venv; only the dylib is used
export ORT_DYLIB_PATH=$(python -c "import onnxruntime, glob, os; print(glob.glob(os.path.join(os.path.dirname(onnxruntime.__file__), 'capi', 'libonnxruntime.*.dylib'))[0])")
cargo test -p headroom-core --test magika_idle_threadsTo compare, repeat with onnxruntime==1.30.0.
Expected Behavior
The test passes, as it does on Linux and with ORT 1.30.0 on macOS.
Actual Behavior
# ORT 1.31.0
---- magika_session_does_not_spawn_intra_op_pool stdout ----
thread 'magika_session_does_not_spawn_intra_op_pool' panicked at crates/headroom-core/tests/magika_idle_threads.rs:53:5:
magika session left 8 extra OS thread(s) running (before=2, after=10)
test result: FAILED. 0 passed; 1 failed
# ORT 1.30.0, same machine and commit
test result: ok. 1 passed; 0 failed
The count was 9 on one of the runs.
What the extra threads are
I added a throwaway probe test that runs the same magika_detect call. It prints ps -M before the call, 200 ms after, and after 3 s idle, and it runs sample against the idle process.
threads after magika_detect |
extra | |
|---|---|---|
| ORT 1.30.0 | 4 | 2 |
| ORT 1.31.0 | 10 | 8 |
Under ORT 1.31.0, sample attributes the threads like this:
- 5 ×
ANEServicesThread:ANE::ANEServicesThreadStart(inANEServices) →CFRunLoopRun→mach_msg2_trap. They block on a Mach port, run at priority 58 inps -M, and show 0.0% CPU after 3 s idle. The process hasCoreML,Espresso,AppleNeuralEngineandANECompilerloaded. - 1 ORT telemetry worker:
Microsoft::Applications::Events::PlatformAbstraction::WorkerThread::threadFunc(inlibonnxruntime.1.31.0.dylib), waiting on a condition variable. This is one of the two background threads the test already allows for. - 2 libdispatch work-queue threads (
start_wqthread→__workq_kernreturn), idle.
No thread is a spinning intra-op pool worker, which is what the test guards against.
Both the 1.30.0 and 1.31.0 dylibs link CoreML.framework (otool -L), so linking isn't the difference. My guess is that 1.31.0 initializes CoreML and the Neural Engine at environment or session setup, perhaps while discovering execution-provider devices, and 1.30.0 doesn't. I haven't confirmed that in ORT's source.
Possible fixes
- Count only threads that use CPU while idle, instead of all threads. For example, sample each thread's CPU time across the idle period and fail when one advances. That checks the real symptom (fix(onnx): magika detector session still spins idle cores (#2540 covered only the Python factory) #4187 is about spinning) and doesn't depend on how many framework threads the OS starts.
- On macOS, leave
ANEServicesThreadout of the count (ps -Mdoesn't show thread names, so this would needproc_pidinfoorthread_info), or raise the macOS allowance. Simpler, but brittle the next time Apple or ORT adds a thread. - Leave the assertion as it is and document that it's only meaningful on Linux (
cfg(target_os = "linux")). The macOS branch already notes that a small pool can hide inside the allowance.
I'd lean toward 1, and I'm happy to send a PR if that direction sounds right.
Environment
- Headroom:
mainatf1bc2609(also reproduced on a branch rebased on it) - OS: macOS 27.0.1, Apple M5, 10 cores
- Rust: 1.95.0 (repo
rust-toolchain.toml) - ONNX Runtime: 1.31.0 fails, 1.30.0 passes (dylibs from the PyPI
onnxruntimewheels)
Additional Context
- Without
ORT_DYLIB_PATH(and no piponnxruntimefor the loader to find), the test skips withmagika runtime unavailable on this host: no pip onnxruntime native library was found …. So the failure only shows up when ORT 1.31 is loaded. - CI (
rust.yml) installsonnxruntime>=1.24and runs only on Linux, where the count comes from/proc/self/task.
Activity
rominparekh commentedon Oct 14, 2026
@KennethWKZ : thank you for reporting the issue with ONNX Runtime 1.31.0, I am testing
mainatf1bc2609against this version to reproduce the issue and see if any of your suggested fixes can be quickly applied to fix.test(onnx): check magika for spinning threads by CPU time, not thread…
KennethWKZ commentedon Oct 14, 2026
@rominparekh thanks! To save you the work: #4193 implements option 1. The test now checks per-thread CPU over a light-load and idle window instead of counting threads. On an M5 it passes with ORT 1.30.0 (busiest other thread 0 ms) and 1.31.0 (1.5–3.5 ms). With #4189's fix reverted, it still fails on both: four pool workers at 331–377 ms each. The ANE thread stacks are in the issue above. Happy to adjust if you'd rather go another way.
rominparekh commentedon Oct 14, 2026
Thanks for the very thorough write-up, @KennethWKZ. I wrote that test in #4189, so I dug in. I can reproduce it: ORT 1.31.0 shows 8 extra threads against an allowance of 2, and 1.29.0 passes. I also found what starts the Neural Engine, which confirms your guess.
Why 1.31 starts the Neural Engine
PREFER_NPU"). In the source,ep_library_internal.ccgainedinternal_eps.push_back(CreateCoreMLEp()). On 1.30 nothing claimed the NPU, which is why linkingCoreML.frameworkmade no difference.ortcrate prefers the NPU by default.ort2.0.0-rc.12 setsSessionOptionsSetEpSelectionPolicy(MAX_EFFICIENCY)on everySessionBuilder::new()(src/session/builder/mod.rs, underapi-22, which we enable).MAX_EFFICIENCYcurrently meansPREFER_NPU.Measured on macOS with ORT 1.31.0, building the magika model through raw
ortwithintra_threads=1:ortdefault (MaxEfficiency)with_auto_device(PreferCPU)ort::init_from(..).commit()on its own adds no ANE threads. They appear when the session is built, and some exit when it's dropped.Why option 1 wouldn't catch #4187
When I measured #4187, the pool workers spun only briefly after each inference. A fully idle process sat at about 0% CPU with or without the pool. The cost only appeared under steady light load, roughly 190% → 24% of a core at one detection every 50 ms. So an idle-CPU check would pass even with the pool back. A load-based CPU check would catch it, but it's timing-dependent and likely flaky on shared runners.
Proposed fix (test and CI)
Build a pool-free control session (
magika::Session::builder().with_intra_threads(1)) first, and use the threads it adds as the allowance instead of a fixed2. The control goes through the same EP selection as headroom's session, so whatever its EP starts (Core ML/ANE now, something else on a future ORT or OS) cancels out, and a pool still shows up on top. With the #4187 fix reverted:It passes in all three with the fix, 6/6 repeat runs on 1.31 passed, and your exact repro steps pass.
Rust CI only runs on Ubuntu today, which is why this slipped through. As a precaution, the change also adds a
macos-26entry to thetestjob inrust.yml. It reuses the same ONNX Runtime setup and runscargo testonly; Ubuntu's checks are unchanged.Separate question for maintainers (not in this fix)
This also means that on macOS with ORT ≥ 1.31, headroom's magika detection runs through Core ML on the Neural Engine instead of the CPU EP. Nobody chose that; it comes from
ort's default policy. The visible cost is a one-off ~400 ms session build plus 5–7 idle threads. I haven't measured per-call latency or whether outputs differ. Pinning it to CPU would needwith_auto_device(PreferCPU)on the session, but magika 1.1.0'sBuilderdoesn't expose that. I checked that an environment-level CPU EP (ort::init_from(..).with_execution_providers([CPU])) does not override the policy. If that's worth addressing, we should open a follow-up issue on themagikacrate requesting to add a pass-through.I have the test and CI change ready and will raise a PR for you and maintainers to review.
rominparekh commentedon Oct 14, 2026
@KennethWKZ , our comments overlapped and I believe we have two PRs open for this issue - #4193 and #4194. :)
KennethWKZ commentedon Oct 14, 2026
@rominparekh closed #4193 in favour of #4194, which is approved now and passes the magika check on macOS too. Thanks for running down the Core ML default. The offer stands: happy to open the
magikacrate issue about exposing the EP selection policy on itsBuilder, if pinning magika to CPU is wanted.test(onnx): check magika for spinning threads by CPU time, not thread…