wasm: stream large media from Blob URLs instead of copying into RAM (Tier B)#2148
Open
jcelerier wants to merge 4 commits into
Open
wasm: stream large media from Blob URLs instead of copying into RAM (Tier B)#2148jcelerier wants to merge 4 commits into
jcelerier wants to merge 4 commits into
Conversation
Tier A copies every dropped/imported file into MEMFS (RAM), which is capped
by the 2GB wasm heap and cannot handle multi-GB media. Tier B streams large
files on demand instead: a custom AVIOContext whose read/seek callbacks drive
a QFile opened on the browser "weblocalfile:" Blob URL, so ffmpeg pulls only
the byte-ranges it needs (QWasmFileEngine does true Blob.slice range reads)
and nothing is ever fully buffered.
- Media/AvStreamIO.{hpp,cpp}: AvIoDevice (QFile-backed AVIOContext),
isStreamedMediaPath(), open_input_custom_io(); registers the libossia
libav_handle open hook so the audio-stream path streams too.
- Route the file-open sites through custom IO for weblocalfile: paths:
VideoDecoder, AudioDecoder (open_audio), LibavMediaInfo probe (drop-time
stream/duration detection), DirectVideoNodeRenderer and VideoThumbnailer
(both via the libav hook). Small/normal paths keep the fopen fast path.
- ProcessDropHandler::getDrop: files >48MB on weblocalfile: are no longer
staged into MEMFS; the weblocalfile URL is kept and streamed. Small files
still stage (Tier A). The file-extension drop loop now resolves
weblocalfile: URLs (empty toLocalFile()).
- SoundDrop and relativizeFilePath: accept/preserve weblocalfile: URLs.
Verified headed: a 74MB mp4 drop opens via custom IO reading only ~9% of the
file (header + moov tail seek), proving no full-file copy; a small mp4 still
stages and decodes unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019oPj1zRcxSQX7FNni7EHM6
A weblocalfile: QFile is backed by a browser File/Blob whose emscripten::val handle is bound to the thread that created it (the main thread, at drop time). Reading it from a worker thread dereferences an invalid handle and crashes with "Cannot read properties of undefined (reading 'size')" inside qstdweb::BlobIODevice::size(). The confirmed trigger is Video::VideoThumbnailer, which moveToThread()s onto a worker and demuxes there; the audio waveform and worker-thread decoders hit the same bug. - AvIoDevice now marshals the non-suspending QFile operations (open, seek, size, close) to the main runtime thread via emscripten_proxy_sync, so the Blob val is only ever touched by its owner. - av_io_read fails cleanly (AVERROR(EIO)) when called off the main thread: reading the Blob suspends via JSPI, which only works on the main thread's "promising" call stack, not on a proxied job (it traps with SuspendError). This makes any off-thread decode abort gracefully instead of crashing. - Skip the timeline thumbnail (Gfx/Video/View) and the audio waveform (Media/Sound/SoundView) for streamed sources: they scan on worker threads, and re-scanning a streamed (typically huge) file is undesirable anyway (B3). Full analysis and the rejected alternatives (blocking proxy-to-main is blocked by JSPI; FileSystemSyncAccessHandle is OPFS-only) are in WASM-B2-ANALYSIS.md. Worker-thread streaming playback remains a follow-up (needs an async-proxy + worker-semaphore reader). Verified headed: a 74MB mp4 drop no longer crashes (video + sound processes created, correctly-sized interval), still streams from the main thread; a small staged mp4 keeps full thumbnails and waveform. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019oPj1zRcxSQX7FNni7EHM6
…ead reader Worker-thread decode of a streamed source now reads real bytes instead of failing with AVERROR(EIO). The Blob's JS val is bound to the main thread and the main thread can only suspend (JSPI) on a promising stack, so a worker cannot read the Blob directly nor via a blocking proxy. Instead AvIoDevice drives the reads asynchronously: - Position/size bookkeeping is authoritative in AvStreamState; seek is pure integer math (no I/O), so it needs no thread hop. - Read on the main thread: slice the Blob and arrayBuffer_sync() (JSPI, legal on the main event loop's promising stack). - Read on a worker: emscripten_proxy_async a job to the main thread, then block on a futex. The main job kicks off Blob.arrayBuffer() (no suspend) and its .then copies the bytes into the shared AVIO buffer, records the count, and wakes the worker via Atomics.notify. The Blob val is fetched fresh on the main thread each read (QWasmFileEngineHandler::getFile), so no val ever crosses a thread. - QFile is dropped entirely from the streaming path; media now links Qt::CorePrivate for the wasm file-engine access. Deadlock-safe as long as the main thread keeps pumping its event loop (it does during playback); it must not be join()-blocked on a decode worker mid-read. Validated headed: a 74 MB weblocalfile mp4 plays past the first frame, streaming on demand (~6 MB read during playback), the main thread stays responsive (no freeze), and no crash / SuspendError occurs. The audio decode worker streams through the same path. Small/staged files are unaffected (they never take the streaming path). Analysis and rejected alternatives are in WASM-B2-ANALYSIS.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019oPj1zRcxSQX7FNni7EHM6
Bump the custom AVIO buffer from 256 KB to 4 MB and add a 16 MB read-ahead window to AvStreamState: on a miss, one Blob read prefetches 16 MB at the current position and the following sequential 4 MB reads are served from it with no thread hop. The window is >= the AVIO buffer so ffmpeg's reads hit; seek is pure position math and a straddling read just refills; the window is clamped to the file size at EOF. Measured headed on a 4.3 GB 4K ProRes clip (strobe_scientist.mkv), playback window vs the old 256 KB / no-cache path: main-thread round-trips ~16,150 -> 50 (~323x fewer), bytes fetched 4.0 GB -> 0.78 GB (~5x less), summed I/O-wait ~15.9 s -> 0.81 s (~20x less). Still streams on demand (0.78 GB of 4.3 GB for the portion played), no crash/deadlock. Playback framerate is unchanged (~0.5x realtime) because it is decode-bound: after this change avcodec is ~94% of playback wall time and I/O ~3%. The win here is far lower main-thread contention with WebGL and much lower bandwidth, not the framerate. See WASM-STREAM-PERF-ANALYSIS.md; the framerate lever is multithreaded decode (VideoDecoder.cpp:108 thread_count=1), left as-is pending a teardown-crash fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019oPj1zRcxSQX7FNni7EHM6
|
coverage: 15.303% (-0.008%) from 15.311% — wasm-tier-b into master |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Streams large media on the WebAssembly build instead of copying it into RAM.
Until now every dropped or imported file was copied whole into MEMFS, which is
bounded by the 2 GB wasm heap — so multi-GB media simply could not be opened. This
adds a custom
AVIOContextthat pulls byte ranges out of the browserBlobbehinda
weblocalfile:URL on demand, so ffmpeg reads only what it needs and nothing isever fully buffered.
Four commits, each independently verified in a real browser:
Stream from disk without copying into RAM.
Media/AvStreamIO.{hpp,cpp}addsAvIoDevice,isStreamedMediaPath()andopen_input_custom_io(), and installs thelibossia open hook so the audio path streams too. Drops above 48 MB keep the
weblocalfile:URL rather than staging it; smaller files keep the existing fastpath unchanged. Measured: a 74 MB mp4 opens reading ~9 % of the file — header plus
the moov tail seek.
Fix a cross-thread crash. A
weblocalfile:Blob'semscripten::valis bound tothe thread that created it, so touching it from a worker crashed inside
qstdweb::BlobIODevice::size(). Non-suspending operations are now marshalled to themain runtime thread, and off-thread reads fail cleanly rather than trapping.
Read on worker threads. Workers previously could not read at all (JSPI only
suspends on the main thread's promising stack). Reads are now proxied
asynchronously to the main thread, which starts
Blob.arrayBuffer()and wakes thewaiting worker through a futex — so no
valever crosses a thread.QFiledrops outof the streaming path entirely; media links
Qt::CorePrivatefor file-engine accesson wasm only.
4 MB AVIO reads + 16 MB read-ahead. Measured headed on a 4.3 GB 4K ProRes clip:
main-thread round-trips 16,150 → 50 (~323× fewer), bytes fetched 4.0 GB → 0.78 GB
(~5× less), summed I/O wait 15.9 s → 0.81 s (~20× less).
Framerate is unchanged (~0.5× realtime) because playback is decode-bound — after this
change avcodec is ~94 % of wall time and I/O ~3 %. The win is main-thread contention
and bandwidth, not throughput; the remaining lever is multithreaded decode
(
VideoDecoder.cppthread_count=1), left alone pending a teardown-crash fix.Scope
Desktop is unaffected: the wasm-specific code in
AvStreamIO.cppis behind__EMSCRIPTEN__, and non-wasm paths keep the fopen fast path. The file set here doesnot overlap #2144, #2145, #2146 or #2147, so it can land in any order relative to them.
Requires ossia/libossia#912 — this bumps the submodule to it.
🤖 Generated with Claude Code
https://claude.ai/code/session_019oPj1zRcxSQX7FNni7EHM6