Loaders, Editor: Improve handling of assets with unicode characters.#33301
Loaders, Editor: Improve handling of assets with unicode characters.#33301Mugen87 wants to merge 2 commits intomrdoob:devfrom
Conversation
📦 Bundle sizeFull ESM build, minified and gzipped.
🌳 Bundle size after tree-shakingMinimal build including a renderer, camera, empty scene, and dependencies.
|
|
TIL Unicode Normalization Forms. Because I was still unsure, I've asked AI why decodeURIComponent only undoes percent-encoding — it converts %EC%95%88 back to the bytes ec 95 88, which become the character 안. But it has no opinion about which byte sequence represents that character. The Korean syllable 안 can be encoded in UTF-8 two ways: NFC (composed): ec 95 88 — a single codepoint U+C548 안 decodeURIComponent faithfully reproduces whichever form was percent-encoded. If the glTF JSON had NFD bytes, decoding gives you the NFD string. If the zip entry used NFC, decoding gives NFC. They won't match. This is exactly what happens with this test asset: glTF content (written on macOS, which favors NFD): buffer + NFD jamo → 27 bytes for the Korean part The same applies to pure network requests. If we fetch a URL from the glTF JSON, we must normalize it to NFC since all network requests expect NFC-encoded requests. |
|
@donmccurdy I wasn't aware of the topic "Unicode Normalization Forms" until I studied it the last two days. Did you ever encounter a similar issue like #29963 in context of |
Fixed #29963.
Description
The PR improves the loading of glTF assets with URL-encoded UTF8 characters in the editor via drag'n'drop or via "Import". Both the zipped glTF assets as well as the unzipped versions from #29963 can be imported now as expected. Besides, to fix network requests outside of the editor,
LoadingManagerwas also updated.To further explain: There are different Normalization Forms in Unicode. Only when the normalization is handled consistently, the loading process works as expected. The PR makes sure the editor (zipped and drag'n'drop) as well as plain network requests use a NFC normalization.