Add HTML-in-Canvas APIs by foolip · Pull Request #11588 · whatwg/html

foolip · 2025-08-21T12:51:56Z

At least two implementers are interested (and none opposed):
- Chromium
- …
Tests are written and can be reviewed and commented upon at:
- …
Implementation bugs are filed:
- Chromium: https://issues.chromium.org/u/1/hotlists/6783002
- Gecko: …
- WebKit: …
- Deno (only for timers, structured clone, base64 utils, channel messaging, module resolution, web workers, and web storage): …
- Node.js (only for timers, structured clone, base64 utils, channel messaging, and module resolution): …
Corresponding HTML AAM & ARIA in HTML issues & PRs:
MDN issue is filed: …
The top of this comment includes a clear commit message to use.

(See WHATWG Working Mode: Changes for more details.)

/canvas.html ( diff )
/dom.html ( diff )
/index.html ( diff )
/indices.html ( diff )
/infrastructure.html ( diff )
/references.html ( diff )
/rendering.html ( diff )
/webappapis.html ( diff )

Handwavy things that need fleshing out are marked with 👋

Kaiido

Glad to see this being worked on, thanks.

Not quite sure how much discussion should be held at this stage. So to note, this doesn't seem to fully match the latest state of https://github.com/WICG/html-in-canvas. e.g. the rename to drawHTMLElement. The layoutsubtree attribute is also missing along with the implications to the existing fallback contents.
Still, thanks for making this move.

foolip · 2025-08-29T09:47:43Z

@Kaiido thank you for the review! I've fleshed things out more, renaming to drawHTMLElement() (because drawElement() will probably not be workable in WebGL) and adding the layoutsubtree attribute.

There's still some handwaving going on of course, in particular what causes the subtree to be laid out but not painted.

foolip · 2025-09-17T16:21:14Z

I've fleshed this out some more now, in particular the hit testing.

Kaiido

One common complain with the use of dictionaries in the Canvas2D API is that this makes GC kick in very often during animations which has a non-negligible performance cost.
This API shape makes a big use of such dictionaries with one for the wrapper CanvasElementHitTestRegion and then a nested one for the CanvasHitTestRect, and I guess there will be scenarios where multiple of these will need to be updated at every frame. Since the values are copied over from the passed objects to new internal objects, it's unclear if even a careful author, who would try to reuse the same objects, could avoid GC at all here.

On the other hand, I really like how this API shape enables future additions like using a Path2D, or even a bitmap mask, instead of a CanvasHitTestRect. (btw can we bikeshed on rect for that purpose?)

It's not my area of expertise, but would an actual exposed interface allow for non copy from JS, so that authors can just update the regions instead of setting new ones?

Kaiido · 2025-09-23T01:32:28Z

+
+  <ol>
+   <li>
+    <p>For each <span>hit test region</span> <var>region</var> in <var>canvas</var>'s <span>hit test regions</span>:</p>


Definitely an edge case, and I suppose it's a bit of a gray area (see whatwg/infra#396) but how is this supposed to work if setHitTestRegions is called during this iteration? E.g.

// Add multiple regions ctx.setHitTestRegions([ { element, rect: { x, y } }, { element: anotherElement, rect: { x: anotherX, y: anotherY } } ]); element.onclick = e => ctx.setHitTestRegions([]); // that was a 'once' handler

Should the anotherElement still perform the hit-test?

This kind of problem is sometimes handled in the spec by making a frozen copy of the thing to iterate before starting iteration. Do you think that'd be OK here?

That would certainly be clearer as to what's supposed to happen yes. Now, whether it's the best behavior or not, I don't know and don't have any strong opinion. Both possibilities might come surprising depending on the case. The fact that the timing of hit-testing w.r.t. events propagation isn't well defined doesn't help...

@szager made a good point here which is that really when we start dispatching events hit testing is already done. So rather than making a copy of the list here, the spec here needs to make clear how the list is used in hit testing and that it all happens before event dispatch.

tabatkins · 2026-03-26T21:00:09Z

Drive-by comment about styling/layout of the contents of a layoutsubtree canvas:

Canvas being a replaced element doesn't stop selectors from matching its children; that part already works normally.

It does prevent us from laying out the children, absent instructions on how to do so. I think what's expected here is that the elements lay out as normal children of the canvas, with the appropriate containments applied by the UA style sheet, right? Like, the canvas is just an ordinary block container as far as they're concerned. We can just specify that, and further say (already present in the spec) that they don't paint, but they do respond to hit-tests, selections, etc as normal. The element snapshot is then them doing a special paint operation.

Or does each child lay out independently, as if it's the sole child of the canvas, so that when you draw we can just snapshot from the canvas's 0,0 point and place that at the draw location? That works too, it just needs some clarifying text if so.

AutoSponge · 2026-03-28T18:57:37Z

Can we get a canvas method that renders CSS anchors at (X,Y) positions as shadowdom elements? Then, canvas children just anchor position to their respective points by name and we get to use actual HTML on canvas instead of images of HTML in canvas? Or is there another use case here that I don't see?

progers · 2026-03-28T22:14:22Z

Can we get a canvas method that renders CSS anchors at (X,Y) positions as shadowdom elements? Then, canvas children just anchor position to their respective points by name and we get to use actual HTML on canvas instead of images of HTML in canvas? Or is there another use case here that I don't see?

Hi @AutoSponge , are you asking why we don't use anchor positioning to move elements on top of the canvas, like https://github.com/shuding/cobe? This is a good question, as this API ultimately does need to keep the DOM synchronized with what is being drawn into the canvas. Our goal with this API is to enable patterns that are just not possible today, such as interleaving canvas drawing commands with html (e.g., jelly slider). The explainer lists the use cases this API is solving.

AutoSponge · 2026-03-29T16:39:25Z

In the explainer, what you're calling the <canvas> fallback others have used to build the semantic proxy layer (a hack that <canvas> never intended) that makes their UI accessible, and claiming there's some accessibility improvement/capability by this proposal. I've tried and I still don't see it. What's the % of scenarios where "fallback" content is perceived or interacted with visually?

I want to help devs use <canvas> but I don't want to see a proliferation of graphical UIs with no concern for accessibility. The explosion of TUIs and "GPU-powered" web interfaces are almost complete garbage for accessibility. If this is purely about putting CSS/browser-rendered content in <canvas>, then I'd recommend removing the accessibility claims. If I'm missing something, please help me understand. Thanks.

Kaiido · 2026-03-31T23:29:32Z

+  data-x="dom-context-2d-drawElementImage">drawElementImage()</code>. An <span
+  data-x="concept-element-image-snapshot">element image snapshot</span> has a <dfn
+  data-x="concept-element-image-snapshot-width">width</dfn> and <dfn
+  data-x="concept-element-image-snapshot-height">height</dfn> (dimensions in <span>output


This should probably reference the "parent" canvas so that we can link to its output bitmap.

Kaiido · 2026-03-31T23:49:40Z

+   <li><p>If <var>dw</var> and <var>dh</var> are given and either are infinite or NaN, then
+   return.</p></li>
+
+   <li><p>Let <var>snapshot</var> be the result of <span>get element image snapshot</span> given


Other Canvas2D API tend to silently not draw rather than throwing. But I don't know what to think about that legacy API design. Here in particular, since it also does return an object and isn't void like e.g. drawImage, it might be better to have it throw.
However I think other invalid attributes should also throw (maybe RangeErrors?) rather than silently return undefined (which will potentially cause throwing lower down the authors' code).

+1, to specifying that the error conditions throw. here are the associated tests for this. Can you update "get element image snapshot" to throw a InvalidStateError if called prior to paint?

Kaiido · 2026-04-01T00:04:21Z

  <span>ImageBitmap</span> <span data-x="dom-OffscreenCanvas-transferToImageBitmap">transferToImageBitmap</span>();
  <span data-x="idl-Promise">Promise</span>&lt;<span>Blob</span>> <span data-x="dom-OffscreenCanvas-convertToBlob">convertToBlob</span>(optional <span>ImageEncodeOptions</span> options = {});

+  <span>DOMMatrix</span> <span data-x="dom-OffscreenCanvas-getElementTransform">getElementTransform</span>((<span>Element</span> or <span>ElementImage</span>) element, <span>DOMMatrix</span> drawTransform);


Element should be removed here.
At first I was wondering if it would work with Web-IDL in workers, but that doesn't matter, the get element image snapshot algorithm would anyway not work with an Element since the context is bound to an OffscreenCanvas, not a <canvas>.

Kaiido · 2026-04-01T00:05:20Z

+
+   <li><p><span>Assert</span>: <var>element</var> is an <code>Element</code>.</p></li>
+
+   <li><p>Let <var>canvas</var> be the <code>canvas</code> element to which <var>context</var> is


Should there be an assert canvas is a <canvas> and not an OffscreenCanvas?

Oh and context is actually undefined here, unlike draw an element above, this algorithm doesn't take a context input, and can be called from <canvas>'s or OffscreenCanvas's getElementTransform() methods directly, before there is any context.

Kaiido · 2026-04-01T00:09:16Z

+    <var>element</var>.</p></li>
+
+    <li><p>Return the result of <span>get element transform</span> with <span>this</span>'s <span
+    data-x="offscreencanvas-placeholder">placeholder <code>canvas</code> element</span>,


What when there is no placeholder?
And even when there is one, can this work synchronously across realms?

Kaiido · 2026-04-01T00:14:49Z

+
+   <dd>
+    <p>Returns a <span>frozen array</span> of the elements that have changed, or the
+    <code>ElementImage</code> objects representing those elements for


Does it still fire on OffscreenCanvas?

hybridherbst · 2026-04-07T18:28:27Z

Is it correct that so far only Chromium has shown intent to ship, or are there responses from Firefox/Webkit/… as well?

Kaiido · 2026-04-09T11:44:08Z

+  <dl>
+   <dt>All content
+   <dd><span>CORS-cross-origin</span> data.
+   <dd>Non-default colors, fonts, themes, and preferences.


What is a non-default font? Or rather, what is a default font? If it must be a font that's available on all platforms, wouldn't that prevent basically all the fonts?
Moreover, aren't fonts already exposed by fillText() (minus a few settings that are already exposed through <foreignObject>)?

Also regarding this paragraph, are all these testable?

+1, can you remove "fonts" from this list? This is already readily available via fillText/layout.

Also regarding this paragraph, are all these testable?

The cross-origin parts are testable. The theme colors are not testable (web-platform-tests/wpt#29309).

progers · 2026-04-28T21:07:56Z

+
  attribute <span>EventHandler</span> <span data-x="handler-offscreencanvas-oncontextlost">oncontextlost</span>;
  attribute <span>EventHandler</span> <span data-x="handler-offscreencanvas-oncontextrestored">oncontextrestored</span>;
+  attribute <span>EventHandler</span> <span data-x="handler-offscreencanvas-onpaint">onpaint</span>;


Can you remove the offscreen canvas paint event (here and line 76636)? We can let developers implement this themselves by listening to the canvas element's paint event, and forwarding this information to the offscreen canvas. For example, in the case of an offscreen canvas in a worker thread, the developer can postMessage the updated ElementImage snapshots to the worker thread.

progers · 2026-04-28T22:26:47Z

+  <ol>
+   <li><p><span>Ensure non-nested canvas</span> given the canvas element to which <var>context</var>
+   is bound.</p></li>
+


I think we need to specify that we only allow drawing Elements into their parent canvas, and also that we only allow drawing ElementImages into a context of a canvas that they are associated with. (I have a CL in review to implement the latter in blink). Can this be specified with a "validate the Element/ElementImage" step that works for both Element and ElementImage, and that can be referenced by both drawElementImage and getElementTransform? Would this allow us to remove the "Ensure non-nested canvas" section?

The CL mentions security reasons, could you expand on it?

To me, the sizing issue seems to reflect more that the sizing rules in general are fragile, and I don't see how this solution would handle e.g. the case of a source canvas that's been resized between the time the ElementImage has been generated and the time it's drawn.

progers · 2026-04-28T23:01:21Z

+
+  <dl>
+   <dt>All content
+   <dd><span>CORS-cross-origin</span> data.


Can you expand on this? Suggestion: This includes cross-origin data in embedded content (e.g., <iframe>, <img>), references (e.g., background-image, clip-path), <canvas> elements tainted with cross-origin data, and SVG (e.g., <use>, <pattern>, <feImage>). Note that same-origin iframes would still paint, but cross-origin content in them would not.

progers · 2026-04-28T23:29:34Z

+    <p>If <var>element</var> is an <code>ElementImage</code>:</p>
+
+    <ol>
+     <li><p>If <var>element</var>'s <span>[[Detached]]</span> internal slot's value is true, then


We need to fix WICG/html-in-canvas#116 in the implementation. Do we need to update the spec for this too?

Stub out canvas.drawElement()

e181de1

Handwavy things that need fleshing out are marked with 👋

foolip marked this pull request as draft August 21, 2025 12:52

Kaiido reviewed Aug 25, 2025

View reviewed changes

Comment thread source Outdated

Comment thread source Outdated

Comment thread source Outdated

Comment thread source Outdated

Comment thread source Outdated

Comment thread source Outdated

foolip mentioned this pull request Aug 28, 2025

Drawing only part of an element (cropping) WICG/html-in-canvas#37

Closed

foolip added 5 commits August 28, 2025 14:25

Fix typ: y->h

8634ebe

Modernize the style somewhat

afc4a4d

Replace handwavy check with real check and drop "update the rendering"

f151c83

Add the layoutsubtree attribute

74d67ae

Rename to drawHTMLElement()

f7d320a

foolip changed the title ~~Stub out canvas.drawElement()~~ Stub out canvas.drawHTMLElement() Aug 29, 2025

foolip mentioned this pull request Sep 2, 2025

HTML-in-Canvas #10650

Open

Fix the layoutsubtree UA style sheet

3cad6d4

foolip commented Sep 2, 2025

View reviewed changes

Comment thread source Outdated

foolip changed the title ~~Stub out canvas.drawHTMLElement()~~ Stub out canvas.drawElementImage() Sep 16, 2025

foolip added 6 commits September 16, 2025 17:06

Rename to drawElementImage()

4367aac

Define sensitive information

84ec8e6

Expose children to AT

f351c3d

grammar

aed994e

Use containment to make children containing blocks

8c3ce46

Define setHitTestRegions()

ff066c9

foolip changed the title ~~Stub out canvas.drawElementImage()~~ Add HTML-in-Canvas APIs Sep 17, 2025

Kaiido reviewed Sep 23, 2025

View reviewed changes

foolip commented Sep 30, 2025

View reviewed changes

Comment thread source Outdated

foolip added 5 commits October 7, 2025 15:26

Merge remote-tracking branch 'origin/main' into foolip/html-in-canvas

5e3c3d0

Wrap algorithms

f8f119a

Fix typo and reserved word

202adae

Add a note about not using CTM

be22310

Fix reference to hit test regions

9866034

foolip added 4 commits March 27, 2026 09:40

CanvasPaintEvent's changedElements can only be Element

0a158e6

Capture the scaling factor when creating snapshots

2df6477

Subpixel rendering is sensitive information

c5e2815

Say that selection and find-in-page highlight colors are sensitive

2850d56

foolip added 3 commits March 31, 2026 12:30

Remove ElementImage's id

f4f738b

Add ElementImage close() method

09b811d

Throw on drawing a detached ElementImage

542723b

Kaiido reviewed Apr 1, 2026

View reviewed changes

Make ElementImage width/height double

7aee3ee

foolip added 2 commits April 9, 2026 12:39

Make non-default fonts sensitive information

2c95e37

Don't expose platform-specific form appearance

0a476ef

Kaiido reviewed Apr 9, 2026

View reviewed changes

foolip mentioned this pull request Apr 9, 2026

Specify private/sensitive information when drawing <svg><foreignObject> to canvas #12350

Open

Require snapshots to be scalable (not bitmaps)

71347b5

olivermrose mentioned this pull request Apr 9, 2026

Add html-in-canvas proposal Fyrd/caniuse#7507

Open

foolip added 3 commits April 10, 2026 16:46

Disallow same-origin nesting of canvas elements

22b725e

Replace shadow tree bits with simpler rendering rules

959ca6b

Remove stray quotes

3350248

tabatkins reviewed Apr 17, 2026

View reviewed changes

Comment thread source Outdated

foolip added 2 commits April 24, 2026 17:41

Merge remote-tracking branch 'origin/main' into foolip/html-in-canvas

cb6c69d

Add layout containment

ae964a9

progers reviewed Apr 28, 2026

View reviewed changes


		<li><p><span>Assert</span>: <var>element</var> is an <code>Element</code>.</p></li>

		<li><p>Let <var>canvas</var> be the <code>canvas</code> element to which <var>context</var> is

Conversation

foolip commented Aug 21, 2025 • edited by pr-preview Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kaiido left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

foolip commented Aug 29, 2025

Uh oh!

Uh oh!

foolip commented Sep 17, 2025

Uh oh!

Kaiido left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tabatkins commented Mar 26, 2026

Uh oh!

AutoSponge commented Mar 28, 2026

Uh oh!

progers commented Mar 28, 2026

Uh oh!

AutoSponge commented Mar 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kaiido Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hybridherbst commented Apr 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

progers Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

foolip commented Aug 21, 2025 •

edited by pr-preview Bot

Loading

Kaiido left a comment •

edited

Loading

Kaiido Apr 1, 2026 •

edited

Loading

progers Apr 28, 2026 •

edited

Loading