Guidance about exposing on Window vs Worker only (and other contexts)

[LeaVerou]

This came up in w3ctag/design-reviews#612 (comment) and it was requested that the TAG reaches consensus on guidance on this.

[jan-ivar]

Isn't this going to depend heavily on context?

[chrisn]

I support TAG developing guidance, as there's clearly a question of overall architecture direction here.

The Media WG is now running a call for consensus on this question for WebCodecs. One of the points raised in the discussion is that other APIs expected to be used together with WebCodecs, such as WebUSB and Web Audio (among others), are not available in Worker context. Another interesting argument put forward in the WG discussions was to make WebCodecs Worker-only in order to create incentive for other APIs to also be enabled in Worker contexts.

Leaving the specifics of WebCodecs to w3ctag/design-reviews#612, I think TAG guidance on these broader questions would be valuable.

[torgo]

Hi @chrisn - we're just picking this up now. I think we were too late to help you with your call for consensus. However, is there anything that came out of that process that could help us craft a recommendation / design principle? Do you have any further thoughts about what that could look like? One point of consideration is that workers should remain light-weight.

[chrisn]

Sorry for the delay in replying. I think that there are a couple of aspects to consider:

One guideline that could be added (although separate to Window vs Worker exposure) is: APIs that potentially involve lengthy computation should be async and offload to a worker thread to avoid blocking the main thread. I guess many specs do this already, but could still be worthwhile to capture as a design principle.

On the question of Window vs Worker in Web Codecs, the Media WG did reach a decision. One of the arguments for Window exposure was limited availability of other related APIs in Worker context. So I think a design principle that says that where possible APIs should be made available in both Window and DedicatedWorker could be useful, to encourage spec authors to consider this earlier in the process, and so over time there's less of a need to retrofit Worker support to existing APIs.

[jan-ivar]

So I think a design principle that says that where possible APIs should be made available in both Window and DedicatedWorker could be useful, to encourage spec authors to consider this earlier in the process, and so over time there's less of a need to retrofit Worker support to existing APIs.

A simpler principle of "where possible, APIs should be made available in DedicatedWorker" should suffice to cover that specific concern.

[alvestrand]

The debate is ongoing in webrtc; the issue in the version that is discussed there can be formulated as "is it appropriate to use non-exposure of an API on Window as a way to force developers to not do things on the main thread?".

My personal opinion is that it's not appropriate; developers should be using Workers where appropriate, and we should aim for minimum friction in them doing so, but making it impossible to use an API on Window hinders the testing and experimentation that is a critical part of web development and letting developers learn new APIs, and prejudges the question of whether there are applications for which running on the main thread is reasonable.

Carrot, not stick.

[dontcallmedom]

I agree with @alvestrand 's characterization of the issue.

My current thinking of the answer differs - while I agree workers are currently somewhat painful from a developer experience (making it harder to test and experiment), I don't think that trumps the risk of developers shipping code that run fine in high-end devices (as developers typically have) but very poorly on lower end ones.

The workers-are-painful-to-develop-with aspect should be looked into no matter what - I think there is no disagreement that they're the right approach most of the time, and so we should ensure the friction to do the right thing is as low as it can be.

[chrisn]

The debate is ongoing in webrtc; the issue in the version that is discussed there can be formulated as "is it appropriate to use non-exposure of an API on Window as a way to force developers to not do things on the main thread?".

I also agree with that formulation. I chose not to include it in my previous comment (#325 (comment)) as it wasn't clear to me whether the WGs wanted the TAG to consider that as a design principle, or whether we want to leave it to the WGs.

[LeaVerou]

We picked this up again today in our virtual f2f. We are all in agreement that if an API is exposed on Workers, it should be exposed on Window as well, to facilitate experimentation and iteration.

On the topic of which APIs to expose on Workers that are already exposed on Window, there is a tension here: adding too many APIs on Workers would make them slow, but each API that is exposed on Workers affords author more capabilities. Therefore, we would like to hear more from implementers before we make any recommendations about this.

[dontcallmedom]

We picked this up again today in our virtual f2f. We are all in agreement that if an API is exposed on Workers, it should be exposed on Window as well, to facilitate experimentation and iteration.

This sounds similar to the reasoning behind initially allowing sync XHR in the main thread, which did facilitate experimentation and iteration by developers, but to the detriment of end users who suffered from janky Web apps - I think the priority of constituencies doesn't support this view.

Another case of an APi exposed only in workers was mentioned in webmachinelearning/webnn#229 (comment) - FileReaderSync would be a really bad idea to expose in a non worker context (which I think invalidates that the principle as stated suffices)

(in terms of facilitating experimentation and iteration, Secured Context I think is showing a way where this can be done by special casing contexts needed for experimentation - localhost in the case of Secure Context - there may be similar approaches to consider if experimentation with workers is such a high barrier)

[alvestrand]

For the sync APIs, isn't that a special case of "if you feel that you have to expose a badly designed API, minimize its exposure"?

[youennf]

As the web gets more powerful, it seems that more and more new worker-only APIs are being designed: AudioWorkletProcessor, FileSystemSyncAccessHandle, RTCRtpScriptTransformer.

We are also seeing more and more window only APIs getting exposed to Workers (MSE, data channel, media stream tracks), and for good reasons too. But it is harder to expose them in workers once implementations were done with the assumption these APIs would be window only.

In general, it seems hard to provide a global recommendation.
On the other, it might be interesting to list typical questions a WG could ask itself to decide where to expose an API.

For instance, is the API deeply tied to user input (say prompt the user)? If so, recommendation could be to expose it to a window environment, maybe DediactedWorker, but not to a ServiceWorker/SharedWorker environment.
Would the equivalent native API be run in high priority threads on a native implementation? If so, it probably belongs to specific environments like Worklet/DedicatedWorker.

By doing such an exercise early in the API design, a WG could decide from the start the end goal in terms of exposure, even if implementors might first ship window-only support.

[alvestrand]

There is a dispute about whether it is appropriate to make RTCRtpScriptTransformer worker-only (w3c/webrtc-encoded-transform#89). In my opinion, it isn't, for the reasons stated in both this thread and that thread.

AudioWorkletProcessor is the implementation of AudioWorklet, so that naturally belongs on the AudioWorklet (only).

That leaves FileSystemSyncAccessHandle as the sole uncontested (AFAIK) example of an API that might have been exposed on main thread, but isn't.

I agree that it is useful to have a discussion about this when designing new APIs. I just happen to disagree strongly with Youenn on the appropriate disposition of the APIs we are discussing at the moment.

[jan-ivar]

Would the TAG block deprecation of ScriptProcessorNode? Or push to expose AudioWorkletProcessor on Window?

If not, how does the TAG's rationale apply in one case but not the other?

I think to form any general principle requires having looked at multiple APIs. In WebCodecs, the decision had multiple reasons, some specific to that API. But in MediaCapture-transform (CfA) exposure on DedicatedWorker is the only thing that has WG consensus.

The latter API exposes the browser's realtime video pipeline to JS, much like AudioWorklet does for audio (in fact a "VideoWorklet" was an early consideration). This makes exposure rules that apply to Worker but not Worklet seem arbitrary.

[youennf]

This makes exposure rules that apply to Worker but not Worklet seem arbitrary.

It might seemed arbitrary right now but we should still be able to come up with rules to decide worker vs. worklet.

A worklet is probably superior in realtime oriented usecases as it would allow implementations to consistently set a higher priority to the corresponding thread compared to regular worker threads.

We discussed this possibility for RTCRtpScriptTransformer. Using a worker was a compromise as it allowed to not be departing too much from the original proposal we were based upon.
The drawbacks I heard against worklet at the time were increased API surface and increased implementation complexity, these are of course debatable.

[kenchris]

Maybe some text like:

Where possible APIs should be made available in DedicatedWorker

When exposing a feature, please consider whether it makes sense to expose the feature to DedicatedWorker as well. A DedicatedWorker is a non specialized worker type that allows developers to run code off the main thread.

Many features could work out of the box on a DedicatedWorker and not enabling the feature there could limit the ability for users to run their code in a non blocking manner.

Certain challenges can exist when trying to enable a feature on DedicatedWorker, especially if the feature requires user input by asking for permission, or showing a color picker or device selector. Even though this might discourage spec authors to support workers, we still recommend designing the feature with DedicatedWorker support in mind, in order to not add assumptions that will later make it unnecessarily hard to expose these APIs to DedicatedWorker.

[chrisn]

Thanks @kenchris, this is the kind of guidance I was thinking of. One minor suggestion: we may not need to mention the color picker and device selector examples which are very specific, I think it's enough to just mention user input or permissions.

[alvestrand]

Based on the closing of this issue, I take it that the TAG has issued no guidance saying that there are interfaces that should not be exposed on Window.

Thanks.

[annevk]

I hope you don't take that to mean they should be exposed on Window though, as that would be the wrong conclusion to draw.

[dontcallmedom]

I wonder if a separate issue should be opened on do-not-expose-on-Window, since the resolution of this issue very specifically only addressed expose-also-on-Worker.

[youennf]

Thanks @kenchris, this is the kind of guidance I was thinking of.

In the same vein, I thing guidance could be added to consider 'NOT adding' APIs on SharedWorker/ServiceWorker where a prompt or user input is needed.

I wonder if a separate issue should be opened on do-not-expose-on-Window, since the resolution of this issue very specifically only addressed expose-also-on-Worker.

Agreed.

[kenchris]

@dontcallmedom sure, do you mind filing?

@youennf we would need to think about that yes, currently there are cases where you can ask permission/pick device from the "main thread" and send that to the worker. I believe @reillyeon used that in one of his specs - maybe he has comments

[youennf]

currently there are cases where you can ask permission/pick device from the "main thread" and send that to the worker.

getUserMedia flow is typically doing the permission request in a Document, then web application can send the result (MediaStreamTrack) to workers. That is fine.

Having a service worker/shared worker trigger a permission prompt is difficult.
Querying permission from a service worker/shared worker is fine.

[dontcallmedom]

I've opened #360 as a place to continue the discussion on do-not-expose-on-Window

[reillyeon]

I would like there to eventually be a pattern for triggering permission requests from workers in a way that makes sense to users (the UI shows up in the right place) and satisfies our existing patterns around user activation requirements.

Right now APIs like WebUSB are similar to getUserMedia in that the permission request must be made in a Document and then there is a way to query for existing permissions in a worker. Transferability of USBDevice instances would be more ergonomic but we haven't gotten around to defining the semantics. Either of those solutions however are more work for the developer than allowing the permission request to simply happen in the worker to begin with.