EyeDropper API

Question

EyeDropper API

Closed this issue 3 years ago · 37 comments

ipopescu93 commented 4 years ago

HIQaH! QaH! TAG!

I'm requesting a TAG review of EyeDropper API.

The EyeDropper API enables developers to use a browser-supplied eyedropper in the construction of custom color pickers.

Explainer: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/EyeDropper/explainer.md
Security and Privacy self-review²: https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/EyeDropper/security-privacy.md
GitHub repo (if you prefer feedback filed there): https://github.com/MicrosoftEdge/MSEdgeExplainers/
Primary contacts (and their relationship to the specification):
- Ionel Popescu (@ipopescu93), Microsoft,
- Bo Cupp (@BoCupp-Microsoft), Microsoft
Organization/project driving the design: Microsoft
External status/issue trackers for this feature (publicly visible, e.g. Chrome Status): https://chromestatus.com/features/6304275594477568

Further details:

I have reviewed the TAG's API Design Principles
The group where the incubation/design work on this is being done (or is intended to be done in the future): WICG
The group where standardization of this work is intended to be done ("unknown" if not known): Unknown
Existing major pieces of multi-stakeholder review or discussion of this design: Unknown
Major unresolved issues with or opposition to this design: None at the moment
This work is being funded by: Microsoft

We'd prefer the TAG provide feedback as (please delete all but the desired option):
🐛 open issues in our GitHub repo for each point of feedback

Answer 1 · 2021-01-13T09:58:35.000Z

First, there is a need for this in the developer community and I'm happy to see this work. However, I see a number of issues with the API in its current form.

Hex colors are in sRGB, which cannot represent all on-screen colors

The most serious issue I see with this proposal is that the API returns hex colors, which on the Web platform are in sRGB. What this means is that the CSS color #ff0000 is not the brightest red the screen can produce, it's sRGB red, which is typically far less saturated than device RGB red for most modern screens (e.g. for a P3 screen, like most modern Apple devices, it would be equivalent to device 91.8% red, 20% green, 13.9% blue). This proposal seems to suggest returning a hex encoding of device RGB coordinates. We are trying to avoid exposing device RGB on the Web platform these days, as there is no way to ensure consistent results. This means that, as currently defined, this API will return colors that are not actually the color picked in any Web platform technology, defeating the purpose of an eyedropper. Furthermore, there is not even a way to convert them to the color the user actually selected.

I'm not sure what is the best way to deal with this, since none of the currently widely supported color formats can express all on-screen colors. Ideally, the API should return one of the CSS supported device independent color formats, such as lab() or lch() but support is currently limited. One may suggest returning hex when the color is within sRGB and another format when it is not, akin to computed values. However, not only is this inconsistent and would cause developer pain when parsing the value, but the cases where on-screen colors fall outside sRGB are not exactly rare: in modern P3 screens, 33% of colors are outside sRGB and thus, cannot be specified with hex colors.

Eyedropper starting point

Given the way the API is currently defined, there is no way to specify the starting point for the eyedropper, which would be jarring for end users. Typically, in most UIs of that sort, the eye dropper starts from the location of the user interaction (where applicable), e.g. see on OSX:

picker-location.mp4

Event Design

It is unclear to me why eyeDropper.open() is asynchronous with the current design, since color selection happens via the colorselect event, and not via a promise. It does not appear to tick any of the criteria for an asynchronous operation (see TAG Design Principle 5.7).

However, stepping back from the details for a moment, it appears that a lot of the complexity here has to do with using an event-based model, which has been chosen to enable selection of multiple colors. However, I am missing a list of use cases that require such multiple color selection. In most cases of eyedropper UI in the wild that I have come across, the eyedropper typically closes after color selection, which also serves as useful feedback to the end user that their selection has been accepted. If the eyedropper did not close, it is difficult to communicate to the user that their selection has been recorded, or display what their selection was, not to mention that it would be externally inconsistent with most other eyedropper interfaces. If such use cases exist for multiple selection, one could always re-open the picker after selection of the first color, since there is arbitrary open/close functionality. The spec could ensure that calling open() right after the eyedropper closes is seamless. Prioritizing the single selection common case also means that developers don't need to explicitly call close() to close the picker every time (which I can see causing many confusing bugs, e.g. when an error is thrown before the call to close(), so the picker is left open).

Playing around with these ideas, the code sample listed in the explainer could look like this with a promise-based design and ability to specify start coordinates:

// Create an EyeDropper object
let eyeDropper = new EyeDropper();

// Enter eyedropper mode
let icon = document.getElementbyId("eyeDropperIcon")
icon.addEventListener('click', async e => {
    let selection = await eyeDropper.open({left: e.clientX, top: e.clientY});
    // if we are here, the picker has closed, either by selection, or via the user choosing to exit
    // we can call eyeDropper.open() to re-open it, or do work with selection

    if (selection.color !== null) { // a color was selected
        console.log(selection.color); // logs selected color

        if (selection.position !== null) { 
            // The selected pixel came from our own document (or a document of same origin)
            // The selected pixel is located at:
            console.log(`${selection.position.left}, ${selection.position.top}`)
        }
    }
});

Please note that this is just a quick sketch to make the feedback above more concrete, and not necessarily the way you should go!

Containing color selection

This proposal does not currently define a mechanism to allow developers to hide or show the eyedropper's pixel selection UI while remaining in eyedropoper mode, but a future version may allow that, for example, to facilitate clicking on application UI elements instead of selecting a color value.

Note that to replicate the UIs that do support multiple color selection with an eyedropper, merely preventing the eyedropper from closing or re-opening it does not suffice. These UIs typically exclude an area of the screen from selection, and this area contains some of the color-related UI and provides feedback for the color selection. Take a look at this capture from Adobe Photoshop:

picker-photoshop.mp4

Note that the actual color picker dialog does not participate in the eyedropping.

Furthermore, authors may want to contain the selection within their application, or even a part of their application (e.g. the Photoshop picker does not select colors outside of the Photoshop window).

If you want to replicate these interactions on the Web, it may be worth exploring ways to specify elements to exclude, and/or contain the selection within the viewport or specific elements.

Btw

This proposal does not currently define an object model for a color, though it seems like something that would be a good addition to the web platform.

There is some work on that.

Answer 2 · 2021-01-13T10:07:30.000Z

It is unclear to me why eyeDropper.open() is asynchronous with the current design, since color selection happens via the colorselect event, and not via a promise. It does not appear to tick any of the criteria for an asynchronous operation (see TAG Design Principle 5.7).

The Explainer mentions that UAs might implement a permission prompt.

Answer 3 · 2021-01-13T14:06:52.000Z

The Explainer mentions that UAs might implement a permission prompt.

Ah, I missed that. I'm not sure why a permissions prompt would be needed however. The user is in control the entire time, and the only color actually exposed to script is the selected color, if the user does not abort. This seems analogous to a file input to me (or the font picker we were discussing in breakout B this week), both of which do not require a permission prompt.

Answer 4 · 2021-01-14T05:23:35.000Z

Eyedropper starting point

Is it desirable for the API to specify the starting point, or should the starting point be chosen by the implementation?

It seems to me that for mouse-based interfaces, the starting point is probably the mouse pointer location since the eyedropper essentially is the mouse cursor -- though I wonder if this is still true if the eyedropper mode is initiated via a keyboard interaction.

That said, thinking about this made me curious how this works for non-mouse-based interactions, whether keyboard navigation on a desktop or whether a touchscreen.

Answer 5 · 2021-01-14T09:24:02.000Z

Thanks for the great comments, @LeaVerou.

Regarding this point you made:

Hex colors are in sRGB, which cannot represent all on-screen colors

We looked for a better representation of color while writing the explainer and found some of the GitHub issues you mentioned. It seems like it might take a while to close on a proper definition of a color type that can serve all the web platform's needs. Our thinking was that we could decouple from that work by returning the same color value strings that the input[type=color] element returns today from its value property. When a better color type is available we can then add a new property to the ColorSelectEvent - similar to what I'm sure we'll do for input[type=color].

Let me know if you agree with that thinking.

@dbaron, regarding the EyeDropper starting point, as you suggest, it should begin at the cursor location. For keyboard accessibility the same should be true and the the arrow keys should move the mouse cursor. Actually at any time the user should be able to move the cursor location with the mouse or with the arrow keys and go back and forth. You can see this pattern in the Snip and Sketch tool for Windows and with the new Smart Copy feature recently released for the Edge browser.

@LeaVerou, regarding this comment on selecting multiple colors:

However, I am missing a list of use cases that require such multiple color selection

You can find an example in Photoshop. The eyedropper is just another tool in the tool pallete and it remains in eyedropper mode until you switch to a different tool. I find the multiple selection model useful when the app shows a preview of what you selected as you select it. When selecting from an anti-aliased region or wherever some color dithering has happened, the user may not prefer the first color they select and will select another nearby color until they get the desired result.

@LeaVerou, regarding your comments on containing color selection:

Note that to replicate the UIs that do support multiple color selection with an eyedropper, merely preventing the eyedropper from closing or re-opening it does not suffice. These UIs typically exclude an area of the screen from selection, and this area contains some of the color-related UI and provides feedback for the color selection.

You are absolutely right. We currently have this written in the explainer:

This proposal does not currently define a mechanism to allow developers to hide or show the eyedropper's pixel selection UI while remaining in eyedropper mode, but a future version may allow that, for example, to facilitate clicking on application UI elements instead of selecting a color value.

I have it in the non-goals section, but agree the multiple color selection experience would be lacking without some mechanism to tag portions of the UI as not color selectable. Since we don't have the mechanism for doing that sorted out yet we may split the proposal into two-levels - the first being a single select model that we can deliver while we figure out the API surface for excluding portions of the document to support the multi-select model.

Note I also have an issue already opened on this topic here with one thinly defined proposal. If you have any thoughts on how you think it should work feel free to jump in.

Thanks!

Answer 6 · 2021-01-14T12:06:11.000Z

@dbaron

It seems to me that for mouse-based interfaces, the starting point is probably the mouse pointer location since the eyedropper essentially is the mouse cursor though I wonder if this is still true if the eyedropper mode is initiated via a keyboard interaction.

Ah yeah, that makes perfect sense. Even if a position is still needed for accessibility, it would make an excellent default.

@BoCupp-Microsoft

We looked for a better representation of color while writing the explainer and found some of the GitHub issues you mentioned. It seems like it might take a while to close on a proper definition of a color type that can serve all the web platform's needs. Our thinking was that we could decouple from that work by returning the same color value strings that the input[type=color] element returns today from its value property. When a better color type is available we can then add a new property to the ColorSelectEvent - similar to what I'm sure we'll do for input[type=color].

Let me know if you agree with that thinking.

<input type=color> does not have to match any specific on-screen color, it can just display an sRGB picker and suffer from no such inconsistency. With an eyedropper API, it's quite important that the color the user picked from the screen is the actual color displayed in the web application. If you generate a hex color based on device RGB, which is then interpreted as sRGB on the Web platform, it's not only the colors outside the sRGB gamut that will be different, every color that the user picks will be different. The difference is already fairly noticeable on today's P3 screens (up to 7.1 ΔΕ2000 units) and will increase further as screen gamuts expand beyond P3 in the near future.

The discussion about color types is orthogonal, as you are currently returning a string anyway, and there are many other CSS color formats beyond hex.

As an aside, even for <input type=color> there is discussion about hex values being insufficient. Regardless, as I explain above, it's not as much of a problem there because the picker UI is constrained, and not the entire screen gamut. If you are able to pick any displayable on-screen color, you need to be able to represent any displayable on-screen color.

Answer 7 · 2021-01-14T20:36:39.000Z

@LeaVerou regarding this comment:

<input type=color> does not have to match any specific on-screen color, it can just display an sRGB picker and suffer from no such inconsistency.

Depending on the implementation, <input type=color> does have the ability to use an eyedropper to pick the color that will be turned into an RGB hex string. Here's a screencast showing that in Chromium. Isn't that the same issue?

Answer 8 · 2021-01-16T02:42:29.000Z

@LeaVerou regarding this comment:

<input type=color> does not have to match any specific on-screen color, it can just display an sRGB picker and suffer from no such inconsistency.

Depending on the implementation, <input type=color> does have the ability to use an eyedropper to pick the color that will be turned into an RGB hex string. Here's a screencast showing that in Chromium. Isn't that the same issue?

The UI for <input type=color> is largely left up to the implementation. Since no spec defines this UI, the fact that an eyedropper appears in (an unreleased version of) one browser should not set a precedent about what makes it into the Web platform.

To answer your question, yes, this suffers from the same issues. I've tried to illustrate with a video below.

chrome-picker.mp4

The gradient on the right is a gradient from P3 red to gray (you only see part of it). At first, I'm using OSX's native color meter to show you device RGB coordinates. As you can see, these are all different colors and the coordinates smoothly change as I move my pointer downwards. Then, I use the Chrome picker to consecutively pick colors lower and lower down. As you can see, they all yield #ff0000 and I need to select several until I start getting a different hex value, because apparently Chrome seems to convert the P3 coordinates to sRGB and then just clips them to the [0, 255] range. In practice, these means that the most saturated 33% of on-screen colors would be squashed onto the same hex value.

Answer 9 · 2021-01-19T16:20:13.000Z

Does the TAG have concrete suggestions for how the input element could be extended to support device independent colors in a backwards compatible way? Maybe that could be used as model here.

E.g. the <input type=color>'s value property would need to continue to be sRGB hex to avoid breaking sites, but maybe there could be a color property added which is only not undefined for color inputs, and is a CSSOM type (?) that provides forward-compatibility to access/translate colors between color spaces in a defined way. I don't think such a thing exists yet, though.

In that case, this API could follow that model - value on the event continues to be an sRGB hex, but once the device-independent, forward-compatible color type is added to the platform it could be added to the event as color.

This is a half-baked idea; I'm hoping the TAG can suggest something more practicable.

Answer 10 · 2021-01-23T09:22:53.000Z

@LeaVerou re: this comment:

the fact that an eyedropper appears in (an unreleased version of) one browser should not set a precedent about what makes it into the Web platform.

I wasn't trying to be deceptive... just showing some new UX one of my devs recently created. You can see another screencast here I took showing Firefox, an older Chrome, and Safari all using the Mac's color sample to take a color from the screen and then expose it through the value property of input[type=color].

Answer 11 · 2021-01-26T10:23:20.000Z

We discussed this today in our VF2F and have the following feedback.

a) The biggest issue is color output. There are four ways to address this, none ideal:

The Web Platform is currently in a transitional state wrt color. Possibly the best solution is to wait until there are more implementations of non gamut restricted color formats and/or a Color object.
You could clip or gamut map the selected color to sRGB and return an sRGB hex value (or other sRGB color format). The problem is that color selection will not roundtrip for at least 1/3 of onscreen colors in today's displays (and as display technology improves, this will increase). Given that many eyedropper use cases need precision, we do not think this is a good solution. For example, a user could have a Photoshop window side-by-side with a web application and use the eyedropper to select a color from the mockup to apply to their document in the web application. The color should be identical to serve this use case. We also think returning strings is suboptimal.
You could return an lab() or lch() string, which can specify any visible color. This does roundtrip, though is also a string. Another problem is that implementations of these formats are currently limited.
You could build a special-purpose color API, with srgb and hex properties. We do not think this is ideal, as it duplicates the Color API that the Web Platform will eventually get, and it's still unclear what the primary returned value should be.

b) As long as the eyedropper cannot be used to scrub the screen, and only communicates the user's final selection back to the web application, this does not need a permissions prompt, just like the file input does not need a selection prompt, so open() does not need to be asynchronous (unless you move to a promise-based design, see below).

c) The API is designed around using a PointerEvent but it is unclear what the use cases are for reading coordinates, especially since their presence cannot be guaranteed and selection cannot be constrained to only areas where coordinates can be returned.

d) The API is designed around multiple color selection as the primary use case, requiring use of events and explicit close(). However, as pointed out by the authors, multiple color selection is primarily intended for fine tuning, but the UA can take care of the fine tuning by showing appropriate UI and only communicating the final color selection back to the app, which is also more privacy preserving. This would allow a promise based design and prevent bugs where the eyedropper mode is left open because something (e.g. an exception) intercepted the call to .close().

Answer 12 · 2021-03-01T07:10:16.000Z

Hi Lea @LeaVerou

You mentioned above in part:

"...I'm not sure what is the best way to deal with this, since none of the currently widely supported color formats can express all on-screen colors. Ideally, the API should return one of the CSS supported device independent color formats, such as lab() or lch() but support is currently limited...."

The following idea is only partially baked, kinda like a raw strawberry pop-tart. Nevertheless, it is how we deal with super wide gamut image data in film/TV: use the same Rec709 primaries but be unbounded linear. This is the default for .EXR with primaries that are assumed to be Rec709/sRGB (though others can be specified).

I know you know this Lea but for others reading along at home: The entire visible gamut can be encoded this way, and converted to any arbitrary display with the appropriate LUT. And useful LUTs and related libraries are freely available at OpenColor IO

So the not fully baked idea is, why not use the now very standard half float like EXR, use the same sRGB primaries for value 1.0, (or an ACES space, or possibly the 32 bit LogLuv space), and then use the free open libraries of OCIO to handle converting from or to any given display space, to a neutral, HDR space?

Then the concern about future proofing and "unknown" display spaces is abstracted, needing only a change of LUT, which can happen on the client machine without without any privacy issues (similar to a media query).

And now I'm thinking about pop tarts, I wonder if I have any....

A

Answer 13 · 2021-03-03T07:29:57.000Z

@LeaVerou thanks again for the comments and apologies for taking longer to reply.

You wrote:

The biggest issue is color output. There are four ways to address this:
...
option 2. You could clip or gamut map the selected color to sRGB and return an sRGB hex value (or other sRGB color format). The problem is that color selection will not roundtrip for at least 1/3 of onscreen colors in today's displays (and as display technology improves, this will increase).

Option 2 is what we have proposed. Assuming the author renders the selected color back via CSS in a Chromium browser, there's no way to round trip the colors beyond sRGB regardless of what color format we return. Until we have a more comprehensive representation of color that can be used throughout the web platform, I'm proposing we follow the existing precedent established by input[type=color]'s value property.

This is the output already produced by eye droppers on the web today (for browsers that provide an eye dropper as part of their input[type=color] implementation). I think this approach avoids jumping ahead of groups like this one to create another color representation that will be specific to eye dropper output and potentially misaligned with other web color inputs. Once we have a better color representation, we can add that representation to the event we dispatch when a color is selected. I also imagine we'll be updating the interface for input[type=color] in a similar way.

You wrote:

b) As long as the eyedropper cannot be used to scrub the screen, and only communicates the user's final selection back to the web application, this does not need a permissions prompt, just like the file input does not need a selection prompt, so open() does not need to be asynchronous (unless you move to a promise-based design, see below).

As you mention in point d, the API is designed to accommodate selecting multiple colors, so an async API seems appropriate for now. Note that I realize the explainer calls out some open issues around the selection of multiple colors, so maybe we won't initially ship a multiple color picking implementation until those are resolved, but it seems like a good idea to shape the API so that a multiple color picking implementation could be delivered. Do you agree?

You wrote:

c) The API is designed around using a PointerEvent but it is unclear what the use cases are for reading coordinates, especially since their presence cannot be guaranteed and selection cannot be constrained to only areas where coordinates can be returned.

There are two pieces of information in the explainer related to this:

Goal 2 says: Provide coordinate information in addition to a color value so that web apps may utilize any data known for the selected pixel whenever the selected color is from a document having a similar origin, e.g. layering information in a painting web app.
The second paragraph in the Solution section says: The position of the selected color is included to facilitate scenarios where a web app using the eyedropper samples a pixel color from its own document. The web app could, for example, include an alpha channel for the selected pixel or create a palette of colors associated with a pixel's location based on layer information known to the web app. The color value would otherwise be the final composited color as seen by the user.

You wrote:

d) The API is designed around multiple color selection as the primary use case, requiring use of events and explicit close(). However, as pointed out by the authors, multiple color selection is primarily intended for fine tuning, but the UA can take care of the fine tuning by showing appropriate UI and only communicating the final color selection back to the app, which is also more privacy preserving. This would allow a promise based design and prevent bugs where the eyedropper mode is left open because something (e.g. an exception) intercepted the call to .close().

I agree there's risk of bugs in web apps and missing the call to close(). We have the following text in the explainer to try to mitigate getting stuck in eye dropper mode: "provide the means for the user to exit that mode, for example, by pressing an ESC key and not allowing the behavior to be cancelled by the author." The UA may decide to show UI educating the user that ESC will exit the mode, similar to what happens for fullscreen.

Please let me know if any of my comments successfully addressed any of your concerns. :-) I'd like to find a way to ship this API without needing to wait for wide gamut colors on the web to be a solved problem. Other points seem more negotiable to me.

Thanks for your help and feedback!
Bo

Answer 14 · 2021-03-08T17:16:18.000Z

I agree with the explainer that not extending <input type="color"> to create an eyedropper API is a good idea.

In previous WICG discussion on Allow <input type="color"> to give an alpha channel and/or colors beyond sRGB?, @tabatkins noted that, even for a trivial extension to support alpha,

Agree that a new input type is likely necessary due to the existing consumers that likely depend on the 6-hex format.

In that discussion I also noted that the 8-bit sRGB-only hex format suffers from two issues:

The Web has been moving from sRGB-only since 2016. A significant percentage of phones, tablets,and laptops not have a wider gamut RGB space as native (display-p3, derived from DCI P3, seems to be the most common) while much video content is delivered in rec2020 RGB
Implementations are moving from 8 to 10, 12, or half-float per color component, because wider gamuts need more precision to get the same banding-free performance.

So making a new API in 2021 which is limited to sRGB-only looks like a very short-term exercise. However, making an extensible API which delivers sRGB as a default (or as the only option in the first version, but is clearly labelled as sRGB) makes it possible to extend (to at least display-p3) soon, without re-writing or abandoning the API.

I note that the eyedropper explainer makes no mention of colorspace at all, which should be corrected.

I fully agree with the TAG feedback that early clipping of the input color to sRGB is going to cause a compatibility headache - its hard to then not-clip when other color spaces are supported, because this is a change in eyedropper behavior.

Also as the example from @LeaVerou showed the user experience is confusing if visibly different on-screen colors result in the same picked color.

And since the explainer states that

Provide access to the color values of one or more user-selected pixels, including pixels rendered by different origins, or outside of the browser.

the eyedropper will need to pick up colors displayed by native applications, which will not necessarily be in sRGB, even if the browser does not yet support them.

I can see several options:

return a tuple of a color and a colorspace, so for example #1278CF and sRGB, which can be easily extended to support other colorspaces
return two values, one an sRGB fallback and one the actual color, in Lab
return an unclipped sRGB color (so negative values, and values greater than 100%, would be legal and represent colors out of gamut)

(I'm assuming strings are returned, since the CSS Typed Object Model is not yet ready to be deployed for colors.)

I suspect the first option is both easier and better, for a near-term implementation; and re-reading the earlier WICG discussion I see @tabatkins proposed a colorspace="" attribute so the content author can say what colorspace they want results returned in

By the way I appreciated, from the explainer,

This proposal does not currently define an object model for a color, though it seems like something that would be a good addition to the web platform.

Yes, it would be a great addition to the Web platform but there isn't one yet that is ready for deployment, so for now slinging around strings is the only way; and I appreciate that a quick color object model wasn't developed solely for eyedropper use. A color model for the Web needs to handle CSS, Canvas, WebGPU as well as just colors in HTML.

Answer 15 · 2021-03-09T00:48:36.000Z

@BoCupp-Microsoft

On color spaces

I don't think it's particularly contentious that an eyedropper that returns an incorrect color for about 1/3 (!) of on-screen colors is pretty seriously flawed. To reiterate, the color input is not a good analogy; a color selection tool can just constrain its selection to in-gamut colors. An eyedropper (especially with this proposed design) does not have this luxury. And even for the color input, the decision to go with hex colors is now seen as a design mistake, as @svgeesus points out above.

There is however a way forwards, as @svgeesus suggested, which we hadn't realized earlier. This is actually how Chrome and other UAs plan to ship other color-related features, such as color-mix(): a mandatory colorspace argument. It could be a (dictionary) parameter of the open() method. If the author explicitly declares that they want results in sRGB, it should be no surprise that colors outside sRGB will be gamut mapped (not clipped, which can cause large hue shifts). Once wide gamut colors are implemented, an appropriate default can be chosen (e.g. Lab) so that the argument is no longer mandatory. This also allows authors control over the returned format so that it better suits their use case.

On privacy

As you mention in point d, the API is designed to accommodate selecting multiple colors, so an async API seems appropriate for now. Note that I realize the explainer calls out some open issues around the selection of multiple colors, so maybe we won't initially ship a multiple color picking implementation until those are resolved, but it seems like a good idea to shape the API so that a multiple color picking implementation could be delivered. Do you agree?

I think you misunderstood this point in our feedback. The permissions prompt isn't needed for multiple color selection either, unless the API continuously communicates the color the eyedropper is on, essentially "scrubbing" the screen, but that would be poor design for reasons beyond privacy. As long as the only color(s) that are communicated back via the API are those intentionally selected through explicit user action, it does not need a permissions prompt, in the same way that the file input doesn't need a permissions prompt even for selecting multiple files.

On coordinates

It's clear that there are use cases where coordinates are useful. What is unclear is if there are such use cases that do not also require constraining the selection area to areas that can produce such coordinates, which is currently explicitly listed as a non-goal. Allowing selection from anywhere on the screen when you really mean to let the user select a color from their graphics document seems like a footgun, which will create poor, buggy user experiences.

a) In many cases it will end up being completely unhandled because developers only tested by picking colors within their document
b) Even when developers realize they need to handle selection out of bounds, since they cannot prevent it, they will need to handle it as an error condition post-hoc. Creating error conditions that could have been prevented is a usability antipattern.

There is only one use case listed in the explainer for coordinates, and that is also a use case that requires constraining selection to the document.

On single vs multiple color selection

I agree there's risk of bugs in web apps and missing the call to close(). We have the following text in the explainer to try to mitigate getting stuck in eye dropper mode: "provide the means for the user to exit that mode, for example, by pressing an ESC key and not allowing the behavior to be cancelled by the author." The UA may decide to show UI educating the user that ESC will exit the mode, similar to what happens for fullscreen.

You are discussing how these bugs could be mitigated by the UA, but not any rationale for the current design that would justify its added complexity and bug potential over a far simpler single-selection-by-default promise based design. Note that multiple color selection is still possible with such a design, either via the UA allowing fine-tuning, or even by the developer triggering eyedropper mode again immediately after selection.

Answer 16 · 2021-03-31T17:32:43.000Z

We discussed this again this week.

We have consensus that a way to move forward with this and work around the transient issues relating to the lack of wide gamut support on the Web and the lack of a Color object on the Web Platform would be for the property that returns the selected color to be named in such a way that makes what it is explicit, and that allows room for future properties to be added, once these problems are solved. The name should indicate both that the color is a string (to allow room for a property that returns an object in the future), and that it is in sRGB (to allow room for an accurate color in the future). For example names like hex, hexString, srgbString would be explicit in that regard, whereas names like color, value, srgb, colorString are not (the first two because they are too generic, the last two because they are explicit in one axis but not the other).

We are still concerned about the event based design and think a promise-based design would be more suitable. The use cases described do not seem to warrant the added complexity of an event based design. A promise-based design would provide a considerably simpler API, and would prevent any issues with the eyedropper mode being left hanging due to a missed call to close().

With a promise based design, open() can allow for a permissions prompt or not, based on what the UA wants to expose. E.g. if drag and scrub is allowed, it would make sense to have a permissions prompt, but if the UI only allows selecting individual colors by clicking, it can be omitted. Note that a promise based design does not preclude events from also being fired in the future, if there are compelling use cases.

Security and privacy also came up. You should ensure that sites are not able to maliciously open the eyedropper to sniff, otherwise it could be used for nefarious reasons by advertisers.

Answer 17 · 2021-05-11T06:55:18.000Z

@ipopescu93 @BoCupp-Microsoft Hi there, this was just brought up in one of our VF2F breakouts today, we were wondering if you had any more thoughts on developments to share?

Answer 18 · 2021-05-27T15:35:19.000Z

For the returned named values, It might make sense to look at the color and colord JavaScript packages which both have explicit calls for different color spaces and object or string notation. These are very popular so it would be nice to match APIs where possible.

Answer 19 · 2021-06-01T20:56:16.000Z

@LeaVerou thanks again for the comments and apologies for taking longer to reply.

I have updated the explainer to address the feedback received:

the API surface is simplified by moving to a promise based design
the ColorSelectEvent.value property is renamed to sRGBHex to make it clear that the color is a string and it is in sRGB. it is also mentioned that is expected that a color object will be the primary mechanism by which authors will access sampled color data in the future
providing selection coordinates is moved to a non-goal for now since when using them it would be good to have the option to constrain the selection area to areas that can produce such useful coordinates

Please let me know if these changes successfully addressed your concerns.

Answer 20 · 2021-06-01T20:58:51.000Z

sRGBHex

Fascinating, this is an edge case the design principles casing rules do not anticipate. I wonder whether sRGBHex or srgbHex is better... probably whatever is decided on here should be documented in the design principles so that future APIs align with this.

Answer 21 · 2021-06-02T06:56:02.000Z

sRGBHex

Fascinating, this is an edge case the design principles casing rules do not anticipate. I wonder whether sRGBHex or srgbHex is better... probably whatever is decided on here should be documented in the design principles so that future APIs align with this.

Hey @domenic Domenic,

I don't think it's "that much" of an edge case, it seems to be anticipated in the third row of the table:

So the form of sRGBHex appears to me to be the style/form defined in this table-row rule on initialisms.

THAT SAID (implying that tangential opinion follows): for me personally, for camelCaseInitialisms, I've always preferred the first word after an initial to be lowercase, such as sRGBhex or something like sRGBtoLinear() and not sRGBHex but I suppose that's rabbit-stew if these rules are already set...

Answer 22 · 2021-06-02T13:45:36.000Z

The edge case here is that, whereas "BG" gets de-capitalized to "bg" or "HTML" gets de-capitalized to "html", it's not clear whether "sRGB" should be decapitalized to "srgb".

Answer 23 · 2021-06-02T19:49:28.000Z

The edge case here is that, whereas "BG" gets de-capitalized to "bg" or "HTML" gets de-capitalized to "html", it's not clear whether "sRGB" should be decapitalized to "srgb".

Hi @domenic

Right, but the "s" is lowercase by convention, so the first character would always be lower-case, and the intent of being lowercase when the first word of a method or property is to be consistent with the previous rule of camelCase for methods and properties, as the first character in camelCase is always lowercase as opposed to PascalCase.

The edge then as I see it is if sRGB is used as the first term in a class or mixin which is PascalCase, so then would the lowercase s be forced to uppercase like SRGBobject or SRGBObject ...?

The other edge case then is when sRGB is not the first term — then should the "s" be uppercased? Such as ClassSRGB or still be ClasssRGB or maybe then invert as ClassSrgb

Special Convention?

Since the underlying intent of case conventions is to improve readability of the code, perhaps there is room for a convention for terms where the case is already set/standardized, as it is with sRGB (other examples are NaN, Lab, Luv ...)

The convention would be to use an underscore when the case of the term should not be changed, but the other tenets of the given convention indicates that the case needs to be changed. Examples:

camelCase

sRGB_Color or _LabColor or _Luv_sRGB_Color

PascalCase

_sRGB_Color or LabColor or Luv_sRGB_Color

Or would that be too confused with JSON keys ...

Unrelated side note

Should the table in https://w3ctag.github.io/design-principles/#casing-rules also include a row for GLOBAL_VARIABLES, which are frequently all-uppercase underscore delimited?

Answer 24 · 2021-09-02T11:09:28.000Z

Glad to see the improvements in both API shape and spec text!

We discussed this again in our plenary yesterday and while we thought this was an improvement over the previous iteration, we did have a few thoughts and questions.

While it's good that there is some guidance for UAs on when to exit eyedropper mode without a selection (e.g. pressing ESC), it might be good to also provide a way for the application to explicitly close the eyedropper, e.g. a close() method or an AbortSignal parameter. Since no UI events are dispatched this would not be useful for closing the eyedropper through user interaction (e.g. via additional keyboard shortcuts), but it could be useful for closing the eyedropper (with no selection) in case another high-priority event occurs in the application, e.g. a modal dialog.

Some of us were surprised by the promise rejection when no selection is made (vs resolving with null), since this is ordinary use and not an unusual bad state. Do you consider exiting without a color selection an error condition? Did you choose this design to be consistent with another API?

While we think the promise-based design is an improvement, we did notice that it does not address your original use case of picking colors from a drawing application. Only the final composited color will be picked with no means of tracing it back to the top-most color that the user may have intended to pick since no UI events are fired throughout the selection. What are your thoughts on this? Is there another way to accomplish this with the new design or did you decide that this is a non-goal for now?
Also note that for multiple color selection to be possible with a promise-based API like this, subsequent (synchronous) open() calls would need to open the eyedropper in the same place it was before. I don't see anything about this in the explainer or the spec.

Btw while sRGBHex definitely addresses our feedback on future-proofing this API, do note that a simpler hex property would also do, since all hex colors are in sRGB anyway. That also avoids the casing debates raised above.

Relevant, note that there is now an effort to standardize a Color API for the Web Platform by a subset of the CSS WG per recent CSS WG resolution: https://github.com/WICG/color-api and serving this API is listed as an explicit goal.

Answer 25 · 2021-09-02T11:33:06.000Z

Some of us were surprised by the promise rejection when no selection is made (vs resolving with null), since this is ordinary use and not an unusual bad state. Do you consider exiting without a color selection an error condition? Did you choose this design to be consistent with another API?

I was not involved at all, so can't speak for the authors, but as a data point the File System Access API likewise throws an AbortError should the user cancel the picker.

Answer 26 · 2021-09-02T14:24:07.000Z

Some of us were surprised by the promise rejection when no selection is made (vs resolving with null), since this is ordinary use and not an unusual bad state. Do you consider exiting without a color selection an error condition? Did you choose this design to be consistent with another API?

I was not involved at all, so can't speak for the authors, but as a data point the File System Access API likewise throws an AbortError should the user cancel the picker.

Yup, this was mentioned in the discussion (and in w3ctag/design-principles#55 ), hence the last question.

Answer 27 · 2021-09-02T17:22:32.000Z

I'm becoming more convinced that AbortError in the event of the user canceling the operation is the wrong response (and the File System Access API is wrong too).

Even given the case if it should be agreed that user cancellation is an exceptional situation (which I do not agree with), a different error should be thrown. I believe AbortError should be exclusively used in response to an AbortSignal.

I'm also beginning to strongly feel that the open() method should accept an AbortSignal via an options dictionary, e.g. open({signal: signal}). This allows not only a clear mechanism for the caller to abort the operation, but for surrounding code to pass in an AbortSignal of its own.

Answer 28 · 2021-09-02T17:38:29.000Z

a different error should be thrown. I believe AbortError should be exclusively used in response to an AbortSignal.

I strongly disagree with that. We chose AbortError to be used with AbortSignal because of its long precedent of being used for any cancelation.

Answer 29 · 2021-09-02T18:09:06.000Z

We chose AbortError to be used with AbortSignal because of its long precedent of being used for any cancelation.

Fair enough, I wasn't considering other (systemic) reasons a process may be aborted.

My bottom line is that I'm looking for a general principle to inform these decisions, and I think there's some subtlety that should be captured here between an operation getting aborted for 'reasons' vs the user explicitly opting out of a choice.

Aside from the File Access API, do you have other examples where AbortError is thrown in response to user cancellation of an operation?

Answer 30 · 2021-09-02T18:14:52.000Z

Sure:

When the user presses the browser stop button to stop loading, media elements reject pending play() promises with AbortError
When the user cancels a share operation with web share, the promise rejects with an AbortError.
When the user aborts a payment request dialog, the promise rejects with an AbortError.

This was the result of a quick Google search, but I suspect there are more.

Answer 31 · 2021-09-02T18:33:41.000Z

We discussed this again in our plenary yesterday and while we thought this was an improvement over the previous iteration, we did have a few thoughts and questions.

Thanks for your feedback!

While it's good that there is some guidance for UAs on when to exit eyedropper mode without a selection (e.g. pressing ESC), it might be good to also provide a way for the application to explicitly close the eyedropper, e.g. a close() method or an AbortSignal parameter. Since no UI events are dispatched this would not be useful for closing the eyedropper through user interaction (e.g. via additional keyboard shortcuts), but it could be useful for closing the eyedropper (with no selection) in case another high-priority event occurs in the application, e.g. a modal dialog.

We haven't seen evidence that this is a common scenario, but I don't see any harm in providing an optional AbortSignal parameter to facilitate such a scenario.

Some of us were surprised by the promise rejection when no selection is made (vs resolving with null), since this is ordinary use and not an unusual bad state. Do you consider exiting without a color selection an error condition? Did you choose this design to be consistent with another API?

Since selecting a color is a user initiated action, I think exiting without a color selection should be treated as an exceptional case.
This is also consistent with how other APIs behave when the user aborts the operation by dismissing the UI.

While we think the promise-based design is an improvement, we did notice that it does not address your original use case of picking colors from a drawing application. Only the final composited color will be picked with no means of tracing it back to the top-most color that the user may have intended to pick since no UI events are fired throughout the selection. What are your thoughts on this? Is there another way to accomplish this with the new design or did you decide that this is a non-goal for now?

We would have to include the position of the selected color to facilitate this scenario and we have decided that this is a non-goal for now.

Relevant, note that there is now an effort to standardize a Color API for the Web Platform by a subset of the CSS WG per recent CSS WG resolution: https://github.com/WICG/color-api and serving this API is listed as an explicit goal.

I think that a Color API would be a great addition to the Web Platform and we expect that a color object will be the primary mechanism by which authors will access the selected color in the future.

Answer 32 · 2021-09-15T06:39:59.000Z

Hi @ipopescu93!

@cynthia and I looked at this during a breakout in our Gethen VF2F today. We are happy with the direction it's going and we are going to go ahead and close it.

Thanks for flying TAG!

Answer 33 · 2022-01-12T20:40:45.000Z

Hi @ipopescu93,

I noticed that the EyeDropper API was shipped in Chrome and decided to take it for a spin. I see that it is returning a hex with device RGB coordinates, despite it being labelled sRGBHex. For example, in the linked codepen, even though I'm sampling sRGB #00ff00, the color I get back is #02fe00, which is the pure sRGB green, converted to the RGB space of my (wide gamut) screen. This means that no picked color can roundtrip correctly. Is this intentional or a bug I should report?

Answer 34 · 2022-01-12T20:47:23.000Z

Hi @LeaVerou,

This looks like a bug. Could you please file a bug on http://crbug.com/ and assign it to me (iopopesc@microsoft.com)?

Answer 35 · 2022-01-12T21:03:01.000Z

Hi @ipopescu93, just filed https://bugs.chromium.org/p/chromium/issues/detail?id=1286862 but I cannot assign it to you.

Answer 36 · 2022-01-12T23:40:03.000Z

I came across a Chrome extension using the EyeDropper API by @captainbrosset and tried it out. I immediately noticed that on my Wide Color Gamut screen, the results were being measured in device coordinates, but reported back as sRGB hex thus giving the wrong color.

For example, displaying a patch of #009900 the eyedropper reported #56981C which is clearly incorrect. However, using a browser that does not do color management on CSS (Firefox, which simply throws whatever RGB data at the screen) displaying #56981C gives the exact same color, visually, as displaying #009900 does in Chrome, Edge etc which do handle sRGB values in CSS correctly.

Answer 37 · 2022-01-12T23:44:46.000Z

So basically the EyeDropper API is pulling raw values out of the framebuffer and then pretending those are already in sRGB