Null handling
japgolly opened this issue · 63 comments
What do we do in cases where a type in facade is nullable? We have js.UndefOr but sadly there's no js.NullOr. In scalajs-react I add | Null to facade types but the problem is it's non-trivial to get rid of the null (outside of Scala 3's experimental explicit-nulls), and I've had to add helpers with casts.
I absolutely hate the idea of just saying AudioTrack and the user having to somehow know that it's nullable. I want more clarity for users: types would be the best, information is probably the fallback (eg. annotations, worse: doc).
What do to? It might be an idea to add something like NullOr to scala-js-dom itself.
Yes, I ran into this too when reviewing a PR. The idiom here so far seems to be not to indicate the possibly null type. OTOH, so far this library has essentially been a pure facade and adding opinionated nice-ities is arguably out of scope?
Btw, #414 (comment) made me think critically about what the purpose of this library is. I don't mean to open a can-of-worms over null but it is helpful if we clarify the goals/scope of scala-js-dom going forward.
this library has essentially been a pure facade and adding opinionated nice-ities is arguably out of scope
I agree with that sentence but I don't agree that clarifying nulls in types goes beyond that scope. In the same way the js.UndefOr is a facade over A | undefined (with a few helper methods for nice Scala usage) I think a similar concept over null is just as in scope. It adds precision to the facade types.
Plus I'd also argue that in addition to simply being pure facades, we also provide utilities for convenient Scala usage, and there's already precedent for that. (Check out the [S_] tagged items in the api report.) As an example, one of the most annoying things ever, is getting a NodeList back from a facade, and not being able to iterate over it via typical Scala collections. I think I saw somewhere recently that this is fixed in our pending PR queue? But regardless, I'd argue we've got to go beyond providing the most minimal use. I think we should hard-limit our scope to JS DOM and browser API, but within that scope I think we should make it as nicely usable as possible, and as safe/precise as possible.
Re #414, yeah I don't know about generating this code from TS. Seems like a great idea if everything's up-to-date, and we have a means of overriding certain generated decisions. It's interesting but I'd also say it's orthogonal to the goals/purpose of this project. Like regardless of whether we hand-write or generate the code in this repo, we still need to make decisions about how to make the facades as precise as possible, and ensure they have good-enough dev ergonomics, so I'm happy to separate the decisions.
I think we should make it as nicely usable as possible, and as safe/precise as possible.
So, why not return Option for example? I guess I'm confused where the line is drawn.
Option is a Scala type, we need a type that behaves like Option but represents X | Null at runtime in JS world. Option is None | Some
Plus different subtle semantics too, like you can't represent Some(None) as X | Null because it would be reinterpreted as just None.
Plus different subtle semantics too, like you can't represent
Some(None)asX | Nullbecause it would be reinterpreted as justNone.
Right, but as long as we rule out nested Options I don't think that should crop up :)
I guess the question I'm asking is, what's the difference between having a facade that uses X | Null but then has an implicit conversion to/from Option for convenience sake, versus defining the interfaces in terms of Option to begin with and implementing them via a raw facade hidden from users.
Option isn't a JS type. It's a sum type of two Scala classes None and Some. From a JS pov, having two Scala classes is not the same as having a nullable JS type. Eg. None != null and Some(1) != 1. Theory aside you basically can't use Scala types in facades as a blanket rule.
We could have an implicit conversion from A | Null to Option but I don't think an implicit conversion is a good idea. It's non-obvious, users have to Just Know that they can call option methods on a non-option, plus they're require a special import (which I'd like to avoid as much as possible).
The reason I'm wondering about NullOr is that it wouldn't require any imports, it's mandatory and explicit, and it would be identical to js.UndefOr which is a pattern that's in place already everywhere.
Apologies, perhaps I'm misunderstanding, but js.UndefOr[A] is just a type alias for A | Unit, so then wouldn't be your proposed NullOr[A] just be A | Null with all its caveats?
Theory aside you basically can't use Scala types in facades as a blanket rule.
Right, of course it doesn't go on the facade itself :) absolutely not! But in theory this library could provide a higher level interface with Options rather than a raw facade.
I haven't thought deeply about the solution, I'm just clarifying the problem and the limitations here :) But yeah if we end up with just a type alias over X | Null then I want to ensure that whatever's been done to make js.UndefOr function as more than just a type alias, get's done for NullOr too.
that whatever's been done to make js.UndefOr function as more than just a type alias, get's done for NullOr too.
See my comment above about implicit conversions ;)
Although you make it good point, it's an implicit conversion to add syntax rather than an implicit conversion to another type. Important distinction, my bad!
But in theory this library could provide a higher level interface with Options rather than a raw facade.
That's a nightmare! As an example scalajs-react is mostly that, the facades aren't as safe and nice to use as typical Scala stuff, so here's a better layer on top of that, and it's the most time-consuming library to maintain that I have. Maybe I'm misunderstanding you but if you're suggesting having double the types with one always performing minor conversions to the other, I don't think that's the way to go.
I agree, now I see better where you're going with this! NullOr with implicit syntax like UndefOr is the way to go 👍
Although you make it good point, it's an implicit conversion to add syntax rather than an implicit conversion to another type. Important distinction, my bad!
No no you're right to highlight that! I think the former is much more acceptable than the latter, and we shouldn't rule out any kind of implicit. Thanks for clarifying. So like I said I haven't put any thought into the solution yet but no implicit would be better than syntax via implicits, which would be better than implicit type conversion.
So is js.UndefOr syntax provided simply be implicits? I hadn't really thought about it before but it's surprising. I'd bet that they aren't implicitly in-scope, I'm curious to try and use one with out import scalajs.js._. Whatever's going on there is very good prior-art so we should definitely look into it. I might be able to come up with some hacks to get implicit syntax, into implicit scope (so no import required) if that where this is going.....
I'd bet that they aren't implicitly in-scope,
They are :P see here https://github.com/scala-js/scala-js/blob/8a0655b2ddf3047c13d49814c4debb7f10b88931/library/src/main/scala/scala/scalajs/js/Union.scala#L108
Which sadly means for us, they cannot be, because we do not have access to the Union package companion object.
Oh! That's cheating!! In the union companion object!
Alright we'll figure something out. There are always ways hehehe. We should make our own companion object!
We should make our own companion object!
I think you are right, this would work!
eg idea:
sealed trait NullOr[+A] extends js.Any
object NullOr { magic here }I think it could also be type NullOr[+A] = A | Null, and the companion object trick still works, pretty sure cats does this.
I feel like I've tried that before and it didn't work but happy to try again.
package org.scalajs.dom
import scala.scalajs.js
import scala.scalajs.js.|
import scala.annotation.unchecked.uncheckedVariance
sealed trait NullOr1[+A] extends js.Any
object NullOr1 {
@inline implicit final class Ops[A](private val self: NullOr1[A]) extends AnyVal {
@inline def asUnion: A | Null = self.asInstanceOf[A | Null]
def toOption: Option[A] = if (self eq null) None else Some(self.asInstanceOf[A])
}
}
object ModuleNullOr2 {
type NullOr2[+A] = (A @uncheckedVariance) | Null
object NullOr2 {
@inline implicit final class NullOr2Ops[A](private val self: NullOr2[A]) extends AnyVal {
def toOption: Option[A] = if (self eq null) None else Some(self.asInstanceOf[A])
}
}
}package external
trait NullOr1Test {
import org.scalajs.dom.NullOr1
def x1: NullOr1[Int]
def y1 = x1.toOption
}
trait NullOr2Test {
import org.scalajs.dom.ModuleNullOr2._
def x2: NullOr2[Int]
def y2 = x2.toOption
}objects aren't implicit scopes for type aliases sadly:
[error] value toOption is not a member of org.scalajs.dom.ModuleNullOr2.NullOr2[Int]
[error] def y2 = x2.toOption
[error] ^
My understanding is that from a compiler pov, a type alias should be identical to it's value, which means for every known type, the compiler would have to be aware of every equivalent type alias on the classpath and check all of their companion objects. Considering you typically add to the classpath in chunks (eg. module A depends on module B) you could have code compile in one scope and then suddenly break downstream as a new type alias companion object gets globally added to the type's scope.
You are absolutely right, this was marked wontfix in scala/bug#9770. I was confusing it with an another issue that I read about in a comment in cats.
Oh yeah, that other one's frustrated me many times
I strongly advise against introducing anything like NullOr. The reason is that whatever you manage to do for Scala 2 will immediately break in Scala 3.
In Scala 3, | is a true union type. As long as we have nullable reference types by default in the language (i.e., no explicit nulls), any T | Null where T is a reference type will collapse at the type system level to T. This means that you will immediately lose any API that you have built for NullOr.
You could sidestep this issue with a sealed trait (or opaque type in Scala 3), combined with more implicit conversions. But if you do that you'll end up with the same problems that js.UndefOr had in 0.6.x, i.e., that it doesn't compose.
This is why there is no js.NullOr in the standard library.
If Scala 3 ever gets explicit nulls by default, we will be able to address those issues. In the meantime, we have to use T for something that is T or null in JS types.
Feel like it's ugly? Well, it's exactly the same for Java APIs that return nullable types. There's simply nothing we can do about it.
You could sidestep this issue with a sealed trait (or opaque type in Scala 3), combined with more implicit conversions. But if you do that you'll end up with the same problems that js.UndefOr had in 0.6.x, i.e., that it doesn't compose.
Yeah sealed trait is exactly where I was going but with implicit ops, not implicit conversions. What do you mean about it not composing? Composing with what?
Feel like it's ugly? Well, it's exactly the same for Java APIs that return nullable types.
Java APIs are much better known that random pieces of JS all over the place.
There's simply nothing we can do about it.
Are you trying to motivate me to solve this problem? If not, that's the worst thing you could ever say to me. There's always something you can do about something. It just depends on resources and tradeoffs.
Yeah
sealed traitis exactly where I was going but with implicit ops, not implicit conversions. What do you mean about it not composing? Composing with what?
It doesn't compose with other union types, with type inference, with parameter type inference for functions, etc. In a sense it all boils down to type inference.
Java APIs are much better known that random pieces of JS all over the place.
That's unfair on the JS ecosystem. I would say that DOM APIs, documented on MDN, are much better documented and "known" than random pieces of Java libraries all over the place.
There's simply nothing we can do about it.
Are you trying to motivate me to solve this problem? If not, that's the worst thing you could ever say to me. There's always something you can do about something. It just depends on resources and tradeoffs.
If you actually solve it, in a way that provides good type inference and allows composability with other union types, in Scala 2 and in Scala 3, then all the better. I'm just saying that I didn't find a way so far, otherwise we would have put it in the standard library.
But my advice remains: a lot of time has been lost on this issue already, it's probably better to stick with the status quo, until the true solution naturally comes with explicit nullable types in some future Scala 3 version.
My opinion here is that we should leave the status quo as is; yes it's not as strict as I would like, but the everything-is-nullable sloppiness in Scala matches reasonably well with the everything-is-nullable sloppiness in Javascript. scala-js-dom is meant to expose JS APIs using standard Scala constructs; it's not meant to come up with our own bespoke abstractions on top of the DOM, no matter how imperfect it is.
When Scala 3's across-the-board improvements to Scala's null handling becomes ubiquitous, we can see how to use that, but until then it doesn't make sense to come up with out own thing only to have to get rid of it when we adopt Scala 3s null handling
it doesn't make sense to come up with out own thing only to have to get rid of it when we adopt Scala 3s null handling
This is @sjrd 's prediction, not mine.
If you actually solve it, in a way that provides good type inference and allows composability with other union types, in Scala 2 and in Scala 3, then all the better.
The idea in my head seems to do just this, without being a problem for Scala 3. I'll sketch it up in the coming days when I get a chance and you can see what you think. That will at least give us something concrete to debate rather than all of these hypertheticals.
And the record, I don't agree with this whole "let's do nothing, let's not make any improvements because one day: Scala 3" line of thinking. Who cares what Scala 3 does? We're gonna be cross-compiling for years and years to come, and we're gonna be bound by Scala 2 for a long time yet. I don't see the logic in not improving the status quo because one day things will change. A bit of migration in a few years (literally the worse case as I see it, it seems pretty easy to even avoid this) is a completely acceptable trade-off for years of improvement. Seriously I don't get all the fear.
I started some research into working more conveniently with T | undefined | null, T | undefined and T | null some time ago, see ScalablyTyped/Runtime#1 . I left it because it was quite unknown at the time how the new union types would work
@oyvindberg you do use T | Null in ST's generated facades though, correct? How/why did you decide on that?
Yes, the three cases are translated into js.UndefOr[T | Null], js.UndefOr[T] and T | Null, respectively. I just wanted to see how Scala 3 with strict nulls would play out before doing anything more with it.
Feel like it's ugly? Well, it's exactly the same for Java APIs that return nullable types. There's simply nothing we can do about it.
Well, what is often done about this, is a Scala wrapper around the ugly Java API that injects Option in all the right places. Like, I hope we all might agree this is the idiomatic thing to do, although whether it is the right thing to do for scala-js-dom is of course a different matter. Which leads to my second question...
scala-js-dom is meant to expose JS APIs using standard Scala constructs; it's not meant to come up with our own bespoke abstractions on top of the DOM, no matter how imperfect it is.
Forgive me if I'm a broken record, but what is the goal of this library? If it's simply to provide a raw facade, why don't we just auto-generate it? Presumably because of some clunkiness with an auto-generated facade, but given the Null problem clunkiness in this situation seems inevitable? This is a genuine question: it's obvious that manually maintaining a facade for something as vast as DOM will be a never-ending job, so I'm curious what we are buying/what the trade-off is by doing this manually vs automatically.
There's always something you can do about something. It just depends on resources and tradeoffs.
On the other hand, if our goal is to provide a user-friendly library for DOM that uses the type system to communicate nullable types etc., why aren't we building a wrapper (as one might around a Java library) that uses Option etc.? As @japgolly suggested above, is this purely a resource constraint?
@armanbilge this library was originally auto generated from typescripy definition files, though the generation script and the quality of the translation was poor enough that we continued maintaining the generated code manually after that. Note that this is from the 2014 era, long before things like ScalablyTyped appeared on the scene.
The original plan was for the raw.* APIs to faithfully represent the Javascript APIs, and for the ext.* APIs to be more clever/highlevel. For whatever reason, the former has succeeded while the latter has failed, and if i'm not mistaken we are phasing out the ext.* APIs going forward.
Bringing us back to the present, it's worth askinh what this library should be, regardless of the historical baggage. One strawman proposal could be as follows:
-
We move the raw.* APIs into the dom.* namespace, as discussed earlier. These have a clear purpose, have been stable for years, and I don't think we should start overhauling it lightly e.g. by changing null handling
-
We do not discard the ext.* namespace, but instead use it as an experimental scratch space where the maintainers can try more ambitious improvements, to make a DOM API that we may consider "idiomatic" rather than being a direct wrapper.
-
We experiment with code generation, to see if we can get it to a level of quality and compatibility enough that we can stop hand maintaining the code. Note that this may never happen, but with things like ScalablyTyped now available it could be worth an experiment
These are just my own opinions; despite authoring the library, I haven't been involved in maintenance for years, and I can now only be considered a user. It's up to you guys to decide how you would like to move forward. My only ask is that we preserve the good stuff that we have (i.e. the directly exposed DOM APIs) and not break them unnecessarily even while experimenting with new approaches
@lihaoyi Thank you for the history, definitely helps me to better understand and appreciate the library as it is today.
We move the raw.* APIs into the dom.* namespace ... We do not discard the ext.* namespace, but instead use it as an experimental scratch space where the maintainers can try more ambitious improvements
👍 I like this proposal.
We experiment with code generation, to see if we can get it to a level of quality and compatibility enough that we can stop hand maintaining the code.
I also think this is definitely worth experimenting with. Would you mind specifying some of your quality concerns, so I can get a better idea of where an auto-generation scheme needs improvement? See also #486 (comment).
Would you mind specifying some of your quality concerns, so I can get a better idea of where an auto-generation scheme needs improvement?
The original code generator was a pair of Scala programs, one that translated typescript definition files to Scala code, another that scraped https://developer.mozilla.org/en-US/docs/Web/API to fill in the scaladoc. Both were half-baked, and generated broken code in many cases. They were also never source controlled, and the typescript definition generator has been lost to the mists of time, although the doc scraper I managed to fish out of my archive https://gist.github.com/lihaoyi/86eba5e0956861350b5b98bbb87e6516.
Basically, it didn't work well at all. But it was a weekend project from someone new to Scala back when Scala was younger and Scala.js was still pre-release. I'm sure with infrastructure like ScalablyTyped to build upon you'd be able to get a much higher level of quality than I did, though it remains to be seen whether it'll be enough to match the current hand-maintained facades. The current ones are really pretty good, but who knows maybe @oyvindberg's code generator is just as good!.
If you can get code-gen working well, then it'll also make it trivial to code-gen wrappers, e.g. if someone wants a different encoding for nulls, we could just tweak the code-generator and re-generate. That's probably the path forward if we want to easily experiment with different encodings and wrappers, as manually wrapping the thousands of DOM APIs everytime we want to try something new is totally infeasible
I have no doubt that code generation can get us very good results.
As you allude to @lihaoyi once we have the code generator with full knowledge about all types, sky is the limit for the what we can generate. For now a lot of effort has been spent on generating boilerplate to use third party react components in an effortless way, see for instance usage of the antd component library in a scalajs-react demo .
It's easy to envision code transformations which for instance could move all members of javascript types to extension methods and translate T | Null to Option[T], js.Promise[T] to Future[T], wrap everything in IO/ZIO, wrap event streams in fs2 streams or any other similar thing. A whole lot of fun experimentation can be done in this direction. I invite you all to the ScalablyTyped gitter channel to dream up some cool things there.
However, let's not go overboard here. scala-js-dom is at the top of most projects' dependency trees for pretty much any Scala.js project. One of the huge strengths of the Scala.js ecosystem is its unparalleled stability and strong compatibility story over time.
Reiterating what I said in #486 (comment), I'd personally favor a conservative direction for scala-js-dom. It has very few problems now, and any interested party can voluntarily try a more experimental DOM wrapper. My two cents anyway
Thanks all, this was very informative for me :) I see the value in preserving what we have here while possibly augmenting with copy-pasta from ST, very similar to how this library was born too as far as I can tell.
That's probably the path forward if we want to easily experiment with different encodings and wrappers, as manually wrapping the thousands of DOM APIs everytime we want to try something new is totally infeasible
@japgolly I do agree with this, that's definitely a tricky thing with your NullOr proposal as it would be a lot of work to go through the source and find all the places we need to make this change. At the very least, a 2.x issue (this issue is currently in 1.x milestone).
I believe that properly considering ambitious ideas is the best way to operate
💯 thrilled to be working with someone who has this philosophy 😁
I still want to push for this because I believe it's in my, and everyone's best interests, and I still want to sketch up my ideas and have a more concrete debate
👍 but imho we shouldn't put this in a 1.2 release but definitely consider it for 2.0.
Should we open a new issue to discuss code gen?
I will do this.
Instead of an issue, I made "discussion" #487 for code gen, hopefully the discussion format will work better for this topic!
Here's a quick sketch of my proposal:
import scala.scalajs.js
import scala.scalajs.js.|
sealed trait NullOr[+A] extends js.Any
object NullOr {
@inline def apply[A](a: A): NullOr[A] =
a.asInstanceOf[NullOr[A]]
@inline def empty: NullOr[Nothing] =
null
// Non-commutative. Do properly later.
@inline def fromUnion[A](a: A | Null): NullOr[A] =
a.asInstanceOf[NullOr[A]]
@inline implicit final class Ops[A](private val self: NullOr[A]) extends AnyVal {
@inline def asUnion: A | Null =
self.asInstanceOf[A | Null]
@inline final def get: A =
self.asInstanceOf[A]
@inline final def toOption: Option[A] =
if (self eq null) None else Some(get)
// ++ all the same methods as exists in js.UndefOrOps
}
}and some example usage
@js.native
object SomeFacade extends js.Object {
val blah: NullOr[Int] = js.native
}
def usageExample() = {
SomeFacade.blah.toOption: Option[Int]
SomeFacade.blah.asUnion : Int | Null
SomeFacade.blah.get : Int
SomeFacade.blah.orNull : Integer
}I think this is great
- no boxing or custom Scala data type - it's just literally a more precise representation of a JS type
- ops are always implicitly available
- convert back and forth to unions and/or null-less types
- no problem with Scala 3, when explicit nulls become a stable feature then
.asUnionis right there - provides more safety for the many, many years to come where we're supporting Scala 2 and cross-compiling everything
@sjrd @lihaoyi does seeing this clarify my original intent? Would you still have concerns with this kind of solution?
This proposal looks straightforward to me! Curious to hear @sjrd and @lihaoyi's thoughts.
It doesn't compose with other union types, with type inference, with parameter type inference for functions, etc. In a sense it all boils down to type inference.
I think this is still true ... I'm just not sure why it's so important, that we can't/shouldn't have NullOr at all 🤔
@japgolly I think it clarifies the original intent, but I don't think it really alleviates my concerns.
Really, it comes down to not wanting the "raw" API for scala-js-dom to have anything that's not already widely used and ubiquitous. Nulls, for better or worse, are widely used in both Javascript and JVM and ubiquitous to both platforms. There's a time and place for experimenting with new encodings and new core data types, and I don't think it should be in the most widely depended on Scala.js package in the entire ecosystem.
If NullOr was already in broad usage, part of the standard library like UnderOr is, a common idiom for other Scala.js code, or commonly used for other facades e.g. on ScalablyTyped, then I'd be more willing to use it here. But none of these is currently the case.
As I said above, I think the path for trying novel ways to improve things with significant breakage (and source incompatibility with every single null-returning DOM API is a very significant breakage!) is to play around in the .ext package, possibly using codegen to ease on the maintainability, and have people organically adopt the new-and-improved way of doing things as it stabilizes and proves its worth. That seems like a much better path than rolling out this kind of experimental improvements in-place in a codebase that has traditionally been stable and widely depended upon
If NullOr was already in broad usage, part of the standard library like UnderOr is, a common idiom for other Scala.js code, or commonly used for other facades e.g. on ScalablyTyped, then I'd be more willing to use it here. But none of these is currently the case.
Another idea, what if NullOr was published as a mini-library and we depend on it for 2.0+. We'd end up going first but then presumably other projects and facades would start using it too. The end result would be the same as what you're describing, it's just that this could be the first bit of mainstream use. And to be clear, I wouldn't be making this argument for anything radical, it's just that a find NullOr so universally useful and trivial in its implementation, so personally I'm finding those properties acceptable tradeoffs in this specific case.
@japgolly yes, having it be part of scala-js itself would make me ok with it being in scala-js-dom, but I can't speak to the technical reasons for whether that would be a good idea or not
Moving it out to a mini library nobody else is using does not address my concerns. I care about existing adoption and standards, not about how we organize jars and poms. If it was a mini library many people were already using across the ecosystem, then I'd be ok with it.
As i've said elsewhere, scala-js-dom is effectively part of the Scala.js standard library. Literally everyone depends on it, often through deep dependency trees. It should be the last to adopt any new innovation, not the first! Let application code with zero dependencies be the first, and other libraries be second. scala-js-dom should be the last in line for adoption, after the consequences and characteristics are widely known and any unexpected issues have been worked out.
NullOr's definition is useful and trivial. Such a large breakage across the entire Scala.js ecosystem (a lot of DOM APIs can return nulls!) is very much not trivial. Neither are the unexpected consequences and edge cases; if we end up having to iterate on NullOr as part of scala-js-dom, that means N large breakages across the entire Scala.js ecosystem, with multiple incompatible versions floating around, and people getting random versions resolved based on their dependency tree. That would be a disaster.
I'm not opposed to you trying this out, but you can try it out without breaking anyone by doing it in an .ext package, iterate to your hearts contents, and people can adopt when they're ready as it naturally stabilizes. Even without a specialized code generation tool, it should be pretty easy to set up SBT to copy-paste the source folder and regex-mangle some definitions so we can easily keep both versions in sync without additional maintenance overhead. It would be a bit of build-tool hackiness, but that definitely beats widespread breakage and incompatibility
I guess I just have a very different view of risk and change so we're probably not going to see eye to eye. That being said I'd like to keep exploring this discussion a little longer and see what comes out of it. See for me, I'm not greatly worried about risk here because no one's gonna die if we end up with an improvement but not The Best Improvement In The World, and the risk mitigation as I see it, should it be necessary, is literally, just a little migration script.
It should be the last to adopt any new innovation, not the first!
Why? That's not at all axiomatic or universally agreed upon. We can already clearly see that this is an improvement upon the status quo. Why should a majorly used library be the last to get an improvement? The whole fear here seems to be "what if we come up with a different encoding in the future". Like we know how to handle those types of migrations with minimal effort to users, we have the tools, so again: why should improvement only come from the fringe edges and not from the core?
Such a large breakage across the entire Scala.js ecosystem (a lot of DOM APIs can return nulls!) is very much not trivial
2.0 will have breaking changes regardless, plus this is a binary-compatible change, just not source compatible.
unexpected consequences and edge cases; if we end up having to iterate on NullOr as part of scala-js-dom
The underlying details may change but the API wouldn't. The API would be pretty much exactly the same as UndefOr which is already prior-art.
multiple incompatible versions floating around, and people getting random versions resolved based on their dependency tree. That would be a disaster
This seems reactionary rather than well considered to me. The API would be stable, we can tinker with the implementation under the hood if we need to without breaking compatibility. What would "disaster" really be in this case? I'd agree with your words 100% in other cases but the point I'm trying to stress here is that to me, this specific case when you really consider it concretely, doesn't seem to warrant these types of concerns which are normally valid.
I'm not opposed to you trying this out, but you can try it out without breaking anyone by doing it in an .ext package
I hadn't considered half-way, (or maybe "both world" is a better word), solutions, but that does open up new doors. The first two ideas that come to mind:
- Automatedly duplicate the entire library under a different package, where one package tree is explicit nulls and the other is transparent. Users could choose whichever they like. Potential problems would be that multiple instances of the same types would be considered distinct by the type system. Probably solvable via automation and creativity.
- Two distinct modules (eg.
scala-js-dom&scala-js-dom-experimental-or-something-similar) in which the API in the later module is exactly the same with the exception of nullable types (which also gives binary compatibility between both. Would have to think about the implications of both being on the classpath + transitive composition via libraries)
Any other ideas for good compromises?
Playing devil's advocate: a reasonable counter-argument to my "hey what's a little migration if needed" is that the Scala community has put up with an insane amount of required downstream maintenance over the years, and one the points of Scala 3 that I'm starting to appreciate the most, is that upgrading everything after minor Scala versions will be a thing of the past. So maybe maintenance-fatigue is a fair critique?
We can already clearly see that this is an improvement upon the status quo.
I don't see it as a clear improvement, maybe a sideways change. It's stricter, but also more verbose, and makes Scala.js diverge from both Scala-JVM and Javascript itself in how platform nulls are handled. Not obvious to me that it's an improvement on the status quo.
Why should a majorly used library be the last to get an improvement?
The total amount of migration work on each breaking change is proportional to the number of downstream projects. Thus it makes sense to try and minimize the churn in the most upstream projects: this is normally done by introducing new ideas in the most downstream projects where churn is cheap (change the API, no dependents to migrate!) and letting them stabilize before introducing them into the more upstream projects (change the API, lots of dependents to migrate)
In this case we aren't going to be the one paying the full migration cost, since we don't maintain all the downstream projects, but it still exists in the community. I know a lot of people complain about churn and breaking changes in the Scala ecosystem, and the maintenance burden of publishing libraries; this is our chance not to unnecessarily exacerbate that problem.
Separately, the more upstream a project is, the more likely diamond dependencies become an issue. I don't actually know how diamond dependencies work on Scala.js, but on Scala-JVM they are always a headache
Like we know how to handle those types of migrations with minimal effort to users, we have the tools
I'm not sure what tools you've been using to do minimal effort migrations, but I do plenty of migrations at work with large Scala-JVM dependency trees, and we spend time (hours-days-weeks) wrestling with breaking changes and diamond dependency issues on an ongoing basis. And the more upstream the dependency, the bigger the headache. As a user of a lot of Scala libraries, I'm definitely not seeing this "minimal effort" haha
But I think I've said my piece, can see what others think
Any other ideas for good compromises?
Yes! I think you are already there actually :)
Two distinct modules (eg.
scala-js-dom&scala-js-dom-experimental-or-something-similar) in which the API in the later module is exactly the same with the exception of nullable types (which also gives binary compatibility between both.
The BIG win here is if the experimental version is kept 100% binary compatible.
Would have to think about the implications of both being on the classpath + transitive composition via libraries)
Here's how we fix this:
- for a library,
scala-js-dom-experimentalshould NEVER be a dependency that is inherited transitively by downstreams. If you wish to use it (internally), add it as a compile-time-only (provided?) dependency. - If you transitively depend on
scala-js-domin your classpath, but prefer to use the syntax available inscala-js-dom-experimental, then simply excludescala-js-domand add a dependency toscala-js-dom-experimentalfollowing rule (1) as needed.
I think this gets us 90% of the way there. Two caveats I can think of:
- Ok, admittedly this is pretty painful for those who want to use
scala-js-dom-experimental. But if the prospect of null handling for years and years to come is even more painful, then hopefully that should mitigate a one-time SBT setup? - Even following (1), a library should not expose
NullOrin a public API, because that would break the abstraction. The way to solve this would be to makeNullOrinto a seperate microlib, so you can excludescala-js-dom-experimentalwithout excludingNullOr.
I hope this could be a reasonable compromise, thoughts?
I feel like the "less is more" applies here and perhaps delaying this until 3.x has been out awhile and this lib has been published under new management. Like many, I also tried a bunch of ways and I don't think I ever liked any of them enough to feel strongly that there was an obvious choice.
I don't see it as a clear improvement, maybe a sideways change. It's stricter, but also more verbose, and makes Scala.js diverge from both Scala-JVM and Javascript itself in how platform nulls are handled. Not obvious to me that it's an improvement on the status quo.
Wow. I have no words. If this is honestly the way that some people view this situation then fine, but I'm done putting in any more effort towards it, at least for now. I'm not gonna waste time trying to convince people that "nulls are bad and no-nulls good". If we can't even agree on that then this is by far a lost cause.
Not obvious to me that it's an improvement on the status quo.
Seriously, just wow.
@japgolly I'm hopeful that once the community sees this library is back on track with the basics (i.e., merging PRs and releasing regularly) it will inspire more engagement and support for advancement and feature development :)
@armanbilge Yeah maybe. I'll just close this for now, we can always re-open it later.
This thread popped back up into my mind today and I just wanted to say sorry @lihaoyi for the way I responded earlier. I'm kind of going through some really hard times this year and I let some misplaced emotion rule and I can see now that the way I responded above is actually pretty rude. I shouldn't have responded to you like that, and I shouldn't have made this space oddly-heated so I'm also really sorry to the entire thread. I'm such a huge proponent of big, warm, welcoming, accepting, environments and upon reflection, my behaviour here did the exact opposite. I'll keep striving to be better but for now I just wanted to say sorry :)