Skipping compiling with dart2js because some of its transitive libraries have sdk dependencies that are not supported on this platform:.
Opened this issue · 28 comments
Asking here first per dart-lang/build#4179 (comment)
CC and thanks to @davidmorgan
There should be a way to override the unsupported platform check. dart-lang/build#2200 (comment) mentions flutter has this disabled, how?
The code paths in my library have no problems with web yet I cannot use webdev serve to compile a js library.
Many parts of analyzer work on web yet it seems to be blacklisted. How can I force this to allow a package through and compile the code?
If I dart2js then webdev serve, even though webdev serve skips building, the file built through dart2js works. If I must use dart2js, how can I fully package all the dart dependencies? These packages work fine in flutter too, so I should be able to build them with webdev serve.
Can you clarify what you are asking for? Are there libraries that your application depends on that webdev reports are invalid for the web and blocks compilation? Can you provide the specific libraries that are are having problems with?
Can you clarify what you are asking for? Are there libraries that your application depends on that webdev reports are invalid for the web and blocks compilation? Can you provide the specific libraries that are are having problems with?
@nshahan the restrictions between webdev and dart build js are inconsistent.
The best example is probably the analyzer package which https://pub.dev/packages/dart_eval depends on, it has many paths that can work on web.
Webdev needs to allow an override to stay consistent with dart2js instead of skipping all compilation off of basic library presence detection.
It appears we removed the ability to skip platform checks here: dart-lang/build@6a93e68
Maybe @jakemac53 has additional context?
So, this is a story as old as time, originating in a desire to launch a product quickly instead of correctly.
The ability to skip the check was added in only for flutter web, as soon as flutter web stopped using build_runner we removed the unused functionality. I believe the dart2js/DDC support also was only added to support the flutter web launch because the flutter web team asked for it.
It is my strongly held opinion that the support for compiling dependencies with dart:io or other unsupported imports should have never existed, it opens up your application to arbitrary runtime errors that cannot be caught at compile time. Some library could at any time start using a function from dart:io without warning and you would be broken, even if your code happens to work today. Or maybe you just have some edge cases which aren't tested on web (but are tested on other platforms) and so they only fail on web and only at runtime after you deploy to your users.
The fix here is to ask for platform independent libraries to import from the analyzer team.
Unfortunately, if we have this flag, then the vast majority of users will just end up using it instead of keeping the ecosystem and their own code platform agnostic, or separating out their libraries in such a way that the platform agnostic parts can be imported separately.
Unless there is some huge outcry from a very large number of users over this, everybody is better served by not allowing this, it forces the ecosystem as a whole to be better.
I understand the choice to restrict this but disagree, developers should have the choice as it is entirely possible to use a library with restricted dependencies with flutter. The behavior is inconsistent with flutter, giving flutter a competitive advantage over other frameworks that want to develop on dart. The restriction is superficial in that it does not actually analyze if the code path would generate an error on a given platform, but instead broadly restricts without override. If a dev can build using code paths that do not cause errors or conflicts and test these paths, I should be able to compile/use the code.
I do agree that it would make more sense if dart provided core libraries that gracefully fail on unsupported apis to all platforms. Precedent for at least IO has been established with https://pub.dev/packages/universal_io, but I cannot refactor analyzer or other third party packages that use IO (intentionally working still on web).
Always better to give users a choice than to police the ecosystem with a presumptive restriction. It's the developer's responsibility to test their code on the provided platforms and not many devs are compiling to JS these days anyway, I do not see how webdev supporting an override would pollute the ecosystem much if at all.
Essentially this does more to restrict those that want to produce a JS package from the dart package ecosystem than it helps to keep the dart package ecosystem clean. My intended use is to offer a JS SDK using dart packages, not to produce a JS bundle published in a dart package.
If I understand your request correctly, you would like a mechanism to allow compilation of applications that contain imports of libraries that are not supported on the target platform. I'm going to keep this issue open but we don't expect to add support here at this time. Please feel free to 👍 if you discover this issue and also need the support so we can track this need.
I understand this is confusing since you can observe some tools supported by the Dart and Flutter teams allow this but the inconsistencies are simply an example of long term technical debt that one day we hope to clean up. It is extremely difficult to clean up usages that have proliferated through the ecosystem but we still believe it is a useful goal to work towards so new tooling, compilers, and infrastructure can be built without having to implement bespoke support for unsupported libraries.
The recent history of the Dart language has been to move more errors from runtime to compile time (examples: Dart 2.0 static type system and 2.12 null safety). Similarly, rather that allowing a platform unsupported library to be compiled into an application and cause runtime errors we have opted for a library level granularity for supported platforms.
Always better to give users a choice than to police the ecosystem with a presumptive restriction.
I wish it was that easy. In practice, our teams have found success in trying to find the delicate balance of choice, guidance, and sometimes enforcing prescriptive decisions. User feedback is helpful here, thank you for this issue and discussion. There have been many users confused by how some APIs in an unsupported library can work while others mysteriously fail at runtime without any warning or documentation.
It's the developer's responsibility to test their code on the provided platforms
And it is our job to help make this as easy and intuitive as possible. Currently there is no mechanisms to tell users if an API is supported on any given platform other than writing code that exercises the API fully and test it. It could be argued that member level granularity would allow more code to compile and run but supporting that would put the burden on all package authors to define and document support for every single API and ensure that stays correct even when code they depend on changes. There are not any mechanisms currently to make this possible aside from the library level support.
Maybe this should be a language level request for a mechanism to formally define platform support for a package, library, or some other granularity.
If I understand your request correctly, you would like a mechanism to allow compilation of applications that contain imports of libraries that are not supported on the target platform.
Not quite, even with proper stubbing to prevent the application from accessing unsupported apis in those libraries, the general restriction denies the compilation for the webdev tool but not dart2js. It's not a matter of trying or allowing it to compile unsupported paths, it's a matter of if the library is included at all for native support, it cannot be compiled with the webdev tool. There is no way to avoid this blanket restriction no matter how you architect your package if you aim to support native as well, you have to maintain two entirely separate dart packages, one for web, one for native, requiring developers of apps to do the same, overall worse for the ecosystem. With this decision you are asking developers of packages to choose support for native or web and as mentioned this only applies to the dart ecosystem, flutter ignores this check entirely and compiles mixed packages given they are properly stubbed.
Hopefully the unwillingness to support an override flag is reconsidered until the strategy regarding packages is fully defined. The technical complexity of supporting a flag that was previously available seems minor compared to ironing out the graceful failure of native APIs on web.
I do not think many will stumble on this issue as there are few compiling dart to JS outside of flutter. It is not such a common use case to produce a JS SDK from a dart package, but it could be if not so restricted.
There is no way to avoid this blanket restriction no matter how you architect your package if you aim to support native as well, you have to maintain two entirely separate dart packages, one for web, one for native, requiring developers of apps to do the same, overall worse for the ecosystem.
While we don't make it easy, it is very much possible to write a platform agnostic package, which has a single import yet works on all platforms it supports (either through dependency injection or conditional imports).
The problem is pushing the ecosystem to actually develop in this way. When we introduced this check originally, there was initially some pain but we did manage to get the entire (angular at the time) ecosystem migrated to use these patterns including all their transitive dependencies.
but I cannot refactor analyzer or other third party packages that use IO (intentionally working still on web).
Have you filed an issue on the analyzer?
The analyzer was never designed to be used on the web and does not support the web today. It might happen to run on the web if you inject a special file system etc, but that is mostly just by accident and general good library design (which allows you to inject custom file systems etc).
While we don't make it easy, it is very much possible to write a platform agnostic package, which has a single import yet works on all platforms it supports (either through dependency injection or conditional imports).
Yes, it is possible and I have published several packages that support both, a platform interface or stubs can be used. The problem here is that webdev does a nested dependency search and cares nothing about the proper way to build packages with both.
The analyzer was never designed to be used on the web and does not support the web today. It might happen to run on the web if you inject a special file system etc, but that is mostly just by accident and general good library design (which allows you to inject custom file systems etc).
Analyzer was the example, this problem exists for many packages. It does a nested search for any banned dependencies and prevents compilation regardless of the actual code paths used. If I use a package that includes IO in a stub or a platform implementation, webdev will just prevent compilation to JS.
The majority of packages in dart that compile to JS are for flutter, flutter skips this check and allows the libraries like IO so it would go unnoticed by anyone developing for Flutter. The problem arises when I try to compile a package that has both native AND web paths using webdev. It simply will not allow it because it stops when it detects a native dependency regardless of the actual path used, EVEN in nested platform specific packages. I cannot skip this check. This means I must use dart2js to compile my dart dependencies to JS outside of flutter or maintain a separate web package to superficially satisfy webdev or only use dartjs. The current check and inability to provide an override flag prevents packages from compiling with webdev to JS if there is even mention of the wrong package in nested platform specific package dependencies.
If I stub with conditional imports native/web packages, webdev will fail. If I create separate packages entirely for native and web, webdev will fail because a banned package is detected in the pubspec, even if that particular package is not used.
It sounds to me like the check today is too strict: it is currently rejecting web programs that have native code in their transitive packages, when it could instead only reject web programs that have native code in their transitive imports.
Micheal, did I understand that correctly? If so I think this is different to what Nick understood.
Perhaps it would help the discussion if you could give a specific example of a program that should work and does not work today.
And perhaps we should also have a concrete example of a program we all agree should not compile :)
It sounds to me like the check today is too strict: it is currently rejecting web programs that have native code in their transitive packages, when it could instead only reject web programs that have native code in their transitive imports.
This is not accurate, it looks at the imports not the dependencies. There are actual imports to dart:io in this case. It supports conditional imports etc fully.
FWIW, I do think there is some benefit in being able to allow for importing unsupported libraries that simply throw at runtime. This is what allows for the Flutter widget previewer (a Flutter web application) to preview widgets in libraries that do import dart:io but don't actually touch any of the dart:io APIs directly. We're then able to display an error message at runtime if a particular preview does invoke a dart:io API without preventing the entire preview environment from loading.
Do I think that behavior should be disabled by default? Yes. However, I do think there should be an escape hatch that prints a suppressable warning about the possibility of encountering runtime errors.
If flutter did not enable this by default without even so much as a warning, the number of flutter packages out there importing dart:io unconditionally would be significantly lower. They would have been forced to be platform agnostic from the start, and the ecosystem would be in a much better place.
Expanding that support outside of flutter for a handful of use cases is just not worth the potential harm to the core ecosystem. This core Dart package ecosystem is generally developed platform agnostic, directly as a result of this check. There is no way it would have happened otherwise, and this in itself is proof that it is very much possible to develop packages in such a way that this is not an issue.
This is not accurate, it looks at the imports not the dependencies. There are actual imports to
dart:ioin this case. It supports conditional imports etc fully.
Was mistaken on this, sorry to add confusion on this point. I tested again today the compilation of a few packages with webdev and it did ignore io when stubbed/packaged properly. For me it seems the only code path this prevents me from easily shipping is a dart interpreter that relies on analyzer as a JS bundle. Trying to add a dart-scripted CPU stage for my gpu pipeline library.
Expanding that support outside of flutter for a handful of use cases is just not worth the potential harm to the core ecosystem. This core Dart package ecosystem is generally developed platform agnostic, directly as a result of this check. There is no way it would have happened otherwise, and this in itself is proof that it is very much possible to develop packages in such a way that this is not an issue.
I think maybe the harm by an override is being over-estimated here, especially if the long horizon goal is for a unified dart with paths that work on both or gracefully fail. There still seems to be this arbitrary division at the core of dart between web and native support that confuses devs on a larger scale. This check in my opinion does more to create segmentation in the ecosystem (flutter packages that work on web but do not work for dart web), sure we can blame flutter but the cat's out of the bag on that and they are the largest part of this ecosystem. Maybe I am ignorant on this, but I do not see many JS packages being developed in dart, definitely not more than packages being developed for flutter web, indicating the pollution on the dart side would be minimal.
I'm sympathetic to both sides.
It feels to me like dart:io being platform specific was a mistake: it should itself be multiplatform, which would solve the problem once instead of requiring everyone to solve it over and over again.
Could we have a compile mode you can turn on where we replace dart:io with the platform-suitable package of your choice?
Could we have a compile mode you can turn on where we replace
dart:iowith the platform-suitable package of your choice?
A way through pubspec to upsert dependencies is a great idea that could solve other problems with broken packages too.
Connecting discussions together - here are 3 issues from the dart-lang/sdk repo relevant to this discussion:
Could we have a compile mode you can turn on where we replace
dart:iowith the platform-suitable package of your choice?
Instead of doing this you should just use a platform agnostic package that can use dart:io or whatever else as needed.
Note that existing packages (like package:file) don't do this properly because they still use the types from dart:io regardless.
A separate platform agnostic API needs to be added on top, which delegates to dart:io instead of implementing the types from it.
Could we have a compile mode you can turn on where we replace
dart:iowith the platform-suitable package of your choice?Instead of doing this you should just use a platform agnostic package that can use
dart:ioor whatever else as needed.
But you can't do that if you want to use a package that is not platform agnostic.
Dart's capability to compile cross platform is its killer feature--the idea that we intentionally limit it seems thoroughly broken to me.
Edit: it's also broken to provide a way to compile with very sharp edges, I agree there :)
We have three modes:
- Don't support
dart:ioon Web at all, as DDC today. - Support compiling with
dart:iobut bad failure modes: dart2js and Flutter today. - Proposed: support compiling with a different package swapped in for
dart:io.
I think we could move everything to 3, including Flutter, and get rid of the broken mode that nobody likes but is there for pragmatic reasons.
I don't think it's reasonable to follow any path that leaves a different state for one of the three, so if Flutter is not going to remove the dart:io hack then I think we need to come up with something that works and is not a hack.
Which is what 3 is supposed to be :) ... any other ideas?
Proposed: support compiling with a different package swapped in for dart:io.
This would also allow developers to provide a compatible package that supports isolates on web via web workers, another big division between native and web that limits cross-platform compatibilities.
- Don't support dart:io on Web at all, as DDC today.
Note that DDC fits (2), just like dart2js. It is webdev/build_web_compilers what imposes (1).
It is easy to change DDC and dart2js to impose the restriction (1), which would make it consistent across the board, including flutter. Of course, this would be a breaking change that requires a migration effort first.
Such migration would likely require:
- introducing a package that uses conditional imports to provide a web-safe implementation. This could be exactly the implementation you'd use for (3).
- replacing all imports of
dart:ioto point to that package - change behavior from (2) -> (1) in all of our tools.
I believe this is also what @jakemac53 is suggesting with a platform agnostic API.
Your suggestion of (3) seems to be asking for a new feature: the ability to replace a library with another during compilation. It is a form of external dependency injection. Conditional imports require that you make this choice when authoring the package, whereas (3) suggests we make that choice later, just before compiling. It appears this could be handled by tooling, like build_web_compilers fabricating an artificial package containing a conditional import and replacing all references to dart:io in user's code to point to this new package before it delegates works to other SDK tools. We can also achieve (3) with a proper language feature, which make the CFE/analyzer aware about these late import binding.
In any case, comparing the (2->1) migration vs (3): it seems (3) would in essence implement the kind of migration we need for (2->1), but rather than landing it in the package ecosystem, it does it on every compile.
My take - I'd prefer to see the ecosystem evolve to a cross platform package solution via some migration tactic, I worry that a late dependency injection approach spreads the friction among many developers rather than addressing the issue at its source.
Hmmmmmmmm.
If dart:io was a normal package, you could just pull it from a difference source in your pubspec and all the tools would work fine. It's not import binding or a language feature--it's just part of how building works that you can pick your dependencies.
Publishing if you do that would not work well, but I don't think that's for any fundamental reason, just because we haven't explored this space yet.
I think the actual problem is that 1) SDK packages work less well than normal packages: you can't override them in the pubspec; and 2) dart:io was written as a single platform package, if we could go back 12 years we could design it as cross platform.
So I don't really see why this is a problem the ecosystem should deal with--when it's a problem that is completely due to us.
I think it's a reasonable thing for a compiler to pick the platform libraries at startup, and for that to be configurable. If we try it and there are rough edges then that would be a reason to rethink, but at first glance it doesn't sound too hard--and if it works well, we're done? And nobody has to migrate anything.
The concept of a simple flag or pubspec entry to switch dart:io to something else is exciting to me--it would give a lot of new multiplatform code, immediately and pretty much for free.
I also think it's reasonable to provide some sort of override at compile time, but the main issue would be that the override won't be able to implement dart:io fully and will always have situations where runtime errors will need to be thrown.
If we do decide to investigate this further at a more fundamental level, Flutter still needs to be able to run web applications that depend on dart:io for developer tooling purposes, so fully imposing this restriction across all of our tools is a non-starter.
Given the existence of core libraries that are platform specific and the current state of the ecosystem, I think it's completely reasonable to throw UnsupportedErrors when trying to access unsupported platform specific functionality.
Until we migrate away from having platform specific core libraries, I propose that:
- We allow for importing platform specific libraries for unsupported platform targets
- All tools (
flutter run,dart compile, andwebdev) print a warning when compiling projects that reference an unsupported platform specific library - We provide flags to silence this warning or make it a fatal compilation error
As @davidmorgan has already said, this situation is due to decisions made by our team, and we shouldn't needlessly increase friction for the community when we have a viable workaround for a common situation in the ecosystem.
I fully agree we should aim to avoid or reduce friction for the community, but I see aspects of the workarounds that worsen the ecosystem and add more friction over time. Personally, I'm sad we added the workaround for allowing dart:io many years ago, I think it made the ecosystem worse as a result and exacerbated the issue we are seeing today.
Back to brainstorming - thanks @bkonyi, your points highlight for me that we are conflating a couple problems together:
- (a) needing cross-platform support for certain libraries (e.g. IO) to get a platform agnostic solution to a problem.
- (b) needing to run code in one platform, where the code has transitive dependencies on platform-specific code of another platform, but such platform-specific logic is not used or relevant for subset of the code you will run (so the code practically behaves as if it is platform agnostic).
Reviewing the (1), (2), and (3) solutions I see how much we conflate both problems:
- (1) A way to solve both is to move towards an ecosystem where all packages are platform agnostic, and platform-specific dependencies are hidden by conditional imports. This has been the traditional angle we have pursued and leans on (a) to eliminate (b). Tools already report static errors when trying to use platform-specific libraries in unsupported platforms (except
dart:io). If we pursue (1) we makedart:ioconsistent with all other libraries and we push towards the general direction where more packages become platform agnostic. The big drawback here is the migration cost. Unless we can automate it, this is a big friction. - (2) is like the approach @bkonyi suggested: we are providing a default cross-platform fallback implementation for platform-specific libraries, that is safe to depend on as long as it is unused. We aren't solving (a) here, we are only solving (b). Unfortunately, if the fallback is reached during code execution (which we won't know statically) the program behavior is unspecified. This is not aligned with Dart's philosophy of detecting issues early. We could say the default could be an static error with a flag to turn it into a warning/info (a small variant of (2)) to align better with Dart's philosophy. In practice, the worry here is that over time the flag may end up being specified always, as developers will ignore and forget that the flag was set in their local configuration.
- (3) Finally, option (3) provides a path to solve (a) and (b) together, but also dynamically. This option also has frictions: you need to override the implementation of
dart:iobefore running tools. This friction is spreadout among users of the tools, rather than among package authors. If we provide a default implementation fordart:io, so most users don't have to think about it, and that default would throw on all APIs, then this looks a lot like (2) with more general purpose uses. Testing that the backup implementation works properly is also not trivial and a friction we'd be spreading among users, rather than hardening the packages that provide the functionality.
Since these decisions have impact well beyond just webdev, I think it's worth trying to reach some alignment also from a language and ecosystem perspective.
I see (1) having potential if we can solve the migration problem. Maybe there is a low cost solution here? If I have to chose between (2) and (3), I lean towards (2). It has a smaller scope overall compared (3). Also, if we aren't sure about this direction and we are looking for a short term way to unblock our users, restricting (2) to dart:io only may be preferable. Even though I'd like a consistent solution for all platform-specific libraries, dart:io is the only library that doesn't adhere to (1) already and the one causing the most friction today.
With platform specific imports an author only needs to worry about re-implementing the subset of dart:io that they use. A mechanism for replacing the library entirely sets a very high bar on publishing an alternative. We do not have any supported mechanisms to validate that any given replacement for dart:io is correct, or even that it has the same API statically. I'd expect to end up with at best a number of partial implementations published where the UnimplementedErrors are unpredictable and variable.
Having multiple alternatives for dart:io which each supporting some subset of the API would be an even more confusing situation for app authors. Leaving the choice for what implementation to plug in and how to handle compatibility puts the complexity on app developers, while platform specific imports shifts the complexity to pub package maintainers. It isn't feasible to implement a version of dart:io for browsers that will satisfy every use case.
Note also that allowing people to replace dart:io or any core library with a package of their choice would make evolving the core libraries incredibly difficult - even more so than it already is. Altering the API surface in any way whatsoever would now be a breaking change for all of these packages that implement that core library.
Some of us got together to discuss the options here, we seem to be leaning towards a hybrid of (1) and (2) with new ideas to explore. Here is a brief summary from our brainstorming:
-
We want both: to gradually fix the ecosystem and ensure we don't block progress on tools like widget previews.
-
On unblocking tools: a restricted form of (2)
- For now, continue to allow DDC/dart2js to ignore imports to
dart:iofor longer. - Evolve this into a general purpose capability to ignore platform imports (like
dart:ffitoo), but restricted to selective tools (like widget previews).
- For now, continue to allow DDC/dart2js to ignore imports to
-
On fixing the ecosystem: a less restricted form of (1)
-
We already made the Dart ecosystem hide
dart:iobehind conditional imports and we would like to keep that. For that reason, we prefer only expose the capability to ignore platform imports to the tools that need it and wont regress the ecosystem in the process (e.g. Widget Previews is OK, but we feardart cliorwebdevwon't be). -
Fixing the Flutter ecosystem is not trivial and we think we can do more in the language and libraries, beyond conditional imports, to make it tractable to fix. We need additional analysis of the package ecosystem to answer which strategies will be most helpful here. New ideas we mentioned:
- Type imports: ability to import types without the implementation. Many packages that aim to be a platform agnostic IO implementation only import
dart:iounconditionally to be able to reuse types. - Split platform agnostic declarations. For example, portions of
Platform, enums and values like http error codes are not really platform specific. Today, some of these declarations are combined into types that can't be split, but it may be possible to split them in the future with the use of static extensions (when that feature becomes available).
- Type imports: ability to import types without the implementation. Many packages that aim to be a platform agnostic IO implementation only import
-