flutter_driver pauses all isolates at their startup (even ones started from compute()), rather than only pausing initial isolate
faustinoribeiro opened this issue · 62 comments
Isolates spawned using "compute" don't execute properly while executed in an integration test which in turn results in incorrect app behavior.
How to reproduce:
- use example from https://flutter.io/docs/cookbook/testing/integration
- in lib/main.dart
- add:
import 'package:flutter/foundation.dart';
- add function:
String _test(String s) {
return s;
}- modify _incrementCounter function:
void _incrementCounter() async {
final s = await compute(_test, 'hello');
print(s);
setState(() {
_counter++;
});
}The app works properly when executing flutter run, but the integration test fails (flutter drive --target=test_driver/app.dart)
Tests use https://pub.dartlang.org/packages/fake_async by default.
Try with https://docs.flutter.io/flutter/flutter_test/WidgetTester/runAsync.html to get real async behavior.
Please consider asking support questions in one of the other channels listed at http://flutter.io/support .
Thanks for the quick response.
I will go to support first next time instead of wrongly assuming it is a bug.
It would be nice to have a "gotcha" section in the documentation about cases like this one.
Keep the great work, Flutter is the first mobile framework I really enjoy working with. I have with 3 other frameworks in the past and it was rarely smooth as Flutter.
Getting back to this issue (please let me know if I should move this to support).
The suggested solution using runAsync applies to widgets testing if I understand correctly.
As I mentioned in my initial comment, I am in early stage of integration testing with flutter_driver.
I am probably missing the obvious, but how do I use runAsync (or something else) to get "compute" isolates to execute properly in that context?
I tried to follow the examples you provide by wrapping my app with runAsync with the hope that the integration tests run smoothly, but the app doesn't even launch. I don't have time to run further tests at this time, so I will come back to this in a couple of weeks. Thanks for the help.
@Hixie who might know if this is supposed to work?
I tried GitHub search but couldn't find unit tests for compute() that could be used as example.
Could you give an example of how to wrap the app under test in runAsync ?
This is what I tried:
import 'package:flutter_driver/driver_extension.dart';
import 'package:flutter_test/flutter_test.dart';
import '../lib/my_app.dart';
void main() {
enableFlutterDriverExtension();
testWidgets(
'app test',
(WidgetTester tester) async {
final app = MyApp();
await tester.runAsync(() async {
await tester.pumpWidget(app);
await tester.idle();
});
}
);
}Generally driver tests are not expected to use flutter_test. The device side is expected to just be the app, and the driver side is expected to be a command-line app. Using testWidgets with driver tests is going to cause a world of pain.
Yes, I guessed that much, but just gave a shot.
This is still surprising that driver tests would make isolates fail (assuming I am not doing something really wrong). How is that even possible? I might need (once I get back to working on integration tests) to use some type of flag to not use compute() while testing. This would however mean that I am not doing proper testing.
I am experiencing this as well in my application when attempting to test with the driver. My application loads assets at startup and ends up hanging on a call to AssetBundle.loadString which has a compute call.
The code is open source so I can share that but if it helps I'm happy to reproduce it in an isolated repo.
Meanwhile using a workaround that probably doesn't suit for everybody's needs
import 'package:flutter/foundation.dart' as flutter show compute;
import 'package:flutter/foundation.dart' show ComputeCallback;
Future<R> compute<Q, R>(ComputeCallback<Q, R> callback, Q message) async {
if (isInDebugMode) {
return callback(message);
}
return await flutter.compute(callback, message);
}I implemented a similar approach when trying to figure out why my tests were failing. It works, but it's far from perfect though because you're not testing your app as it will be shipped. It's acceptable if you have a couple of simple compute operations, but not so much when you have a lot more of those in critical places (like I currently have in many of my app API calls).
Seems like there isn't any workaround for Isolate with async function because of #25890
I have case with authentication in Isolate (with async function inside) and it turns out I don't have a way to login to my app during integration test.
@Hixie Hey, is anything planned to resolve this issue ?
Hey @zoechi I think this issue should be added to the Goals milestone. It prevents me from adding integration tests to my which is really annoying for a production app. Thanks for your help!
This is really annoying and came totally unexpected when using the screenshot package and the Future from DefaultAssetBundle.loadString() just never happened to return. Using .load() instead and manually doing the conversion fixed it, but no way to find out why without looking into the source code.
Also, this can break basically anything, as we have no idea what calls the compute() function under the hood.
I would like to clarify, the problem appears when using FlutterDriver itself. Not necessarily when running tests with package:flutter_test or package:test.
Even when running the following code directly...
void main() async
{
FlutterDriver driver = await FlutterDriver.connect();
await driver.waitFor(find.byValueKey("AssetLoadingButton"));
await driver.tap(find.byValueKey("AssetLoadingButton"));
}
... the AssetBundle.loadString inside the app will hang indefinitely (if the string asset is larger than 10*1024 bytes in size).
@eikob is absolutely right. It is only a matter of time when a particular package accidentally breaks the UI automation.
If this behavior of FlutterDriver is "a feature, not an issue", I would like to see at least a universal workaround. The official Flutter docs does not mention anything about running applications "really asynchronously" in the right way.
Anynews on this issue ?
After looking at the dart VM Observatory I found out, that the isolate is paused at start. When you manually start the isolate, it works perfectly. Maybe that helps figuring out the issue.
Can you show a example how to start the Isolate @The-Redhat ? (I use compute from flutter sdk btw)
You have to start the isolate in the browser. This isn't a workaround.
When you run flutter drive --target=test_driver/main.dart -v it gives you the url of the observatory ('Connecting to Flutter application at ...'). Inside you select the isolate which isn't main. In there you see that the isolate is paused. To start it press debug in the top right corner and type continue in the command line.
Do you think there are a way to make it automatic ? Because test should be run be himself. I will try your way first.
Thanks :) !
As I sat my answer isn't a workaround. It should only help the flutter team to finally fix this issue.
Our app is heavily reliant on isolates, and we wanted to use integration testing for performance analysis of optimizations. This issue means we are unable to do that.
I agree @JaspervanRiet , it's need to be fix. It's real pain.
I (and I assume others) reported this issue almost 8 months ago. This is a major concern for anyone wanting to develop a robust application. I suppose that it has not been resolved yet because there is some deep architecture implementation that makes it impossible or would cause major breaking changes.
I am the sole developer of a somewhat complex app and not having at least some basic automation testing makes me really really nervous. I started to do some automation-like testing by testing BLoCs in various sequences (not using flutter_driver), but it means I am entirely skipping UI testing (I will try to test individual pages though whenever possible).
It would be nice to at least know why no solution has been provided so far. It is an architecture issue? Lack of resources? Is something being worked? Or is it simply not a priority?
I really love Flutter and I am interested in where Flutter is going with web and desktop implementations, but if that means diverting resources from Flutter itself I am afraid it will scare even the strongest supporters away because of a promising platform that never gets "there".
I totally agree with you. For me it's especially annoying, that nobody from the flutter team answers.
Can we have at-least a response from the flutter team ?
Am facing the exact same behavior. I've moved the scrypt-hashing of the user's password for local storage to the background. The isolate only starts when triggered manually from the observatory, otherwise, the test times out.
EDIT: Also... are people really either running everything in their app on the UI thread, or not writing any integration tests? I don't see why this issue only seems to concern a tiny handful of developers.
Any way the bug is still here...
Any timeline on this? Currently, I'm not able to write a single test as my app is dependent on isolate from dart SDK. (Otherwise I could at least stub the implementation from test side)
I've just run into this, using compute to load something on startup. I was wondering for past few hours why my instrumentation is stuck at the splash screen and then found this issue... and wow, this is pretty terrible.
This essentially means whenever you use compute you should code some workaround otherwise your app instrumentation will fail.
I can also confirm that you can just unpause the isolate in the observatory and then it all works... so my question would be why the pause is happening?
my workaround for the startup hanging:
dev_dependencies:
vm_service_client: ^0.2.6+2 # this is discontinued BTWthen in the driver code...
FlutterDriver driver;
VM vm;
// Connect to the Flutter driver before running any tests.
setUpAll(() async {
driver = await FlutterDriver.connect();
vm = await driver.serviceClient.getVM();
// unpause any paused isolates
for (final isolateRef in vm.isolates) {
final isolate = await isolateRef.load();
if (isolate.isPaused) {
isolate.resume();
}
}
});unsure yet if newly created isolates down the line will get paused as well
also depending what your instrumentation is doing you might need to push the resume part a bit later (e.g. for vishna@2f858fc it would crash if called from setUpAll)
...anyway looking forward to hearing from flutter team on this one
I'm using compute and have the same problem.
For open my application in testing, i must "continue" for every spawn that created by compute...
Flutter 1.7.8+hotfix.4 • channel stable • https://github.com/flutter/flutter.git
Framework • revision 20e59316b8 (6 weeks ago) • 2019-07-18 20:04:33 -0700
Engine • revision fee001c93f
Tools • Dart 2.4.0
I'm using the onIsolateRunnable stream with the code below so that it automatically resumes isolates.
FlutterDriver driver;
StreamSubscription streamSubscription;
setUpAll(() async {
// Connect to a running Flutter application instance.
driver = await FlutterDriver.connect(printCommunication: true);
streamSubscription = driver.serviceClient.onIsolateRunnable.asBroadcastStream().listen((isolateRef) {
print('Resuming isolate: ${isolateRef.numberAsString}:${isolateRef.name}');
isolateRef.resume();
});
});
tearDownAll(() async {
if (driver != null) await driver.close();
if (streamSubscription != null) streamSubscription.cancel();
});@Gerrel your solution didn't work for me as my isolates are already paused during start and this callback is never reached but I kept your code around in case new wild isolates appear 🐗
...also modified it a bit to check if the isolate you're resuming is paused indeed, like so:
streamSubscription = driver.serviceClient.onIsolateRunnable
.asBroadcastStream()
.listen((isolateRef) async {
final isolate = await isolateRef.load();
if (isolate.isPaused) {
isolate.resume();
print(
'Resuming isolate: ${isolateRef.numberAsString}:${isolateRef.name}');
}
});you'll find in resume documentation that it
Throws an [rpc.RpcException] if the isolate isn't paused
Wrapped the fix in a IsolatesWorkaround class: https://gist.github.com/vishna/03c5d5e8eb14c5e567256782cddce8b4
FlutterDriver driver;
IsolatesWorkaround workaround;
setUpAll(() async {
driver = await FlutterDriver.connect();
workaround = IsolatesWorkaround(driver);
await workaround.resumeIsolates();
});
tearDownAll(() async {
if (driver != null) {
await driver.close();
await workaround.tearDown();
}
});@vishna This solved my problem 😁 Hoping for a native solution down the line, but that workaround is fine for now 😁
The workaround seems to break if you have several groups inside your test file... Or did I do something wrong?
Any progress? I don't think anybody posted any generic workaround (not mentioning an actual solution) here, to what seems to be a major flaw for (I assume) any suffieciently complex app.
For example, this bit me when my app, calling AssetBundle.loadString(), started hanging under flutter_driver after the file loaded this way grew past the magic barrier of 10KB.
my temp solution for ci.
replace
flutter driver test_driver/app.dart
to two comands
- start test_driver/app.dart on device
bash$ flutter run --observatory-port 8888 --disable-service-auth-codes --pid-file=/tmp/flutter.pid test_driver/app.dartsleep for wait start
- start test_driver/app_test.dart
bash$ VM_SERVICE_URL=http://127.0.0.1:8888/ dart test_driver/app_test.dart after tests
kill /tmp/flutter.pid for clean resource to the next test
I noticed this by accident when I ran integration tests in android studio, and connected to the running application.
they worked.
and these tests did not work through
flutter driver.
We should make flutter_driver not pause any of the isolates except the first isolate (which it needs to pause to be able to hook into before the dart code starts).
Thanks for the workaround. It always work in Android but it seems to be inconsistent in iOS for me. I really hope there is an official fix for this!!!!
Adding performance label as it may affect driver performance tests.
When I try to run flutter drive --target=test_driver/app.dart --flavor sandbox -d 9A311FFAZ00C03 --profile --cache-sksl --write-sksl-on-exit sksl_cache/android.sksl.json on android, the test never runs. The app is installed and the splash screen shows but the dart isolate never starts.
...
Running Gradle task 'assembleSandboxProfile'... Done 109.4s
✓ Built build/app/outputs/flutter-apk/app-sandbox-profile.apk (65.7MB).
Installing build/app/outputs/flutter-apk/app.apk... 5.1s
I/flutter (26379): Observatory listening on http://127.0.0.1:39715/yP9wvTmO1Pw=/
00:00 +0: Bottlepay app SkSL warmup (setUpAll)
Connecting driver
VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:62204/yP9wvTmO1Pw=/
VMServiceFlutterDriver: Isolate found with number: 3846210052022527
VMServiceFlutterDriver: Unknown pause event type VMNoneEvent. Assuming application is ready.
E/FlutterFcmService(26379): Fatal: failed to find callback
VMServiceFlutterDriver: Flutter Driver extension is taking a long time to become available. Ensure your test app (often "lib/main.dart") imports "package:flutter_driver/driver_extension.dart" and calls enableFlutterDriverExtension() as the first call in main().
It then hangs here indefinitely. Is this related?
According to debug printing, it is hanging on driver = await FlutterDriver.connect();.
group('Bottlepay app SkSL warmup', () {
FlutterDriver driver;
IsolatesWorkaround workaround;
// Connect to the Flutter driver before running any tests.
setUpAll(() async {
print('Connecting driver');
driver = await FlutterDriver.connect();
print('Driver connected');
if (driver.appIsolate.isPaused) {
print('Waiting for IsolatesWorkaround to resume isolates');
workaround = IsolatesWorkaround(driver);
await workaround.resumeIsolates();
print('Isolates resumed');
}
print('Waiting for first frame to rasterize');
await driver.waitUntilFirstFrameRasterized();
print('First frame rasterized');
});
// Close the connection to the driver after the tests have completed.
tearDownAll(() async {
if (driver != null) {
await driver.close();
await workaround?.tearDown();
}
});I am unable to run driver tests on android for both a simulator and physical device because of this. On the simulator it freezes on await driver.waitUntilFirstFrameRasterized();.
I got back to this and reproduced the problem.
I suspect https://github.com/flutter/gallery/blob/master/test_driver/isolates_workaround.dart is related. Maybe a conflict between the two fixes for the same bug?
What do folks think of this approach to fixing the issue?
- fix flutter/gallery integration test isolate_workaround.dart to be OK with something else unpausing the isolate. Release that change.
- fix flutter driver to unpause isolates, as in #61841. Verify that the devicelab tests pass this time. Release that change.
- celebrate. Bug is fixed.
- wait a while for older versions of flutter driver to go out of circulation.
- delete flutter/gallery isolate_workaround.dart, since it's no longer needed with new flutter versions.
- and 5. will be complete with flutter/gallery#256
The gallery integration tests which are run from flutter/flutter are pinned to the commit in
https://github.com/flutter/flutter/blob/master/dev/devicelab/lib/versions/gallery.dart. The reland PR for 2. will have to update that hash.
Looks like this is getting reverted again due to Google test failures, so re-opening.
@speaking-in-code see also b/77244607
/cc @rmacnak-google since we were discussing this back in April 2018. At the time, we mused about the ability to send a command to the vm service saying "change the vm behavior such that we don't start new isolates paused anymore".
@tvolkert Is this now really fixed in dev or was it again reverted?
@HerrNiklasRaab this was re-landed and re-closed in #65703 (7a4d8e1), which was included in 1.24.0-3.0.pre, currently on dev.
I have run into a situation that may be related.
I have a 1.7MB json asset that I am loading as a string with rootBundle.loadString(path) and it's making my tests (we're not using driver) hang indefinitely. A 4kb json file going through the exact same code is working flawlessly. The only difference appears to be the size of the asset. I have tried running this on beta and dev and no change.
I tried loading the string outside of a compute but that didn't resolve the issue.
final data = await rootBundle.load(path);
final string = utf8.decode(data.buffer.asUint8List());Is loading large string assets causing non-driver tests to hang indefinitely a known issue?
EDIT:
If I swap testWidgets for test and then fire TestWidgetsFlutterBinding.ensureInitialized(); it works. Is this working as expected? Is this even related to this issue?
@lukepighetti yeah that sounds like it's related to this issue.
- We have code in the framework that, when loading a string asset, spawns an isolate to do the UTF8 decoding if the raw bytes is over 50K in length. This was done because UTF8 decoding can be expensive, so for loading very large strings (e.g. the Flutter LICENSE file), we choose not to block the main isolate.
- The Dart VM, when told to start its isolates in a paused state (
--pause_isolates_on_start), will do so for all isolates as long as the VM is running -- even after the VM receives the message to un-pause all isolates (which is what this bug is filed about) - Driver tests use
--pause_isolates_on_start, and the host-based driver code un-pauses all isolates as the driver extension comes up.
I'm using this wrapper to allow my own compute calls to execute during tests
/// Because Isolates are started "paused" during tests (--start-paused) and don't execute at all, this function
/// wraps [compute] and executes the computation on the same [Isolate].
Future<R> testableCompute<Q, R>(ComputeCallback<Q, R> callback, Q message, {String? debugLabel}) async {
// During tests, don't start a new isolate, instead execute the computation on the same isolate
if (!kIsWeb && Platform.environment.containsKey('FLUTTER_TEST')) {
// fake Isolate.spawn delay
await Future.delayed(Duration(milliseconds: 500));
return await callback(message);
}
return await compute(callback, message, debugLabel: debugLabel);
}This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.