Call tests via `flutter test` instead of `flutter drive` possible?
golane-august opened this issue · 3 comments
Can you use integration_test package instead of driver package to test the app?
Use:
flutter test integration_test/gherkin_suite_test.dart
Instead of:
flutter drive --driver=test_driver/integration_test_driver.dart --target=integration_test/gherkin_suite_test.dart
But I get the error with flutter test:
flutter test integration_test/gherkin_suite_test.dart
00:00 +0: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart R00:11 +0: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart 11.7s
✓ Built build/app/outputs/flutter-apk/app-debug.apk.
00:12 +0: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart I00:13 +0: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart 656ms
00:15 +0 -1: loading /Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart [E]
Failed to load "/Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.dart": Bad state: Can't call group() once tests have begun running.
package:test_api Declarer.group
package:flutter_test/src/test_compat.dart 189:13 group
package:flutter_gherkin/src/flutter/runners/gherkin_integration_test_runner.dart 117:5 GherkinIntegrationTestRunner.runFeature
Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.g.dart 34:5 _CustomGherkinIntegrationTestRunner.testFeature0
Users/august/Coding/git/jonsamwell/flutter_gherkin/example_with_integration_test/integration_test/gherkin_suite_test.g.dart 26:5 _CustomGherkinIntegrationTestRunner.onRun
package:flutter_gherkin/src/flutter/runners/gherkin_integration_test_runner.dart 94:5 GherkinIntegrationTestRunner.run
00:16 +0 -1: (tearDownAll) - did not complete [E]
00:16 +0 -1: Some tests failed.
I used the example in integration_test__package_support: https://github.com/jonsamwell/flutter_gherkin/tree/integration_test__package_support/example_with_integration_test
I also tested flutter run with tagExpression @failure-expected, but then the result doesn't match the failure:
flutter run integration_test/gherkin_suite_test.dart
...
I/flutter (16379): FAILED: Scenario Failed expect() should be added to json report # ./integration_test/features/failure.feature:0
...
I/flutter (16379): 00:03 +3: All tests passed!
Also calling this with tagExpression: '@failure-expected':
flutter drive --driver=test_driver/integration_test_driver.dart --target=integration_test/gherkin_suite_test.dart -d chrome
results in:
FAILED: Scenario Failed expect() should be added to json report # ./integration_test/features/failure.feature:0
00:03 +2: (tearDownAll)
2 scenarios (2 failed)
4 steps (2 passed, 2 failed)
0:00:03.088000
00:03 +3: All tests passed!
Application finished.
Note the "All tests passed!", which sould not occur.
The problem seems to be that the generated executeTestSuite method is synchronous, but the run method is asynchronous:
void executeTestSuite({
required FlutterTestConfiguration configuration,
required StartAppFn appMainFunction,
Timeout scenarioExecutionTimeout = const Timeout(const Duration(minutes: 10)),
AppLifecyclePumpHandlerFn? appLifecyclePumpHandler,
LiveTestWidgetsFlutterBindingFramePolicy? framePolicy,
}) {
_CustomGherkinIntegrationTestRunner(
configuration: configuration,
appMainFunction: appMainFunction,
appLifecyclePumpHandler: appLifecyclePumpHandler,
scenarioExecutionTimeout: scenarioExecutionTimeout,
framePolicy: framePolicy,
).run();
}
Also the main method is done before the test setup is complete:
@GherkinTestSuite(
useAbsolutePaths: false,
)
void main() {
executeTestSuite(
appMainFunction: appInitializationFn,
configuration: gherkinTestConfiguration,
);
}
But Flutter assumes that with the end of the main method the test declaration is finished and does not allow any more calls on the test api (group, test, testWidgets, ...)
You could probably make the executeTestSuite method asynchronous and await it.
Future executeTestSuite({
required FlutterTestConfiguration configuration,
required StartAppFn appMainFunction,
Timeout scenarioExecutionTimeout = const Timeout(const Duration(minutes: 10)),
AppLifecyclePumpHandlerFn? appLifecyclePumpHandler,
LiveTestWidgetsFlutterBindingFramePolicy? framePolicy,
}) async {
return _CustomGherkinIntegrationTestRunner(
configuration: configuration,
appMainFunction: appMainFunction,
appLifecyclePumpHandler: appLifecyclePumpHandler,
scenarioExecutionTimeout: scenarioExecutionTimeout,
framePolicy: framePolicy,
).run();
}
@GherkinTestSuite(
useAbsolutePaths: false,
)
void main() async {
await executeTestSuite(
appMainFunction: appInitializationFn,
configuration: gherkinTestConfiguration,
);
}
What do you think @jonsamwell ?