GD-392: `GdUnitTestSuiteScannerTest#test_load_parameterized_test_suite` is failing running as a single test suite but succeeds when running in bundle with others.

Question

GD-392: `GdUnitTestSuiteScannerTest#test_load_parameterized_test_suite` is failing running as a single test suite but succeeds when running in bundle with others.

Closed this issue 3 months ago · 1 comments

MikeSchulze commented 3 months ago

The used GdUnit4 version

4.2.3 (Pre Release/Master branch)

The used Godot version

v4.2.1.stable.official [b09f793f5]

Operating System

Windows 11, Ubuntu 22.04.4

Describe the bug

The test GdUnitTestSuiteScannerTest#test_load_parameterized_test_suite is failing running as single test suite but succeeds when running in bundle with others.
There is an unknown issue with the text execution, it results unexpected to success in case of:

running all tests over the core package res://addons/gdUnit4/test/core/ no failure is reported and the test test_load_parameterized_test_suite passes.

Run Test Suite res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd 
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path_project :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path_project :PASSED 14ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path_plugins :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path_plugins :PASSED 15ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path__no_test_root :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path__no_test_root :PASSED 16ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path__path_contains_src_folder :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path__path_contains_src_folder :PASSED 15ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path__path_not_contains_src_folder :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path__path_not_contains_src_folder :PASSED 16ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_test_suite_exists :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_test_suite_exists :PASSED 17ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_test_case_exists :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_test_case_exists :PASSED 27ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_create_test_suite_pascal_case_path :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_create_test_suite_pascal_case_path :PASSED 45ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_create_test_suite_snake_case_path :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_create_test_suite_snake_case_path :PASSED 43ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_create_test_case :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_create_test_case :PASSED 25ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_build_test_suite_path :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_build_test_suite_path :PASSED 15ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_parse_and_add_test_cases :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_parse_and_add_test_cases :PASSED 38ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_scan_by_inheritance_class_name :STARTED
Scanning for test suites in: res://addons/gdUnit4/test/core/resources/scan_testsuite_inheritance/by_class_name/
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_scan_by_inheritance_class_name :PASSED 36ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_scan_by_inheritance_class_path :STARTED
Scanning for test suites in: res://addons/gdUnit4/test/core/resources/scan_testsuite_inheritance/by_class_path/
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_scan_by_inheritance_class_path :PASSED 34ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_get_test_case_line_number :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_get_test_case_line_number :PASSED 17ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test__to_naming_convention :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test__to_naming_convention :PASSED 14ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_is_script_format_supported :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_is_script_format_supported :PASSED 14ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_load_parameterized_test_suite :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_load_parameterized_test_suite :PASSED 49ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path :PASSED 12ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path_with_src_folders :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path_with_src_folders :PASSED 16ms
PASSED 486ms
Statistics: | 534 tests cases | 0 error | 0 failed | 0 skipped | 0 orphans |

running the test suite onlye res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd

Run Test Suite res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd 
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path_project :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path_project :PASSED 13ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path_plugins :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path_plugins :PASSED 14ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path__no_test_root :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path__no_test_root :PASSED 14ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path__path_contains_src_folder :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path__path_contains_src_folder :PASSED 13ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path__path_not_contains_src_folder :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path__path_not_contains_src_folder :PASSED 15ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_test_suite_exists :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_test_suite_exists :PASSED 17ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_test_case_exists :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_test_case_exists :PASSED 32ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_create_test_suite_pascal_case_path :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_create_test_suite_pascal_case_path :PASSED 41ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_create_test_suite_snake_case_path :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_create_test_suite_snake_case_path :PASSED 42ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_create_test_case :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_create_test_case :PASSED 23ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_build_test_suite_path :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_build_test_suite_path :PASSED 9ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_parse_and_add_test_cases :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_parse_and_add_test_cases :PASSED 38ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_scan_by_inheritance_class_name :STARTED
Scanning for test suites in: res://addons/gdUnit4/test/core/resources/scan_testsuite_inheritance/by_class_name/
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_scan_by_inheritance_class_name :PASSED 80ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_scan_by_inheritance_class_path :STARTED
Scanning for test suites in: res://addons/gdUnit4/test/core/resources/scan_testsuite_inheritance/by_class_path/
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_scan_by_inheritance_class_path :PASSED 52ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_get_test_case_line_number :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_get_test_case_line_number :PASSED 12ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test__to_naming_convention :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test__to_naming_convention :PASSED 17ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_is_script_format_supported :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_is_script_format_supported :PASSED 17ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_load_parameterized_test_suite :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_load_parameterized_test_suite :FAILED 44ms
	Report:
		line 350: Expecting contains exactly elements:
		 '[tuple(["test_no_parameters", false]), tuple(["test_parameterized_success", false]), tuple(["test_parameterized_failed", false]), tuple(["test_parameterized_to_less_args", false]), tuple(["test_parameterized_to_many_args", false]), tuple(["test_parameterized_to_less_args_at_index_1", false]), tuple(["test_parameterized_invalid_struct", false]), tuple(["test_parameterized_invalid_args", false])]'
		 do contains exactly (in any order)
		 '[tuple(["test_no_parameters", false]), tuple(["test_parameterized_success", false]), tuple(["test_parameterized_failed", false]), tuple(["test_parameterized_to_less_args", true]), tuple(["test_parameterized_to_many_args", true]), tuple(["test_parameterized_to_less_args_at_index_1", true]), tuple(["test_parameterized_invalid_struct", true]), tuple(["test_parameterized_invalid_args", true])]'
		but some elements where not expected:
		 '[tuple(["test_parameterized_to_less_args", false]), tuple(["test_parameterized_to_many_args", false]), tuple(["test_parameterized_to_less_args_at_index_1", false]), tuple(["test_parameterized_invalid_struct", false]), tuple(["test_parameterized_invalid_args", false])]'
		and could not find elements:
		 '[tuple(["test_parameterized_to_less_args", true]), tuple(["test_parameterized_to_many_args", true]), tuple(["test_parameterized_to_less_args_at_index_1", true]), tuple(["test_parameterized_invalid_struct", true]), tuple(["test_parameterized_invalid_args", true])]'
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path :PASSED 5ms
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path_with_src_folders :STARTED
	Run Test: res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd > test_resolve_test_suite_path_with_src_folders :PASSED 14ms
FAILED 520ms
Statistics: | 20 tests cases | 0 error | 1 failed | 0 skipped | 0 orphans |

Steps to Reproduce

./addons/gdUnit4/runtest.sh --add res://addons/gdUnit4/test/core/ -> succeeds
./addons/gdUnit4/runtest.sh --add res://addons/gdUnit4/test/core/GdUnitTestSuiteScannerTest.gd -> fails

Minimal reproduction project

No response

Answer 1 · 2024-04-09T12:48:08.000Z

This is only an internal test setup issue, when using GdAssertReports.expect_fail() it is not reset by default and results into failure reporting issues.