Add standalone testMode
timcassell opened this issue · 9 comments
Context
Because the IL2CPP runtime is much more fragile than Mono, and doesn't support the full suite of .Net functionalities, it is useful to run tests on that runtime. This can only be done with a built player, not in Editor (it uses Mono, even if IL2CPP is selected for the build). I managed to get it working in my own repo by using NUnitLite instead of unity-test-runner, then building the player and running the built player directly. But it only allows me to run basic NUnit tests and not any [UnityTest]s.
Suggested solution
Add standalone option for testMode.
I asked on the forums if the Unity test runner could have a build-only option to then run the executable directly, and I was pointed to docs that show how to do it already. The standalone option will add the necessary Editor code to override the build options to only build like the docs show, and runtime TestRunCallback code to save out the test results (looks like it needs to be done manually, from the docs) and quit the application with the proper error code.
Considered alternatives
Using NUnitLite instead of unity-test-runner (but cannot run [UnityTest]).
Doing all this manually.
Additional details
In the past, I tried to use the built-in option to run tests on a targetPlatform, but it does not work in Docker seemingly due to the PlayerConnection never connecting. So building the player separately and running it directly to circumvent the PlayerConnection is the only way to make it work until Unity officially supports Docker.
I was only able to get my NUnitLite standalone tests to work on a Windows runner. I am not versed in Ubuntu, so I'm not sure how to get the built player to run properly (in my Ubuntu testing with NUnitLite, the test code never runs and it never quits until it times out). So I'm not sure if this is even possible on Ubuntu (maybe running it inside the Docker instance will make a difference instead of directly on the Ubuntu machine? I'm really not sure...).
Great research and thank you very much for linking the appropriate resources.
I'd support this unless other maintainers have concerns, so open for contributions.
A quick note: eventually this'll have to be built into the cli, which will be the basis for test runner in the future.
Well, I found out how to get the player build working in Ubuntu... I just needed to change this
build/StandaloneLinux64/StandaloneLinux64 {args}
to this
xvfb-run build/StandaloneLinux64/StandaloneLinux64 {args}
That's what I get for not knowing Ubuntu. I think building it headless would also work, but I didn't try it.
And with that, I think this is absolutely feasible. [Edit] Though, I haven't tried a mac build at all.
Great!
xvfb-run
Perhaps you can use the same settings as the other xvfb runs for (hopefully) maximum compatibility.
I would like to work on this, but I am unfamiliar with the project and how to set up my local to work with it. I imagine the necessary scripts could be installed similar to how the builder script is installed in unity-builder, and updating the run_tests scripts to run the built player if necessary. Does that sound right? Can you give me guidance?
@timcassell It sounds like you've got generally the right idea.
On thing I'll note is that getting GitHub Actions to run locally isn't trivial, so when I've worked on actions in the past, I've always just had my own fork. I would work on the action by pushing my code changes to my fork and seeing it run on the GitHub-hosted runners.
@davidmfinol It looks like some of the files are auto-generated, and I'm not sure what tool(s) do that, and what files I should be manually editing to then re-generate those files. That's what I meant. I'm unfamiliar with js.
Ah, gotcha. You need to download and install Yarn, and then run yarn before you commit. There should be pre-commit hook that automatically does this.
Edit: the files you need to edit will be in the src/model folder. When you run yarn, it will take your source files and use them to update the dist folder. You will need to include the updated dist in your commit.
I get an error when I try to commit. What am I missing?
git commit
yarn run v1.22.19
$ C:\Users\Tim\Documents\git\unity-test-runner\node_modules\.bin\lint-staged
[STARTED] Preparing...
[SUCCESS] Preparing...
[STARTED] Running tasks (chunk 1/2)...
[STARTED] Running tasks for *.{js,jsx,ts,tsx}
[STARTED] Running tasks for *.{json,md,yaml,yml}
[SKIPPED] No staged files match *.{js,jsx,ts,tsx}
[STARTED] prettier --write
[SUCCESS] prettier --write
[SUCCESS] Running tasks for *.{json,md,yaml,yml}
[SUCCESS] Running tasks (chunk 1/2)...
[STARTED] Running tasks (chunk 2/2)...
[STARTED] Running tasks for *.{js,jsx,ts,tsx}
[STARTED] Running tasks for *.{json,md,yaml,yml}
[SKIPPED] No staged files match *.{json,md,yaml,yml}
[STARTED] prettier --write
[SUCCESS] prettier --write
[STARTED] eslint
[SUCCESS] eslint
[STARTED] jest --findRelatedTests
[SUCCESS] jest --findRelatedTests
[SUCCESS] Running tasks for *.{js,jsx,ts,tsx}
[SUCCESS] Running tasks (chunk 2/2)...
[STARTED] Applying modifications...
[SUCCESS] Applying modifications...
[STARTED] Cleaning up...
[SUCCESS] Cleaning up...
Done in 13.26s.
yarn run v1.22.19
$ prettier --check "src/**/*.{js,ts}" && eslint src --ext .js,.ts --max-warnings=0
Checking formatting...
[warn] src\index.ts
[warn] src\main.ts
[warn] src\model\action.test.ts
[warn] src\model\action.ts
[warn] src\model\docker.test.ts
[warn] src\model\docker.ts
[warn] src\model\image-tag.test.ts
[warn] src\model\image-tag.ts
[warn] src\model\index.test.ts
[warn] src\model\index.ts
[warn] src\model\input.test.ts
[warn] src\model\licensing-server-setup.ts
[warn] src\model\output.test.ts
[warn] src\model\output.ts
[warn] src\model\platform.test.ts
[warn] src\model\platform.ts
[warn] src\model\results-check.test.ts
[warn] src\model\results-check.ts
[warn] src\model\results-meta.ts
[warn] src\model\results-parser.test.ts
[warn] src\model\results-parser.ts
[warn] src\model\results-report.ts
[warn] src\model\unity-version-parser.test.ts
[warn] src\model\unity-version-parser.ts
[warn] src\post.ts
[warn] Code style issues found in the above file(s). Forgot to run Prettier?
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
husky - pre-commit hook exited with code 1 (error)
The pre-commit hook also runs unit tests and does linting/checks formatting.
That hook failed because prettier found some formatting issue.
You should be able to manually run prettier, which should fix the formatting issue, and you should include the fixed file in your commit.