Mismatch in TestReports versions when using properties
Closed this issue · 4 comments
If a package has TestReports as a test dependency, the version of TestReports that is used to generate the XML in the cmd may be different from the version that calls the command.
E.g. if PkgA has TestReports fixed at v0.4.0, then TestReports v0.5.4 is loaded and used to do TestReports.test("PkgA"), v0.4.0 will actually be used by the test environment. I'm not sure how often this will have an affect and how bad it will be. At the very least, it stops the TestWithProperties test package from being tested successfully during unit tests when I'm trying to make changes for Julia 1.7, as it uses an older version from the registry that is only 1.6 compatible.
Could
- Run a check for this an error if different versions are being used (might cause some CI headaches but probably safest)
- Force it to the version set by the package being tested, with a warning
- Force it to the version being used to do the report generation, with a warning
- Check to see if a compat field is added in
PkgA, if yes do 1, if not do 3 without a warning
This is basically a bug in Julia itself.
It should error or at least warn you when you activate an environment containing a different version of a package from one you have already loaded.
I think there is an issue be open for this.
I don't think anything can be done about this in this package
I think the issue is that TestReports uses stacked environments here, and Julia doesn't promise to resolve anything except what is in the primary environment (I think). If the package being tested has TestReports as a dependency/test dependency, this version will be used ahead of version that is running and in the stacked environment.
If pushfirst! is used here instead of push, then that will stop this happening. But that was something that changed with #53 as it broke other things.
There might be a better solution by using some more of Pkg internals to copy the test environment to a temporary location and check/add the correct version of TestReports, and then pass this temporary location to sandbox.
I don't understand why pushfirst! loadpath doesn't work
I see the comment saying it doesn't but it just says ''per discussion on slack"
#53 (comment)
Been doing some checking and it wasn't the pushfirst! that didn't work on #53, more that the dependencies of TestReports were not being resolved and therefore couldn't be loaded (they weren't in the manifest). The changes in the pr that closed that (#64) manually constructed a new temporary environment with all of the dependencies of TestReports resolved.
As for why pushfirst! and not push!, I think it was so that the primary environment was the test environment of the package being tested. However changing to pushfirst! would solve the problem here as long as it was accompanied by a check that ensured the package being tested did not specify a version of TestReports that is incompatible with the one being used to do the testing (either in package or test deps). I think push first would be okay, as it would just ensure that TestReports and any dependencies (only EzXML at this point) are the versions required for TestReports to function, and all other dependencies for test would be resolved in the secondary environment already by Pkg.