adapt tests to less capable libcurl versions

[bagder]

• trurl can get built with a restricted libcurl
• trurl might execute with a restricted libcurl

(where "restricted" means it might not have all the features or might suffer from a known bug.)

These needs to be taken into account when running tests, since not all features can evaluate and be performed in all situations.

[bagder]

One way we could do it:

1. provide a command line option to trurl that outputs its limitations in a list
2. make tests that require a special feature to be marked accordingly
3. have the test runner skip the tests in (2) for which (1) says can't be run.

[krishean]

I have a potential solution to this for the python script, I'll put in a request when I get back to my computer.

It adds a "minversion" attribute to the relevant tests in tests.json, and gets the libcurl version from the trurl version string on starting the script.

With minor changes instead of "minversion" it could be something like "features" with an array of the required features in the json, and like you previously mentioned have a command line switch to output a list of supported features that could be read into an array.

[bagder]

It adds a "minversion" attribute to the relevant tests in tests.json, and gets the libcurl version from the trurl version string on starting the script.

That can work. At least if we also provide information about which libcurl version that was used when trurl was built - some functionality are limited by built-time conditions and some by run-time.

[bagder]

I made #156 to help.

[krishean]

Would we need two minversion attributes in the json then, one for build-time and another for run-time? That would be easy enough with something like "minruntime" and "minbuildtime".

Or would one minversion attribute be checked against both verions?

[krishean]

Have a look at: https://github.com/krishean/trurl/tree/built-version, if it's what you're looking for I'll open a request.

Otherwise I can change it if needed.

If you could provide some examples of which tests are dependant on certain build/run-time versions I can add the attributes to the json. I know the "--accept-space" tests require version 7.78.0 or above, but I'm unsure if that's a run-time or build-time dependency (I'm guessing run-time?) I've also noticed a couple of the tests with ipv6 addresses fail on ubuntu 18.04/20.04, but I'm unsure what version that requires (all tests pass flawlessly when built against 8.0.1.)

Here's the ones that fail with libcurl/7.58.0 on ubuntu 18.04:

29: failed	'--url https://10.1/we/are.html --get "{host}"'
30: failed	'--url https://[fe80::0000:20c:29ff:fe9c:409b]:8080/we/are.html --get "{host}"'
53, 54	(--accept-space)
74: failed	'--url [2001:0db8:0000:0000:0000:ff00:0042:8329]'

[havbon]

Another thing to consider is that some tests will fail on Windows builds, because of the {} issues. We should adapt Windows tests accordingly. Find a way to exclude those tests on Windows for example

[krishean]

because of the {} issues.

I'm not sure what you mean, I have all the tests passing on windows right now?

[havbon]

essentially .\trurl.exe localhost --get '{host}' just returns host.
see #129

this is one of the test cases when i directly run python.exe .\test.py

82: failed      'https://curl.se?name=mr%00smith --get {query:name}'
--- stderr ---
expected:
''
got:
''
--- returncode ---
expected:
'0'
got:
'0'
--- stdout ---
expected:
'mr.smith'
got:
'query:name'

[bagder]

I know the "--accept-space" tests require version 7.78.0 or above, but I'm unsure if that's a run-time or build-time dependency

It's a build-time requirement, but then it also does not work for earlier run-times either.

[bagder]

I've also noticed a couple of the tests with ipv6 addresses fail on ubuntu

See URL-QUIRKS.md. That's the "Normalizing IPv6 addresses" in 7.81.0, a run-time req.

[bagder]

Another thing to consider is that some tests will fail on Windows builds

I consider that a separate issue. Also: "I have all the tests passing on windows" makes it more complicated than just "on Windows". Is it because of running in powershell vs otherwise?

[krishean]

With windows 10 (I don't have a readily available windows 11 vm to test right now) I tried running python3 test.py with the test.py version over in my branch on cmd, powershell 5, and powershell core and was unable to replicate the failing tests. When directly running trurl in powershell with bare {} parameters it did fail, but that is an expected powershell behavior. Running trurl with quoted parameters on powershell the tests passed for me as well