Question about EARL reports
hsolbrig opened this issue · 8 comments
When I run python tests/runtests.py ../json-ld-api/tests/ ../json-ld-framing/tests/ -e pyld-earl.jsonld, I get
FAILED (failures=18, errors=168, skipped=7)
This doesn't seem to line up with the summary printed in https://w3c.github.io/json-ld-api/reports/
Digging into the report itself, the first failure shows at:
{
"@type": "earl:Assertion",
"earl:assertedBy": "https://github.com/dlongley",
"earl:mode": "earl:automatic",
"earl:test": "https://w3c.github.io/json-ld-api/tests/compact-manifest#tc009",
"earl:result": {
"@type": "earl:TestResult",
"dc:date": "2021-05-25T21:31:04Z",
"earl:outcome": "earl:failed"
}
},
Which, if I understand the reports correctly, appeared to have passed previously. Help!
At least some of the failures seem to fit the pattern below:
EXPECTED: {
"@context": {
"@vocab": "http://example/",
"Foo": {
"@context": {
"baz": {
"@type": "@vocab"
}
}
}
},
"@type": "Foo",
"bar": {
"baz": "buzz"
}
}
ACTUAL: {
"@context": {
"@vocab": "http://example/",
"Foo": {
"@context": {
"baz": {
"@type": "@vocab"
}
}
}
},
"@type": "Foo",
"bar": {
"baz": {
"@id": "http://example/buzz"
}
}
}```
I haven't looked into this, but it may in part be a case of the the test suites changing and pyld hasn't caught up. But looks like you got a bunch of failures and errors so maybe it's something else too.
The reports really should be clear that results are for a test suite and implementation as of a certain date.
As far as tracking down and fixing the issues, help would be appreciated.
Happy to pitch in -- just wanted to be certain I understood the starting point.
Ok - part of the issue is that URL's of the form https://json-ld.org/test-suite/tests/frame/0005-in.jsonld are no longer valid.
curl https://json-ld.org/test-suite/tests/frame/0005-in.jsonld
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at json-ld.org Port 80</address>
</body></html>
If I do
curl https://json-ld.org/test-suite/
I get
<div class="hero-unit">
<h1>JSON-LD Test Suite</h1>
<br>
<p>Please use the Working Group version of the test suite at <a href="https://w3c.github.io/json-ld-api/tests/">https://w3c.github.io/json-ld-api/tests/</a> and <a href="https://w3c.github.io/json-ld-framing/tests/">https://w3c.github.io/json-ld-framing/tests/</a> instead.</p>
</div>
So - do we update the manifest and any embedded links? How to proceed?
Tests should now all be at https://w3c.github.io/json-ld-api/tests/ and /json-ld-framing/tests/. Those in json-ld.org are obsolete.
Will update the manifests accordingly.
Ah - I see what is going on. The instructions for running the tests are a bit ambiguous....
Was using https://github.com/digitalbazaar/pyld/blob/master/CONTRIBUTING.rst , which wasn't clear on where the tests came from. Will file a separate issue