Update mapping schemes barely works

[stefandesu]

The SPARQL query to update mapping schemes seems to be pretty heavy and does not work for me anymore. This is the URL that it is outputting on the console is this:

https://query.wikidata.org/sparql?format=json&query=%23%20getMappingSchemes%0ASELECT%20DISTINCT%0A%20%20%3Fproperty%20%3Fscheme%20%3FschemeLabel%20%3FshortName%0A%20%20%3Fbartoc%20%3FTEMPLATE%20%3FnotationPattern%20%3Fidentifier%20%3Fextent%0AWITH%20%7B%0A%20%20SELECT%20%3Fproperty%20%3Fscheme%20%3Fbartoc%20%3FTEMPLATE%20%3FnotationPattern%20%7B%0A%20%20%20%20%3Fproperty%20wdt%3AP1921%20%3FTEMPLATE%20.%0A%20%20%20%20%3Fproperty%20wdt%3AP1793%20%3FnotationPattern%20.%0A%20%20%20%20%7B%20%7B%20%3Fproperty%20wdt%3AP1629%20%3Fscheme%20%7D%20UNION%20%7B%20%3Fscheme%20wdt%3AP1687%20%3Fproperty%20%7D%20%7D%0A%20%20%20%20%3Fscheme%20wdtn%3AP2689%20%3Fbartoc%20.%0A%20%20%7D%0A%7D%20AS%20%25properties%20WHERE%20%7B%0A%20%20INCLUDE%20%25properties%20.%0A%20%20%0A%20%20OPTIONAL%20%7B%0A%20%20%20%20SELECT%20%3Fproperty%20%28SAMPLE%28%3Fnumber%29%20AS%20%3Fextent%29%20%7B%0A%20%20%20%20%3Fproperty%20wdt%3AP4876%7Cwdt%3AP1114%20%3Fnumber%0A%20%20%20%20%7D%20GROUP%20BY%20%3Fproperty%0A%20%20%7D%0A%0A%20%20OPTIONAL%20%7B%0A%20%20%20%20SELECT%20%3Fscheme%20%28GROUP_CONCAT%28%3Fname%29%20AS%20%3FshortName%29%20WHERE%20%7B%0A%20%20%20%20%20%20%3Fscheme%20wdt%3AP1813%20%3Fname%0A%20%20%20%20%7D%20GROUP%20BY%20%3Fscheme%0A%20%20%7D%0A%0A%20%20OPTIONAL%20%7B%0A%20%20%20%20SELECT%20%3Fscheme%20%28GROUP_CONCAT%28%3Furi%29%20AS%20%3Fidentifier%29%20WHERE%20%7B%0A%20%20%20%20%20%20%3Fscheme%20wdt%3AP2888%7Cwdt%3AP1709%20%3Furi%20.%0A%20%20%20%20%7D%20GROUP%20BY%20%3Fscheme%0A%20%20%7D%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%0A%20%20%20%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22.%0A%20%20%7D%0A%7D%20ORDER%20BY%20%3Fscheme

Can we do this in a more reliable way?

Edit: It took about 10-15 tries for it to work.

[stefandesu]

It seems like this broke even further, now I'm getting this message:

❯ npm run update

> wikidata-jskos@0.4.0 update
> ./bin/wdjskos update && cp cache/mapping-schemes.ndjson assets

undefined:1
SPARQL-QUERY: queryStr=# getMappingSchemes
^

SyntaxError: Unexpected token 'S', "SPARQL-QUE"... is not valid JSON
    at JSON.parse (<anonymous>)
    at parseJSONFromBytes (node:internal/deps/undici/undici:5584:19)
    at successSteps (node:internal/deps/undici/undici:5555:27)
    at fullyReadBody (node:internal/deps/undici/undici:1665:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async specConsumeBody (node:internal/deps/undici/undici:5564:7)
    at async httpRequest (file:///Users/stefan/repos/gbv/wikidata-jskos/lib/request.js:11:18)

Node.js v20.12.2

[nichtich]

I think it's better to not rely on a perfect query but hard-code the list of supported terminologies.

[stefandesu]

Now it worked for me, for some reason. However, I had to add a workaround to get the correct BARTOC URIs: 8ae6729

In the Wikidata BARTOC ID property, there are two formatting URLs: formatter URL and formatter URI for RDF resource. The former is set to the https version of the BARTOC URIs (people always change that), the latter has the correct pattern with http. The latter is supposed to be used to assemble the BARTOC URI in the SPARQL query, but for some reason, it is not. There's not even a reference to P1630 in the query, so I have no idea why it doesn't work.