SOLR is down
Closed this issue · 4 comments
OAI is not working
changed instance
Expected Behaviour
Solr should be working during indexing and when users harvest doi metadata.
Current Behaviour
Solr crashes while users are harvesting doi metadata in oai
Steps to Reproduce
Make this query
https://solr.datacite.org/api?q=*:*&rows=50&start=9555400&sort=updated asc&fq=updated:[0001-01-01T00:00:00Z TO 9999-12-31T23:59:59Z]&qt=/api&wt=javabin&version=2
Context (Environment)
it occured twice in jan 9 at the same time of a SOLR indexing and another time during harvesting doi on jan 13 (there were no logs because oai moved to fargate ithink).
some of the queries seem from only one client TIB.AWI
/oai?verb=ListRecords&resumptionToken=[1547270829286-1547343979088],0001-01-01T00:00:00Z,9999-12-31T23:59:59Z,[50-5750],TIB.AWI,oai_datacite
and some times the y seem to go into very deep pages even when small
request: https://solr.datacite.org/api?q=*:*&rows=50&start=9555400&sort=updated asc&fq=updated:[0001-01-01T00:00:00Z TO 9999-12-31T23:59:59Z]&qt=/api&wt=javabin&version=2
Seems to be going to document 477,770,000
Hypothesis
deep paginating is killing solr.
currently TIB.AWI doesn't seem to realise the service is down and it keep deepaginating even if they get errors.
By deterring deep pagination the index wont die.
Detailed Description
Block deep paginating for solr.
Possible Implementation
it solved the issue. We had to reindex both instances
For reference:
to release a new version of the Solr container one needs to change the tag version in the vpc folder
https://github.com/datacite/mastino/blob/1acb68cc0daca8e2c9ed4ec0574f21cfe8315b0b/prod-eu-west/vpc/_search.auto.tfvars#L7
but to change the instance to be shown in the elb the change has to be made in search folder