PASS Resource Duplicate Checker (PASS-RDC)
The PASS Resource Duplicate Checker traverses a Fedora repository containing PASS resources, and for each resource, determines if a duplicate exists by performing queries of the Elastic Search index. Results from the process are persisted in a local Sqlite database. The identity properties used for determining duplication are documented below.
Design-wise, PASS-RDC could have (should have?) been coded to work primarily with the index, using it as a source of truth. Since the index does not contain auditing information like dates and creators, the duplicates would need to be retrieved from Fedora at the end of the process to fill in those necessary details used for analysis. A future TODO :)
Known Issues and TODOs
Presented in approximate priority order with respect to resolution.
Queries that target multivalued fields require all values be present to be considered a duplicate
For example, given a Journal
with two ISSNs, a query for duplicate journals would require the duplicate to have both ISSNs. Consequently, a potential duplicate that possesses only one of the ISSNs in question would not be found by this query. This issue affects queries for the Journal
(issn
), Submission
(preparers
), and User
(locatorIds
) type: fields that allow multiple values are affected by this issue.
For example, given the two journals below, Journal A and B are equivalent: they have the same name, and share at least one ISSN.
- Journal A,
name
: "Journal of Foo",issn
: Print:ABCD-1234, Online:WXYZ-6789 - Journal B,
name
: "Journal of Foo",issn
: Print:ABCD-1234
PASS-RDC has a shortcoming: the Elastic Search query for Journal A will require a match on both of its ISSNs, and will not discover Journal B. The Elastic Search query for Journal B will require a match on its single ISSN, and will match Journal A. This implementation shortcoming should not affect the ultimate result (after all, the query for Journal B correctly identifies the duplicate), but ought to be fixed to be more robust.
Conceivably this issue will cause a problem where there are more than two permutations of a multi-valued attribute. Let's assume PASS is updated to support Linking ISSNs along with Print and Electronic ISSNs. The following is a contrived, but feasible situation:
- Journal A,
name
: "Journal of Foo",issn
: Print:ABCD-1234, Online:WXYZ-6789 - Journal B,
name
: "Journal of Foo",issn
: Print:ABCD-1234, Linking:JKLM-8765
In this scenario, the duplicate Journals will not be discovered, because the query for Journal A will insist on a match for the Print and Online ISSN, while the query for Journal B will insist on a match for the Print and Linking ISSN. Since neither Journal has the full complement of ISSNs, they won't be discovered by their respective queries.
User resources might also be affected by this issue, where there may be up to three values contained in locatorIds
. Some User
resources possess two values for locatorIds
(missing a hopkinsId
), while others possess all three. I haven't seen any User resources with only one value for locatorIds
.
Pausing and resuming is not implemented
While the persistence layer and associated models support the ability to resume, the production code does not persist the progress of PASS-RDC in the local database. Completing the resume implementation was de-prioritized since the process runs reliably and within a reasonable amount of time. Finishing the implementation is a TODO.
Sqlite database lifecycle management is limited
PASS-RDC does very little with respect to the management of the Sqlite database used to store the results of its analysis. If a database is present at the supplied DSN, it is used. If a database is not present, it will attempt to be created by default, unless otherwise instructed (e.g. by supplying a mode
to a URI DSN, see the documentation for opening a new connection and the URI format). After running PASS-RDC, move the database for safekeeping; PASS-RDC does not make any accommodations for re-using a previously initialized database (when resumption is implemented, using an existing database will be explicitly supported).
It is likely that attempting to re-use an existing database will result in a slew of table constraint violations.
Query plans may not be modified at runtime
PASS-RDC comes packaged with an internal query plan that cannot be modified at runtime.
Query plans only support boolean OR logic
Query plans support boolean OR processes when executing and analyzing Elastic Search results. Other boolean operators (e.g. AND or NOT) are not supported at this time.
Requirements and Running
PASS-RDC must be executed against a running instance of Fedora and Elastic Search. It assumes the index is the source of truth for the presence of duplicate objects, therefore the index must be complete and uncorrupted. That is, PASS-RDC is not meant to assess the integrity of the index, but the integrity of the PASS resources contained in the Fedora repository.
Storage requirements are modest: less than 1 MiB for storing duplicate information, and less than 500 GiB if full state supporting resumption is persisted.
Usage
./passrdc -starturi <base repository uri> -indexuri <base elasticsearch uri> -dsn <sqlite dsn>
Example: ./passrdc -starturi http://fcrepo:8080/fcrepo/rest -indexuri http://elasticsearch:9200/pass/_search -dsn file:/passrdc.db?mode=rwc
Overview
Invoking passrdc
will launch parallel goroutines to walk the resources in the Fedora repository, starting from the URI specified by -starturi
.
PASS-RDC can be invoked on a subset of the repository, e.g. only check for duplicate users by providing
http://fcrepo:8080/fcrepo/rest/users
as the-starturi
As repository resources are encountered, a query plan will be executed against the Elastic Search index based on the PASS type of the resource. Depending on the query plan for a given resource, multiple queries may be performed against the index for a single repository resource. Query plans are based on the type of PASS resource, and informed by the identity properties documented below.
Query plans are JSON documents that enumerate the identity properties of a given type, associate them with an Elastic Search query, and support simple boolean operations between query results
The results of the query plan execution are aggregated, then filtered, with suspected duplicates being persisted in the database.
If a query only returns one result, then the object does not have a duplicate, so the result is not useful for analysis and is discarded
Identity Properties
The identity properties of PASS resources were originally documented here, but they are reproduced below with an additional commentary with respect to their usage in PASS-RDC. PASS-RDC uses these identity properties to determine if two different resources are equivalent. For example, if two Journal
resources have the exact same title
and have at least one issn
in common, then they would be considered equivalent - i.e. a duplicate Journal
exists.
PASS-RDC does not check the following object types for duplication:
Institution
,Policy
,Repository
: these objects are loaded manually by application administrators. It is assumed that the manual loading process does not introduce duplicate objects.SubmissionEvent
,File
,Deposit
: these objects are created during the lifecycle of the submission process, and are within the scope of a singleSubmission
. It is assumed that if a duplicateSubmission
is found, the objects associated with its lifecycle may be remediated.Contributor
: this object is associated with aPublication
; if thePublication
is deemed a duplicate, the associatedContributor
may be remediated.
Object types checked for duplication, and their associated identity properties are:
- Publisher
name
ANDpmcParticipation
Publisher
objects are considered equal if they have the samename
andpmcParticipation
- Journal
nlmta
OR (journalName
AND at least oneissn
in common)Journal
objects are considered equal if they have the samenlmta
or if they have the samejournalName
and share at least oneissn
- Publication
- (
doi
ORpmid
) ORtitle
Publication
objects are considered equal if they have the samedoi
orpmid
, or if they have the sametitle
.
- (
- Funder
localKey
Funder
objects are considered equal if they have the samelocalKey
- Grant
localKey
Grant
objects are considered equal if they have the samelocalKey
- RepositoryCopy
accessUrl
OR (repository
† ANDpublication
†)RepositoryCopy
objects are considered equal if they have the sameaccessUrl
, or if they have the samerepository
† andpublication
†.
- User
- share at least one
locatorIds
User
objects are considered equal if their set oflocatorIds
intersect; e.g. User A and User B only need to share one locatorId in common to be considered equivalent.
- share at least one
- Submission
publication
† AND (share a commonpreparers
† ORsubmitter
†)Submission
objects are considered equal if they have the samepublication
† URI and share a common preparer or have the same submitter.
† values of these properties are URIs to other PASS resources
When an identity property is a URI referencing another PASS resource, all potential equivalent resources identified by the URI need to be considered. For example, given two publishers where <Foo>
denotes the URI of object Foo:
- Given
<Publication A>
and<Publication B>
are considered equivalent (e.g. their resources have the samedoi
) <Submission A>
and<Submission B>
share a commonsubmitter
- (note that
submitter
is also a URI; in this example thesubmitter
URI is identical for both submissions, so it does not need further consideration: they are byte-for-byte identical)
- (note that
<Submission A>
publication
references<Publication A>
and<Submission B>
publication
references<Publication B>
- (unlike
submitter
, thepublication
for the submissions are not identical)
- (unlike
The equivalence of Submission A
and Submission B
cannot be determined without knowing whether their respective Publications are equivalent.