obis-qc

Quality checks

The following checks are performed on each occurrence record in order to add quality flags, tag as absence data, or drop records from the main index:

Check	Fields	Flags	Absence	Dropped	Vandepitte et al. flag number
`occurrenceStatus` should be present.	`occurrenceStatus`
`occurrenceStatus` should be `absent` or `present`.	`occurrenceStatus`		x
If `individualCount` equals 0, record is absence.	`individualCount`		x
`basisOfRecord` should be present.	`basisOfRecord`
`basisOfRecord` should be `PreservedSpecimen`, `FossilSpecimen`, `LivingSpecimen`, `MaterialSample`, `Event`, `HumanObservation`, `MachineObservation`, `Taxon` or `Occurrence`.	`basisOfRecord`
`eventDate` should be present.	`eventDate`				7, 11, 12, 13
`eventDate` should conform to ISO 8601.	`eventDate`
`eventDate` should not be in the future.	`eventDate`	`DATE_IN_FUTURE`
`eventDate` should be later than the set minimum date (year 0)	`eventDate`	`DATE_BEFORE_MIN`
`decimalLongitude` and `decimalLatitude` should be present.	`decimalLongitude`, `decimalLatitude`	`NO_COORD`		x
`decimalLatitude` and `decimalLongitude` should not be zero.	`decimalLongitude`, `decimalLatitude`	`ZERO_COORD`		x	4
`decimalLatitude` and `decimalLongitude` should be within range. The range of `decimalLongitude` is between -180 to 180 and the range of `decimalLatitude` is between -90 and 90.	`decimalLongitude`, `decimalLatitude`	`LON_OUT_OF_RANGE`, `LAT_OUT_OF_RANGE`, `NO_COORD`		x	5
`coordinateUncertaintyInMeters` should be present.	`coordinateUncertaintyInMeters`
`coordinateUncertaintyInMeters` should be within the range of 0 and 10000000.	`coordinateUncertaintyInMeters`
`minimumDepthInMeters` and `maximumDepthInMeters` should be present.	`minimumDepthInMeters`, `maximumDepthInMeters`	`NO_DEPTH`
`minimumDepthInMeters` and `maximumDepthInMeters` should be within the range of -100000 and 11000.	`minimumDepthInMeters`, `maximumDepthInMeters`	`MIN_DEPTH_EXCEEDS_MAX`
`minimumDepthInMeters` should not exceeds `maximumDepthInMeters`.	`minimumDepthInMeters`, `maximumDepthInMeters`	`NO_DEPTH`
`minimumDepthInMeters` and `maximumDepthInMeters` should be less than or equal to the bathymetric depth.	`minimumDepthInMeters`, `maximumDepthInMeters`	`DEPTH_EXCEEDS_BATH`			19
Is the occurrence located on land? Based on OpenStreetMap land polygons.	`decimalLongitude`, `decimalLatitude`	`ON_LAND`			6
`scientificName` should be present.	`scientificName`
`scientificNameID` should be present.	`scientificNameID`
`scientificNameID` should be valid WoRMS LSID.	`scientificNameID`				2
Taxon should unambiguously match with WoRMS.	`scientificName`, `scientificNameID`	`NO_MATCH`		x
An accepted name should exist in WoRMS.	`scientificName`, `scientificNameID`	`NO_ACCEPTED_NAME`
Taxon should not be exclusively freshwater or terrestrial according to WoRMS.	`scientificName`, `scientificNameID`	`NOT_MARINE`, `MARINE_UNSURE`		in case of `NOT_MARINE`
`measurementType` and `measurementTypeID` should be present if extended Measurement or Fact extension is being used.	`measurementType`, `measurementTypeID`

Vandepitte et al. flags not implemented: 3, 9, 14, 15, 16, 10, 17, 21-30.

Installation

pip install -r requirements.txt
python setup.py install

Usage

See tests.

Using a cache to speed up taxonomy checks

By default the taxonomy component fetches information from WoRMS the WoRMS API by AphiaID. If you have a local cache of WoRMS information, you can use that instead of the API connection by providing an object that implements the fetch() and store() methods. The WoRMS information objects provided to the cache are constructed like this:

record = pyworms.aphiaRecordByAphiaID(aphiaid)
classification = pyworms.aphiaClassificationByAphiaID(aphiaid)
bold_id = pyworms.aphiaExternalIDByAphiaID(aphiaid, "bold")
ncbi_id = pyworms.aphiaExternalIDByAphiaID(aphiaid, "ncbi")
distribution = pyworms.aphiaDistributionsByAphiaID(aphiaid)
bold_id = bold_id[0] if bold_id is not None and isinstance(bold_id, list) and len(bold_id) > 0 else None
ncbi_id = ncbi_id[0] if ncbi_id is not None and isinstance(ncbi_id, list) and len(ncbi_id) > 0 else None
aphia_info = {
    "record": record,
    "classification": classification,
    "bold_id": bold_id,
    "ncbi_id": ncbi_id,
    "distribution": distribution
}

Run tests

nosetests --with-coverage --cover-package=obisqc --cover-html