A set of Python3 script parsing MARC21 to FOLIO inventory format.
- Instance Metadata Elements
- Recommended MARC mapping to Inventory Instances
- Recommended MFHD to Inventory Holdings mapping
- Holdingsrecord JSON Schema
- FOLIO Instance storage JSON Schema
- FOLIO Intance (BL) JSON Schema
- Inventory elements - Beta
This template plays a vital part in a process together with other repos allowing you to perform bibliographic data migration from a legacy ILS into FOLIO. For more information on the process, head over to the linked repos below. In order to perform migrations according to this process, you need to clone the following repositories:
The scripts requires a FOLIO tenant with reference data setup properly. The script will throw messages telling what reference data is missing. One way to set up reference data is to use service_tools to download the reference data from one FOLIO tenant and then upload it to the target tenant.
The Scripts also relies on a Folder with a set of mapping files. Look in the migration_repo_template repo to understand what is needed.
MARC mapping for Bib level records is based on the mapping-rules residing in a FOLIO tenant. Read more on this in the Readme in the Source record manager Module repo.
This processing does not store the MARC records anywhere since this is not available in FOLIO yet. Only FOLIO Holdings records are created. MFHD-to-Inventory mapping also relies on mapping based on a similar JSON structure. This is not stored in the tenant and must be maintained by you. A template/example is available in migration_repo_template
For holdings mapping, you also need to map legacy locations to FOLIO locations. An example map file is available at migration_repo_template
Items-to-Inventory mapping is based on a json structure where the CSV headers are matched against the target fields in the FOLIO items. The mapping is limited currently, but will be built out as work progresses. For an example go to the migration_repo_template repository
For Item mapping, you also need to map legacy locations to FOLIO locations. An example map file is available at migration_repo_template
For Item mapping, you also need to map legacy item types to FOLIO equivalents. An example map file is available at migration_repo_template In order to set this up, you need to have a concept of how the FOLI circulation rules will look like.
For Item mapping, you also need to map legacy loan types to FOLIO equivalents. An example map file is available at migration_repo_template In order to set this up, you need to have a concept of how the FOLI circulation rules will look like.
There is a test suite for Bibs-to-Instance mapping.
- Install the packages in the Pipfile
- pipenv run python3 -m unittest test_rules_mapper.TestRulesMapper
For actual examples of the output, go to the migration_repo_template
pipenv run python3 main_bibs.py PATH_TO_FOLDER_WITH_MARC_FILES RESULTS_FOLDER OKAPI_URL TENANT_ID USERNAME PASSWORD RECORD_SOURCE_NAME
The above will fetch the mapping-rules from the FOLIO tenant specified and transform the supplied MARC21 record files into FOLIO Instance
Example:
pipenv run python ~/code/MARC21-To-FOLIO/main_bibs.py ~/code/migration_repo_template/example_files/data/bibs ~/code/migration_repo_template/example_files/results/ https://okapi-bugfest-honeysuckle.folio.ebsco.com fs090000
00 folio folio voyager
time will write out some statistics about time elapsed and memory usage. This is optional but useful.
pipenv run python instructs to run the script in a virtual environment using python.
~/code/MARC21-To-FOLIO/main_bibs.py is the script we want to run.
~/client_data/hogwartslibrary/goldenrod/data/bibs is the folder where we’ll fetch the file with MARC records to transform.
~/client_data/hogwartslibrary/goldenrod/results is the folder where we’ll place the result files (created instances and transformation report).
https://okapi-hogwartslibrary.folio.ebsco.com fs00000000 admin 'password' iii is a string containing the okapi URL, tenant ID, username and password for the tenant, as well as a code indicating the legacy ILS.
| tee ~/client_data/hogwartslibrary/goldenrod/results/instance_transformation.log prints the log that is printed in the terminal during the running and also prints it to a file. This is optional but useful.
For actual examples of the output, go to the migration_repo_template
pipenv run python3 /codez/MARC21-To-FOLIO/main_holdings.py ~/code/migration_repo_template/example_files/data/holdings ~/code/migration_repo_template/example_files/results https://okapi-bugfest-honeysuckle.folio.ebsco.com fs09000000 folio folio voyager -m ~/code/migration_repo_template/mapping_files
pipenv run python3 /codez/MARC21-To-FOLIO/main_items.py ~/code/migration_repo_template/example_files/data/items ~/code/migration_repo_template/example_files/results https://okapi-bugfest-honeysuckle.folio.ebsco.com fs09000000 folio folio -m ~/code/migration_repo_template/mapping_files
In order for SRS record loading to run, you need a snapshot object in the FOLIO database. The snapshot ID (jobExecutionId) is hard coded into the SRS records by the transformation scripts. To do this, do the following:
Make a POST request to your FOLIO tenant to this endpoint:
{{baseUrl}}/source-storage/snapshots
with the following payload:
{
"jobExecutionId": "67dfac11-1caf-4470-9ad1-d533f6360bdd",
"status": "PARSING_IN_PROGRESS",
"processingStartedDate": "2020-12-30T14:33:50.478+0000",
"metadata": {
"createdDate": "2020-12-30T14:33:50.478+0000",
"createdByUserId": "0280835d-b08d-4187-969d-9b4ecc247eae",
"updatedDate": "2020-12-30T14:33:50.478+0000",
"updatedByUserId": "0280835d-b08d-4187-969d-9b4ecc247eae"
}
}
Download the HRID handling settings from the tenant. If there are HRID handling in the mapping rules:
- The HRID is set on the Instance
- The 001 in the MARC21 record (bound for SRS) is replaced with this HRID.
If the mapping-rules specify no HRID handling or the field designated for HRID contains no value:
- The HRID is being constructed from the HRID settings
- Pad the number in the HRID Settings so length is 11
- The 001 in the MARC21 record (bound for SRS) is replaced with this HRID.