A repository for processing udot parcel images and extracting parcel numbers using machine learning. Currently, this tool processes pdf's and images looking for circles. These circles are extracted and tiled and stored to be run against the Google Cloud DocumentAI optical character recognition processor.
This project is organized to work with build pack and Google Cloud Run Jobs and to run the commands locally via a CLI.
To work with the CLI,
- Create a python environment and install the
requirements.dev.txt
into that environment - Execute the CLI to see the commands and options available
python row_cli.py
-
generate an index of all files
-
filter the index to remove non image files and deeds
python row_cli.py index filter ./data/elephant/remaining_index.txt
-
put the index in storage
-
run the job referencing the index location (edit the job name, file size, and task count)
-
generate another index from the resulting job
python row_cli.py storage generate-index --from=gs://ut-dts-agrc-udot-parcels-dev --prefix=elephant/mosaics/ --save-to=./data/elephant
-
use a logging sink to add files with 0 circles detected and query for the file names and add that to the index generated in the previous step to avoid double processing files.
-
generate a remaining index between the original and the prior
python row_cli.py storage generate-remaining-index --full-index=gs://ut-dts-agrc-udot-parcels-dev --processed-index=./data/elephant --save-to=./data/elephant
assuming the index in the bucket is the last remaining index for comparison
-
filter out the deeds which have no circles
python row_cli.py index filter ./data/elephant/remaining_index.txt
-
move the current index into the job and replace with the remaining index renamed as index.txt
-
repeat 4-9 until there are no more files left to process
-
Authentication for document ai job
-
activate your terminal as a service account
gcloud auth activate-service-account email@address --key-file=/path/to/sa.json
-
-
start the job
python row_cli.py process circles --job=elephant --from=gs://bucket--save-to=bucket --index=gs://bucket --task-index=0 --file-count=1 --instances=1 --project=1234 --processor=123abc