Library to aid in organizing, running, and debugging regular expressions against large bodies of text.
The goal of this library is to simplify the deployment of regular expression on large bodies of text, in a variety of input formats.
To get a local copy up and running follow these simple steps.
- Python 3.8+
- runrex package: https://github.com/kpwhri/runrex
- Clone the repo
git clone https://github.com/kpwhri/runrex.git
- Install requirements (
requirements-dev
is for test packages)pip install -r requirements.txt -r requirements-dev.txt
- If you wish to read text from SAS or SQL, you will need to install additional requirements. These additional requirements files may be of use:
- ODBC-connection:
requirements-db.txt
- Postgres:
requirements-psql.txt
- SAS:
requirements-sas.txt
- ODBC-connection:
- Run tests.
set/export PYTHONPATH=src pytest tests
- Create 4 files:
patterns.py
: defines regular expressions of interest- See
examples/example_patterns.py
for some examples
- See
test_patterns.py
: tests for those regular expressions- Why? Make sure the patterns do what you think they do
algorithm.py
: defines algorithm (how to use regular expressions); returns a Result- See
examples/example_algorithm.py
for guidance
- See
config.(py|json|yaml)
: various configurations defined inschema.py
- See example in
examples/example_config.py
for basic config
- See example in
Accepts a variety of input formats, but will need to at least specify a document_id
and document_text
. The names are configurable.
By default, the input document text is expected to have each sentence on a separate line. If a sentence splitting scheme is desired, it will need to be supplied to the application.
For more details, see the example config or consult the schema
- Recommended output format is
jsonl
- The data can be extracted using python:
import json
with open('output.jsonl') as fh:
for line in fh:
data = json.loads(line) # data is dict
-
Output variables are configurable and can include:
- id: unique id for line
- name: document name
- algorithm: name of algorithm with finding
- value
- category: name of category (usually the pattern; multiple categories contribute to an algorithm)
- date
- extras
- matches: pattern matches
- text: captured text
- start: start index/offset of match
- end: end index/offset of match
-
Scripts to accomplish useful tasks with the output are included in the
scripts
directory.
Uses SEMVER.
See https://github.com/kpwhri/runrex/releases.
See the open issues for a list of proposed features (and known issues).
Any contributions you make are greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature
) - Commit your Changes (
git commit -m 'Add some AmazingFeature'
) - Push to the Branch (
git push origin feature/AmazingFeature
) - Open a Pull Request
Distributed under the MIT License.
See LICENSE
or https://kpwhri.mit-license.org for more information.
Please use the issue tracker.