cic1988/odata-edc-scanner

Odata scanner for EDC

Custom Scanner: odata-edc-scanner

Odata scanner for EDC

Notice

The main branch is since 06.06.2022 set to v2 branch as v2 is the main supported Odata version.

Overview

✅ (supported) 🚧 (on-going) 🔜 (plan) 🚫 (backlog)

• V2 Objects ✅ (see v2 and main branch)

  • ✅ basic authentication
  • ✅ entity type
  • ✅ key
  • ✅ property
  • ✅ navigation property
  • ✅ association + navigation property
  • ✅ function import
  • ✅ custom attributes

• V3 Objects 🚫

• V4 Objects 🚧 (see v4 branch)

  • 🚫 basic authentication
  • ✅ entity type
  • ✅ key
  • ✅ property
  • ✅ navigation property
  • 🚫 association + navigation property
  • 🚫 function import
  • 🚫 custom attriburtes

• Feature

  • ✅ metadata
  • ✅ lineage
  • ✅ profiling (configurable + filter) + data domain discovery
  • ⚠️ currently only basic authentication supported by v2 or main branch

• Screenshot (example of https://services.odata.org/V2/Northwind/Northwind.svc)

Screenshot 1	Screenshot 2

Usage

Import model

1. Download the model.zip file from the latest release: https://github.com/cic1988/odata-edc-scanner/releases
2. Import this model in EDC admin panel

3. Create a custom resource with imported data model

4. Use this newly create resource to setup scanner

Initial setup

Option One: if running the build binary:

1. Download the binary files from the latest release (currently CentOS/Redhat and OSX are supported): https://github.com/cic1988/odata-edc-scanner/releases
2. Unpack the zip file in a target folder
3. chmod u+x odata-edc-scanner
4. cp config.ini.sample config.ini
5. Edit the config.ini with required information (see comment in the file)
6. Run the command ./odata-edc-scanner -f

Option Two: if running the python source code:

1. Clone this repository via git clone https://github.com/cic1988/odata-edc-scanner.git
2. (optional) yum install python3-devel -y
3. cd odata-edc-scanner
4. python3 -m venv .
5. source bin/activate
6. pip install --upgrade pip
7. pip install -r requirements.txt
8. cp config.ini.sample config.ini
9. Edit the config.ini with required information (see comment in the file)
10. chmod u+x main.py

Generate the needed files manually (if you want to run scanner off-EDC):

1. Run ./main.py -f
2. Upload the metadata.zip to the scanner configuration
3. Upload the ProfileableClassTypes.csv to the field Classes enabled for Profiling
4. (optional - when profiling enabled) Point the SFTP location and login to the folder, where the script has created the files

Example configuration in EDC (if you want to automate scanning in EDC):

Step	Screenshot
General setup. for example /home/infa/custom_scanner/odata/odata-edc-scanner/ is where you have cloned the repository. ODATASAMPLE is the folder specified in the config.ini file
Upload the generated ProfileableClassTypes.csv file
SFTP configuration for the generated profiling files

Other standard scanner configurations (data domains, etc.) are similar here. Then run the job.