conff

Simple config parser with evaluator library.

https://travis-ci.com/kororo/conff.svg?branch=master

https://coveralls.io/repos/github/kororo/conff/badge.svg?branch=master

Why Another Config Parser Module?

This project inspired of the necessity complex config in a project. By means complex:

Reusability
- Import values from file
- Reference values from other object
Secure
- Encrypt/decrypt sensitive values
Flexible
- Make logical expression to derive values
Powerful
- Add custom functions in Python
- Link name data from Python

Important Notes

In Python 3.5, the dict data type has inconsistent ordering, it is STRONGLY recommended to use OrderedDict if you manually parse object. If you load from YAML file, the library already handled it.

Install

[sudo] pip install conff

Basic Usage

To get very basic parsing:

import conff
r = conff.parse({'math': '1 + 3'})
r = {'math': '4'}

load YAML file

import conff
r = conff.load('path_of_file.yaml')

import files

import conff
## y1.yml
# shared_conf: 1
## y2.yml
# conf: F.inc('y1.yml')

r = conff.load('y2.yml')
r = {'conf': {'shared_conf': 1}}

Examples

More advances examples:

Parse with simple expression

import conff
r = conff.parse('1 + 2')
r = 3

Parse object

import conff
r = conff.parse({"math": "1 + 2"})
r = {'math': 3}

Ignore expression (declare it as string)

import conff
r = conff.parse('"1 + 2"')
r = '1 + 2'

Parse error behaviours

import conff
errors = []
r = conff.parse({"math": "1 / 0"}, errors=errors)
r = {'math': '1 / 0'}
errors = [['1 / 0', ZeroDivisionError('division by zero',)]]

Parse with functions

import conff
def fn_add(a, b):
    return a + b
r = conff.parse('F.add(1, 2)', fns={'add': fn_add})
r = 3

Parse with names

import conff
r = conff.parse('a + b', names={'a': 1, 'b': 2})
r = 3

Parse with extends

import conff
data = {
    't1': {'a': 'a'},
    't2': {
        'F.extend': 'R.t1',
        'b': 'b'
    }
}
r = conff.parse(data)
r = {'t1': {'a': 'a'}, 't2': {'a': 'a', 'b': 'b'}}

Parse with updates

import conff
data = {
    't1': {'a': 'a'},
    't2': {
        'b': 'b',
        'F.update': {
            'c': 'c'
        },
    }
}
r = conff.parse(data)
r = {'t1': {'a': 'a'}, 't2': {'b': 'b', 'c': 'c'}}

Parse with extends and updates

import conff
data = {
    't1': {'a': 'a'},
    't2': {
        'F.extend': 'R.t1',
        'b': 'b',
        'F.update': {
            'a': 'A',
            'c': 'c'
        },
    }
}
r = conff.parse(data)
r = {'t1': {'a': 'a'}, 't2': {'a': 'A', 'b': 'b', 'c': 'c'}}

Create a list of Values

This creates a list of floats, similar to numpy.linspace

import conff
data = {'t2': 'F.linspace(0, 10, 5)'}
r = conff.parse(data)
r = {'t2': [0.0, 2.5, 5.0, 7.5, 10.0]}

This also creates a list of floats, but behaves like numpy.arange (although slightly different in that it is inclusive of the endpoint).

import conff
data = {'t2': 'F.arange(0, 10, 2)'}
r = conff.parse(data)
r = {'t2': [0, 2, 4, 6, 8, 10]}

Parse with for each

One can mimic the logic of a for loop with the following example

import conff
data = {'t1': 2,
        'F.foreach': {
            'values': 'F.linspace(0, 10, 2)',
            # You have access to loop.index, loop.value, and loop.length
            # within the template, as well as all the usual names
            'template': {
                 '"test%i"%loop.index': 'R.t1*loop.value',
                 'length': 'loop.length'
                 }
            }
       }
r = conff.parse(data)
r = {'length': 3, 't1': 2, 'test0': 0.0, 'test1': 10.0, 'test2': 20.0}

Encryption

This section to help you to quickly generate encryption key, initial encrypt values and test to decrypt the value.

import conff
# generate key, save it somewhere safe
names = {'R': {'_': {'etype': 'fernet'}}}
etype = conff.generate_key(names)()
# or just
ekey = conff.generate_key()('fernet')

# encrypt data
# BIG WARNING: this should be retrieved somewhere secured for example in ~/.secret
# below just for example purposes
ekey = 'FOb7DBRftamqsyRFIaP01q57ZLZZV6MVB2xg1Cg_E7g='
names = {'R': {'_': {'etype': 'fernet', 'ekey': ekey}}}
# gAAAAABbBBhOJDMoQSbF9jfNgt97FwyflQEZRxv2L2buv6YD_Jiq8XNrxv8VqFis__J7YlpZQA07nDvzYwMU562Mlm978uP9BQf6M9Priy3btidL6Pm406w=
encrypted_value = conff.encrypt(names)('ACCESSSECRETPLAIN1234')

# decrypt data
ekey = 'FOb7DBRftamqsyRFIaP01q57ZLZZV6MVB2xg1Cg_E7g='
names = {'R': {'_': {'etype': 'fernet', 'ekey': ekey}}}
encrypted_value = 'gAAAAABbBBhOJDMoQSbF9jfNgt97FwyflQEZRxv2L2buv6YD_Jiq8XNrxv8VqFis__J7YlpZQA07nDvzYwMU562Mlm978uP9BQf6M9Priy3btidL6Pm406w='
conff.decrypt(names)(encrypted_value)

Real World Examples

All the example below located in data directory. Imagine you start an important project, your code need to analyse image/videos which involves workflow with set of tasks with AWS Rekognition. The steps will be more/less like this:

Read images/videos from a specific folder, if images goes to (2), if videos goes to (3).

Analyse the images with AWS API, then goes (4)

Analyse the videos with AWS API, then goes (4)

Write the result back to JSON file, finished

The configuration required:

Read images/videos (where is the folder)

Analyse images (AWS API credential and max resolution for image)

Analyse videos (AWS API credential and max resolution for video)

Write results (where is the result should be written)

1. Without conff library

File: data/sample_config_01.yml

Where it is all started, if we require to store the configuration as per normally, it should be like this.

job:
  read_image:
    # R01
    root_path: /data/project/images_and_videos/
  analyse_image:
    # R02
    api_cred:
      region_name: ap-southeast-2
      aws_access_key_id: ACCESSKEY1234
      # R03
      aws_secret_access_key: ACCESSSECRETPLAIN1234
    max_res: [1024, 768]
  analyse_video:
    # R04
    api_cred:
      region_name: ap-southeast-2
      aws_access_key_id: ACCESSKEY1234
      aws_secret_access_key: ACCESSSECRETPLAIN1234
    max_res: [800, 600]
  write_result:
    # R05
    output_path: /data/project/result.json

import yaml
with open('data/sample_config_01.yml') as stream:
    r1 = yaml.safe_load(stream)

Notes:

R01: The subpath of "/data/project" is repeated between R01 and R05

R02: api_cred is repeatedly defined with R04

R03: the secret is plain visible, if this stored in GIT, it is pure disaster

2. Fix the repeat

File: data/sample_config_02.yml

Repeating values/configuration is bad, this could potentially cause human mistake if changes made is not consistently applied in all occurences.

# this can be any name, as long as not reserved in Python
shared:
  project_path: /data/project
  aws_cred:
    region_name: ap-southeast-2
    aws_access_key_id: ACCESSKEY1234
    # F03
    aws_secret_access_key: F.decrypt('gAAAAABbBBhOJDMoQSbF9jfNgt97FwyflQEZRxv2L2buv6YD_Jiq8XNrxv8VqFis__J7YlpZQA07nDvzYwMU562Mlm978uP9BQf6M9Priy3btidL6Pm406w=')

job:
  read_image:
    # F01
    root_path: R.shared.project_path + '/images_and_videos/'
  analyse_image:
    # F02
    api_cred: R.shared.aws_cred
    max_res: [1024, 768]
  analyse_video:
    # F04
    api_cred: R.shared.aws_cred
    max_res: [800, 600]
  write_result:
    # F05
    output_path: R.shared.project_path + '/result.json'

import conff
# ekey is the secured encryption key
# WARNING: this is just demonstration purposes
ekey = 'FOb7DBRftamqsyRFIaP01q57ZLZZV6MVB2xg1Cg_E7g='
r2 = conff.load(fs_path='data/sample_config_02.yml', params={'ekey': ekey})

Notes:

F01: it is safe if the prefix '/data/project' need to be changed, it will automatically changed for F05

F02: no more duplicated config with F04

F03: it is secured to save this to GIT, as long as the encryption key is stored securely somewhere in server such as ~/.secret

3. Optimise to the extreme

File: data/sample_config_03.yml

This is just demonstration purposes to see the full capabilities of this library.

# this can be any name, as long as not reserved in Python
shared:
  project_path: /data/project
  analyse_image_video:
    api_cred:
      region_name: ap-southeast-2
      aws_access_key_id: ACCESSKEY1234
      aws_secret_access_key: F.decrypt('gAAAAABbBBhOJDMoQSbF9jfNgt97FwyflQEZRxv2L2buv6YD_Jiq8XNrxv8VqFis__J7YlpZQA07nDvzYwMU562Mlm978uP9BQf6M9Priy3btidL6Pm406w=')
    max_res: [1024, 768]
job:
  read_image:
    root_path: R.shared.project_path + '/images_and_videos/'
  analyse_image: R.shared.analyse_image_video
  analyse_video:
    F.extend: R.shared.analyse_image_video
    F.update:
      max_res: [800, 600]
  write_result:
    output_path: R.shared.project_path + '/result.json'

For completeness, ensuring data is consistent and correct between sample_config_01.yml, sample_config_02.yml and sample_config_03.yml.

# nose2 conff.test.ConffTestCase.test_sample
fs_path = 'data/sample_config_01.yml'
with open(fs_path) as stream:
    r1 = yaml.safe_load(stream)
fs_path = 'data/sample_config_02.yml'
ekey = 'FOb7DBRftamqsyRFIaP01q57ZLZZV6MVB2xg1Cg_E7g='
r2 = conff.load(fs_path=fs_path, params={'ekey': ekey})
fs_path = 'data/sample_config_03.yml'
r3 = conff.load(fs_path=fs_path, params={'ekey': ekey})
self.assertDictEqual(r1['job'], r2['job'], 'Mismatch value')
self.assertDictEqual(r2['job'], r3['job'], 'Mismatch value')

Test

To test this project:

# default test
nose2

# test with coverage
nose2 --with-coverage

# test specific
nose2 conff.test.ConffTestCase.test_sample

TODO

[X] Setup basic necessity
- [X] Stop procrastinating
- [X] Project registration in pypi
- [X] Create unit tests
- [X] Setup travis
- [X] Setup coveralls
[ ] Add more support on Python versions
- [ ] 2.7
- [ ] 3.4
- [X] 3.5
- [X] 3.6
[ ] Features
- [X] Add more functions for encryption
- [ ] Test on multilanguage
- [ ] Add circular dependencies error
- [ ] Ensure this is good on production environment
- [X] Add options to give more flexibility
- [ ] Check safety on the evaluator, expose more of its options such as (MAX_STRING)
- [ ] Improve F.extend to allow list to be extended
- [ ] Allow conff to update existing config object
- [ ] Have more converter from/to excel, xml
- [ ] Feature to display warning and error messages (for example, missing parameters, logging purposes)
- [ ] Add options to extend structure in template (for example, https://github.com/defunkt/pystache), the idea is render the values in string, convert it to object and attach it back. This will also inherit lots of feature such as loop, decision blocks
- [ ] Separate current functions into built-in and restructure it (first break-changes to organise all the built-in functions into more better way)
[ ] Improve docs
- [ ] Add more code comments and visibilities
- [ ] Make github layout code into two left -> right
- [X] Put more examples
- [ ] Setup readthedocs
- [ ] Add code conduct, issue template into git project.

Other Open Source

This project uses other awesome projects:

Who uses conff?

Please send a PR to keep the list growing, if you may please add your handle and company.

kwrobert/conff