Overview ======= conf package is pure-tcl package for loading a textual configuration data from file, channels and strings. It supports conf syntax with hierarchical and ini-style sections and key-value pairs where value can be string or list. The features are: * conf syntax: hierarchical sections support * conf syntax: ini-style sections support * conf syntax: comments support * conf syntax: file inclusion support * callback can be used on every key-value assignment to change a default behaviour drastically(you can use a package only as a parser and process a parsed key-value as you want even without a creation of resulting dict with a parsed conf) * conf is parsed into 2 dicts: one is a parsed conf and another is a conf specification with info about value types. This is very important for working with parsed conf correctly. See Rationale section in the README. * there are utility routines to help with conf spec checking. conf package contains 3 main routines for text configuration file loading: * conf::load_from_file - to load a conf from a file * conf::load_from_fh - to load a conf from an open file handler * conf::load_from_str - to load a conf from a string And the next utility routines: * conf::get_key - get a specified key from a parsed conf * conf::escape_value - escape a specified value according to conf syntax rules * conf::spec_key_existence - return key existence state * conf::spec_cmp - compare a cspec against a reference cspec * conf::spec_key_get - get a value of a specified key Synopsis ======== load_from_file ?-hd STR? ?-default CAS? ?-path STR? ?-cb CB_AND_PRIV? FILE_NAME load_from_fh ?-hd STR? ?-default CAS? ?-path STR? ?-cb CB_AND_PRIV? CHAN load_from_str ?-hd STR? ?-default CAS? ?-path STR? ?-cb CB_AND_PRIV? ?-s START_INDEX? ?-e END_INDEX? CONF_STR get_key CAS NAME ?TYPE? escape_value VALUE spec_key_existence CSPEC_OR_PATTERN NAMES ?_OUT? spec_cmp CSPEC_OR_PATTERN CSPEC spec_key_get CSPEC_OR_PATTERN NAMES Description =========== load_from_file command parse a specified file and returns a result as a list. Where: - element #1 is a dict with a parsed conf - element #2 is a dict with a conf specification for a parsed conf - element #3 (exists only if -cb is used) is callback private data This command accepts the next options: -hd STR use STR as hierarchy delimiter in key names and group names -default CAS use CAS(list with 2 elements: parsed conf and its spec) as an initial(default) conf. -path STR use STR as file path prefix for every included file -cb CB_AND_PRIV CB_AND_PRIV is a list: {CALLBACK PRIV}. Use specified CALLBACK proc as value set/append callback; and specified PRIV will be used as initial value for priv in a context. load_from_fh command parse a data from a specified file handle(chan) and returns the same value as load_from_file. Options are the same as for load_from_file. load_from_str command parse a data from a specified string and returns the same value as load_from_file. Options are the same as for load_from_file with addition of: -s START_INDEX start parsing of a string from specified position(in chars) -e END_INDEX end parsing of a string at specified position(in chars); this will be the last character we read get_key command find a specified key in a specified CAS(conf and its spec) and returns a value of this key. CAS is a value returned from any of load proc. Key is specified by list with names(e.g. {s1 s2 k1} for key k1 in section s2 which resides in section s1). If a specified key isn't found, then error is generated. Optionally, a key type(S for string or L for list) can be specified; in this case, if a specified key has a different type, then error is generated. escape_value command returns an escaped form(according to conf syntax rules) of specified value. spec_key_existence command tries to find a specified key(NAMES) in a specified conf spec or conf spec pattern(CSPEC) and returns a result(key existence state). In addition, for some ret codes an additional data saved in a specified variable(_OUT - a name of a var for this data). This command returns: -2 - specified key doesn't exists and some predecessor key is a leaf key; $_out var contains existent part of key path(a part of names list) -1 - specified key doesn't exists 0 - specified key exists and it is a leaf key $_out var contains a value of a specified key 1 - specified key exists, but it is not a leaf key spec_cmp command compares a specified cspec(CSPEC) against another cspec or a cspec pattern(PATTERN) and returns a list with mismatch entries. Mismatch entry is a list with 2 elements, where 1 item is a mismatch type and 2 item is full path to a key: {M KEY_PATH} - miss, KEY_PATH from pattern is missed from cspec {T KEY_PATH} - wrong type, KEY_PATH from pattern is different type than in cspec {E KEY_PATH} - excess, KEY_PATH from cspec is missed from pattern spec_key_get command tries to find a specified key(NAMES) in a specified conf spec or conf spec pattern and returns its value. If a specified key isn't found, then spec_key_get returns "-"; if a specified key is a section, then spec_key_get returns "sect". This command mostly useful as utility proc for assisting in making an error messages for user. Conf syntax =========== Conf consists from key-value pairs, which can be grouped in sections. Whitespaces are ignored. Comment can be started by # symbols at almost any place(excluding # inside a quoted string) and lasts until the end of the line. Key, value and section name can consist of any chars exclude any space chars, '=', '#', '"', '[', ']', '{', '}', '+', '?'. If a key, value or section name contains any of these characters, then the entire key, value or section name must be enclosed in double quotes. In this case, if double quotes appears inside the value, it must be escaped with a backslash(\). Value can be a string(quoted or not) or a list. A list is started with '[' and is ended with ']'. List elements are separated with spaces. Assignment ---------- Assignment a string to a key: k1 = some_word k2 = "some string \"with\" spaces" Assignment a list to a key: k1 = [v1 v2 v3] k2 = [v1 "some string \"with\" spaces" v2] Assignment a string/list to a key only if this key isn't defined yet: k1 ?= some_word k2 ?= [v1 v2] Append a string/list to a key value(if previous value is a string, then it is converted to a list where the previous value becomes the only element): k1 = v1 k1 += some_word k1 += [v2 v3] the same as: k1 = [v1 some_word v2 v3] File including -------------- Conf file can include another conf files with syntax: < FILENAME Where FILENAME is glob pattern. Sections -------- A section can be defined in two ways. With [] syntax: [SECT_NAME] Or with {} syntax: SECT_NAME { } If a section is defined with `[SECT_NAME]` syntax, then current section prefix(top of a section prefixes stack) is replaced with SECT_NAME. If a section is defined with `SECT_NAME {` syntax, then SECT_NAME section prefix is pushed to a section prefixes stack and becomes a current section prefix until we reach corresponding "}", in which case SECT_NAME is poped from a section prefixes stack and previous value becomes a top of a stack and a current section prefix. In other words, []-sections is not stacked and is replaced by both []-section and {}-section that come after it; {}-sections is stacked and both []-section and {}-section coming after is appended to it. E.g. if we call any of load proc with "-hd .", then this confs are equal: variant 1: sect1.sect2.key1 = val1 sect1.sect2.key2 = val2 sect1.sect2.sect3.key3 = val3 sect1.sect2.key4 = val4 variant 2: [sect1.sect2] key1 = val1 key2 = val2 sect3.key3 = val3 key4 = val4 variant 3: [sect1.sect2] key1 = val1 key2 = val2 [sect1.sect2.sect3] key3 = val3 [sect1.sect2] key4 = val4 variant 4: sect1.sect2 { key1 = val1 key2 = val2 sect3 { key3 = val3 } key4 = val4 } Conf syntax ABNF ================ conf = *(key-value / section / include-stmt / WSP0) key-value = key ("=" / "+=" / "?=") (value / list) key = word / str value = word / str list = WSP0 "[" *((word / str / list) WSP) "]" section = ("[" (word / str) "]" *conf) / ((word / str) "{" *conf "}") include-stmt = "<" WSP0 (word / str) word = WSP0 (%d33 / %d36-60 / %d62-90 / %d92 / %d94-122 / %d124 / %d126 - %d255) WSP0 ; all except any space (belong to [:space:] char class), =, #, ", ; +, ?, [, ], {, } str = WSP0 DQUOTE %d01-%d255 DQUOTE WSP0 ; any chars in double quotes, where double quotes inside string ; can be escaped with \ char WSP0 = *WSP WSP = SP / HTAB / CR / LF / CRLF / AND_ANY_UNICODE_WHITESPACE_CHAR Callback ======== If -cb option is specified for any of load proc, then a specified callback proc is called on every parsed key and value from a conf. Callback must accept the next parameters in order: - ctx var name - conf var name - operation (=S, =L, +=S, +=L, ?=S or ?=L) - var name with key full name(list with sect names and a key name) - value var name Where operation is "=S" for assigning a string, "=L" for assigning a list, "+=S" for adding a string, "+=L" for adding a list, "?=S" for assigning a string if no such key yet, "?=L" for assigning a list if no such key yet. Callback can change a key name and/or a key value. In this case, new key and/or value will be saved in a parsed conf dict. Callback must ret value type(S or L) or "" if value shouldn't be saved(we don't want it or we already saved it ourselves in a callback). In the last case, a value type info also isn't saved(into cspec). Conf specification ================== A configuration specification("conf spec" or "cspec" to be short) is a data structure with data types info for parsed keys/values. cspec is returned from load_* proc together with a parsed conf. cspec and parsed conf are dicts. cspec purpose is to checking values types and key existence(the single conf by itself is insufficient for this - see Rationale). cspec contains the same keys as a conf. But values can be only "S" or "L". Where "S" means a string value and "L" means a list value. Thus, if your cspec dict has "S" value for some key, then corresponding parsed conf dict has a string value for the same key. E.g. for this conf, parsed with "-hd .": sect1.sect2 { key1 = val1 key2 = val2 sect3 { key3 = val3 } key4 = val4 } The parsed conf will be the dict: sect1 {sect2 {key1 val1 key2 val2 sect3 {key3 val3} key4 val4}} And the cspec will be the dict: sect1 {sect2 {key1 S key2 S sect3 {key3 S} key4 S}} To simplify a work with cspec several routines exists: spec_key_existence, spec_cmp, spec_get_key. Conf specification pattern ========================== A configuration specification pattern("conf spec pattern" or "cspec pattern" to be short) is a data structure with data types info for parsed keys/values. When data type info can be actual data type or a data type pattern. cspec isn't returned from any proc - it's created by hands. cspec pattern purpose is to comparing cspec against it. This is simplify a procedure of checking values types and key existence after we parse a conf. spec_cmp proc is the main consumer of cspec pattern. The all syntax rules of cspec is also applies to cspec pattern with one additions: value can be not only "S" and "L", but also "A", "c", "C". Where: - "A" mean "S" or "L" - "c" mean any conf hierarchy - "C" mean any conf hierarchy or "S" or "L" Rationale ========= Due to shimmering in tcl we can't reliably determine the type of a value in a parsed conf. For example, we expect the next conf: n1 = VALUE But a user mistakenly(due to copy-pasting or something else) made n1 a section: n1 { n2 = VALUE } After parsing with any of load proc we will have the next parsed conf: n1 {n2 VALUE} Which from our perspective has key n1 with value "n2 VALUE". If we do no acrobatics with ::tcl::unsupported::representation (it is unsupported, isn't it :-)?), then we can't differentiate these 2 cases. Due to implicit type convertion(shimmering) we can simplify a situation with data types to "our values without data types". Thus, we have a some conf with data types(section vs string vs list) and we have a dict with a parsed conf where no data types info attached to values. Therefore, in order not to lose data type info, we should parse a conf into 2 data structures: one is a parsed conf(with keys and values), another is a struct with data types info for parsed keys/values. The last structure is called conf specification(or cspec for short) in this package. Thus, any load_* proc return a list with these 2 structures after work: a dict with a parsed conf and a dict with a conf spec. And workflow with some conf is like the next: - read a conf - compare a conf spec against a conf spec pattern - show some errors about mismatches - if no error, work with parsed conf as with usual dict Examples ======== Simple conf ----------- k1=v1 k2="val with many lines" k3 = "val with \" inside" k4 = [ [many values] [inside list] [for one key]] loaded with `load_from_file my.conf` give a conf: k1 v1 k2 {val with many lines} k3 {val with " inside} k4 {{many values} {inside list} {for one key}} Conf with sections ------------------ k1 = v1 [g1] k2 = v2 [g2] k3 = v3 loaded with `load_from_file my.conf` give a conf: k1 v1 g1 {k2 v2} g2 {k3 v3} Conf with nested sections ------------------------- k1 = v1 g1 { k2 = v2 g2 { k3 = v3 } } loaded with `load_from_file my.conf` give a conf: k1 v1 g1 {k2 v2 g2 {k3 v3}} Conf with hierarchy delimiter specified --------------------------------------- g1.k1 = {1 2 3 {4 5}} g2.k2 = v2 loaded with `load_from_file -hd . my.conf` give a conf: g1 {k1 {1 2 3 {4 5}}} g2 {k2 v2} Conf monster ------------ g1.k1 = [1 2 3 [4 5]] [g2] k2 = v2 k3.k4 = v3 [g2.g3] k4 = v4 g4.g5 { k5.k6 = v6 [g7.g8] k7.k8 = v7 } loaded with `load_from_file -hd . my.conf` give a conf: g1 {k1 {1 2 3 {4 5}}} g2 {k2 v2 k3 {k4 v3} g3 {k4 v4}} g4 {g5 {k5 {k6 v6} g7 {g8 {k7 {k8 v7}}}}} Conf with comments ------------------ # here is some comments k = v k1 = v1 # another comments k2 = v2 k3 = # this is k3 v3 # this is k3 value k4 = "v4 with # inside" # end of file loaded with `load_from_file -hd . my.conf` give a conf: k v k1 v1 k2 v2 k3 v3 k4 {v4 with # inside} See ex*.tcl for examples of code. Errors ====== conf error at m.conf:1: unexpected token sequence: "'k2'#6(L1) 'v2'#6(L1)". Want: KEY = VAL or KEY = [ or GROUP_NAME { or } or [GROUP_NAME] There is parser error for token sequence started at line 1. First token is 'k2' with token code 6 at line 1(L1), second token is 'v2' with tokne code 6 at line 1. E.g.: k2 v2 conf error at m.conf:1: Possible unclosed quotes at line 1. unexpected token sequence: "'k2'#6(L1) '='#1(L1)". Want: KEY = VAL or KEY = [ or GROUP_NAME { or } or [GROUP_NAME] Double quotes is opened, but is not closed. E.g.: k2 = "v2 Syntax error in cspec pattern for key 'k1': value is 'W' If you call spec_cmp, then there is probably a misspelled key type in cspec pattern. E.g.: spec_cmp {k1 W} {k1 {k2 S}} Syntax error in cspec for key 'k1': value is 'W' If you call spec_cmp, then there is probably a misspelled key type in cspec. E.g.: spec_cmp {k1 {k2 S}} {k1 W}