Slapd-viper Database Backend for OpenLDAP

slapd-viper is a flexible database backend for OpenLDAP. It has been in use since 2008.

It implements all the usual LDAP operations, as well as provides elaborate features for responding with dynamic and auto-computed data on a level not seen in traditional LDAP installations.

Quick Start

Install system dependencies:

apt install slapd libfile-find-rule-perl libnet-ldap-perl libtext-csv-xs-perl liblist-moreutils-perl

Obtain and run Viper with an example config file that provides suffix dc=example,dc=com:

cd /etc/ldap
git clone https://github.com/crystallabs/slapd-viper

/usr/sbin/slapd -d 256 -f ./slapd-viper/etc/ldap/slapd.conf

While the server is running, load example data into the tree:

ldapadd -x -D cn=admin,dc=example,dc=com -w nevairbe -c -f slapd-viper/ldifs/example.ldif

And query the added data:

ldapsearch -x -b dc=example,dc=com

That's it for the most basic example.

Overview

slapd-viper is a custom backend for OpenLDAP built on top of OpenLDAP's slapd-perl.

It stores the directory tree in files and directories on disk, by default under /var/lib/ldap/viper/.

It can be used as a normal backend — reading and writing data is just the same as for any backend. However, its primary use is for LDAP trees which contain highly dynamic data and where elaborate and configurable server-side behavior is needed.

Dynamic functions possible with slapd-viper are:

ADD operation can overwrite existing entries without throwing an error
ADD operation can ignore adds for entries which already exist without throwing an error
DN of new entries can be modified (entries can be "relocated") before they are saved
When adding Debconf templates, a prompt can be opened on a remote machine, asking the admin for Debconf answer
When MODIFYing entries, no-ops can be detected and no changes made
When MODIFYing (and in combination with other functionality), Copy on Write is supported
DELETE can be allowed to delete whole subtrees if entries to delete are non-leaves
SEARCH queries can be arbitrarily rewritten
SEARCH can fallback to multiple other locations if no results are found
Attribute values can expand to content of disk files
Attribute values can expand to value of other entries and attributes
Attribute values can expand to value of another SEARCH
Attribute values can evaluate Perl code
Returned entries can be appened with additional attributes or values in various ways

Configuration Reference

Each database backed by Viper should be configured with something similar to the following block in slapd.conf:

database           perl
suffix             "dc=example,dc=com"
perlModulePath     "/etc/ldap/slapd-viper"
perlModule         "Viper"

directory          "/var/lib/ldap/viper/"
treesuffix         "dc=example,dc=com"

The first four lines are required by slapd and its back-perl backend to configure the suffix and initialize Viper.

The last two lines are required by the Viper backend. The value of treesufix must be equal to suffix. This small duplication cannot be avoided because suffix directive is consumed by slapd and is not passed onto our backend.

After the above standard lines, the following directives can be used:

Configuration Directives

The list is split into simple and complex directives, simple first.

Within each group, the list is sorted alphabetically, with each heading specifying configuration directive name and its usage syntax.

Where applicable, the first value listed indicates the default value.

Simple Configuration Directives

addIgnoredups 0|1

Specify whether LDAP ADD operation should ignore adds on existing entries without throwing LDAP_ALREADY_EXISTS error. Applicable if addoverwrites is 0.

addOverwrites 0|1

Specify whether LDAP ADD operation should overwrite existing entries without throwing LDAP_ALREADY_EXISTS error.

cacheRead SPEC

Specify for how many LDAP operations to cache LDIF reads from disk. No specification implies no cache.

SPEC can be:

X    - X LDAP operations

Overall best value, which minimizes risk of serving stale data while reaching noticeable optimization improvement (up to 25%), is 1 operation, specified as 1.

cacheRead and overlayConfig cache can be used together, amplifying the effect.

clean

Invoke removal of all saved stack files from disk.

deleteTrees 0|1

Specify whether Viper should allow deleting non-leaf elements (deleting the entry and everything under it in one go).

See notes for DELETE under "LDAP Operations - Notes".

enableBind 0|1

Whether to allow binding as entries under this suffix. By default this is disabled.

See notes for BIND under "LDAP Operations - Notes".

extension .ldif

Specify file extension to use when storing server data on disk.

Viper's data is kept in a directory tree that corresponds to the LDAP tree, where DN components are directories, and leaf nodes are files. Each file contains one LDAP entry in LDIF format.

File extension must be specified to make directories distinguishable from files, and the default value should rarely be changed.

load FILE [PATTERN REPLACEMENT ...]

Load and process configuration stack from FILE. FILE is always relative to suffix base directory.

If list of PATTERN REPLACEMENTS is specified, the s/PATTERN/REPLACEMENT/ substitution is done on all loaded lines before sending them to the config processor.

Example: load default_opts

message TEXT

Print TEXT to the log. The log will be a console if slapd is started with option -d (such as -d 256) to run in the foreground.

Example: message Test

modifyCopyOnWrite 1|0

When a MODIFY request is issued for an entry that does not really exist (i.e. it comes from a fallback), specify whether Viper should copy the entry to the expected location and then modify it, or return LDAP_NO_SUCH_OBJECT.

modifySmarts 1|0

Specify whether Viper should ignore MODIFY requests that do not result in any real change within the entry.

This is useful to enable to detect "no-op" modifications and avoid writing to disk, preserving meaningful modification timestamps on existing entries.

parse 1|0

Specify whether in the lines that follow, variable and directive expansion should be performed.

This includes expanding ${variable} to variable values and %{directive} to configuration directive values.

reset

Reset current stack in memory to empty list.

save FILE

Save current stack to FILE, always relative to suffix base directory.

Example: save default_opts

schemaFatal 0|1

Specify whether a missing or inaccessible schemaLDIF file should trigger a fatal error.

It is vital for Viper to be aware of server's schema (which comes from the schemaLDIF config option). The server won't work optimally if the schema file in LDIF format is missing, or is not up to date with the server's schema.

However, we issue a warning and allow startup without it, because you are then expected to use scripts/schema.pl to connect to the server right away and obtain the schema in LDIF format, saving it to the expected location. Then, restart the server to pick it up.

This must be done because the server's schema is not available to slapd-perl.

schemaFatal value should be set to 1 when you are sure you do have the schema.ldif file in the correct location.

schemaLDIF FILE

Specify location of server's schema in a single file, in LDIF format.

See the previous option for more info.

Note that the schema in LDIF format does not eliminate the need to have the real schema files in /etc/ldap/schema/*.schema. Schema files are read by slapd, and schema LDIF file is read by Viper. LDIF is created on the basis of real schema files, and at all times, Viper's LDIF schema should be in sync with slapd.

Example: schemaLDIF /etc/ldap/schema/schema.ldif

treeSuffix SUFFIX

This value should always match the value of suffix configured for the database.

This small duplication cannot be avoided because suffix directive is consumed by slapd and is not passed onto our backend.

var VARIABLE "VALUE STRING"

Assign "VALUE STRING" to variable VARIABLE. Variables, in this context, are visible only within the suffix where they are defined, and their value is expanded with ${variable} if option "parse" is enabled.

Complex Configuration Directives

addPrompt

addRelocate

entryAppend ATTRIBUTE PATTERN ... -> attr ATTRIBUTE [ATTRATTR [ATTR...]]

entryAppend ATTRIBUTE PATTERN ... -> append PATTERN REPLACEMENT [ATTR...]

Specify an entryAppend rule, allowing adding extra attributes into an entry before returning it to the client.

When all ATTRIBUTE-PATTERN pairs match, Viper looks to append the entry with a set of default attributes.

The entry from which to import the attributes can be specified in two ways:

With "attr ATTRIBUTE" (usually "attr seeAlso"). In that case, the attribute seeAlso is looked up in the current entry. It is expected to contain the DN of the entry whose attributes should append the current entry.

If ATTRATTR and ATTRs are unspecified, the entry is appended with all allowed attributes. Otherwise, it is appended only with attributes listed in the ATTRATTR attribute within the entry and/or in the literal list of ATTRs.

With "append PATTERN REPLACEMENT", where s/PATTERN/REPLACEMENT/ is performed on the original DN, and the result is used as the entry from which to pull the extra attributes.

With the 'append' method, there is no ATTRATTR field, so you cannot append the entry with the values of attributes listed in the entry, but you do have the option of specifying ATTRs to append with.

If left unspecified, the entry is appended with all allowed attributes.

Examples from production config:

entryAppend  objectClass "^dhcpHost$"                      \
             ->                                            \
             append .+ cn=dhcpHost,ou=objectClasses,ou=defaults

entryAppend  objectClass "^dhcpSubnet$"                    \
             ->                                            \
             append .+ cn=dhcpSubnet,ou=objectClasses,ou=defaults

entryAppend  dn          "^cn=default,ou=networks"         \
             objectClass "^ipNetwork$"                     \
             ->                                            \
             attr seeAlso

exp MATCH_REGEX NON_MATCH_REGEX

Specify regexes that each entry DN must and must not match respectively, to have overlay "exp" run on its attributes.

The "exp" overlay enables expansion into values of other attributes, in the current or other entry.

Example which always matches, and so enables the exp overlay: exp . ^$

file MATCH_REGEX NON_MATCH_REGEX

Specify regexes that each entry DN must and must not match respectively, to have overlay "file" run on its attributes.

The "file" overlay enables expansion into values of on-disk files, always relative to the suffix base directory.

Example: file . ^$

find MATCH_REGEX NON_MATCH_REGEX

Specify regexes that each entry DN must and must not match respectively, to have overlay "find" run on its attributes.

The "find" overlay enables internal re-invocation of the search function, and using the values retrieved in constructing the original value.

This overlay shares many similarities with "exp", but contains a crucial difference -- with "exp", you generally know where the entry and attribute to expand to are located. With "find", you generally don't, so you perform a search to find them.

Example: find . ^$

loadDump FILE

THIS IS A DEBUG OPTION .

Load direct Perl Storable dump of configuration hash from FILE, always relative to the suffix base directory.

This is an advanced option that should not be called from slapd.conf.

It is intended for scenarios where Viper is at least once initialized by slapd (and configured via slapd.conf), and config then dumped as Storable object using saveDump.

After that, you can run Viper "standalone", directly under the Perl interpreter using scripts/viper.pl, and instead of re-parsing slapd.conf for configuration, simply send "loadDump FILE" to the config processor, to load the exact state as had by slapd/Viper.

This is almost always needed only when you want to run Viper under the Perl interpreter directly, to specify Perl debug or profiling options.

overlayConfig OVERLAY OPTION VALUE ...

Specify default overlay options.

OVERLAY can be an overlay name (perl, exp, file, find) or "default".

OPTION can be "cache", "prefix" or "if".

cache SPEC - specify cache expiry time.

Caching overlay results improves performance enormously in situations
where multiple entries are returned and all produce the same dynamic
values for certain attributes.

In such cases, operations of complexity O(n) are reduced to O(1) level.

Syntax is the same as listed under "cacheRead", and 1o is again the
overall best setting.

NOTE: due to deficiencies in Memoize::Expire module, time- and
uses-based methods of expiry do not work correctly when caching non-scalar
values (such as multiple values for an attribute). It is therefore suggested
to always use the number-of-operations cache (like 1o).

Example: cache 1o

prefix PREFIX - generic prefix option, used where applicable. Currently
only the "file" overlay honors it, where it is a prefix to prepend on
all file specifications.

Directory separator is not added automatically,
so to prefix with a directory, include "/" at the end.

Example: prefix subdir/

perl MATCH_REGEX NON_MATCH_REGEX

Specify regexes that each entry DN must and must not match respectively, to have overlay "perl" run on its attributes.

By default, Perl overlay is disabled as it is in fact an interface for "eval", and is considered dangerous. To activate it, open Viper.pm and enable constant PERLEVAL.

Example: perl . ^$

saveDump FILE

THIS IS A DEBUG OPTION .

Save direct Perl Storable dump of configuration hash to FILE, always relative to the suffix base directory.

This is an advanced option that should usually be called as the last line of slapd.conf configuration for a particular suffix.

This is almost always needed only when you want to run Viper under the Perl interpreter directly, to specify Perl debug or profiling options.

searchFallback PATTERN REPLACEMENT

Specify search fallback rule, effectively implementing default entries.

When a specific search base is requested, and it does not exist in the searched location, it is possible to fallback to a chain of default entries. The first entry found wins.

Examples: production examples defaulting to site-wide and global defaults

# Fallback 1: site defaults tree.
searchFallback  cn=.[^,\\s]+,ou=hosts         ou=hosts,ou=defaults
searchFallback  cn=.[^,\\s]+,ou=templates     ou=templates,ou=defaults

# Fallback 2: global defaults tree.
searchFallback  cn=.[^,\\s]+,ou=hosts,.+      ou=hosts,ou=defaults
searchFallback  cn=.[^,\\s]+,ou=templates,.+  ou=templates,ou=defaults

searchSubst KEY PATTERN ... -> KEY PATTERN REPLACEMENT ...

Specify searchSubst rule, allowing rewrite of any part of the search request.

When the incoming search request matches all KEY PATTERN pairs, Viper performs the specified KEY=~ s/PATTERN/REPLACEMENT/ actions to rewrite the incoming search.

Search rewriting is completely free-form, and it is possible to rewrite searches to a completely different Viper suffix, as long as both are located in the same base directory.

This is a legitimate feature of the rewrite model, and is officially used to rewrite incoming DHCP search queries under ou=dhcp to appropriate places and with appropriate options under ou=clients.

KEY can be one of base, scope, deref, size, time, filter, attrOnly. Rewriting one last element of a search, the list of attributes to return, is currently not possible, but the feature is on the way.

Examples: production examples used in rewriting ou=dhcp to ou=clients

Example 1:

# Solve lack of flexibility in ISC DHCP3 LDAP patch by
# plainly specifying ldap-base-dn "ou=dhcp" in DHCP's
# config, and then here, rewriting DHCP ethernet address
# lookup to the ou=clients tree under which all clients
# are defined.

searchSubst  base        "^ou=dhcp$"                       \
             filter      "^\\(&\\(objectClass=dhcpHost\\)\\(dhcpHWAddress=ethernet [\\dabcdef:]+\\)\\)$" \
             -&gt;                                            \
             base   .+   ou=clients


Example 2:

# Solve lack of flexibility in ISC DHCP3 LDAP patch by
# rewriting a search in any shared network, tree
# ou=dhcp, to a proper location,

searchSubst  base        "^ou=\\w+,ou=dhcp$"                \
             scope       "^1$"                             \
             filter      "^\\(objectClass=\\*\\)$"         \
             -&gt;                                            \
             base   .+   "ou=clients"                      \
             filter .+   "(&amp;(objectClass=dhcpSubnet)(!(cn=default)))" \
             scope  .+   2

LDAP Operations - Notes

BIND operation is supported, but only in a trivial way. Therefore, binding using a DN under this part of DIT is not encouraged nor enabled by default unless enable_bind 1 is present in the config file. It is expected that users should authenticate using some other suffixes or via e.g. GSSAPI, so that this bind() is never called or needed.
DELETE operation is supported, but does not receive an indication from OpenLDAP whether subtree delete was requested or not. Therefore, currently the way to control whether a DELETE will delete whole subtrees or refuse to work on non-leaf values is controlled using the config option deleteTrees.

docelic/slapd-viper

Slapd-viper Database Backend for OpenLDAP

Quick Start

Overview

Configuration Reference

Configuration Directives

Simple Configuration Directives

addIgnoredups 0|1

addOverwrites 0|1

cacheRead SPEC

clean

deleteTrees 0|1

enableBind 0|1

extension .ldif

load FILE [PATTERN REPLACEMENT ...]

message TEXT

modifyCopyOnWrite 1|0

modifySmarts 1|0

parse 1|0

reset

save FILE

schemaFatal 0|1

schemaLDIF FILE

treeSuffix SUFFIX

var VARIABLE "VALUE STRING"

Complex Configuration Directives

addPrompt

addRelocate

entryAppend ATTRIBUTE PATTERN ... -> attr ATTRIBUTE [ATTRATTR [ATTR...]]

entryAppend ATTRIBUTE PATTERN ... -> append PATTERN REPLACEMENT [ATTR...]

exp MATCH_REGEX NON_MATCH_REGEX

file MATCH_REGEX NON_MATCH_REGEX

find MATCH_REGEX NON_MATCH_REGEX

loadDump FILE

overlayConfig OVERLAY OPTION VALUE ...

perl MATCH_REGEX NON_MATCH_REGEX

saveDump FILE

searchFallback PATTERN REPLACEMENT

searchSubst KEY PATTERN ... -> KEY PATTERN REPLACEMENT ...

LDAP Operations - Notes