Leverages ISLE to provide a local development environment for the IDC stack, with particular focus on development/testing of the Drupal site.
- Our Drupal site is in
codebase.- Use
composerto add, remove, or update dependencies incodebase/composer.jsonandcodebase/composer.lockwhen developing - Dependencies are not vendored, so you need to do a composer install. This is included in
make bootstrap
- Use
- IDC development-specific environment variables are in
.envanddocker-compose.env.yml - An idc-specific Makefile
idc.Makefiledefines additional make targets available for `make
To start the IDC development environment, run
make up
This will build a docker-compose file, run composer install to locally install all dependencies for our site (which will
take a few minutes when done the first time, but will be much quicker subsequent times), and start the stack. The stack will
start from a known snapshot state, which currently is an entirely empty (but initialized) Drupal.
To reset to a known Drupal state, run
docker-compose down -v
docker-compose up -d
This will remove all content from volumes that you may have added, launch using the snapshot as its initial state.
To dump the site's configuration so that it can be committed to git, do
make config-export
To take a snapshot of Drupal's current content, do
make snapshot
See snapshots for more information on how to make and publish snapshots
There are several Make targets in the Makefile, and its idc-specific companion idc.Makefile (which are included by default,
so no need to do anything special other than make to invoke them). A few useful targets are as follows:
- make bootstrap Burn everything down and create a fresh installation from scratch, deleting any pre-existing data, and starting from a completely empty state. Only the list of modules in
composer.json(and dependencies incomposer.lock) survives the process. - make reset Burn everything down and create a fresh installation from the snapshot image. Unlike
make bootstrap, modules and dependencies do not survive; they will be installed when the drupal container starts. Does not pull in configuration from config/sync, will use the active configuration present in the snapshot. - make composer-install Use the Drupal container to run a
composer install. This avoids having to install composer on your local system. - make cache-rebuild Uses Drush inside the Drupal container to rebuild Drupal's cache.
- make config-export Exports all current active Drupal config to the
codebase/config/syncdirectory, so that it can be committed to git. - make snapshot Create a snapshot of the current Drupal state (db, content files, etc), so that you can reset to this state at will, or push it so that others can.
- make snapshot-push Push the current snapshot image to the container registry
- make up Brings up the development environment, including running
composer install.
A few specialized targets are:
- make static-docker-compose.yml Make a docker-compose.yml based off non-development "static" environment. Notably:
- A
drupal-staticimage is built or pulled, which hascodebasebaked in, and is used in place of the normaldrupalimage codebaseis no longer bind mounted
- A
- make static-drupal-image builds (or pulls, if published) a "static" drupal image suitable for deployment in the cloud. This image:
- Has the contents of
codebasebaked into it, as well as all dependencies viacomposer install - Will load its config from
config/syncupon startup - Is named
drupal-staticand is tagged based ongit describe --tags.
- Has the contents of
Snapshots are Docker images that contain Drupal state (content files, database, SOLR indexes, Fedora files, etc). When Docker starts, all Docker volumes will be populated with files from the snapshot image. The net result is that an environment will start quickly, from a known state, with pre-populated content.
After Docker starts from a snapshot, data subsequent in Docker's volumes is ephemeral. It will persist across stop and down, but can be wiped out by
docker-compose down -v
When docker subsequently starts, it will start from the known snapshot state. You are free to take a snapshot whenever you want a checkpoint you can reliably reset Drupal to.
The image used for the snapshot is specified via environment variables in .env. For example:
# Docker image and tag for snapshot image
SNAPSHOT_IMAGE=birkland/snapshot
SNAPSHOT_TAG=upstream-20201007-739693ae-12-ga409e4d8.1602146397
When the docker-compose.yml make target is run, that image and tag will be specified in the docker-compose file. The images contain data that are
copied to Docker volumes upon initial startup of the stack (i.e. snapshots are deployed only once, until all volumes are wiped out via docker-compose down -v). Because they are just regular docker images, they can be pushed and puled from container registry as usual.
To take a snapshot, run
make snapshot
This will do the following:
- stop the docker-compose stack
- dump the contents of the volumes
- create a new image from the contents of the volume
- give the image a unique tag based on the current git commit, and the date
- update the
.envfile to specify the just-takenSNAPSHOT_TAG - rebuild the
docker-compose.ymlfile to specify that tag - start docker-compose
If you want to commit that snapshot so that others can use it, you need to commit .env (which contains the tag of the snapshot image),
and publish the snapshot image to a Docker registry via
make snapshot-push
Make sure you do both steps! You need to push the image (so others can pull it), and push .env (so others can check out and run it).
Drupal incorporates its own SAML Service Provider using SimpleSAMLphp; an Apache-based SP proxy (c.f. PASS SAML configuration) is not used, and would be incompatible with Drupal. The SimpleSAMLphp SP will be used in development and production environments (including all cloud-based instances).
The developer environment includes a mock Shibboleth Identity Provider (IdP) and LDAP user store to serve as a backend for user attributes. These services would also be used when running integration tests, e.g. as a part of a PR or locally-executed tests. The production environment would not deploy these mock services.
SimpleSAMLphp has an administrative web interface which supports:
- diagnostics and configuration testing
- testing logins, including viewing attribute assertions by the IdP
- SP metadata generation (used to generate SP
EntityDescriptorXML used by Shibboleth IdPs) - IdP XML metadata conversion (used to transform Shibboleth IdP metadata for use by SimpleSAMLphp)
Testing authentication and attributes presented by the IdP is best performed using this admin interface.
Runtime parameterization of the IdP and SP is accomplished using a mixture of environment variables and secrets. Two containers use these secrets: the idp and drupal containers. Secrets are defined in docker-compose.yml from the consitutient service definitions in docker-compose.saml.yml and docker-compose.local.yml.
For a deeper understanding of SAML certificate roles, this primer may help.
| Allocation | URN | OID |
|---|---|---|
| eduPersonAffiliation | urn:mace:dir:attribute-def:eduPersonAffiliation | 1.3.6.1.4.1.5923.1.1.1.1 |
| eduPersonUniqueId | urn:oid:1.3.6.1.4.1.5923.1.1.1.13 | 1.3.6.1.4.1.5923.1.1.1.13 |
| eduPersonPrincipalName | urn:mace:dir:attribute-def:eduPersonPrincipalName | 1.3.6.1.4.1.5923.1.1.1.6 |
| eduPersonScopedAffiliation | urn:mace:dir:attribute-def:eduPersonScopedAffiliation | 1.3.6.1.4.1.5923.1.1.1.9 |
| eduPersonAffiliation | urn:mace:dir:attribute-def:eduPersonAffiliation | 1.3.6.1.4.1.5923.1.1.1.1 |
| employeeNumber | urn:mace:dir:attribute-def:employeeNumber | 2.16.840.1.113730.3.1.3 |
| displayName | urn:mace:dir:attribute-def:displayName | 2.16.840.1.113730.3.1.241 |
| givenName | urn:mace:dir:attribute-def:givenName | 2.5.4.42 |
| departmentNumber | urn:mace:dir:attribute-def:departmentNumber | 2.16.840.1.113730.3.1.2 |
The authsources.php file contains the requested attributes. These are hardcoded, and are not parameterized.
See the SimpleSAMLphp User Sync page for how these attributes may be mapped to Drupal user attributes or role selection.
These are truly environment variables (in the sense that they are not secrets). Cloud-based installations will need to update these to expected values.
| Variable Name | Default Value | Location | Description |
|---|---|---|---|
| SP_BASEURL | https://islandora-idc.traefik.me | .env file |
Base URL for SP services. Cloud-based installations will need to update these to expected values. |
| SP_ENTITYID | https://islandora-idc.traefik.me/sp/shibboleth | .env file |
SP entity ID (a URI). Cloud-based installations will need to update these to expected values. |
| IDP_BASEURL | https://islandora-idp.traefik.me:4443 | .env file |
Base URL for IdP services. Cloud-based installations will need to update these to expected values. |
| IDP_ENTITYID | https://islandora-idp.traefik.me/idp/shibboleth | .env file |
IdP entity ID (a URI). Cloud-based installations will need to update these to expected values. |
| DRUPAL_DEFAULT_SITE_URL | https://islandora-idc.traefik.me | docker-compose.env.yml |
Use to derive the SimpleSAMLphp baseurlpath |
| DRUPAL_SP_TECHCONTACTNAME | Moo Cow | docker-compose.env.yml |
Some information about the technical persons running this installation. The email address will be used as the recipient address for error reports, and also as the technical contact in generated metadata. |
| DRUPAL_SP_TECHCONTACTEMAIL | moo@cow.org | docker-compose.env.yml |
Some information about the technical persons running this installation. The email address will be used as the recipient address for error reports, and also as the technical contact in generated metadata. |
| DRUPAL_SP_PROTECTINDEXPAGE | false | docker-compose.env.yml |
Set this options to true if you want to require administrator password to access the web interface or the metadata pages, respectively. |
| DRUPAL_SP_PROTECTMETADATAPAGE | false | docker-compose.env.yml |
Set this options to true if you want to require administrator password to access the web interface or the metadata pages, respectively. |
| DRUPAL_SP_CHECKFORUPDATES | true | docker-compose.env.yml |
Set this option to false if you don't want SimpleSAMLphp to check for new stable releases when visiting the configuration tab in the web interface. |
| DRUPAL_SP_ASSERTIONALLOWEDCLOCKSKEW | 180 | docker-compose.env.yml |
Set the allowed clock skew between encrypting/decrypting assertions |
| DRUPAL_SP_SESSION_DURATIONSECONDS | 28800 | docker-compose.env.yml |
This value is the duration of the session in seconds. Make sure that the time duration of cookies both at the SP and the IdP exceeds this duration. |
| DRUPAL_SP_SESSION_DATASTORETIMEOUTSECONDS | 14400 | docker-compose.env.yml |
Sets the duration, in seconds, data should be stored in the datastore. As the data store is used for login and logout requests, this option will control the maximum time these operations can take. |
| DRUPAL_SP_SESSION_STATETIMEOUTSECONDS | 3600 | docker-compose.env.yml |
Sets the duration, in seconds, auth state should be stored. |
| DRUPAL_SP_SESSION_COOKIENAME | SimpleSAMLSessionID | docker-compose.env.yml |
Option to override the default settings for the session cookie name |
| DRUPAL_SP_SESSION_COOKIELIFETIMESECONDS | 0 | docker-compose.env.yml |
Expiration time for the session cookie, in seconds. Defaults to 0, which means that the cookie expires when the browser is closed. |
| DRUPAL_SP_SESSION_COOKIEPATH | / | docker-compose.env.yml |
Limit the path of the cookies. Can be used to limit the path of the cookies to a specific subdirectory. |
| DRUPAL_SP_SESSION_COOKIESECURE | false | docker-compose.env.yml |
Set the secure flag in the cookie. Set this to TRUE if the user only accesses your service through https. If the user can access the service through both http and https, this must be set to FALSE. |
It should be said immediately that not every key/value pair in the saml-secrets.yml file is strictly a secret. For example, public keys (in the form of X509 certs) would not be considered a secret. However, to facilitate uniform processing of private/public key pairs and other values, using a single secrets file together with the confd file backend makes sense. saml-secrets.yml uses a structure well-known to the containers that process its contents. Renaming, removing, or adding keys may impact the processing of its content by the idp or drupal containers.
In the Docker environment, there are unfortunate layers of indirection when it comes to the use of secrets.
First and foremost, secrets are defined in docker-compose.yml, which is formed by constituent service files including docker-compose.local.yml (which defines the drupal service) and docker-compose.saml.yml (which defines the idp service). The first indirection decouples the location of the secret from the name of the secret: the secrets are exposed to the containers by their name, independent of their location in the repository.
Generally, a named secret defines a single value. The exception is the secret named saml_secrets; it contains multiple secrets. This is the second indirection, whereby a single named secret can securely expose multiple secret values. (This is in order to leverage the file backend of confd) Values in saml-secrets.yml may themselves be environment variables, which must be processed before they are used. This is the third indirection.
Finally, some values from saml_secrets may be written out to files. This depends on the capabilities of the consumer: some read configuration values inline, others read them from files.
| Composer Secret Name | Composer Location | Confd Key | Container(s) | Used By | Referenced As | References | Notes |
|---|---|---|---|---|---|---|---|
| saml_secrets | /secrets/saml-secrets.yml | /saml-secrets/idp/signing-key | idp | Shibboleth IdP | file | idp.properties |
Ends with: 'opm/rxS83hCrTsIX3Il3T8Fpb97kdF+unCiWEaxrPEurjW8lB506' |
| saml_secrets | /secrets/saml-secrets.yml | /saml-secrets/idp/signing-cert | idp,drupal | Shibboleth IdP, SimpleSAMLphp | file, inline | idp.properties, idp-metadata.xml, shib13-idp-remote.php |
Ends with: 's00xrv14zLifcc8oj5DYzOhYRifRXgHX' |
| saml_secrets | /secrets/saml-secrets.yml | /saml-secrets/idp/encryption-key | idp | Shibboleth IdP | file | idp.properties |
Ends with: 'qX7ZsBuOT72RwVEa8fpT6IZ6IpOOEPmUid/f2VM2aAcXgaF//vMjxA==' |
| saml_secrets | /secrets/saml-secrets.yml | /saml-secrets/idp/encryption-cert | idp,drupal | Shibboleth IdP, SimpleSAMLphp | file, inline | idp.properties, idp-metadata.xml, shib13-idp-remote.php |
Ends with: 'p+tGUbGS2l873J5PrsbpeKEVR/IIoKo=' |
| idp_sealer | /secrets/idp/sealer.jks | - | idp | Shibboleth IdP | file | idp.properties |
- |
| idp_backchannel | /secrets/idp/idp-backchannel.p12 | - | idp | Jetty | file | backchannel.ini |
Runs on port 8443, may not be used |
| saml_secrets | /secrets/saml-secrets.yml | /saml-secrets/idp/backchannel-signing-cert | idp,drupal | Shibboleth IdP, SimpleSAMLphp | inline | idp-metadata.xml, shib13-idp-remote.php |
PEM version of the PKCS 12 encoded idp_backchannel; ends with: 't6Lf23Kb8yD6ZR7dihMZAGHnYQ/hlhM=' |
| idp_browser | /secrets/idp/idp-browser.p12 | - | idp | Jetty | file | ssl.ini |
Runs on port 4443; this is the cert presented to HTTPS clients for IdP requests |
| saml_secrets | /secrets/saml-secrets.yml | /saml-secrets/idp/browser-cert | - | - | - | - | PEM version of the PKCS 12 encoded idp_browser, seems unused; ends with '7vawjZs0YP5qGifhos34g2GKW81m6sjoxpstLMK7pNQRy/pR/kv/jiXEn8xHRE6s' |
| saml_secrets | /secrets/saml-secrets.yml | /saml-secrets/sp/admin-pw | drupal | SimpleSAMLphp | inline | config.php |
Contains the administrative password for the SimpleSAMLphp interface |
| saml_secrets | /secrets/saml-secrets.yml | /saml-secrets/sp/signing-cert | idp,drupal | Shibboleth IdP, SimpleSAMLphp | inline | sp-metadata.xml, authsources.php |
Ends with: 'lQQUhxyEXTBJx3luLlpIjoloFKIute9K7pE5qAENjg==' |
| saml_secrets | /secrets/saml-secrets.yml | /saml-secrets/sp/hashing-salt | drupal | SimpleSAMLphp | inline | config.php |
Used by SimpleSAMLphp to create secure hashes |
| saml_secrets | /secrets/saml-secrets.yml | /saml-secrets/sp/signing-key | drupal | SimpleSAMLphp | file | authsources.php |
Ends with: 'HPxmTTEX5graPtXeDM3hz5A=' |
Two containers use SAML env vars and secrets: the idp container (the mock Shibboleth Identity Provider) and the drupal container (SimpleSAMLphp Service Provider). Secrets are defined in their accompanying docker-compose files, docker-compose.local.yml and docker-compose.saml.yml.
There are a number of files which contribute to the SAML environment, distributed across the idc-isle-dc and idc-isle-buildkit repositories, and multiple methods used for processing. The complexity is high for a couple of reasons:
- multiple applications with different runtimes (java, nginx/php) consume the same SAML parameters (e.g. the cert used to sign SAML assertions is referenced by both the SP and IdP)
- a single application will use the same parameter inline as well as consume it from a file (so if you want to change a value, and only want to change it in one place, some hoops need to be jumped through)
An example may help: you wish to know how/where the IdP signing key is used. Using the table above, a starting point would be to search the idc-isle-buildkit repo for occurrences of idp/signing-key key. The only hits will be confd configuration, where you see that it is written out to /tmp/idp_signing. Perform a second search for /tmp/idp_signing, and you'll see it used in the idp.properties confd template (which in turn writes idp.properties to /opt/shibboleth-idp/conf/idp.properties).
A second, more complex example: you wish to know how/where the IdP signing cert is used. Using the table, start by searching the idc-isle-buildkit repo for occurrences of idp/signing-cert key. You'll see it referenced by a couple of confd templates which in turn will lead you to idp.properties (references the value in the file /tmp/idp_signing.crt) and idp-metadata.xml (the value is used inline). You will also want to perform a similar search of the idc-isle-dc repository. To cast a wide net, search for the string signing-cert (you'll get some hits from both the SP and IdP). You'll note that the file shib13-idp-remote.php uses PHP to parse the value from the saml_secrets YAML structure (essentially inline use). As things evolve, I do not expect the table below to be kept up-to-date, so it is worth understanding how to discover where and how various secrets are used.
Tips:
- Search in both repositories
- Cast a wide net: searching for
/saml-secrets/idp/signing-certis not going to find hits in PHP code that reference[saml-secrets][idp][signing-cert] - If
confdwrites the value to a file, perform a subsequent search for references to the file. - Having a general understanding of the SAML protocol helps: for example, encryption or signing keys shouldn't be shared between two different containers, but public keys (i.e. those used for verifying signatures) would be.
The IdP entrypoint uses confd to interpolate a number of configuration files that dictate SAML request processing. Because saml_secrets can use environment variables as secret values, the configuration files are subsequently processed through envsubst.
Note that the IdP configuration files may refer to any named secrets that are exposed by Docker, not just the secrets present in saml_secrets.
Finally, processing logic occurs in the idp entrypoint which resides in the idc-isle-buildkit repository. Secrets are defined in the idc-isle-dc repository. The names of secrets are shared between the two repositories; if the name of a secret changes, it must be changed in both.
The Service Provider (SimpleSAMLphp) runs in the drupal container, installed as a vendored composer dependency. A number of PHP configuration files have been modified to reference secrets and environment variables where it makes sense. For example, in config.php 'assertion.allowed_clock_skew' => 180 has been changed to 'assertion.allowed_clock_skew' => intval(getenv('DRUPAL_SP_ASSERTIONALLOWEDCLOCKSKEW')). Instead of inlining certificates, shib-13-idp-remote.php parses saml_secrets. These modifications reside in a patch file managed in the idc-isle-dc repository.
The PHP interpreter executes under PHP-FPM, therefore environment variables must be explicitly passed from NGINX. Confd is used to process SP-related environment variables (using the env backend) and expose them to PHP-FPM via the include file drupal_sp_fastcgi_params. Confd is executed on container start, and is managed in the idc-isle-buildkit repository.
======= Note: In order for the push to be successful, you need to be "logged in" to the container registry via
docker login ghcr.io
See the guide to the github container registry for more information.