Suggestions for simplifying Docker Compose

[sandertan]

Hi @vladd-bit , in addition to #19 I think there are some more simplifications possible for services.yml that make it easier to do custom deployments while also making it easy to regularly pull updates from this repository's master branch.

1. Move env variables to YML files.
   Are there any specific reasons to keep some ENV vars in the docker-compose, while having other configuration for the same services in the YML / properties files? In elasticsearch-1, -2 and kibana services there are quite a number of environment variables that can also be specified in the respective YML-files. Also the nifi service contains some env variables that can be moved to nifi/conf/nifi.properties I think (although I've not tested this). Apparently some NiFi properties can only be set using ENV vars https://stackoverflow.com/a/55266528/4141535.

2. Create git tracked -EXAMPLE files for configuration files
   Just like I suggested earlier with deploy/.env-example (git tracked) and deploy/.env (git ignored), we can use this way of working with the OpenSearch YML and NiFi properties files. Custom deployments can copy the example file and tailor it to their needs. This makes it easy to pull new changes, and maintainers can inspect (e.g. using a diff-tool) the differences between the example and used file to see whether properties are added/changed/deleted. This way of working was quite effective in the previous projects I collaborated in (example).

3. Remove container and network names, and rely on $COMPOSE_PROJECT_NAME (https://docs.docker.com/compose/reference/envvars/#compose_project_name). I documented how to use this in #19 .

4. There are a lot of commented and uncommented lines regarding ElasticSearch and Kibana mounted security files. Perhaps this can be simplified by using a single ENV var, e.g. $ELASTICSEARCH_SECURITY_DIR, which we can also put it .env-example and refer to ../security/es_certificates/opensearch/.

5. In our deployments we set all hosts ports in the .env outside of the docker-compose. For example, - ${KIBANA_HOST_PORT}:5601. This makes it easy to switch between local (fine if port is open) and server deployments (we set port to 127.0.0.1:5601 and let the reverse proxy on the host machine regulated traffic). What are your thoughts on moving this configuration to .env?

6. When ports: is set, expose: no longer has any informative meaning, and can be removed (https://stackoverflow.com/a/40801773/4141535). Or do you include them for a different reason?

For a new user doing a new deployment, it would be nice to require the least amount of actions to start the Docker containers. Perhaps only creating a .env file from the .env-example and executing docker-compose up is enough. We can configure the .env-example to point it to all the other examples files, which the user can at a later time change to his deployment specific configuration files.

I'd rather discuss this with you before creating a PR, since your workflows probably depend on the current way of working.

By the way, congratulations on releasing v1.0.0 :)

[vladd-bit]

Hello,

I will address this in the next release and focus on having a more adaptable ENV var config.

Some things that I should mention about the points made:

1. The reason I kept some vars in the docker compose for now was because of supporting multiple ES versions (opensearch and native), it totally makes sense to have it the way you mentioned. However, on the NiFi env var side of things, when I did the upgrade from 1.14 to 1.1(5/6).x I had noticed most variables were not picked up by the container as they are not configurable via env vars(reported as improvements on their tracker, https://issues.apache.org/jira/browse/NIFI-7060?jql=project%20%3D%20NIFI%20AND%20status%20%3D%20Open%20AND%20text%20~%20%22Environment%22 ), it required .prop file being changed (for example for the proxy context path), this is definitely on the scope though and I will just adapt the changes mentioned there.

2. Agreed.

3. Will review and merge.

4. Will revise this, need a new structure that fits both versions of ES.

5. Agreed.

6. I've only included those as they were specified in the default ES/open-ES setup samples. Will be removing them and including them under the ports section instead as they were used for other logging purposes.

[vladd-bit]

All of the things here have been addressed in the past commits. Documentation updates to follow. But now it is possible to fully dinamically switch ES versions by simply modifying the .env files in ./deploy and then executing the export_env_vars.sh, settings for the other services are now present in their respective .env files.

[sandertan]

Thanks a lot @vladd-bit ! I was planning to rebase our local branch on master this week, so this is perfect timing :)