deployment automation of ckan as a vm with docker
Closed this issue · 10 comments
provisioning of ckan vm is automatic but lacks of installation of needed software and config files for ckan
- cifs share mount
- install docker, docker-compose
- docker-compose file for ckan only and its cifs share as data volume
- confiure env for docker-compose
The generated json for whole resource group required some more parametrization and modifications to be run elsewhere.
I almost finished in deploying a complete resource group with custom parameters (for hostnames, resources, so on).
- docker registry ok
- file shares ok
- postgres deploys correctly
- redis deploys correctly
- private networking looks ok
- solr deployment is not working in this way but we can always use the script as it remained a container (in a new environment its image must be built).
- the virtual machine which has to host docker and ckan has some issues in deploying which I'm working on
- fileshares are actually not working as I thought yesterday I will check
- virtual machine for ckan deploy and I am able to inject a bash script to install docker and checkout repository
I made a pull request for my automatic approach to deploy everything unattended here #4
started working to fix deployment issues:
- check fix environment variables, evidence in README any global resources which must have unique name.
- we are possibly afflicted by this bug: Azure/azure-cli#16499 and Azure/azure-cli#16705 preventing clean deployment of solr container
- test each connection on private network
@etj I was able to reproduce the issue we had yesterday with init db as well on the environment where we have the running ckan (resource group CREA_test_20210120), the very same sql alchemy url behaves differently when application does db init (that is a command done at application yet not running) than when application starts, in this latter case the sqlalchemy url is good and the '@' escaping we worked out works but in the init db phase:
invalid interpolation syntax in 'postgresql://ckan%40creapostgresqltest :XXXXXXXXXXXXXXX@crea-pg.privatelink.postgres.database.azure.com/ckan' at position 17
I was able to make it work patching ckan like this:
def setup_migration_version_control(self):
self.reset_alembic_output()
alembic_config = AlembicConfig(self._alembic_ini)
dburl = str(self.metadata.bind.url).replace('%', '%%')
alembic_config.set_main_option(
"sqlalchemy.url", str(dburl)
)
try:
sqlalchemy_migrate_version = self.metadata.bind.execute(
u'select version from migrate_version'
).scalar()
except ProgrammingError:
sqlalchemy_migrate_version = 0
# this value is used for graceful upgrade from
# sqlalchemy-migrate to alembic
alembic_config.set_main_option(
"sqlalchemy_migrate_version", str(sqlalchemy_migrate_version)
)
# This is an interceptor for alembic output. Otherwise,
# everything will be printed to stdout
alembic_config.print_stdout = self.add_alembic_output
self.alembic_config = alembic_config
I committed in this branch here
All fixes should be in place, my only problem to release for the client is that I changed submodule in the branch where I placed the fixes to use my ckan patch, anyway, it seems I can updated the submodule once the git url changed, so the script which configures git repository in azure for the ckan docker image is not working and requires manual intervention because the database is not initalized correctly.
Deployment was tested from scratch a few times.
For "deployment" I mean whole resources are up, I can login with a default password as admin in ckan, I can run the script of #12 and set 3 orgs and 3 datasets as initial data, human interactions are:
- edit parameters.json
- deploy the big part of resources with Azure ARM template
- run first two scripts
- copy .env on ckan vm
- run script forl solr and startup
- run script for populating data.
I have all needed modifications were pushed to branch envoronment-fixes on geosolutions-it/C195-azure-workspace repository.
I needed to patch ckan init module in models here in order to populate ckan database with initial data.
In order to proceed to merge everything and have a master branch with all needed modifications and pointing to the correct submodule I'd need help from you @etj and a revision of my work as well.
For the CKAN issue I opened an issue on the official CKAN repo and proposed a PR:
- Issue ckan/ckan#5998
- PR ckan/ckan#5999