osdu-infrastructure

This project is an implementation of the Infrastructure as Code and Pipelines necessary to build and deploy the required infrastructure necessary for the Open Subsurface Data Universe (OSDU). Project Development for this code base is performed and maintained on osdu-infrastructure in GitHub with a mirrored copy located in GitLab.

All patterns for this have been built and leverage to Microsoft Projects, for detailed design principals, operation and tutorials and those patterns it is best to review information directly from those projects.

BootStrap a Pipeline

Eventually a bootstrap process will be handled by an ado terraform provider but for now this is a manual process.

Prerequisites

Azure CLI installed or alternately Azure Cloud Shell.

Assumes CLI Version = azure-cli (2.0.75)

Execute Install Script

The script ./scripts/install.sh will conveniently setup the common things that are necessary to execute a pipeline.

cd scripts
./install.sh <your_subscription> <unique_alphanumeric>

Installed Common Resources

Resource Group
Storage Account
Key Vault
Applications for Integration Testing (2)

Elastic Search Setup

Infrastructure assumes bring your own Elastic Search Instance at a version of 6.8.3 and access information must be stored in the Common KeyVault.

AZURE_VAULT="<your_keyvault>"
az keyvault secret set --vault-name $AZURE_VAULT --name "elastic-endpoint-ado-demo" --value <your_es_endpoint>
az keyvault secret set --vault-name $AZURE_VAULT --name "elastic-username-ado-demo" --value <your_es_username>
az keyvault secret set --vault-name $AZURE_VAULT --name "elastic-password-ado-demo" --value <your_es_password>

# Dump all secrets to output
for i in `az keyvault secret list --vault-name $AZURE_VAULT --query [].id -otsv`
do
   echo "export ${i##*/}=\"$(az keyvault secret show --vault-name $AZURE_VAULT --id $i --query value -otsv)\""
done

Configure Azure DevOps

Create a new ADO Project in your organization called osdu-r2
Import the osdu-infrastructure to the ADO Project Repo from this URL https://github.com/Azure/osdu-infrastructure.git
Configure an ARM Resources Service Connection for the desired subscription.

This will create a Service Principal.
Configure Service Principal API permissions.
- Azure Active Directory Graph - Application.ReadWrite.OwnedBy
- Microsoft Graph - Application.ReadWrite.OwnedBy
Setup and Configure the ADO Library Infrastructure Pipeline Variables
- AGENT_POOL = Hosted Ubuntu 1604
- BUILD_ARTIFACT_NAME = infra-templates
- SERVICE_CONNECTION_NAME = <your_service_connection_name>
- TF_VAR_elasticsearch_secrets_keyvault_name = osducommon<your_unique>-kv
- TF_VAR_elasticsearch_secrets_keyvault_resource_group = osdu-common-<your_unique>
- TF_VAR_remote_state_account = osducommon<your_unique>
- TF_VAR_remote_state_container = remote-state-container
Setup and Configure the ADO Library Infrastructure Pipeline Variables - demo
- ARM_SUBSCRIPTION_ID = <your_subscription_id>
- TF_VAR_cosmosdb_replica_location = eastus2
- TF_VAR_resource_group_location = centralus
Setup and Configure the ADO Library Infrastructure Pipeline Secrets - demo

This should be linked Secrets from Azure Key Vault osducommon<your_unique>-kv
- elastic-endpoint-ado-demo = *********
- elastic-username-ado-demo = *********
- elastic-password-ado-demo = *********
Setup 2 Secure Files

This is for future AKS work but the keys will be located after running install.sh in the scripts/.ssh directory
- azure-aks-gitops-ssh-key
- azure-aks-node-ssh-key.pub
Create a New Azure Pipeline from the Azure Repo using the pipeline /azure-pipeline.yml
Run the Pipeline and the Infrastructure will deploy

Pipelines automatically check for changes. Set a runtime variable of FORCE_RUN=true to bypass Change Detection.

Contributing

We do not claim to have all the answers and would greatly appreciate your ideas and pull requests.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

For project level questions, please contact Daniel Scholl or Dania Kodeih.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0