Self Managed OSDU

This is a prototype used to evaluate the possibility of providing an easy self managed instance of OSDU on Azure to ease the development experience of developing components or extensions that might run on top or inside of the OSDU Platform that runs in Azure.

Goals:

Build an automation layer on top of OSDU on Azure without impacting the official OSDU Platform.
Ease the amount of required knowledge needed to deploy the platform.
Allow for customizations and configuration.
Provide access to and control of all software components installed.

Initialize the project

The solution is designed to operate from a fork instance of the project.

Fork the Project into your own GitHub Account.
Edit the Configuration Settings. (see Configuration)
Add any customizations (Optional) (see Customizations)

Configure GitHub Secrets

Secrets are managed using Github Repository Secrets some secrets are required to be created manually while others are created automatically by running Github Actions

Manually Created Secrets

GH_REPO_TOKEN: A personal access token with repo scope.
ELASTIC_ENDPOINT: The endpoint of the Elasticsearch cluster.

# Sample Format
https://my-osdu.es.southcentralus.azure.elastic-cloud.com:9243

ELASTIC_PASSWORD: The password of the Elasticsearch cluster.
AZURE_LOCATION: The Azure Region to deploy the resources to.
AZURE_CREDENTIALS: The json output of a Service Principal with Owner Subscription Scope.

SUBSCRIPTION_ID=$(az account show --query id --output tsv)
AZURE_CREDENTIALS="self-managed-osdu-azure-credentials-$(az account show --query user.name -otsv | awk -F "@" '{print $1}')"

az ad sp create-for-rbac --name $AZURE_CREDENTIALS \
  --role "Owner" \
  --scopes /subscriptions/$SUBSCRIPTION_ID \
  --sdk-auth \
  -ojson

# Sample Format
{
  "clientId": "00000000-0000-0000-0000-000000000000",                       # Client ID GUID
  "clientSecret": "**********************************",                     # Client Secret
  "subscriptionId": "00000000-0000-0000-0000-000000000000",                 # Subscription ID GUID
  "tenantId": "00000000-0000-0000-0000-000000000000",                       # Tenant ID GUID
  "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
  "resourceManagerEndpointUrl": "https://management.azure.com/",
  "activeDirectoryGraphResourceId": "https://graph.windows.net/",
  "sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
  "galleryEndpointUrl": "https://gallery.azure.com/",
  "managementEndpointUrl": "https://management.core.windows.net/"
}

OSDU_CREDENTIALS: The json output a Service Principal with Contributor Subscription Scope.

SUBSCRIPTION_ID=$(az account show --query id --output tsv)
OSDU_CREDENTIALS="self-managed-osdu-stamp-credentials-$(az account show --query user.name -otsv | awk -F "@" '{print $1}')"

az ad sp create-for-rbac --name $OSDU_CREDENTIALS \
  --role "Contributor" \
  --scopes /subscriptions/$SUBSCRIPTION_ID \
  --sdk-auth \
  -ojson

# Sample Format
{
  "clientId": "00000000-0000-0000-0000-000000000000",                       # Client ID GUID
  "clientSecret": "**********************************",                     # Client Secret
  "subscriptionId": "00000000-0000-0000-0000-000000000000",                 # Subscription ID GUID
  "tenantId": "00000000-0000-0000-0000-000000000000",                       # Tenant ID GUID
  "activeDirectoryEndpointUrl": "https://login.microsoftonline.com",
  "resourceManagerEndpointUrl": "https://management.azure.com/",
  "activeDirectoryGraphResourceId": "https://graph.windows.net/",
  "sqlManagementEndpointUrl": "https://management.core.windows.net:8443/",
  "galleryEndpointUrl": "https://gallery.azure.com/",
  "managementEndpointUrl": "https://management.core.windows.net/"
}

OSDU_CREDENTIAL_OID: The Object ID of the OSDU_CREDENTIALS Service Principal.

az ad sp list --display-name $OSDU_CREDENTIALS --query [].objectId -otsv

OSDU_APPLICATION: The json output of an Azure AD Application.

OSDU_APPLICATION="self-managed-osdu-stamp-application-$(az account show --query user.name -otsv | awk -F "@" '{print $1}')"

az ad app create --display-name $OSDU_APPLICATION \
  --oauth2-allow-implicit-flow \
  --required-resource-accesses @configuration/manifest.json \
  --query '{appId:appId, displayName:displayName, objectId:objectId}' \
  -ojson


  # Sample Format
{
  "appId": "00000000-0000-0000-0000-000000000000",
  "displayName": "osdu-application",
  "objectId": "00000000-0000-0000-0000-000000000000"
}

Generated Secrets

RAND: Unique Random String - Created by Action `Stamp Initialize'
SSH_PASSPHRASE: AKS SSH Key - Created by Action `Stamp Initialize'
SSH_PRIVATE_KEY: AKS SSH Key - Created by Action `Stamp Initialize'
SSH_PUBLIC_KEY: AKS SSH Key - Created by Action `Stamp Initialize'
BUILDER_GROUP: Resource Group Name - Created by Action `Stamp Builder'

Execute Github Actions

Deployment of a self managed osdu instance is performed by executing github actions to work with a Deployment Stamp. Currently there is only support for the deployment of 1 stamp.

Stamp Initialize: This action initializes any neccesary items in github that are necessary to begin (ie: Randomizer Secrets, ssh keys). (Time: ~30s)
Stamp Builder: This action provisions resources necessary to support a provisioning process (ie: Terraform State and Secret Storage). (Time: ~3m)
Stamp Provision: This action provisions resources for the Deployment Stamp. (Time: ~1h)
Stamp Configure: This action initializes the Software Configuration process of the Deployment Stamp. (Time: ~20m)

Note: The pipeline creates the software configuration definition which is performed by Flux and will complete in ~2m. Flux will then manage the installation process from within the cluster and will complete in ~18m.

Stamp Load: This action loads any necessary data into the Stamp in order to properly function. (ie: Entitlements TenantInit, Public Schemas) (Time: ~20m)

Note: The pipeline brings airflow online which is not currently done by Stamp Configure due to compatability issues with Airflow and Flux v3.

Azure Resources

The following resources are created in Azure for a Deployment Stamp.

Builder

Azure Key Vault (Standard)
Azure Storage Account - Table (StorageV2-LRS)

Control Plane

Log Analytics Workspace - 2 Solutions - Container & KeyVault (Pay-as-you-go)
Application Insights (Pay-as-you-go)
Azure Key Vault (Standard)
Azure Storage Account - Table (StorageV2-LRS)
Azure Cosmos DB Account - Gremlin (4000 Max RU's - Shared)
Azure Container Registry - (Standard)
User Managed Identity

Data Plane

Azure Virtual Network
Azure Storage Account - Qty2 (StorageV2-LRS)
Event Grid System Topic
Application Gateway - (WAFv2 w/ Autoscale (2-10))
Public IP Address
Azure Database for Postgres (General Purpose - 4vCore 5GB)
Azure Cosmos DB Account - Core (4000 Max RU's - Shared)
Azure Cache for Redis (Standard 1Gb)
Azure Kubernetes Service - System NodePool (2 Standard_E4s_v3), User NodePool (5-10 Standard_E4s_v3)
User Managed Identity - Qty3

Data Partition

Service Bus Namespace (Standard)
Event Grid Domain w/ 5 Topics (Basic)
Azure Cosmos DB Account - Core (12000 Max RU's - Shared)
Azure Storage Account - Qty2 (StorageV2-GZRS)

Platform Access

Verify the `osdu-azure` services have successfully started.

Verify the `self-managed-osdu` instance is available.

Configure the AD Application for `Web Platform` access.

HTTP Rest Scripts

The simpliest way to execute against the Platform is to leverage the HTTP Rest Scripts that make testing and executing API calls easier. These scripts are compatable with the VS Code Extension REST Client.

Rest Client Settings can be set to create environments and saved in VS Code Settings.

Retrieve the Rest Client Environment Setings

For convenience the Rest Client Environment Settings can be retrieved from the builder keyvault. (An access policy must be created to retrieve secrets)

These settings can now be placed in the .vscode/settings.json file along with a Client Secret that is used to authenticate the Rest Client.

Validate Partition Access

The Partition should be initialized automatically. Validate the Partition is accessible.

Configure RBAC Access

Entitlements needs to be initialized. This can be done by running the manage-user.http script.

Retrieve an OAuth Token

Define Intial OpenID Connect User

Create the Intial OpenID Connect User

Assign the User to the desired RBAC Group

Login the First User

User authentication leverages the Microsoft Identity Platform and OpenID Connect procotol. To enable an easier method of obtaining the required tokens an authcode login page with a rest-client http script are used to obtain API tokens.

Retrieve an Authorization Code

Access the /login page from the self-managed-osdu instance and click the authorize button. Approve the access by the AD Application if required (only required for the first time) and retrieve the code.

Retrieve an Initial Token

Using the Rest-Client HTTP Script auth-token.http add the auth_code obtained from the previous step to the AUTH_CODE parameter and post the authorize request. The returned refresh_token value then can be saved in the .vscode/settings.json file as the INITIAL_TOKEN parameter.

Execute the Platform Core Check Script

Using the Rest-Client HTTP Script check_core.http validate the functionality for the platform core APIs.

roalexan/self-managed-osdu