An Operator based on the Operator SDK to run the Mobile Security Service protected by OAuth on a OpenShift cluster.
By the following commands you will create a local directory and clone this project.
$ git clone git@github.com:aerogear/mobile-security-service-operator.git $GOPATH/src/github.com/aerogear/mobile-security-service-operator
Install Minishift then enable Operators on it by running the following commands.
# create a new profile to test the operator
$ minishift profile set mss-operator
# enable the admin-user add-on
$ minishift addon enable admin-user
# add insecure registry to download the images from docker
$ minishift config set insecure-registry 172.30.0.0/16
# start the instance
$ minishift start
Use the following command to install the Operator, Service and its Database.
ℹ️
|
To install you need be logged in as a user with cluster privileges like the system:admin user. E.g. By using: oc login -u system:admin .
|
$ make install
To verify that the pods has been successful created you can run the following command.
# Go to the project namespace. E.g mobile-security-service
$ oc project mobile-security-service
# Check the pods which are deployed
$ oc get pods
NAME READY STATUS RESTARTS AGE
mobile-security-service-569d4f8d86-vp5hq 1/1 Running 2 1h
mobile-security-service-db-ccb5856f6-8qnvc 1/1 Running 0 1h
mobile-security-service-operator-785cbdbf46-wq2lf 1/1 Running 0 1h
To verify that the installation was successful completed you can check the AppStatus=
status field of Service CR and the DatabaseStatus=OK
status field of Database CR in the cluster. For both are expected this status fields be added with the value "OK" when all installation process is finished and are all created successfully as the following example.
# To check the Service CR
$ oc describe mobilesecurityservices
...
Route Name: route
Size: 1
Status:
App Status: OK
Config Map Name: mobile-security-service-config
Deployment Name: mobile-security-service
...
# To check the Database CR
$ oc describe mobilesecurityservicedbs
...
Size: 1
Status:
Database Status: OK
Deployment Name: mobile-security-service-db
Deployment Status:
...
ℹ️
|
To troubleshooting the installation if it does not complete successfully check the status of its CRs. Note that all objects which are managed by them has an respective status field on it. E.g Deployment Status
|
-
Create a MobileSecurityServiceApp CR as this example.
-
The app name and appId need to be specified into the MobileSecurityServiceApp CR as follows.
# The appName spec defines the name of the app used to bind the service appName: "app" # The appId spec defines the appId of the app used to bind the service appId: "appid"
-
Run the following command to bind the app to the Mobile Security Service
$ make example-app/apply
ℹ️
|
This command will execute kubectl apply -f deploy/crds/examples/mobile-security-service_v1alpha1_mobilesecurityserviceapp_cr.yaml and apply the MobileSecurityServiceApp CR example.
|
-
Run the following command to delete/unbind the app from the service.
$ make example-app/delete
❗
|
This command will execute kubectl delete -f deploy/crds/examples/mobile-security-service_v1alpha1_mobilesecurityserviceapp_cr.yaml and delete the MobileSecurityServiceApp CR example.
|
ℹ️
|
The Rest Service endpoint to delete it is called in the finalizer of the MobileSecurityServiceApp CR and the CR will be just allowed to be removed when the app is no longer available in its Service. |
Use the following command to delete the Operator, the Service and its Database and all related configuration applied by the install
of this project.
$ make uninstall
ℹ️
|
To uninstall you need be logged in as a user with cluster privileges like the system:admin user. E.g. By using: oc login -u system:admin .
|
The Mobile Security Service image and its parameters are configurable and specified by the MobileSecurityService CR.
ℹ️
|
Default values are applied by the operator in mandatory_specs.go are applied for the mandatory specifications which are not made in the MobileSecurityService CR. |
The database image and its parameters are configurable and specified by the MobileSecurityServiceDB CR.
ℹ️
|
Default values are applied by the operator in mandatory_specs.go are applied for the mandatory specifications which are not made in the MobileSecurityServiceDB CR. |
By using the command make install
the default namespace mobile-security-service
, defined in the Makefile will be created and the operator will be installed in this namespace. You are able to install the operator in another namespace if you wish, however, you need to set up its roles (RBAC) in order to apply them on the namespace where the operator will be installed. The namespace name needs to be changed in the Cluster Role Binding file. Note, that you also need to change the namespace in the Makefile in order to use the command make install
for another namespace.
# Replace this with the namespace where the operator will be deployed.
namespace: mobile-security-service
Only namespaces specified in the environment variable APP_NAMESPACES
can be used to apply Apps. If the MobileSecurityServiceApp CR is applied to a namespace that is not specified in APP_NAMESPACES
it will be ignored. Refer to configuration in the operator.yaml file.
❗
|
The values should be split by ; . E.g mobile-security-service-apps;example-namespace-apps
|
ℹ️
|
To run the project locally export the ENV VAR. E.g. export APP_NAMESPACES=mobile-security-service-apps
|
Environment Variables are used to configure the Mobile Security Service Application and Database. For further information on configuration see the Setup and Configuration section.
The application-monitoring stack provisioned by the application-monitoring-operator on Integr8ly can be used to gather metrics from this operator and the mobile security service. These metrics can be used by Integr8ly’s application monitoring to generate Prometheus metrics, AlertManager alerts and a Grafana dashboard.
It is required that the integr8ly/Grafana and Prometheus operators are installed. For further detail see integr8ly/application-monitoring-operator.
The following commands will enable the monitoring service where the operator has been installed in the default namespace with the make commands.
make monitoring/install
❗
|
The namespaces are setup manually in the files ServiceMonitor, Prometheus Rules, Operator Service, and Grafana Dashboard. Following an example from the Prometheus Rules. You should replace them if the operator is not installed in the default namespace. |
expr: |
(1-absent(kube_pod_status_ready{condition="true", namespace="mobile-security-service"})) or sum(kube_pod_status_ready{condition="true", namespace="mobile-security-service"}) != 3
[source,shell]
ℹ️
|
The command make monitoring/uninstall will uninstall the Monitor Service.
|
The backup service is implemented by using integr8ly/backup-container-image. It will do the backup of the database and the APP CRs which are data required to be restore in the case of failures. Following the steps to enable it.
-
Setup the AWS in order to store the backup outside of the cluster. You need to add your AWS details to MobileSecurityServiceBackup as follows or add the name of the secret which has already this data in the cluster.
# --------------------------------- # Stored Host - AWS # ---------------------------- awsS3BucketName: "example-awsS3BucketName" awsAccessKeyId: "example-awsAccessKeyId" awsSecretAccessKey: "example-awsSecretAccessKey"
❗Also, you can add the name of the secret which is created already in the cluster. -
Run the command
make backup/install
in the same namespace where the MobileSecurityService Database is installed in order to apply the CronJob which will do this process. -
Add the label
integreatly-middleware-service=true
in all namespaces defined in the ENV VAR APP_NAMESPACES in the operator.yaml. (E.g oc label ns/mobile-security-service-apps integreatly-middleware-service=true`). It will backup all MobileSecurityServiceApp CR applied on them.
ℹ️
|
To install you need be logged in as a user with cluster privileges like the system:admin user. E.g. By using: oc login -u system:admin .
|
To verify that the backup has been successful created you can run the following command in the namespace where the operator is installed.
$ oc get cronjob.batch/mobile-security-service-backup
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
mobile-security-service-backup 0 * * * * False 0 13s 12m
To check the jobs executed you can run the command oc get jobs
in the namespace where the operator is installed as the following example.
$ oc get jobs
NAME DESIRED SUCCESSFUL AGE
mobile-security-service-backup-1561588320 1 0 6m
mobile-security-service-backup-1561588380 1 0 5m
mobile-security-service-backup-1561588440 1 0 4m
mobile-security-service-backup-1561588500 1 0 3m
mobile-security-service-backup-1561588560 1 0 2m
mobile-security-service-backup-1561588620 1 0 1m
mobile-security-service-backup-1561588680 1 0 43s
ℹ️
|
In the above example the schedule was made to run this job each minute (*/1 * * * * )
|
To check the logs and troubleshooting you can run the command oc logs $podName -f
in the namespace where the operator is installed as the following example.
$ oc logs job.batch/mobile-security-service-backup-1561589040 -f
dumping mobile_security_service
dumping postgres
==> Component data dump completed
/tmp/intly/archives/mobile-security-service.mobile_security_service-22_46_06.pg_dump.gz
WARNING: mobile-security-service.mobile_security_service-22_46_06.pg_dump.gz: Owner username not known. Storing UID=1001 instead.
upload: '/tmp/intly/archives/mobile-security-service.mobile_security_service-22_46_06.pg_dump.gz' -> 's3://camilabkp/backups/mss/postgres/2019/06/26/mobile-security-service.mobile_security_service-22_46_06.pg_dump.gz' [1 of 1]
1213 of 1213 100% in 1s 955.54 B/s done
ERROR: S3 error: 403 (RequestTimeTooSkewed): The difference between the request time and the current time is too large.
Following the steps required to be performed in case of be required do the restore based in the backup service.
-
Install the Mobile Security Service by following the steps in Installing.
-
Restore the database with the dump which was stored in the AWS S3 bucket.
ℹ️To restore we should run gunzip -c filename.gz | psql dbname
-
It is required re-apply all APP CRs which has an backup in the AWS bucket as well.
This operator is cluster-scoped
. For further information see the Operator Scope section in the Operator Framework documentation. Also, check its roles in Deploy directory.
ℹ️
|
The operator, application and database will be installed in the namespace mobile-security-service which will be created by this project.
|
CustomResourceDefinition |
Description |
Packages, manages, installs and configures the Mobile Security Service on the cluster. |
|
Packages, manages, installs and configures the Mobile Security Service Database on the cluster. |
|
Creates and update the app in the REST API. |
|
Packages, manages, installs and configures the CronJob to do the backup using the image backup-container-image |
-
Resource
Description
Define the ConfigMap resources required for the Mobile Security Service Application and its Database. It will create the
mobile-security-service-app
which map the values used in the Environment Variables of both.Define the Deployment resource of Mobile Security Service Application, e.g. container and resources definitions.
Define the route resource required to expose the Mobile Security Service (REST Service and UI).
Define the Service resource of Mobile Security Service Application.
-
Mobile Security Service Database
Resource
Description
Define the Deployment resource of Mobile Security Service Database. (E.g container and resources definitions)
Define the PersistentVolumeClaim resource used by its Database.
Define the Service resource of Mobile Security Service Database.
-
Mobile Security Service Backup
Resource
Description
Define the CronJob resources in order to do the Backup.
Define the database and AWS secrets resources created.
An Oauth Proxy container and the required configuration will be setup by default by the operator to provide authentication to the Mobile Security Service.
The Design Pattern adopted to build the objects in its controllers is Factory Method Pattern.
-
Status
Description
appStatus
For this status is expected the value
OK
which means that all required Kubernetes/OCP objects are created.configMapName
Name of the configMap created with the Environment Variables.
deploymentName
Name of the deployment object created for the App.
deploymentStatus
Deployment Status from ks8 API (v1beta1.DeploymentStatus).
serviceName
Name of the service object created for the App.
serviceStatus
Deployment Status from ks8 API (v1.ServiceStatus).
routeName
Name of the route object created for the App.
routeStatus
Route Status from OCP API (v1.Route).
-
Status
Description
databaseStatus
For this status is expected the value
OK
which means that all required Kubernetes/OCP objects are created.deploymentName
Name of the deployment object created for the Database.
deploymentStatus
Deployment Status from ks8 API (v1beta1.DeploymentStatus).
serviceName
Name of the service object created for the Database.
serviceStatus
Deployment Status from ks8 API (v1.ServiceStatus).
PersistentVolumeClaimName
Name of the PersistentVolumeClaimName object created for the Database.
-
Status
Description
bindStatus
For this status is expected the value
OK
which means that the app was created in the API Service. -
Status
Description
backupStatus
Should show
OK
when everything is created successfully.cronJobName
Name of cronJob resource created by it.
cronJobStatus
CronJob Status from ks8 API (k8s.io/api/batch/v1beta1/CronJobStatus).
dbSecretName
Name of database secret resource created in order to allow the integr8ly/backup-container-image connect to the database .
dbSecretData
Data used into the secret to connect to the database .
awsSecretName
Name of AWS S3 bucket secret resource used in order to allow the integr8ly/backup-container-image connect to AWS to send the backup .
awsSecretData
Data used to in the secret to send the backup files to the AWS S3.
awsSecretDataNamespace
Namespace where the backup image will looking for the of the Aws Secret used.
encryptionKeySecretName
Name of the EncryptionKey used.
encryptionKeySecretNamespace
Namespace where the backup image will looking for the of the EncryptionKey used.
encryptionKeySecretData
Data used into the EncryptionKey.
hasEncryptionKey
Expected true when it was configured to use an EncryptionKey secret
databasePodFound
The value expected here is true which shows that the database pod was found.
servicePodFound
The value expected here is true which shows that the database service was found.
The following command will install the operator in the cluster and run the changes performed locally without the need to publish a dev
tag. In this way, you can verify your code in the development environment.
$ make code/run/local
❗
|
The local changes are applied when the command operator-sdk up local --namespace=mobile-security-service is executed then it is not a hot deploy and to get the latest changes you need re-run the command.
|
By the following commands you are able to connect in the Mobile Security Service Database. You can check it by OpenShift UI in the Database’s pod terminal.
# Login into the the Postgres
psql -U postgres
# To connect into the default database
\c mobile_security_service
# To list the tables
\dt
# To select all data from the app table
SELECT * FROM app;
Follow the below steps to debug the project in some IDEs.
ℹ️
|
The code needs to be compiled/built first. |
$ make setup/debug
$ cd cmd/manager/
$ dlv debug --headless --listen=:2345 --api-version=2
Then, debug the project from the IDE by using the default setup of Go Remote
option.
$ make setup/debug
$ dlv --listen=:2345 --headless=true --api-version=2 exec ./build/_output/bin/mobile-security-service-operator-local --
debug the project using the following Visual Code launch config.
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "test",
"type": "go",
"request": "launch",
"mode": "remote",
"remotePath": "${workspaceFolder}/cmd/manager/main.go",
"port": 2345,
"host": "127.0.0.1",
"program": "${workspaceFolder}",
"env": {},
"args": []
}
]
}
ℹ️
|
The image/tag used from Mobile Security Service is defined in mobile-security-service_v1alpha1_mobilesecurityservice_cr.yaml file. |
Command |
Description |
|
Creates the |
|
Uninstalls the operator and the Service and DB. Deletes the |
|
Applies the Example App CR (Create/Update app in the cluster and Service.). |
|
Deletes the Example App CR (Delete app from the Service). |
|
Deletes and applies the operator in order to refresh the image when a tag is not changed (development use) |
|
Installs Monitoring Service in order to provide metrics |
|
Uninstalls Monitoring Service in order to provide metrics, i.e. all configuration applied by |
|
Installs the backup Service in the operator’s namespace |
|
Uninstalls the backup Service from the operator’s namespace. |
|
Runs the operator locally for development purposes. |
|
Sets up environment for debugging proposes. |
|
Examines source code and reports suspicious constructs using vet. |
|
Formats code using gofmt. |
|
It will automatically generated/update the files by using the operator-sdk based on the CR status and spec definitions. |
|
Runs test suite |
|
Run coverage check |
|
Used by CI to build operator image from |
|
Used by CI to push the |
|
Used by CI to build operator image from a tagged commit and add |
|
Used by CI to push the |
|
Runs test suite |
|
Run coverage check |
ℹ️
|
The Makefile is implemented with tasks which you should use to work with. |
Images are automatically built and pushed to our image repository in the following cases:
-
For every change merged to master a new image with the
master
tag is published. -
For every change merged that has a git tag a new image with the
<operator-version>
andlatest
tags are published.
If the image does not get built and pushed automatically the job may be re-run manually via the CI dashboard.
Following the steps
-
Create a new version tag following the semver, for example
0.1.0
-
Bump the version in the version.go file.
-
Update the the CHANGELOG.MD with the new release.
-
Looking for the SOPs and update the tag for the them in all files (e.g
https://github.com/aerogear/mobile-security-service-operator/blob/0.2.0/SOP/SOP-operator.adoc
) -
Create a git tag with the version value, for example:
$ git tag -a 0.1.0 -m "version 0.1.0"
-
Push the new tag to the upstream repository, this will trigger an automated release by the CI, for example:
$ git push upstream 0.1.0
ℹ️
|
The image with the tag will be created and pushed to the mobile-security-service image hosting repository by the CI. |
|
Do not use letters in the tag such as v . It will not work.
|
This operator was developed using mainly the Kubernetes APIs in order to be compatible with both, however, currently this project requires the usage of the v1.Route to expose the service and OAuth-proxy for authentication which make it unsupportable for Kubernetes.
All contributions are hugely appreciated. Please see our Contributing Guide for guidelines on how to open issues and pull requests. Please check out our Code of Conduct too.
There are a number of ways you can get in in touch with us, please see the AeroGear community.