Support mutable config for all kinds of shadowed services
xxx7xxxx opened this issue · 1 comments
xxx7xxxx commented
Background
As EaseMesh has started to support non-java applications, the shadow service only supports java for now. So we want to expand our product to include as many features as possible for non-java services, shadow service is the next choice.
Shadow service supports changing the addresses of some middleware, for example, the process needs to change production endpoints to staging/testing ones in case of disturbing the production lines.
We inject all containers with sidecar and EaseAgent[1]. But only applications running in JVM will load and launch EaseAgent. And sidecar will notify EaseAgent of the shadowed middleware information through sidecar protocol[2].
On the other hand, there's no explicit information about whether the internal running application is a Java application or not. So we must distinguish the type of applications before passing corresponding configs.
So we can break down the things that we must do to support mutable config for all kinds of shadowed services:
- We must know which type of application running (Java with EaseAgent and others such as golang, python, etc.)
- After knowing the information above, we need to deliver user-defined config to shadowed services running in containers
Proposal
In the need to generalize this feature, we should expand the existing sidecar protocol for the language-insensitive guarantee.
Agent Type
There are mainly two kinds of learning agent type of applications:
- Manual. User give it to use in service spec, such as:
echo 'kind: Service
metadata:
name: service-001
spec:
registerTenant: "tenant-001"
agentType: EaseAgent # GoSDK, PythonSDK...
#...This is the simplest solution, but it actually relies on the users' awareness, which could give us the wrong information.
- Automatically. We expand the sidecar protocol on the agent side for
http://localhost:9900/agent-info, which returns the agent information including agent type, such as
agentType: EaseAgent # GoSDK, PythonSDK, None...
agentVersion: v2.2.1This method would give us the real result and need no awareness from users. But it could bring complexity and make inconsistency among different service instances, which might give inconsistent agent types in some cases (although it's the responsibility of users to prevent it from happening).
So in another perspective, the manual solution is to add static service-level information, but the automatic one is to add dynamic service-instance-level information.
IMHO, the automatic one is better in the case of rigid standards.
Deliver Config
Currently, only EaseAgent can take up config(only the observability part). Besides that, we prepare to support map other configs into the application which means the container in terms of Kubernetes env.
Config categories could be these below:
- Env: They could be copied, and then added, deleted, or mutated by users. (NOTICE: Some env generated by EaseStack are dedicated, which could not be copied)
- ConfigMap: They would be copied into another config map(such as the name of
shadow-xxx-configmap-01), and then be mutated by users. - Secret: Same as ConfigMap.
The more work for ConfigMap and Secret is the extra lifecycle management while deleting Shadow Service. An example would be
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-mesh
spec:
template:
spec:
containers:
name: order-mesh
image: megaease/consuldemo:latest
- env:
- name: DEBUG
value: false
volumeMounts:
- name: cm-01
mountPath: "/etc/config-01"
- name: secret-01
mountPath: "/etc/secret-01"
volumes:
- name: cm-01
configMap:
name: cm-01
- name: secret-01
secret:
name: secret-01shadowed into:
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-mesh-shadow # append suffix -shadow
spec:
template:
spec:
containers:
name: order-mesh
image: megaease/consuldemo:latest
- env:
- name: DEBUG
value: false # changed from false to true
- name: MYSQL_ADDRESS # add a new env
value: mysql://192.168.0.111:13306
volumeMounts:
- name: cm-01
mountPath: "/etc/config-01"
- name: secret-01
mountPath: "/etc/secret-01"
volumes:
- name: cm-01
configMap:
name: cm-01-order-mesh-shadow # append suffix -order-mesh-shadow
- name: secret-01
secret:
name: secret-01-order-mesh-shadow # append suffix -order-mesh-shadowAs we can see, the format of copies is {configmap/secret name}-{deployment/statefulset name}-shadow, which contains the content which is coped and changed by users (if they want).
The reason we add {deployment/statefulset name} into the shadowed config name is that the original configmap/secret may be shared by multiple deployments/statefulsets. As we split them totally, the cleaning of one shadow resource won't affect others. As we see, this brings a certain amount of complexity as a cost.
The added APIs would be:
- Control Plane: add APIs for retrieving deployment/statefulsets specs and their Secrets and ConfigMaps.
- Shadow service: adds the creation API for handling shadowing Secrets and confiConfigMapsgmap besides shadowed component.
Reference
[1] https://github.com/megaease/easeagent
[2] https://github.com/megaease/easemesh/blob/main/docs/sidecar-protocol.md
xxx7xxxx commented
- Retrieval of deployment spec exmaple:
$ curl http://127.0.0.1:32202/apis/v2/mesh/services/order-mesh/deployment
which contains 3 major fields:
- app: objest, deployment spec as k8s Deployment spec.
- configMaps: array, the item is k8s ConfigMap spec.
- secrets: arrays, the item is k8s Secret spec.
{
"app": {
"metadata": {
"name": "order-mesh",
"namespace": "mesh-service",
"uid": "38071fc0-19a3-4a6d-b2c4-70c8d58c5e1b",
"resourceVersion": "11688463",
"generation": 2,
"creationTimestamp": "2022-07-15T13:43:38Z",
"annotations": {
"deployment.kubernetes.io/revision": "2",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"mesh.megaease.com/service-name\":\"order-mesh\"},\"name\":\"order-mesh\",\"namespace\":\"mesh-service\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"order-mesh\"}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"order-mesh\"}},\"spec\":{\"containers\":[{\"env\":[{\"name\":\"SERVICE_NAME\",\"value\":\"order-mesh\"},{\"name\":\"ZIPKIN_SERVER_URL\",\"value\":\"https://dev.megaease.cn/zipkin\"}],\"image\":\"megaease/consuldemo:latest\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"order-mesh\",\"ports\":[{\"containerPort\":80}]}]}}}}\n",
"mesh.megaease.com/service-name": "order-mesh"
},
"managedFields": [
{
"manager": "kubectl-client-side-apply",
"operation": "Update",
"apiVersion": "apps/v1",
"time": "2022-07-15T13:43:38Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:kubectl.kubernetes.io/last-applied-configuration": {},
"f:mesh.megaease.com/service-name": {}
}
},
"f:spec": {
"f:progressDeadlineSeconds": {},
"f:replicas": {},
"f:revisionHistoryLimit": {},
"f:selector": {},
"f:strategy": {
"f:rollingUpdate": {
".": {},
"f:maxSurge": {},
"f:maxUnavailable": {}
},
"f:type": {}
},
"f:template": {
"f:metadata": {
"f:labels": {
".": {},
"f:app": {}
}
},
"f:spec": {
"f:containers": {
"k:{\"name\":\"order-mesh\"}": {
".": {},
"f:env": {
".": {},
"k:{\"name\":\"SERVICE_NAME\"}": {
".": {},
"f:name": {},
"f:value": {}
},
"k:{\"name\":\"ZIPKIN_SERVER_URL\"}": {
".": {},
"f:name": {},
"f:value": {}
}
},
"f:image": {},
"f:imagePullPolicy": {},
"f:name": {},
"f:ports": {
".": {},
"k:{\"containerPort\":80,\"protocol\":\"TCP\"}": {
".": {},
"f:containerPort": {},
"f:protocol": {}
}
},
"f:resources": {},
"f:terminationMessagePath": {},
"f:terminationMessagePolicy": {}
}
},
"f:dnsPolicy": {},
"f:restartPolicy": {},
"f:schedulerName": {},
"f:securityContext": {},
"f:terminationGracePeriodSeconds": {}
}
}
}
}
},
{
"manager": "kube-controller-manager",
"operation": "Update",
"apiVersion": "apps/v1",
"time": "2022-10-01T07:54:55Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:conditions": {
".": {},
"k:{\"type\":\"Available\"}": {
".": {},
"f:type": {}
},
"k:{\"type\":\"Progressing\"}": {
".": {},
"f:lastTransitionTime": {},
"f:status": {},
"f:type": {}
}
}
}
}
},
{
"manager": "kubectl-edit",
"operation": "Update",
"apiVersion": "apps/v1",
"time": "2022-10-01T08:38:24Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:spec": {
"f:template": {
"f:spec": {
"f:volumes": {
"k:{\"name\":\"config\"}": {
".": {},
"f:configMap": {
".": {},
"f:defaultMode": {},
"f:name": {}
},
"f:name": {}
}
}
}
}
}
}
},
{
"manager": "kube-controller-manager",
"operation": "Update",
"apiVersion": "apps/v1",
"time": "2022-10-07T03:38:38Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
"f:deployment.kubernetes.io/revision": {}
}
},
"f:status": {
"f:availableReplicas": {},
"f:conditions": {
"k:{\"type\":\"Available\"}": {
"f:lastTransitionTime": {},
"f:lastUpdateTime": {},
"f:message": {},
"f:reason": {},
"f:status": {}
},
"k:{\"type\":\"Progressing\"}": {
"f:lastUpdateTime": {},
"f:message": {},
"f:reason": {}
}
},
"f:observedGeneration": {},
"f:readyReplicas": {},
"f:replicas": {},
"f:updatedReplicas": {}
}
},
"subresource": "status"
}
]
},
"spec": {
"replicas": 1,
"selector": {
"matchLabels": {
"app": "order-mesh"
}
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "order-mesh"
}
},
"spec": {
"volumes": [
{
"name": "agent-volume",
"emptyDir": {}
},
{
"name": "sidecar-volume",
"emptyDir": {}
},
{
"name": "config",
"configMap": {
"name": "game-demo",
"defaultMode": 420
}
}
],
"initContainers": [
{
"name": "initializer",
"image": "megaease/easeagent-initializer:latest",
"command": [
"sh",
"-c",
"set -e\ncp -r /easeagent-volume/* /agent-volume\n\necho 'name: order-mesh\ncluster-name: easemesh-control-plane\ncluster-role: secondary\ncluster-request-timeout: 10s\ncluster:\n primary-listen-peer-urls: http://easemesh-control-plane-service.easemesh:2380\nlabels:\n mesh-alive-probe: http://localhost:9900/health\n mesh-application-port: 80\n mesh-service-labels: \n mesh-service-name: order-mesh\n' > /sidecar-volume/sidecar-config.yaml"
],
"resources": {},
"volumeMounts": [
{
"name": "agent-volume",
"mountPath": "/agent-volume"
},
{
"name": "sidecar-volume",
"mountPath": "/sidecar-volume"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent"
}
],
"containers": [
{
"name": "order-mesh",
"image": "megaease/consuldemo:latest",
"ports": [
{
"containerPort": 80,
"protocol": "TCP"
}
],
"env": [
{
"name": "SERVICE_NAME",
"value": "order-mesh"
},
{
"name": "ZIPKIN_SERVER_URL",
"value": "https://dev.megaease.cn/zipkin"
},
{
"name": "JAVA_TOOL_OPTIONS",
"value": " -javaagent:/agent-volume/easeagent.jar -Deaseagent.log.conf=/agent-volume/log4j2.xml "
}
],
"resources": {},
"volumeMounts": [
{
"name": "agent-volume",
"mountPath": "/agent-volume"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent"
},
{
"name": "easemesh-sidecar",
"image": "megaease/easegress:server-sidecar",
"command": [
"/bin/sh",
"-c",
"/opt/easegress/bin/easegress-server -f /sidecar-volume/sidecar-config.yaml"
],
"ports": [
{
"name": "sidecar-ingress",
"containerPort": 13001,
"protocol": "TCP"
},
{
"name": "sidecar-egress",
"containerPort": 13002,
"protocol": "TCP"
},
{
"name": "sidecar-eureka",
"containerPort": 13009,
"protocol": "TCP"
}
],
"env": [
{
"name": "APPLICATION_IP",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "status.podIP"
}
}
}
],
"resources": {},
"volumeMounts": [
{
"name": "sidecar-volume",
"mountPath": "/sidecar-volume"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"securityContext": {},
"schedulerName": "default-scheduler"
}
},
"strategy": {
"type": "RollingUpdate",
"rollingUpdate": {
"maxUnavailable": "25%",
"maxSurge": "25%"
}
},
"revisionHistoryLimit": 10,
"progressDeadlineSeconds": 600
},
"status": {
"observedGeneration": 2,
"replicas": 1,
"updatedReplicas": 1,
"readyReplicas": 1,
"availableReplicas": 1,
"conditions": [
{
"type": "Progressing",
"status": "True",
"lastUpdateTime": "2022-10-01T08:38:26Z",
"lastTransitionTime": "2022-07-15T13:43:38Z",
"reason": "NewReplicaSetAvailable",
"message": "ReplicaSet \"order-mesh-65bb5976cc\" has successfully progressed."
},
{
"type": "Available",
"status": "True",
"lastUpdateTime": "2022-10-07T03:38:38Z",
"lastTransitionTime": "2022-10-07T03:38:38Z",
"reason": "MinimumReplicasAvailable",
"message": "Deployment has minimum availability."
}
]
}
},
"configMaps": [
{
"metadata": {
"name": "game-demo",
"namespace": "mesh-service",
"uid": "6c2d970e-cfb2-4c3f-bd6b-92a6356771fb",
"resourceVersion": "11260453",
"creationTimestamp": "2022-10-01T08:37:41Z",
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"data\":{\"game.properties\":\"enemy.types=aliens,monsters\\nplayer.maximum-lives=5 \\n\",\"player_initial_lives\":\"3\",\"ui_properties_file_name\":\"user-interface.properties\",\"user-interface.properties\":\"color.good=purple\\ncolor.bad=yellow\\nallow.textmode=true\\n\"},\"kind\":\"ConfigMap\",\"metadata\":{\"annotations\":{},\"name\":\"game-demo\",\"namespace\":\"mesh-service\"}}\n"
},
"managedFields": [
{
"manager": "kubectl-client-side-apply",
"operation": "Update",
"apiVersion": "v1",
"time": "2022-10-01T08:37:41Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:data": {
".": {},
"f:game.properties": {},
"f:player_initial_lives": {},
"f:ui_properties_file_name": {},
"f:user-interface.properties": {}
},
"f:metadata": {
"f:annotations": {
".": {},
"f:kubectl.kubernetes.io/last-applied-configuration": {}
}
}
}
}
]
},
"data": {
"game.properties": "enemy.types=aliens,monsters\nplayer.maximum-lives=5 \n",
"player_initial_lives": "3",
"ui_properties_file_name": "user-interface.properties",
"user-interface.properties": "color.good=purple\ncolor.bad=yellow\nallow.textmode=true\n"
}
}
],
"secrets": null
}$ curl http://127.0.0.1:32202/apis/v2/mesh/customresources -X POST --data-binary @./ss.json
The added major fields are:
- envs: users added or changed environments. (The come from app container from the deployment spec.)
- configMaps: all configMaps retrieved and changed.
- secrets: all secrets retrieved and changed.
ss.json looks like:
{
"kind": "ShadowService",
"name": "shadow-001",
"namespace": "mesh-service",
"serviceName": "order-mesh",
"envs": {
"MY_ENV": "FreeMine",
"ZIPKIN_SERVER_URL": "http://fake_addr:12030"
},
"configMaps": [
{
"apiVersion": "v1",
"kind": "ConfigMap",
"metadata": {
"labels": {
"esl-component": "nginx1",
"esl-stackId": "easetest2-default-docker_desktop",
"esl-stackName": "easetest2",
"esl-version": "a4b870a2-2a4e-451b-b143-fb0f9a3bf207"
},
"name": "nginx1",
"namespace": "default"
}
}
],
"secrets": [
{
"apiVersion": "v1",
"data": {
"ca.crt": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1ERXhPVEE1TkRrd00xb1hEVE15TURFeE56QTVORGt3TTFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS1czCkdHTzE1OURtNVI0dnMyaUNsUXIxcExMd1lLK2NKc3RBT0pzbGw0eEZQYk5oTUVNYlRjSHJrazhpYnRkclVGemIKVU45WXhEN3RMTU04dWpLWnRXTldFdDMydExJNUhJTzczYTFtRUpFSnI1UG9EcG9GTHdSYWxPWXFNNG54RjI5awp1M1hJbkh6QUV2MzFyakdhQ0pBeFBuTmdLQlZrV0hNL0c2RlBxRWZpSk9WK3hiMXdKRmdkZE1UUEZrNjEyVmtnCktxR0wxRUpUNFRPSkZzS0pvSHk2M2VFblVQOUM5NitJbmZ3RmFGS3huQWdzMENnYnpMYmp1ZlBVa052a3pTU0sKLzk2elRxMkhXc05DMjlNcFhvOGs3clY0L2xYeGhuSWEzdVo1dGZsQnU1dTExVUMzcGR3VWU3ZTFkOTVYZWU0ZwpENFhkbTlFTGNBb0NhbVcvNURVQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZNWFhjMnFGSVZsQVh5NFlXVFBSOXlLWEIzMjFNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQ3dEb1JpQzRRMmFpZTRIK1hHQQpoL2pXOGQ0cmkrL0kzMFhNQ05VNVUvK0Nib290azBTc0JhTjBQOVYxeXBkeDZDOXg3a2FSUGRqSkJVVU1qVGJuClZiZkJKV1VVNFpwcHJHMDRXU1I3NjBYWS91Sk9VVGtKWkdURXlIbG1FMk40eGdTYVpmWUhmZU1YekV5RTdoOWEKTUNYRFE0dFViajZQejl4WTFWZTRXQnJvdldxWFN4WVUwemRGMUNrbG5qa3EwU1lQcnVJRUdDNDdENDcrNks5aQpRSjBhQWFRQ29RY01xdnVDODVUd0VqeHluQnY3NStTYnVkSEkxVk1UQzJCUGFBMWlxRHluNDF4d3hiT2d4SForCnVZdUdtUVB0cnNHYjF4QmRrZ1hXYUlUSmtJOGxHYWpmNTFTN25BWHdpeVp1SE10MS9WTXUzTG00T2tqRU1xWHIKcXNvPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==",
"namespace": "ZGVmYXVsdA==",
"token": "ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNkltWmpNall0Y2xoWVgxSndVRlJuVDFoTlUybDFYMnBrZFdWSlFWRnBTRzVYTWtSS01WZGhjblpUVlRRaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUprWldaaGRXeDBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbVJsWm1GMWJIUXRkRzlyWlc0dFkyUjBjbk1pTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pWkdWbVlYVnNkQ0lzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG5WcFpDSTZJbUZrWVRFd05qWXpMVEJpWTJFdE5EYzRZeTA0TVdFMUxUazJZVE00TXpsbE1tTmhaQ0lzSW5OMVlpSTZJbk41YzNSbGJUcHpaWEoyYVdObFlXTmpiM1Z1ZERwa1pXWmhkV3gwT21SbFptRjFiSFFpZlEuYmFwY093RE5FYlhOaDM1bkdRcnlEU2tFQzh2UmdFenc0OERBNFdXMmxmSWFIN21Zb2RYWGNORS1iek9PMzZ4eldkWndtTm53ODRoYlQ1LWE5NHJtTXB1QlA2MVhucno2VzZNcUFvcmxSMWlGcmxJUFVsb3J3NWhEYS1xUlRDRlMzNDVQN2F5ZjNDM1ZueDRrYVJFYzZUbk5PWGdtNlU0RE1hQXZqbm9NTG5CSWlCVkdzUVhvUWxlREVTLUpVNEp5VUhVeWRKT2kxR244VnV4MjlXcDhtV3BiUHA0UGhOaXhaX2U2dmFjajRWNmF1N0NObm5URE82MVBvZzlDeGdPRXpxbklCZlpJLWs3bUdpQVFNT2RCZG0yWUJJSUUwZjBvTExqWnUwamtBSlk1MFpGMVNVOVZQYVo4NUprVUtLWFNieGo5cWFvUVZrcEd4Q2RsWDJsbUtR"
},
"kind": "Secret",
"metadata": {
"annotations": {
"kubernetes.io/service-account.name": "default",
"kubernetes.io/service-account.uid": "ada10663-0bca-478c-81a5-96a3839e2cad"
},
"name": "default-token-cdtrs",
"namespace": "default"
},
"type": "kubernetes.io/service-account-token"
}
]
}