Support Pod Injection to deploy Layotto as a sidecar in Kubernetes.

[Xunzhuo]

What would you like to be added:

Support pod injection to deploy Layotto as a sidecar in Kubernetes.

Why is this needed:

Deploying Layotto as a sidecar is a common scenario in Kubernetes.

We need to provide such things in Layotto:

Command Line Tool

A command lint tool which supports manually inject like istioctl kube-inject:

lyoctl kube-inject -f kube-resources.yaml \
--injectConfigFile inj-template.tmpl \
--layottoConfigFile layotto.json \
--valuesFile values.json

MutatingWebhook

An optional component to dynamically inject Layotto when target pod is creating.

Auto inject into pods conditions:

• Namespace with label: mosn-injection: enabled
• Pod without label: sidecar.mosn.io/inject: false

Please refer to injection logic in Istio.

[github-actions]

Hi @Xunzhuo,
Thanks for opening an issue! 🎉

[Xunzhuo]

/kind ospp
/area installation
/priority high
/kind hard

[Xunzhuo]

/milestone v0.6.0

[Xunzhuo]

/assign

Assign myself too, to keep an eye on it.

[wenxuwan]

@zhenjunMa 我们可以把这个课题加到开源之夏里面去 #894

[JasonChen86899]

hi layotto member
I saw layotto project recently, it is very good! I am quite familiar with knative and istio so we can add serverless feature to layotto. I hope this is a good feature.

Maybe we need an new issue for this discussion an new project like layotto-cloud-native for this.

[seeflood]

@JasonChen86899 Thanks!
We can start from discussing the design. A design proposal is welcomed. We can use this issue or create a new issue, both are fine

[Xunzhuo]

Thanks @JasonChen86899, welcome to comment your ideas and proposals.

[y-ykcir]

@Xunzhuo Hi, I am interested in this issue and would like to have it as my OSPP 2023 program. Can you suggest where I should start？

[Xunzhuo]

@y-ykcir Can you send an application from OSPP website?

[y-ykcir]

@y-ykcir Can you send an application from OSPP website?

@Xunzhuo Sure, the submission of the application requires a proposal, and I may need some time to prepare it.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue or help wanted) or other activity occurs. Thank you for your contributions.

[Xunzhuo]

cc @xiaoxiang10086, you can use this issue to track the status of the task.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue or help wanted) or other activity occurs. Thank you for your contributions.

[zhenjunMa]

@Xunzhuo Hi, if the solution design has already been completed, please share it first so that we can also refer to it.

[Xunzhuo]

@Xunzhuo Hi, if the solution design has already been completed, please share it first so that we can also refer to it.

cc @xiaoxiang10086

[xiaoxiang10086]

@zhenjunMa

This project mainly implements two modules:

1. A command-line tool that supports manual injection of the Layout sidecar

2. Kubernetes webhook component that support dynamic injection of Layotto sidecar's

I will introduce the current specific design ideas of the two modules in turn.

Command line tool

(1) layoctl kube-inject use case

layoctl kube-inject -f kube-resources.yaml \
--injectConfigFile inj-template.tmpl \
--layottoConfigFile layotto.json \
--valuesFile values.json \
--output kube-resources-injected.yaml

The kube-inject subcommand is used to inject the Layotto sidecar into the Kubernetes payload. The list of available parameters is as follows:

Option	Abbreviation	Description
--filename	-f	Kubernetes resource filename
--injectConfigFile		injection configuration file name
--layottoConfigFile		Layout configuration file name
--valuesFile		Injection values configuration file name
--output	-o	Output resource file name after injection (default value '')

(2) Implementation ideas of command line tools

• Use cobra framework to parse command line arguments
• Read the specified Kubernetes resource file and parse out the Pod object *corev1.PodSpec to be injected
• Use Go text/template package to render injectConfigFile with valuesFile.
• Use injectConfig to update Containers property value of podSpec
• Write updated Kubernetes resources to the output stream

(3) Command line file

inj-template.tmpl

templates:
   sidecar: |-
     spec:
       containers:
       - name: layoutto-proxy
         image: docker.io/layotto/layotto:{{.Values.global.tag}}

values.yaml

global:
   tag: latest

layotto.json

Take the example configuration file: https://github.com/mosn/layotto/blob/HEAD/configs/config.json

(4) Questions

According to Layotto Configuration File Introduction, Layotto starts Need to read a configuration file in json format. This configuration file should already be included in the image file, so is it necessary to provide the configuration parameter layouttoConfigFile here?

Official image of Layotto found: https://hub.docker.com/r/layotto/layotto/tags

MutatingWebhook Component

(1) Background summary

Mutation Webhook is a kind of Webhook in Kubernetes, which is used to intercept and modify the mutation operation of Kubernetes objects. When the Kubernetes API Server receives a user request for an object, Mutation Webhook can intercept the request and modify the request, and then send the modified request to the Kubernetes API Server, thereby realizing dynamic modification of the Kubernetes object.

(2) Injection conditions

The Mutating Webhook component should only inject the Layotto Sidecar on pods that are in a namespace with the label mosn-injection: enabled. The pod itself has no label sidecar.mosn.io/inject: false.

(3) Implementation ideas

• Creates a mutating webhook that is registered with the Kubernetes API server and is called when the Pod is created.
• In MutatingWebhookConfiguration, use namespaceSelector to match namespaces with label mosn-injection: enabled.
• In webhooks, sidecar injections to pods with label sidecar.mosn.io/inject: false are ignored. If the Pod meets the injection conditions, use the specified Laotto configuration file and value template file to generate a Laotto sidecarTemplate and inject it into the Pod as a sidecar.
• Encapsulate the modification operation PatchOperation to return to the Kubernetes API server in AdmissionReview.Response in order to create the Pod.

[Xunzhuo]

@xiaoxiang10086 Great job! Hope this can land successfully : ) Ping me or @zhenjunMa if you have some questions.

Draft PRs are welcomed.

[JasonChen86899]

Hi @xiaoxiang10086 @Xunzhuo a good design for injection but I have a question. In Mosn doc, it is supported by istio, maybe we can also use the same way. The job in this design do the same thing like istio inject function such as istioctl and k8s webhooks.
I read layotto not much but find it dependent on mosn. Is layotto integration with mosn and only need one container? @Xunzhuo
If so use istio existing function maybe another design, just like istioctl manifest apply --set .values.global.proxy.image=${LAYOTTO IMAGE} --set meshConfig.defaultConfig.binaryPath="/usr/local/bin/xxx --set meshConfig.defaultConfig.configPath="/xxx" : )

[wenxuwan]

Hi @xiaoxiang10086 @Xunzhuo a good design for injection but I have a question. In Mosn doc, it is supported by istio, maybe we can also use the same way. The job in this design do the same thing like istio inject function such as istioctl and k8s webhooks. I read layotto not much but find it dependent on mosn. Is layotto integration with mosn and only need one container? @Xunzhuo If so use istio existing function maybe another design, just like istioctl manifest apply --set .values.global.proxy.image=${LAYOTTO IMAGE} --set meshConfig.defaultConfig.binaryPath="/usr/local/bin/xxx --set meshConfig.defaultConfig.configPath="/xxx" : )

@JasonChen86899 Very useful information，Mosn and Layotto is one container.

[zhenjunMa]

According to Layotto Configuration File Introduction, Layotto starts Need to read a configuration file in json format. This configuration file should already be included in the image file, so is it necessary to provide the configuration parameter layouttoConfigFile here?

@xiaoxiang10086
Yes, the configuration file in the Layotto image is only a demo, which includes all the components supported by Layotto, most of which may not be necessary for the application. Therefore, in practical use cases, the application owner should provide this configuration file and only start the components required by the application.

[wenxuwan]

Dapr

dapr当前支持两种配置文件注入：
● 静态配置（--resources-path参数）
● 动态配置（通过operater）

Layotto

Layotto当前并不支持动态注入的能力，最终的形态肯定是上述的形态，Layotto适配Dapr的Operator，然后可以做组建的动态加载：

但目前无法支持，因此需要在启动的时候就注入应用的配置文件，可以injector来实现文件目录的映射：

渲染完成后类似下面的形式:

这样可能会存在一个问题，在Layotto启动完成时，如果Mount操作未完成，Layotto是无法启动完成的，可以包装个start.sh脚本，循环监听特定目录下的文件，只有mount成功后，再启动Layotto：

#!/bin/bash

while true
do
	# 判断 /data 目录下是否有名为 layotto.json 的文件
	if [ -f /data/layotto.json ]; then
	# 如果有，执行 layotto start -c layotto.json 命令
		layotto start -c /data/layotto.json
	fi
done

https://docs.dapr.io/getting-started/tutorials/configure-state-pubsub/ dapr通过静态配置和动态配置启动初始化component
https://docs.dapr.io/operations/hosting/kubernetes/kubernetes-volume-mounts/ dapr的目录映射

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue or help wanted) or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue or help wanted. Thank you for your contributions.

[seeflood]

这样可能会存在一个问题，在Layotto启动完成时，如果Mount操作未完成，Layotto是无法启动完成的，可以包装个start.sh脚本，循环监听特定目录下的文件，只有mount成功后，再启动Layotto：

@wenxuwan 是不是把 json 文件以config map 的形式挂载进 pod就行，这样就不用纠结启动顺序的问题了;

用 config map 来维护组件配置的话，后续可以watch 这个文件变更，有变更就通过 动态配置下发机制 调layotto 接口、变更组件配置
这样就不用维护operator了

---

哦刚看了下实现，现在就是用configmap存的json