This tutorial shows how to deploy confidential applications with the help of SCONE in a Kubernetes cluster.
The tutorial is organized as follows:
- Before you begin
- Hello World!
- Hello World! with remote attestation
- Hello World! with TLS certificates auto-generated by CAS
- Encrypt your source code
An executable version of this code can be cloned as follows:
git clone https://github.com/scontain/hello-world-scone-on-kubernetes.git
!!!info "Note that you need to get access to the images" Send us an email to get access.
You can run this demo by executing:
cd hello-world-scone-on-kubernetes
./run_hello_world.sh
This tutorial assumes that you have access to a Kubernetes cluster (through kubectl
), and that you are deploying everything to a separate namespace, called hello-scone
. Namespaces are a good way to separate resources, not to mention how it makes the clean-up process a lot easier at the end:
kubectl create namespace hello-scone
Don't forget to specify it by adding -n hello-scone
to your kubectl
commands.
SGX applications need access to the Intel SGX driver, a char device located at the host machine. The device name might differ depending on the driver implementation, but the most common ones are /dev/sgx
(DCAP), /dev/sgx/enclave
(in-tree) and /dev/isgx
(IAS/out-of-tree). If you are in a cluster context, you need the right device mounted into the container. In Kubernetes, mounting host devices requires your applications to run in privileged mode.
We provide an SGX device plugin for Kubernetes, which lets you request the SGX driver through the resource.limits
field (just like CPU or RAM). The device plugin handles different driver names transparently, and also allows your applications to run unprivileged (which is always good for security).
The SGX device plugin can be easily installed with the help of Helm:
helm install sgxdevplugin sconeapps/sgxdevplugin
Please refer to Kubernetes SGX plugin documentation for more details and Helm setup instructions.
This tutorial assumes that you have the SGX device plugin running on your cluster.
NOTE: If you can't install the SGX device plugin for some reason, you have to mount the Intel SGX driver device manually into your container. To do so, first figure out which driver is installed in your cluster. Then update all the manifests below by removing the
resource.limits
field, and adding ahostPath
and the respectivevolumeMount
definitions. You also have to run the container in privileged mode by addingprivileged: true
tosecurityContext
. Check a sample manifest.
NOTE: If you don't have Intel SGX in your cluster, you can run SCONE in simulated mode by setting the environment variable
SCONE_MODE=sim
. The application will run encrypted in memory, but Intel SGX security guarantees do not apply.
A lot of files are created through this tutorial, and they have to be somewhere. Choose whatever directory you want to run the commands below, but it might be a good idea to create a temporary one:
cd $(mktemp -d)
We use the following base image:
export BASE_IMAGE=sconecuratedimages/kubernetes:python-3.7.3-alpine3.10-scone4.2
Have fun!
When remote attestation enters the scene, you will need a CAS to provide attestation, configuration and secrets. Export now the address of your CAS. If you don't have your own CAS, use our public one at scone.ml:
export SCONE_CAS_ADDR=scone-cas.cf
Now, generate client certificates (needed to submit new sessions to CAS):
mkdir -p conf
if [[ ! -f conf/client.crt || ! -f conf/client-key.key ]] ; then
openssl req -x509 -newkey rsa:4096 -out conf/client.crt -keyout conf/client-key.key -days 31 -nodes -sha256 -subj "/C=US/ST=Dresden/L=Saxony/O=Scontain/OU=Org/CN=www.scontain.com" -reqexts SAN -extensions SAN -config <(cat /etc/ssl/openssl.cnf \
<(printf '[SAN]\nsubjectAltName=DNS:www.scontain.com'))
fi
Create a CAS namespace for your sessions.
export NS=HelloWorld-$RANDOM-$RANDOM
cat > namespace.yaml <<EOF
name: $NS
access_policy:
read:
- CREATOR
update:
- CREATOR
EOF
curl -v -k -s --cert conf/client.crt --key conf/client-key.key --data-binary @namespace.yaml -X POST https://$SCONE_CAS_ADDR:8081/v1/sessions
Let's start by writing a simple Python webserver: it only returns the message Hello World!
and the value of the environment variable GREETING
.
mkdir -p app
cat > app/server.py << EOF
from http.server import HTTPServer
from http.server import BaseHTTPRequestHandler
import os
class HTTPHelloWorldHandler(BaseHTTPRequestHandler):
def do_GET(self):
"""say "Hello World!" and the value of \`GREETING\` env. variable."""
self.send_response(200)
self.end_headers()
self.wfile.write(b'Hello World!\n\$GREETING is: %s\n' % (os.getenv('GREETING', 'no greeting :(').encode()))
httpd = HTTPServer(('0.0.0.0', 8080), HTTPHelloWorldHandler)
print("Starting server")
httpd.serve_forever()
EOF
To build this application with SCONE, simply use our Python 3.7 curated image as base. The Python interpreter will run inside of an enclave!
cat > Dockerfile << EOF
FROM $BASE_IMAGE
EXPOSE 8080
COPY app /app
CMD [ "python3", "/app/server.py" ]
EOF
Set the name of the image. If you already have an image, skip the building process.
export IMAGE=sconecuratedimages/kubernetes:hello-k8s-scone0.1
docker build --pull . -t $IMAGE && docker push $IMAGE
Deploying the application with Kubernetes is also simple: we just need a deployment and a service exposing it.
Write the Kubernetes manifests:
cat > app.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world
spec:
selector:
matchLabels:
run: hello-world
replicas: 1
template:
metadata:
labels:
run: hello-world
spec:
containers:
- name: hello-world
image: $IMAGE
imagePullPolicy: Always
ports:
- containerPort: 8080
env:
- name: GREETING
value: howdy!
resources:
limits:
sgx.k8s.io/sgx: 1
---
apiVersion: v1
kind: Service
metadata:
name: hello-world
labels:
run: hello-world
spec:
ports:
- port: 8080
protocol: TCP
selector:
run: hello-world
EOF
Now, submit the manifests to the Kubernetes API, using kubectl
command:
kubectl create -f app.yaml -n hello-scone
Now that everything is deployed, you can access your Python app running on your cluster. Forward your local 8080
port to the service port:
kubectl port-forward svc/hello-world 8080:8080 -n hello-scone
The application will be available at your http://localhost:8080:
$ curl localhost:8080
Hello World!
$GREETING is: howdy!
SCONE provides a remote attestation feature, so you make sure your application is running unmodified. It's also possible to have secrets and configuration delivered directly to the attested enclave!
The remote attestation is provided by two components: LAS (local attestation service, runs on the cluster) and CAS (a trusted service that runs elsewhere. We provide a public one in scone-cas.cf).
You can deploy LAS to your cluster through our LAS Helm chart:
helm install las sconeapps/las --set image=sconecuratedimages/kubernetes:las-scone4.2
Alternatively, you can deploy LAS to your cluster with the help of a DaemonSet, deploying one LAS instance per cluster node. As your application has to contact the LAS container running in the same host, we use the current node IP as our LAS address.
cat > las.yaml << EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: local-attestation
labels:
k8s-app: local-attestation
spec:
selector:
matchLabels:
k8s-app: local-attestation
template:
metadata:
labels:
k8s-app: local-attestation
spec:
hostNetwork: true
containers:
- name: local-attestation
image: sconecuratedimages/kubernetes:las-scone4.2
resources:
limits:
sgx.k8s.io/sgx: 1
ports:
- containerPort: 18766
hostPort: 18766
EOF
kubectl create -f las.yaml -n hello-scone
To setup remote attestation, you will need a session file and the MRENCLAVE
, which is a unique signature of your application. Extract MRENCLAVE
of your application by running its container with the environment variable SCONE_HASH
set to 1
:
MRENCLAVE=`docker run -i --rm -e "SCONE_HASH=1" $IMAGE`
Then create a session file. Please note that we are also defining a secret GREETING
. Only the MRENCLAVES
registered in this session file will be allowed to see such secret.
cat > session.yaml << EOF
name: $NS~hello-k8s-scone
version: "0.2"
services:
- name: application
mrenclaves: [$MRENCLAVE]
command: python3 /app/server.py
pwd: /
environment:
GREETING: hello from SCONE!!!
EOF
Now, post the session file to your CAS of choice:
curl -v -k -s --cert conf/client.crt --key conf/client-key.key --data-binary @session.yaml -X POST https://$SCONE_CAS_ADDR:8081/v1/sessions
Once you submit the session file, you just need to inject the CAS address into your Deployment manifest. To showcase that, we're creating a new Deployment:
cat > attested-app.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: attested-hello-world
spec:
selector:
matchLabels:
run: attested-hello-world
replicas: 1
template:
metadata:
labels:
run: attested-hello-world
spec:
containers:
- name: attested-hello-world
image: $IMAGE
imagePullPolicy: Always
ports:
- containerPort: 8080
env:
- name: SCONE_CAS_ADDR
value: $SCONE_CAS_ADDR
- name: SCONE_CONFIG_ID
value: $NS~hello-k8s-scone/application
- name: SCONE_LAS_ADDR
valueFrom:
fieldRef:
fieldPath: status.hostIP
resources:
limits:
sgx.k8s.io/sgx: 1
---
apiVersion: v1
kind: Service
metadata:
name: attested-hello-world
labels:
run: attested-hello-world
spec:
ports:
- port: 8080
protocol: TCP
selector:
run: attested-hello-world
EOF
NOTE: The environment defined in the Kubernetes manifest won't be considered once your app is attested. The environment will be retrieved from your session file.
Deploy your attested app:
kubectl create -f attested-app.yaml -n hello-scone
Once again, forward your local port 8082 to the service port:
kubectl port-forward svc/attested-hello-world 8082:8080 -n hello-scone
The attested application will be available at your http://localhost:8082:
$ curl localhost:8082
Hello World!
$GREETING is: hello from SCONE!!!
The CAS service can also generate secrets and certificates automatically. Combined with access policies, it means that such auto-generated secrets and certificates will be seen only by certain applications. No human (e.g. a system operator) will ever see them: they only exist inside of SGX enclaves.
To showcase such feature, let's use the same application as last example. But now, we serve the traffic over TLS, and we let CAS generate the server certificates.
Start by rewriting our application to serve with TLS:
cat > app/server-tls.py << EOF
from http.server import HTTPServer
from http.server import BaseHTTPRequestHandler
import os
import socket
import ssl
class HTTPHelloWorldHandler(BaseHTTPRequestHandler):
def do_GET(self):
"""say "Hello World!" and the value of \`GREETING\` env. variable."""
self.send_response(200)
self.end_headers()
self.wfile.write(b'Hello World!\n\$GREETING is: %s\n' % (os.getenv('GREETING', 'no greeting :(').encode()))
httpd = HTTPServer(('0.0.0.0', 4443), HTTPHelloWorldHandler)
httpd.socket = ssl.wrap_socket(httpd.socket,
keyfile="/app/key.pem",
certfile="/app/cert.pem",
server_side=True)
print("Starting server")
httpd.serve_forever()
EOF
Our updated Dockerfile looks like this:
cat > Dockerfile << EOF
FROM $BASE_IMAGE
EXPOSE 4443
COPY app /app
CMD [ "python3" ]
EOF
Build the updated image:
export IMAGE=sconecuratedimages/kubernetes:hello-k8s-scone0.2
docker build --pull . -t $IMAGE && docker push $IMAGE
The magic is done in the session file. First, extract the MRENCLAVE
again:
MRENCLAVE=$(docker run -i --rm -e "SCONE_HASH=1" $IMAGE)
Now, create a Session file to be submitted to CAS. The certificates are defined in the secrets
field, and are injected into the filesystem through images.injection_files
:
cat > session-tls-certs.yaml << EOF
name: $NS~hello-k8s-scone-tls-certs
version: "0.2"
services:
- name: application
image_name: application_image
mrenclaves: [$MRENCLAVE]
command: python3 /app/server-tls.py
pwd: /
environment:
GREETING: hello from SCONE with TLS and auto-generated certs!!!
images:
- name: application_image
injection_files:
- path: /app/cert.pem
content: \$\$SCONE::SERVER_CERT.crt\$\$
- path: /app/key.pem
content: \$\$SCONE::SERVER_CERT.key\$\$
secrets:
- name: SERVER_CERT
kind: x509
EOF
Post the session file to your CAS of choice:
curl -k -s --cert conf/client.crt --key conf/client-key.key --data-binary @session-tls-certs.yaml -X POST https://$SCONE_CAS_ADDR:8081/session
The steps to run your app are similar to before: let's create a Kubernetes manifest and submit it.
cat > attested-app-tls-certs.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: attested-hello-world-tls-certs
spec:
selector:
matchLabels:
run: attested-hello-world-tls-certs
replicas: 1
template:
metadata:
labels:
run: attested-hello-world-tls-certs
spec:
containers:
- name: attested-hello-world-tls-certs
image: $IMAGE
imagePullPolicy: Always
ports:
- containerPort: 4443
env:
- name: SCONE_CAS_ADDR
value: $SCONE_CAS_ADDR
- name: SCONE_CONFIG_ID
value: $NS~hello-k8s-scone-tls-certs/application
- name: SCONE_LAS_ADDR
valueFrom:
fieldRef:
fieldPath: status.hostIP
resources:
limits:
sgx.k8s.io/sgx: 1
---
apiVersion: v1
kind: Service
metadata:
name: attested-hello-world-tls-certs
labels:
run: attested-hello-world-tls-certs
spec:
ports:
- port: 4443
protocol: TCP
selector:
run: attested-hello-world-tls-certs
EOF
kubectl create -f attested-app-tls-certs.yaml -n hello-scone
Now it's time to access your app. Again, forward the traffic to its service:
kubectl port-forward svc/attested-hello-world-tls-certs 8083:4443 -n hello-scone
And send a request:
$ curl -k https://localhost:8083
Hello World!
$GREETING is: hello from SCONE with TLS and auto-generated certs!!!
Moving further, you can run the exact same application, but now the server source code will be encrypted using SCONE's file protection feature, and only the Python interpreter that you register will be able to read them (after being attested by CAS).
Let's encrypt the source code using SCONE's Fileshield. Run a SCONE CLI container with access to the files:
docker run -it --rm -e SCONE_MODE=sim -v $PWD:/tutorial $BASE_IMAGE sh
Inside the container, create an encrypted region /app
and add all the files in /tutorial/app
(the plain files) to it. Lastly, encrypt the key itself.
cd /tutorial
rm -rf app_image && mkdir -p app_image/app
cd app_image
scone fspf create fspf.pb
scone fspf addr fspf.pb / --not-protected --kernel /
scone fspf addr fspf.pb /app --encrypted --kernel /app
scone fspf addf fspf.pb /app /tutorial/app /tutorial/app_image/app
scone fspf encrypt fspf.pb > /tutorial/app/keytag
The contents of /tutorial/app_image/app
are now encrypted. Try to cat
them from inside the container:
$ cd /tutorial/app_image/app && ls
server-tls.py
$ cat server-tls.py
$�ʇ(E@��
�����Id�Z���g3uc��mѪ�Z.$�__�!)ҹS��٠���(�� �X�Wϐ�{$���|�זH��Ǔ��!;2����>@z4-�h��3iݾs�t�7�<>H4:����(9=�a)�j?��p����q�ߧ3�}��Hظa�|���w-�q��96���o�9̃�Ev�v��*$����TU/���Ӕ��v�G���T�1<ڹ��C#p|i��AƢ˅_!6���F���w�@
�C��J���+81�8�
You can now exit the container. Let's build the server image with our files, now encrypted:
cat > Dockerfile << EOF
FROM $BASE_IMAGE
EXPOSE 4443
COPY app_image /
CMD [ "python3" ]
EOF
Choose the new image name and build it:
export IMAGE=sconecuratedimages/kubernetes:hello-k8s-scone-0.3
docker build --pull . -t $IMAGE && docker push $IMAGE
Time to deploy our server to Kubernetes. Let's setup the attestation first, so we make sure that only our application (identified by its MRENCLAVE
) will have access to the secrets to read the encrypted filesystem, as well as our secret greeting.
Extract the MRENCLAVE
:
MRENCLAVE=$(docker run -i --rm -e "SCONE_HASH=1" $IMAGE)
Extract SCONE_FSPF_KEY
and SCONE_FSPF_TAG
:
export SCONE_FSPF_KEY=$(cat app/keytag | awk '{print $11}')
export SCONE_FSPF_TAG=$(cat app/keytag | awk '{print $9}')
export SCONE_FSPF=/fspf.pb
Create a session file:
cat > session-tls.yaml << EOF
name: $NS~hello-k8s-scone-tls
version: "0.2"
services:
- name: application
image_name: application_image
mrenclaves: [$MRENCLAVE]
command: python3 /app/server-tls.py
pwd: /
environment:
GREETING: hello from SCONE with encrypted source code and auto-generated certs!!!
fspf_path: $SCONE_FSPF
fspf_key: $SCONE_FSPF_KEY
fspf_tag: $SCONE_FSPF_TAG
images:
- name: application_image
injection_files:
- path: /app/cert.pem
content: \$\$SCONE::SERVER_CERT.crt\$\$
- path: /app/key.pem
content: \$\$SCONE::SERVER_CERT.key\$\$
secrets:
- name: SERVER_CERT
kind: x509
EOF
Post the session file to your CAS of choice:
curl -k -s --cert conf/client.crt --key conf/client-key.key --data-binary @session-tls.yaml -X POST https://$SCONE_CAS_ADDR:8081/session
Create the manifest and submit it to your Kubernetes cluster:
cat > attested-app-tls.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: attested-hello-world-tls
spec:
selector:
matchLabels:
run: attested-hello-world-tls
replicas: 1
template:
metadata:
labels:
run: attested-hello-world-tls
spec:
containers:
- name: attested-hello-world-tls
image: $IMAGE
imagePullPolicy: Always
ports:
- containerPort: 4443
env:
- name: SCONE_CAS_ADDR
value: $SCONE_CAS_ADDR
- name: SCONE_CONFIG_ID
value: $NS~hello-k8s-scone-tls/application
- name: SCONE_LAS_ADDR
valueFrom:
fieldRef:
fieldPath: status.hostIP
resources:
limits:
sgx.k8s.io/sgx: 1
---
apiVersion: v1
kind: Service
metadata:
name: attested-hello-world-tls
labels:
run: attested-hello-world-tls
spec:
ports:
- port: 4443
protocol: TCP
selector:
run: attested-hello-world-tls
EOF
kubectl create -f attested-app-tls.yaml -n hello-scone
Time to access your app. Forward the traffic to its service:
kubectl port-forward svc/attested-hello-world-tls 8084:4443 -n hello-scone
And send a request:
$ curl -k https://localhost:8084
Hello World!
$GREETING is: hello from SCONE with encrypted source code and auto-generated certs!!!
To clean up all the resources created in this tutorial, simply delete the namespace:
kubectl delete namespace hello-scone
Author: Clenimar
© scontain.com, 2020. Questions or Suggestions?