nicholas-fwang/spring-cloud-kubernetes

Kubernetes integration with Spring Cloud

Spring Cloud Kubernetes

Spring Cloud integration with Kubernetes

Features

• DiscoveryClient for Kubernetes
• KubernetesClient autoconfiguration
• PropertySource
• ConfigMap PropertySource
• Secrets PropertySource
• PropertySource Reload
• Pod Health Indicator
• Transparency (its transparent whether the code runs in or outside of Kubernetes)
• Kubernetes Profile Autoconfiguration
• Ribbon discovery in Kubernetes
• Zipkin discovery in Kubernetes
• ConfigMap Archaius Bridge

---

DiscoveryClient for Kubernetes

This project provides an implementation of Discovery Client for Kubernetes. This allows you to query Kubernetes endpoints (see services) by name. A service is typically exposed by the Kubernetes API server as a collection of endpoints which represent http, https addresses that a client can access from a Spring Boot application running as a pod. This discovery feature is also used by the Spring Cloud Kubernetes Ribbon or Zipkin projects to fetch respectively the list of the endpoints defined for an application to be load balanced or the Zipkin servers available to send the traces or spans.

This is something that you get for free just by adding the following dependency inside your project:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-kubernetes</artifactId>
    <version>${latest.version}</version>
</dependency>

To enable loading of the DiscoveryClient, add @EnableDiscoveryClient to the according configuration or application class.

Then you can inject the client in your code simply by:

@Autowire
private DiscoveryClient discoveryClient;

If for any reason you need to disable the DiscoveryClient you can simply set the following property in application .properties:

spring.cloud.kubernetes.discovery.enabled=false

Some Spring Cloud components use the DiscoveryClient in order to obtain info about the local service instance. For this to work you need to align the service name with the spring.application.name property.

Kubernetes PropertySource

The most common approach to configure your Spring Boot application is to create an application.properties|yaml or an application-profile.properties|yaml file containing key-value pairs providing customization values to your application or Spring Boot starters. Users may override these properties by specifying system properties or environment variables.

ConfigMap PropertySource

Kubernetes provides a resource named ConfigMap to externalize the parameters to pass to your application in the form of key-value pairs or embedded application.properties|yaml files. The Spring Cloud Kubernetes Config project makes Kubernetes ConfigMaps available during application bootstrapping and triggers hot reloading of beans or Spring context when changes are detected on observed ConfigMaps.

ConfigMapPropertySource will search for a Kubernetes ConfigMap which metadata.name is either the name of your Spring application (as defined by its spring.application.name property) or a custom name defined within the bootstrap.properties file under the following key spring.cloud.kubernetes.config.name.

If such a ConfigMap is found, it will be processed as follows:

• apply individual configuration properties.
• apply as yaml the content of any property named application.yaml
• apply as properties file the content of any property named application.properties

Example:

Let's assume that we have a Spring Boot application named demo that uses properties to read its thread pool configuration.

• pool.size.core
• pool.size.maximum

This can be externalized to config map in yaml format:

kind: ConfigMap
apiVersion: v1
metadata:
  name: demo
data:
  pool.size.core: 1
  pool.size.max: 16

Individual properties work fine for most cases but sometimes embedded yaml is more convenient. In this case we will use a single property named application.yaml to embed our yaml:

kind: ConfigMap
apiVersion: v1
metadata:
 name: demo
data:
 application.yaml: |-
   pool:
     size:
       core: 1
       max:16

Spring Boot applications can also be configured differently depending on active profiles and it is possible to provide different property values for different profiles using an application.properties|yaml property, specifying profile-specific values each in their own document (indicated by the --- sequence) as follows:

kind: ConfigMap
apiVersion: v1
metadata:
  name: demo
data:
  application.yml: |-
    greeting:
      message: Say Hello to the World
    ---
    spring:
      profiles: development
    greeting:
      message: Say Hello to the Developers
    ---
    spring:
      profiles: production
    greeting:
      message: Say Hello to the Ops

To tell to Spring Boot which profile should be enabled at bootstrap, a system property can be passed to the Java command launching your Spring Boot application using an env variable that you will define with the OpenShift DeploymentConfig or Kubernetes ReplicationConfig resource file as follows:

apiVersion: v1
kind: DeploymentConfig
spec:
  replicas: 1
  ...
    spec:
      containers:
      - env:
        - name: JAVA_APP_DIR
          value: /deployments
        - name: JAVA_OPTIONS
          value: -Dspring.profiles.active=developer

Notes:

• To access ConfigMaps on OpenShift the service account needs at least view permissions i.e.:

  oc policy add-role-to-user view system:serviceaccount:$(oc project -q):default -n $(oc project -q)

Properties:

Name	Type	Default	Description
spring.cloud.kubernetes.config.enabled	Boolean	true	Enable Secrets PropertySource
spring.cloud.kubernetes.config.name	String	${spring.application.name}	Sets the name of ConfigMap to lookup
spring.cloud.kubernetes.config.namespace	String	Client namespace	Sets the Kubernetes namespace where to lookup
spring.cloud.kubernetes.config.paths	List	null	Sets the paths were ConfigMaps are mounted
spring.cloud.kubernetes.config.enableApi	Boolean	true	Enable/Disable consuming ConfigMaps via APIs

Secrets PropertySource

Kubernetes has the notion of Secrets for storing sensitive data such as password, OAuth tokens, etc. This project provides integration with Secrets to make secrets accessible by Spring Boot applications. This feature can be explicitly enabled/disabled using the spring.cloud.kubernetes.secrets.enabled property.

The SecretsPropertySource when enabled will lookup Kubernetes for Secrets from the following sources:

1. reading recursively from secrets mounts
2. named after the application (as defined by spring.application.name)
3. matching some labels

Please note that by default, consuming Secrets via API (points 2 and 3 above) is not enabled for security reasons and it is recommend that containers share secrets via mounted volumes.

If the secrets are found their data is made available to the application.

Example:

Let's assume that we have a spring boot application named demo that uses properties to read its database configuration. We can create a Kubernetes secret using the following command:

oc create secret generic db-secret --from-literal=username=user --from-literal=password=p455w0rd

This would create the following secret (shown using oc get secrets db-secret -o yaml):

apiVersion: v1
data:
  password: cDQ1NXcwcmQ=
  username: dXNlcg==
kind: Secret
metadata:
  creationTimestamp: 2017-07-04T09:15:57Z
  name: db-secret
  namespace: default
  resourceVersion: "357496"
  selfLink: /api/v1/namespaces/default/secrets/db-secret
  uid: 63c89263-6099-11e7-b3da-76d6186905a8
type: Opaque

Note that the data contains Base64-encoded versions of the literal provided by the create command.

This secret can then be used by your application for example by exporting the secret's value as environment variables:

apiVersion: v1
kind: Deployment
metadata:
  name: ${project.artifactId}
spec:
   template:
     spec:
       containers:
         - env:
            - name: DB_USERNAME
              valueFrom:
                 secretKeyRef:
                   name: db-secret
                   key: username
            - name: DB_PASSWORD
              valueFrom:
                 secretKeyRef:
                   name: db-secret
                   key: password

You can select the Secrets to consume in a number of ways:

1. By listing the directories were secrets are mapped:

   -Dspring.cloud.kubernetes.secrets.paths=/etc/secrets/db-secret,etc/secrets/postgresql
   

   If you have all the secrets mapped to a common root, you can set them like:

   -Dspring.cloud.kubernetes.secrets.paths=/etc/secrets
   

2. By setting a named secret:

   -Dspring.cloud.kubernetes.secrets.name=db-secret
   

3. By defining a list of labels:

   -Dspring.cloud.kubernetes.secrets.labels.broker=activemq
   -Dspring.cloud.kubernetes.secrets.labels.db=postgresql
   

Properties:

Name	Type	Default	Description
spring.cloud.kubernetes.secrets.enabled	Boolean	true	Enable Secrets PropertySource
spring.cloud.kubernetes.secrets.name	String	${spring.application.name}	Sets the name of the secret to lookup
spring.cloud.kubernetes.secrets.namespace	String	Client namespace	Sets the Kubernetes namespace where to lookup
spring.cloud.kubernetes.secrets.labels	Map	null	Sets the labels used to lookup secrets
spring.cloud.kubernetes.secrets.paths	List	null	Sets the paths were secrets are mounted (example 1)
spring.cloud.kubernetes.secrets.enableApi	Boolean	false	Enable/Disable consuming secrets via APIs (examples 2 and 3)

Notes:

• The property spring.cloud.kubernetes.secrets.labels behaves as defined by Map-based binding.
• The property spring.cloud.kubernetes.secrets.paths behaves as defined by Collection-based binding.
• Access to secrets via API may be restricted for security reasons, the preferred way is to mount secret to the POD.

Example of application using secrets (though it hasn't been updated to use the new spring-cloud-kubernetes project): spring-boot-camel-config

PropertySource Reload

Some applications may need to detect changes on external property sources and update their internal status to reflect the new configuration. The reload feature of Spring Cloud Kubernetes is able to trigger an application reload when a related ConfigMap or Secret changes.

This feature is disabled by default and can be enabled using the configuration property spring.cloud.kubernetes.reload.enabled=true (eg. in the application.properties file).

The following levels of reload are supported (property spring.cloud.kubernetes.reload.strategy):

• refresh (default): only configuration beans annotated with @ConfigurationProperties or @RefreshScope are reloaded. This reload level leverages the refresh feature of Spring Cloud Context.
• restart_context: the whole Spring ApplicationContext is gracefully restarted. Beans are recreated with the new configuration.
• shutdown: the Spring ApplicationContext is shut down to activate a restart of the container. When using this level, make sure that the lifecycle of all non-daemon threads is bound to the ApplicationContext and that a replication controller or replica set is configured to restart the pod.

Example:

Assuming that the reload feature is enabled with default settings (refresh mode), the following bean will be refreshed when the config map changes:

@Configuration
@ConfigurationProperties(prefix = "bean")
public class MyConfig {

    private String message = "a message that can be changed live";

    // getter and setters

}

A way to see that changes effectively happen is creating another bean that prints the message periodically.

@Component
public class MyBean {

    @Autowired
    private MyConfig config;

    @Scheduled(fixedDelay = 5000)
    public void hello() {
        System.out.println("The message is: " + config.getMessage());
    }
}

The message printed by the application can be changed using a ConfigMap as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: reload-example
data:
  application.properties: |-
    bean.message=Hello World!

Any change to the property named bean.message in the ConfigMap associated to the pod will be reflected in the output. More generally speaking, changes associated to properties prefixed with the value defined by the prefix field of the @ConfigurationProperties annotation will be detected and reflected in the application. Associating a ConfigMap to a pod is explained above.

The full example is available in spring-cloud-kubernetes-reload-example.

The reload feature supports two operating modes:

• event (default): watches for changes in config maps or secrets using the Kubernetes API (web socket). Any event will produce a re-check on the configuration and a reload in case of changes. The view role on the service account is required in order to listen for config map changes. A higher level role (eg. edit) is required for secrets (secrets are not monitored by default).
• polling: re-creates the configuration periodically from config maps and secrets to see if it has changed. The polling period can be configured using the property spring.cloud.kubernetes.reload.period and defaults to 15 seconds. It requires the same role as the monitored property source. This means, for example, that using polling on file mounted secret sources does not require particular privileges.

Properties:

Name	Type	Default	Description
spring.cloud.kubernetes.reload.enabled	Boolean	false	Enables monitoring of property sources and configuration reload
spring.cloud.kubernetes.reload.monitoring-config-maps	Boolean	true	Allow monitoring changes in config maps
spring.cloud.kubernetes.reload.monitoring-secrets	Boolean	false	Allow monitoring changes in secrets
spring.cloud.kubernetes.reload.strategy	Enum	refresh	The strategy to use when firing a reload (refresh, restart_context, shutdown)
spring.cloud.kubernetes.reload.mode	Enum	event	Specifies how to listen for changes in property sources (event, polling)
spring.cloud.kubernetes.reload.period	Long	15000	The period in milliseconds for verifying changes when using the polling strategy

Notes:

• Properties under spring.cloud.kubernetes.reload. should not be used in config maps or secrets: changing such properties at runtime may lead to unexpected results;
• Deleting a property or the whole config map does not restore the original state of the beans when using the refresh level.

Pod Health Indicator

Spring Boot uses HealthIndicator to expose info about the health of an application. That makes it really useful for exposing health related information to the user and are also a good fit for use as readiness probes.

The Kubernetes health indicator which is part of the core module exposes the following info:

• pod name, ip address, namespace, service account, node name and its ip address
• flag that indicates if the Spring Boot application is internal or external to Kubernetes

Transparency

All of the features described above will work equally well regardless of whether your application is running inside Kubernetes or not. This is really helpful for development and troubleshooting. From a development point of view, this is really helpful as you can start your Spring Boot application and debug one of the modules part of this project. It is not required to deploy it in Kubernetes as the code of the project relies on the Fabric8 Kubernetes Java client which is a fluent DSL able to communicate using http protocol to the REST API of Kubernetes Server.

Kubernetes Profile Autoconfiguration

When the application runs as a pod inside Kubernetes a Spring profile named kubernetes will automatically get activated. This allows the developer to customize the configuration, to define beans that will be applied when the Spring Boot application is deployed within the Kubernetes platform (e.g. different dev and prod configuration).

Ribbon discovery in Kubernetes

Spring Cloud client applications calling a microservice should be interested on relying on a client load-balancing feature in order to automatically discover at which endpoint(s) it can reach a given service. This mechanism has been implemented within the spring-cloud-kubernetes-ribbon project where a Kubernetes client will populate a Ribbon ServerList containing information about such endpoints.

The implementation is part of the following starter that you can use by adding its dependency to your pom file:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-kubernetes-netflix</artifactId>
    <version>${latest.version}</version>
</dependency>

When the list of the endpoints is populated, the Kubernetes client will search the registered endpoints living in the current namespace/project matching the service name defined using the Ribbon Client annotation:

@RibbonClient(name = "name-service")

You can configure Ribbon's behavior by providing properties in your application.properties (via your application's dedicated ConfigMap) using the following format: <name of your service>.ribbon.<Ribbon configuration key> where:

• <name of your service> corresponds to the service name you're accessing over Ribbon, as configured using the @RibbonClient annotation (e.g. name-service in the example above)
• <Ribbon configuration key> is one of the Ribbon configuration key defined by Ribbon's CommonClientConfigKey class

Additionally, the spring-cloud-kubernetes-ribbon project defines two additional configuration keys to further control how Ribbon interacts with Kubernetes. In particular, if an endpoint defines multiple ports, the default behavior is to use the first one found. To select more specifically which port to use, in a multi-port service, use the PortName key. If you want to specify in which Kubernetes' namespace the target service should be looked up, use the KubernetesNamespace key, remembering in both instances to prefix these keys with your service name and ribbon prefix as specified above.

Examples that are using this module for ribbon discovery are:

• Spring Cloud Circuitbreaker and Ribbon
• fabric8-quickstarts - Spring Boot - Ribbon
• Kubeflix - LoanBroker - Bank

Remark : The Ribbon discovery client can be disabled by setting this key within the application properties file spring.cloud.kubernetes.ribbon.enabled=false.

Zipkin discovery in Kubernetes

Zipkin is a distributed tracing system which is supported by the project Spring Cloud Sleuth which allows to collect traces or spans from microservice applications.

A Discovery client has been implemented top of Kubernetes in order to fetch the Zipkin service (e.g. zipkin). This client is provided by the spring-cloud-kubernetes-zipkin project that you can use by adding this starter to your maven pom file:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-kubernetes-zipkin</artifactId>
    <version>${latest.version}</version>
</dependency>

This works as an extension of the spring-cloud-sleuth-zipkin project. The name of the Zipkin service and the target Kubernetes namespace/project where the service runs can be specified using the following application.properties properties:

spring.cloud.kubernetes.zipkin.discovery.serviceName=my-zipkin
spring.cloud.kubernetes.zipkin.discovery.serviceNamespace=tracing

By default, the discovery client will look for a Zipkin service named zipkin within the current namespace.

Examples of application that are using Zipkin discovery in Kubernetes:

• Spring Cloud Kubernetes and Zipkin
• fabric8-quickstarts - Spring Boot - Ribbon
• Kubeflix - LoanBroker - Bank

ConfigMap Archaius Bridge

The section ConfigMap PropertySource introduced how to configure a spring boot application via Kubernetes ConfigMap containing your configuration properties file.

If you prefer to use the configuration management library Netflix Archaius instead of using the Spring application.properties|"yaml file, then you can also leverage the ConfigMap feature by using the spring-cloud-kubernetes-archaius project.

To use it, add the following starter spring-cloud-starter-kubernetes-all to your pom file definition.

This module allows you to annotate your application with the @ArchaiusConfigMapSource and archaius will automatically use the Kubernetes configmap as a watched source (get notification on changes).

---

Troubleshooting

Version and compatibility

The current version of Spring Cloud Kubernetes is using version 2.2.x of the Fabric8 Kubernetes Client and is expected to work with version 1.x of Kubernetes and 1.x of Openshift. Note, that Kubernetes and Openshift are for the most part backwards compatible so, its expected that this framework is compatible with both the latest and earlier versions.

Namespace

Most of the components provided in this project need to know the namespace. For Kubernetes (1.3+) the namespace is made available to pod as part of the service account secret and automatically detected by the client. For earlier version it needs to be specified as an env var to the pod. A quick way to do this is:

  env:
  - name: "KUBERNETES_NAMESPACE"
    valueFrom:
      fieldRef:
        fieldPath: "metadata.namespace"

Service Account

For distros of Kubernetes that support more fine-grained role-based access within the cluster, you need to make sure a pod that runs with spring-cloud-kubernetes has access to the Kubernetes API. For example, OpenShift has very comprehensive security measures that are on by default (typically) in a shared cluster. For any service accounts you assign to a deployment/pod, you need to make sure it has the correct roles. For example, you can add cluster-reader permissions to your default service account depending on the project you're in:

oc policy add-role-to-user cluster-reader system:serviceaccount:<project/namespace>:default

Building

You can just use maven to build it from sources:

mvn clean install

Usage

The project provides a "starter" module, so you just need to add the following dependency in your project.

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-kubernetes</artifactId>
    <version>x.y.z</version>
</dependency>