kubernetes_manifest resource warns OpenAPI schema not recognised and updates will force recreation
Closed this issue · 9 comments
Terraform Version, Provider Version and Kubernetes Version
Terraform v1.0.5
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.55.0
+ provider registry.terraform.io/hashicorp/helm v2.2.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.4.1
+ provider registry.terraform.io/hashicorp/vault v2.23.0
Affected Resource(s)
- kubernetes_manifest
Terraform Configuration Files
resource "kubernetes_manifest" "kafka_users" {
for_each = local.kafka_users
provider = kubernetes.kafka
manifest = {
"apiVersion" = "kafka.strimzi.io/v1beta1"
"kind" = "KafkaUser"
"metadata" = {
"labels" = {
"app" = "kafka-user"
}
"name" = each.key
"namespace" = "kafka"
}... It all checks out - just not the OpenAPI warning.
Debug Output
│ Warning: This custom resource does not have an associated OpenAPI schema.
│
│ with kubernetes_manifest.monese_dev_kafka_users["test-user"],
│ on new-kafka-users.tf line 50, in resource "kubernetes_manifest" "kafka_users":
│ 50: resource "kubernetes_manifest" "kafka_users" {
│
│ We could not find an OpenAPI schema for this custom resource. Updates to
│ this resource will cause a forced replacement.
│
│ (and one more similar warning elsewhere)
╵
Steps to Reproduce
terraform apply-->
Expected Behavior
No warning - and for the OpenApi schema defined in CRD to be recognised
Actual Behavior
Warning, and a resource that for me is not usable.
Important Factoids
- Issue was on previous "kubernetes-alpha" provider too.
- CRD created by Strimzi v0.22.0
- CRD uses OpenApi Schema
References
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
any news on this issue? we are facing the same problem with IstioOperator CRD.
@ls-sajad-sameti I'm pretty sure this is an issue with Istio not providing the Schema in the CRD correctly.
Unfortunately had no luck in asking the Istio team to add: istio/istio#33535
We experienced the same with Longhorn or it's CRD altough it is installed in our cluster we get this warning.
Anyone working on this issue and can share some status?
@scbay It's not about the CRD being missing from your cluster (if it was, you would not be able to create the object at all). You receive this warning because some CRD authors unfortunately leave the schema unspecified in the CRD definition (so it's basically just a freeform object), which prevents validation of the object you create with kubernetes_manifest and prevents proper diffing on changes (so the object has to be replaced whenever it is changed).
The "proper" solution is to ask your vendor to please fix their CRDs to follow best practices. Unfortunately many vendors can't or won't produce good quality CRDs.
If the warning is accompanied by some other problem (@Doug-North said "Actual behaviour: Warning, and a resource that for me is not usable" but didn't expand on what "not usable" meant), the other problem is likely unrelated to the warning.
I'd love if there was a way to silence the warning. We have some vendor CRDs which have this problem and the persistent warning is annoying. Something like kubectl_manifest's force_new = true so that you could explicitly opt-in to the replace-on-change behaviour which could also silence this warning.
I have the same problem with the eniconfigs.crd.k8s.amazonaws.com CRDs, terraform complains about This custom resource does not have an associated OpenAPI schema, but the CRDs seems to have an openAPIV3Schema see source code here.
# Source: aws-vpc-cni/templates/customresourcedefinition.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: eniconfigs.crd.k8s.amazonaws.com
labels:
app.kubernetes.io/name: aws-node
app.kubernetes.io/instance: aws-vpc-cni
k8s-app: aws-node
app.kubernetes.io/version: "v1.11.4"
spec:
scope: Cluster
group: crd.k8s.amazonaws.com
preserveUnknownFields: false
versions:
- name: v1alpha1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
x-kubernetes-preserve-unknown-fields: true
names:
plural: eniconfigs
singular: eniconfig
kind: ENIConfig
I'm not familiar with Open API Schema so I don't know what's wrong with this openAPIV3Schema, is terraform right or wrong when it says that there is no OpenAPI schema for this CRD?
There is no schema in that CRD.
The CRD has a '.spec.versions[0].schema.openAPIV3Schema', is not that the schema?
I mean the
schema:
openAPIV3Schema:
type: object
x-kubernetes-preserve-unknown-fields: true
seems like the one of the examples that they give in the Kubernetes Documentation for OpenAPI v3 validation schema.
I though that "openAPIV3Schema" meant that the value must be an object, and nothing else. Do you mean that is not right kind of schema?
Is there any good example of an actual CRD that can be managed with kubernetes_manifest so that I can look to the source code and see what is an actual valid "OpenAPI schema"?
This is not a schema. All it says is that the object is an object. All objects are objects! x-kubernetes-preserve-unknown-fields: true is a hack to avoid having to specify a schema (the fields of the object and their types).
Is there any good example of an actual CRD that can be managed with kubernetes_manifest so that I can look to the source code and see what is an actual valid "OpenAPI schema"?
Do note that you can manage this CRD with kubernetes_manifest. You can also manage the related custom resources with kubernetes_manifest, with the caveat that modifications to them will cause a delete-and-create rather than an in-place update.
Here's an example of the openAPIV3Schema for a CRD that has a schema: https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/install/v2_2_4_full.yaml#L37
As jbg explained, the CRD examples quoted here don't include valid schema information. What ecerulm quoted as schema is actually just the placeholder attribute for the schema, but it's contents are empty (x-kubernetes-preserve-unknown-fields: true is a wildcard to mark the absence of schema).
Whenever we encounter this kind of CRDs, we have no other choice but to force re-creation on updates since we cannot guarantee the consistency of the resource schema across updates, which is a hard requirement by Terraform (checked and enforced after each apply).
I'm going to close this issue, but if anyone thinks the error message is not descriptive enough, please speak up.