[Sandbox] Apicurio Registry
carlesarnal opened this issue · 9 comments
Application contact emails
Project Summary
Apicurio Registry is a runtime server system that stores a specific set of artifacts (primarily schemas and API designs) as files and manages evolution of artifact versions. Apicurio Registry enables you to add, update, and remove the artifacts from the store using a REST API or a Web UI.
Project Description
API Registry
The key use case for the Apicurio community is as an API registry, which is a system that enables you to register your designs (OpenAPI, AsyncAPI, GraphQL, Apache Avro, Google Protocol Buffers, JSON Schema, Kafka Connect schema, WSDL and XML Schema (XSD)) so that potential API consumers can do the following:
- Browse, update, or search the list of APIs available
- Configure, and execute artifact rules
- View details about a single available API
- Learn how to use an API using auto-generated documentation
- Generate client SDKs for an API
Schema Registry
In the messaging and event streaming world, data published to topics and queues often must be serialized or validated using a schema (OpenAPI, AsyncAPI, GraphQL, Apache Avro, Google Protocol Buffers, JSON Schema, Kafka Connect schema, WSDL and XML Schema (XSD)). Schemas can be packaged in each application, but it is often a better architectural pattern to register schemas in an external system and then reference them from each application. A schema registry provides the following benefits:
- Schemas do not have to be bundled together with consumer or producer applications
- Potential to upgrade producers without upgrading consumers thanks to the configuration and execution of both validity and compatibility rules for schemas
- Central location to track all schemas used in a specific environment
- Document the data format required for each use case
- Centralized control of data format and schema evolution
- Enabled the automation of upgrade procedures such as canary/blue-green
Org repo URL (provide if all repos under the org are in scope of the application)
N/A
Project repo URL in scope of application
https://github.com/Apicurio/apicurio-registry
Additional repos in scope of the application
No response
Website URL
https://www.apicur.io/registry
Roadmap
https://github.com/orgs/Apicurio/projects/6
Roadmap context
Over time, all applications face organic growth. We’re no different in that sense, and we’re planning a new major release for the project. For example, the project supports different storage options, like storing the application data in a Kafka topic or storing the information in a relational database. Due to some technical constraints, in the past, we had to release a new container image per each storage option (even for separate database drivers) but these constraints have been removed and, with the new major version, we will be releasing just one container image for all the storage options.
In addition to addressing some technical limitations we also started adding some new major features, like a GitOps implementation that can be used to store the application data.
We have also introduced a new client library for the application that relies on Kiota, an exciting technology developed by Microsoft as an open source tool that allows us to generate client SDKs from OpenAPI definitions.
Contributing Guide
https://github.com/Apicurio/apicurio-registry/blob/main/CONTRIBUTING.md
Code of Conduct (CoC)
https://github.com/Apicurio/apicurio-registry/blob/main/CODE_OF_CONDUCT.md
Adopters
N/A
Contributing or Sponsoring Org
Maintainers file
https://github.com/Apicurio/apicurio-registry/blob/main/pom.xml#L38-L70
IP Policy
- If the project is accepted, I agree the project will follow the CNCF IP Policy
Trademark and accounts
- If the project is accepted, I agree to donate all project trademarks and accounts to the CNCF
Why CNCF?
Community and Ecosystem Support.
Neutral Governance.
Increased Visibility.
Interoperability and Standards.
Marketing and Recognition.
Graduation Path.
Legal and Intellectual Property Support.
Benefit to the Landscape
This question can be resolved by mentioning the multiple reasons why a project like Apicurio Registry is a good “Cloud Native” fit, like the ones described below. Another big benefit to the landscape is the close integration with an existing project like Strimzi.
Cloud Native 'Fit'
A schema registry like Apicurio Registry is considered a good fit for cloud-native architectures for several reasons:
- Decentralized Microservices:
- In cloud-native environments, applications are often built as microservices. Each microservice may use different data formats and schemas. A schema registry helps in maintaining a centralized repository for schemas, making it easier for microservices to understand and communicate with each other.
- Dynamic Scaling:
- Cloud-native applications often need to scale dynamically based on demand. As the number of services or instances increases or decreases, having a centralized schema registry ensures that all services can quickly adapt to changes in the schema without requiring extensive coordination.
- Compatibility and Versioning:
- Microservices are developed and deployed independently, and they may evolve at different rates. A schema registry enables versioning and backward/forward compatibility checks, ensuring that newer versions of services can communicate with older versions without causing data mismatches or errors.
- Contract Testing:
- Cloud-native applications benefit from contract testing, where each service's compatibility is verified against the contracts defined in the schema registry. This ensures that changes to a service don't break the contracts established with other services.
- Schema Evolution:
- Cloud-native systems often require the ability to evolve schemas over time without disrupting existing services. A schema registry facilitates controlled schema evolution, allowing for the introduction of new fields or changes while ensuring backward compatibility.
- Interoperability:
- In multi-cloud or hybrid cloud environments, where different services may be deployed on different platforms, a schema registry promotes interoperability by providing a standardized way for services to share and understand data formats.
- Real-time Data Governance:
- A schema registry provides a central point for managing and governing data schemas. This is crucial for maintaining data quality, ensuring consistency across services, and enforcing data governance policies in real-time.
- Tooling Integration:
- Cloud-native ecosystems often come with a variety of tools for monitoring, logging, and debugging. A schema registry can integrate with these tools, providing insights into data changes and ensuring that the data flowing through the system adheres to predefined schemas.
- Security and Access Control:
- Schema registries typically offer features for managing access control and security. In a cloud-native environment, where security is a top concern, having centralized control over who can access and modify schemas is crucial for maintaining the integrity of the data.
In summary, a schema registry like Apicurio Registry is well-suited for cloud-native architectures because it addresses the challenges associated with the dynamic, decentralized, and scalable nature of microservices-based applications, providing a robust foundation for data interoperability and evolution.
Cloud Native 'Integration'
Apicurio Registry has a “native” clear integration with Strimzi in various ways.
- Apicurio Registry can be configured on the Strimzi Kafka cluster as the Schema Registry. A common integration would look as follows:
- Deploy Apicurio Registry:
- Deploy Apicurio Registry on your Kubernetes cluster.
- Configure Strimzi:
- Configure your Strimzi Kafka cluster to use Apicurio Registry as the schema registry for your Kafka topics. This involves updating your Kafka custom resource (Kafka CR) or KafkaConnect custom resource, depending on your specific use case.
- Update Producer and Consumer Configurations:
- Update your Kafka producer and consumer configurations to point to Apicurio Registry for schema management. This includes specifying the Apicurio Registry URL and any other required configurations.
- Schema Registration:
- When producing messages to Kafka, ensure that the Avro or other schema is registered with Apicurio Registry first. This might involve making API calls to the Apicurio Registry or using client libraries that handle schema registration.
- Using the Kafka Cluster managed and created by Strimzi as the persistent storage for Apicurio Registry. Essentially, with this configuration, Apicurio Registry has an internal database that is used for reads and writes that are made to a Kafka topic and synchronized back into the internal database.
- In a microservices system, Apicurio Registry can be deployed in the target environments maintaining the source of truth of the API contracts produced and consumed by the services. Making it easy and enabling the automation of seamless upgrades.
Cloud Native Overlap
This project does not have an overlap with any other CNCF project.
Similar projects
https://github.com/confluentinc/schema-registry
Landscape
https://landscape.cncf.io/?selected=apicurio-registry
Business Product or Service to Project separation
Even though there is a Red Hat product based on Apicurio Registry, we acknowledge the importance of maintaining independence and transparency. Our approach involves emphasizing the clear distinction between the upstream project and downstream products or services.
The project itself, referred to as the upstream, is maintained as an open-source endeavor with a distinct identity. We commit to keeping it separate from any specific Red Hat product or service, ensuring its autonomy.
Any downstream products or services developed by Red Hat based on the project will have their own organizational structure and development processes. We recognize the significance of avoiding a direct overlap between the project and any commercial offerings.
While Red Hat is currently a significant contributor and maintainer, we recognize the importance of ensuring the project's independence.
We actively encourage and welcome contributions from external contributors. We understand the need for diverse perspectives and are committed to fostering an open environment where regular contributors can become maintainers, ensuring a broader range of opinions and contributions.
Project presentations
No
Project champions
No response
Additional information
No response
Couple of questions:
- Does Apcurio support the CloudEvents standard for messages?
- Are there ways to use Apcurio with the Kubernetes API?
Couple of questions:
- Does Apcurio support the CloudEvents standard for messages?
Yes, we currently support firing events the the CloudEvents format for the different operations happening in Apicurio Registry to both and HTTP sink or to Kafka . Aditionally, we did some PoCs in the past around the standard, like this one.
- Are there ways to use Apcurio with the Kubernetes API?
No, we do have a Kubernetes operator to manage Apicurio Registry, but the artifacts managed by Registry cannot be managed through the Kubernetes API.
@carlesarnal could you and the Apicurio Registry team come present to TAG App Delivery at our general meeting on December 20, 2023?
@carlesarnal could you and the Apicurio Registry team come present to TAG App Delivery at our general meeting on December 20, 2023?
Hi @angellk, yes! I have no problem presenting next week.
Thanks
- Are there ways to use Apcurio with the Kubernetes API?
No, we do have a Kubernetes operator to manage Apicurio Registry, but the artifacts managed by Registry cannot be managed through the Kubernetes API.
I was asking about the other way around: can I use Apicurio to manage the Kubernetes API?
Hello @carlesarnal! I am reaching out on behalf of TAG Runtime. It would be great to have you present Apicurio at our bi-weekly meeting to understand the project roadmap.
The next occurrence of the meeting is on 4th Jan 2024. Let me know if that works for you else we can find some other time!
Hello @rajaskakodkar,
Based on some internal conversations we've been having, I think we should postpone the Apicurio presentation until we can get clarification on some key questions.
I'll write here again once we're ready to present.
Thanks!
Hello all,
For now, we will defer contributing to the project, since we have some branding issues that have to be resolved before contributing it. That said, we might resume the activity in the future whenever we have a more mature version of some other parts of our projects. In any case, for now, we have engaged with the Discovery API working group to get engaged with https://github.com/xregistry initiative, which might create a more organic opportunity for the contribution.
Thanks everyone for your time. For now, I'll close this issue.
- Are there ways to use Apcurio with the Kubernetes API?
No, we do have a Kubernetes operator to manage Apicurio Registry, but the artifacts managed by Registry cannot be managed through the Kubernetes API.
I was asking about the other way around: can I use Apicurio to manage the Kubernetes API?
We have a Kubernetes Operator to manage the application deployment, but nothing beyond that.