external-secrets/kubernetes-external-secrets

Consider merging with aws-secret-operator

max-lobur opened this issue · 19 comments

max-lobur commented

https://github.com/mumoshu/aws-secret-operator is a very similar concept, consider merging two projects.

silasbw commented

Doh -- just when you think you've scoured the internet. Thanks @max-lobur. Yes, we agree, we should definitely consider merging.

  • 👍3
  • 😄2
  • 🚀1
JacopoDaeli commented

Ok ok, but how should we deal with it? What would be the right approach here for considering to merge?

reegnz commented

I believe the right approach would be to reach out to these projects and ask them if they want to unify their efforts in a single project.
Maybe open github issues in each of those projects that links to this issue to start a discussion?

Also given that all of those are written in go, but the godaddy solution is written in JavaScript might proove difficult in merging. ;)

Smallest common goal IMHO would be to could come up with a standard CRD that each of these projects implement as a provider - similar to how storageclass does it.

  • 👍1
riccardomc commented

Hi there, and thank you for reaching out!

I am one of the contributors of https://github.com/ContainerSolutions/externalsecret-operator and I think that we would definitely benefit from aligning on a common CRD spec.

We deliberately wanted to keep it simple, so our CRs currently look like this:

apiVersion: externalsecret-operator.container-solutions.com/v1alpha1
kind: ExternalSecret
metadata:
  name: example-externalsecret
spec:
  Key: example-externalsecret-key
  Backend: asm-example

Our project supports different backends to store secret and that what the Backend key is used for.

What are your ideas on how to unify the spec?

reegnz commented

I played around with the idea of doing something similar to what you get with StorageClass, PersistentVolumeClaim and PersistentVolume.

I wrote the idea down in a gist to start a discussion about that (maybe it's overcomplicated):
https://gist.github.com/reegnz/acc1f97fe3e78746394103e9c96d9c76

riccardomc commented

I like the idea of using annotations.

We had a very similar implementation to support multiple backends on a single operator deployment: ContainerSolutions/externalsecret-operator#6 but didn't consider using annotations.

We decided to simplify this to one operator deployment, one backend. I believe it is easier to reason about and demand namespace isolation to RBACs. See:

A few questions:

  • How would you handle multiple deployments of the same operator with distinct backends?
  • How do you handle configuration of secret backends? Rely entirely on IAM and AWS SDK credentials discovery?
chtompki commented

Note that this project holds the most stars of any of the aforementioned projects, FWIW.

ecout commented

Given my requirements this is the one I was looking for:
https://github.com/mumoshu/aws-secret-operator
I need the Operator to be able to automatically refresh, otherwise there's no real data decoupling and might as well do it with regular secrets and KMS.

Flydiverny commented

@ecout what do you mean by automatically refresh? KES does polling by configurable interval and refreshes the kubernetes secrets with the updated values from the backend.

ecout commented

@Flydiverny Yes, I just tested this one today. It generates VERY verbose logs btw. Also, regarding this issue, why merge two completely different code bases written in different languages.
Please simply UPDATE the READ.me to explicitly say your project supports it.
Thanks!

moolen commented

Hej everyone, i opened #477 with a first draft to continue the discussion on the CRD topic.

Let's assume we've found a common CRD, what's the next step then? Actual git-merging is obviously not an option, do we want to:
A) rewrite this project to keep the precious GitHub stars? ⭐ :trollface:
B) found a new github organization (to be independent from godaddy / or single-person maintainers)?
C) other ideas?

technology-wise i'd strongly recommend using golang for a rewrite due to the 1st class k8s-sdk support, and tooling w/ operator-sdk or kubebuilder.

What do you think?

  • ❤️1
riccardomc commented

First of all, thanks for keeping this effort alive. I even tried to get things rolling together with @slamdev, but other business got in the way. I truly believe there's space to build something cool for the Kubernetes community. Channeling our efforts in one common direction has the potential to have a great tool for secret management and achieve ultimate opensource glory :trollface: .

I believe B is the best option. I also think golang is the right choice.

For the github stars, I think if we play it right we can achieve the same level of popularity in a very short time.

The real difficult part is how can we coordinate our efforts in order to make this happen?

dirtycajunrice commented

I would love for yall to jump in and help with https://github.com/itscontained/secret-manager. The goal is to do exactly what you have stated with a single CRD. itscontained is not a company, simply an org name i thought was fitting of the ecosystem (docker container based) so there are no ties, and we can all share in management of projects.

  • 🚀1
SayakMukhopadhyay commented

As someone just starting out to use KES along with our GitOps setup, I would love to this project thrive. KES has a better idea than Sealed Secrets, which is perhaps the most popular way out there to use secrets along with GitOps, since using managed k8s by cloud providers are becoming the norm and it takes away the human element present in sealed secrets.

Saying that, I want to provide some of my opinions, FWIW:

  1. I would prefer any project that comes out of the merge to work with multiple cloud providers. Right now, KES is the only one which supports the big 3 (AWS, GCP, Azure).
  2. If a plugin approach is chosen, consider creating the major providers in the same org. People wishing to maintain a plugin can be given repo access. This will help is having a single place to manage and would also help in creating docs.
  3. I would love the name of the projects that comes out of the merger to be cloud agnostic with reg. to point 1.

Also, we should have a single issue to have this conversation, as a related conversation is also taking place in #423

SayakMukhopadhyay commented

@moolen @riccardomc Transferring an existing repo to another account can transfer over the stars too. https://docs.github.com/en/github/administering-a-repository/transferring-a-repository#whats-transferred-with-a-repository . If we decide on a new org to have this project, I would suggest transferring this repo over to that account and then rework this repo itself if a conversion to go is planned.

  • 👍4
moolen commented

I think the CRD Spec in #477 is good enough to be implemented in a PoC, let's have a meeting to discuss futher steps on how to channel our efforts, find and organization, move code etc etc.

I made a agenda here for Community Meeting #1 ✨ / https://hackmd.io/utPLB9nkSmCmfCmmzOkQgQ?view
Please add a comment or request write access to add topics to the agenda.

I'll drop a message in slack to find a good date and update this comment.

  • 👍2
Flydiverny commented

Closing this as there are undergoing efforts in https://github.com/external-secrets/external-secrets :)