rudderlabs/config-generator

Add deprecation notice to this repo

gitcommitshow opened this issue · 15 comments

gitcommitshow commented

This open-source control plane repo hasn't been updated for a long time. While the hosted control plane has been moving forward with new updates. As this is not actively maintained and users might face trouble if used with the latest version of the data plane, we will mark this repo as deprecated.

Next steps

  • Announce the deprecation on GH and slack community
  • Gather community feedback
  • Add DEPRECATED notice on README
gitcommitshow commented

We don't have the bandwidth to support the open-source control plane. If anyone from the community wants to pick it up, we are happy to provide help e.g. providing the templates for the configs expected by the data plane.

gitcommitshow commented

tl;dr: don't use this repo, it is highly likely to break. Use RudderStack-hosted free Control Plane instead.

jbergstroem commented

It would be great if the Integration guide was updated to reflect this! https://www.rudderstack.com/docs/user-guides/how-to-guides/how-to-submit-an-integration-pull-request/

gitcommitshow commented

Thanks for reminding @jbergstroem
Will take care of it

mderazon commented

tl;dr: don't use this repo, it is highly likely to break. Use RudderStack-hosted free Control Plane instead.

Are there any limitations on the rudderstack-hoster free control plane ? Like amount of events / destinations / transformers (for people who self host Rudder).

Also, is there an export / import functionality for the hosted control plan to export the config to yaml that can be loaded ?

Thanks

gitcommitshow commented

Are there any limitations on the rudderstack-hoster free control plane ? Like amount of events / destinations / transformers (for people who self host Rudder)?

For self-hosted RudderStack data plane

  • There are no limitations on amount of events or destinations
  • Transformations are available only in the RudderStack-hosted control plane and Open Source users can set up to 5 transformations, same as free RudderStack account. Open Source users can setup transformations only in Cloud mode, not in device or hybrid mode.

Here's a list of all the features available in RudderStack Open Source. And a quick comparison of RudderStack Open Source vs Cloud data plane is here

Is there an export / import functionality for the hosted control plan to export the config to yaml that can be loaded?

Not yet. You can use workspaceToken to get configurations from hosted control plane
Is there any specific reason you need export/import functionality which is not possible usingworkspaceToken?

mderazon commented

Not yet. You can use workspaceToken to get configurations from hosted control plane Is there any specific reason you need export/import functionality which is not possible usingworkspaceToken?

We are using Rudderstack self-hosted with config-generator today. As you said, it is outdated and will probably break in future versions.

If I can use the rudderstack-hosted control plane to create my configuration and then export it and load it to my self hosted rudderstack instance, then it's basically a replacement for config-generator.

Some reasons some companies will opt to not use rudderstack-hosted control plane would be:

  1. Having your service communicate with an external service is not always an option for some companies
  2. In terms of reliability - Having an important piece of your infra depend on 3rd party availability could be a problem. What happens when Rudder's control plane goes down ? Self hosted Rudder services still work ?

You can use workspaceToken to get configurations from hosted control plane

You mean get configuration via an API ?

gitcommitshow commented

You mean get configuration via an API?

The API is already integrated with RudderStack Open Source Data Plane so all you need to do is to set workspaceToken config in your Rudderstack Open Source Data Plane. And the configurations will be updated in your data plane after every time you update them in your RudderStack-hosted control plane.

If you do not want to use this workspaceToken config, and want to fetch configurations manually, you can use API to fetch all of your configurations (from your RudderStack-hosted control plane account). To do this, you'll be using workspaceToken as username for a basic authentication mechanism. Having said that, workspaceToken config method is more future-proof and stable method to fetch the configurations than using the APIs, hence recommended. This is also why the team feels that export feature is redundant but I'd love to make a case for the same if we can present some strong reasoning. Regarding import feature, I can understand it can be an inconvenience to recreate all of your configurations from config-generator to RudderStack-hosted control plane, but this is a one time effort, the team to help you migrate. Please post in slack community if you need help with that.

In terms of reliability - Having an important piece of your infra depend on 3rd party availability could be a problem. What happens when Rudder's control plane goes down ? Self hosted Rudder services still work ?

Self-hosted rudder-server will still work when Rudderstack control plane is down and you had fetched the configurations at least once, anytime before control plane went down. If you had used rudder-server once with RudderStack control plane before the incident, your rudder-server did fetch those configurations.

Having your service communicate with an external service is not always an option for some companies

I am keen to understand those scenarios/use-cases so I can make a case to the team to invest resources on config-generator. There have been comments about GDPR earlier but that can be covered with existing configs that make sure that no data goes to RudderStack control plane from your self-hosted data plane.

Also notice that the config-generator repo is still available and open for community to take a lead on that project. Until now, all the OSS users I talked to, were able to replace config-generator with RudderStack-hosted control plane without any issues and without compromising on any value that config-generator was providing.

Feel free to comment your further thoughts, I am keen to understand

mderazon commented

@gitcommitshow thank you for the detailed answer, it's more clear to me now.

Could you expand a little about this line ?

There have been comments about GDPR earlier but that can be covered with existing configs that make sure that no data goes to RudderStack control plane from your self-hosted data plane.

What part of the config makes events data flow through rudder or not ? According to my understanding, only if you turn on even stream on the control plane would data flow, right ?

Even if data doesn't flow to the hosted control plane, Rudderstack still has all the api keys / secrets etc that are stored for each source / destination, for some companies that is a problem - Not saying it's a deal breaker, just trying to present the other side / how companies see it as a risk

Last question, is the Rudderstack-hosted control plane open source ? If so, might be up to contributing an export feature ourselves (although you made compelling argumemnts it's not so much needed)

gitcommitshow commented

Only case where data flows from the data plane to control plane is for "live events debugging feature" that helps debugging event failures via control plane ui. This feature (and so the data) can be disabled by setting following configs

 RSERVER_SOURCE_DEBUGGER_DISABLE_EVENT_UPLOADS=false
 RSERVER_DESTINATION_DEBUGGER_DISABLE_EVENT_DELIVERY_STATUS_UPLOADS=false
 RSERVER_TRANSFORMATION_DEBUGGER_DISABLE_TRANSFORMATION_STATUS_UPLOADS=false

Thanks for sharing the details. Can understand the concern now. RudderStack-hosted control plane is not open source. config-generator was the only open source control plane we had. If any community member wants to contribute in building OSS control plane or their own private control plane with the features of their choice, config-generator is the best place to get started. RudderStack team will be happy to support. Do check out #open-source channel on slack community, I will share some resources there to facilitate the discussion and support on the topic.

  • 👍2
iamkhalidbashir commented

It seems rudderstack has hijacked open source name in it's product to give illussion of open source product but in reality is the closed source solution

gitcommitshow commented

Thank you for the feedback.
This project config-generator is Open Source. And we decided to not maintain this project anymore as with our growing number of Open Source projects and limited bandwidth, we needed to focus our energy on the most critical projects to serve the need of building a Customer Data Platform for our Open Source product users. These other critical projects include 48+ SDKs and core Data Plane server.
If there's anything that stops you from using RudderStack because of our choice of Open Sourcs projects, I'm here to figure out a solution. Can you please share your use case and what's stopping you to use RudderStack for that use case.

gitcommitshow commented

I got the numbers of Open Source projects we maintain (I was curious and wanted to speak precise numbers)

  • RudderStack team has created 172 public repositories until now
  • More than 88 projects are Open Source and active in 2023 (pushed code)
  • 48/57 Open Source SDKs are active in 2023

Hope this helps you understand why we need to decide the priority of essential Open Source vs less essential Open Source project to maintain

borodiliz commented

@gitcommitshow Firstly, thank you for maintaining these open-source projects. They are incredibly helpful for projects where relying on cloud services is not feasible due to regulatory or security constraints. I really understand the deprecation of this project, and that you need to prioritise what is putting food on your table.

I have some questions regarding other components, like the Android and iOS SDKs, and their dependency on the controlPlaneUrl. What is the plan for them? If they require Control Plane Lite, but the latest rudder-server does not support the format provided by Control Plane Lite... is the only solution to use your RudderStack-hosted free Control Plane?

It would be great to clarify this moving forward to avoid confusion, either by:

  • Clearly stating that this requires a RudderStack-hosted free Control Plane account.
  • Documenting the expected input for each module. For example, for the rudder-server, the JSON format expected in RSERVER_BACKEND_CONFIG_CONFIG_JSONPATH, and for the Android and iOS SDKs, the expected response(s) from controlPlaneUrl.

Many thanks for your time.

gitcommitshow commented

Sincere thank you for the feedback and useful suggestions @borodiliz

Clarifying your questions

  • Yes, RudderStack-hosted free Control Plane is the recommended approach at the moment. Other alternatives/workarounds are not worth the efforts. Alternatives such as - forking control plane lite and adjusting the code to the latest config structure, download the configs from RudderStack-hosted control plane via http request and use them in rudder-server, etc.
  • Documenting the JSON format for configs is a great suggestion, thanks for bringing this up. Let me start a discussion within team on how can we enable this.