/mvp-cms

mvp of a conncetor metadata service

Primary LanguageJavaScript

Airbyte Connector Metadata Service

There are 3 sources of truth as far as connector metadata right now:

  1. actual connector source code in the github repo
  2. the manually maintained definitions files
  3. for cloud connectors, the cloud catalog

Our job is to unify them and serve them quickly from a simple RESTful API that is easy to edit, for consumption from docs, UI, and whatever else.

  • For most people, this is a read-only endeavor which means we can make use of simple infrastructure.
  • For connector maintainers, we need a simple and fast way to update connector metadata for consumption despite not having one source of truth

This means we need a data pipeline from:

  1. 3 sources of truth (external)
  2. manually maintained extra info (maintained here)
  3. combine 1 and 2 into a results.json and serve it from this app/site

We have NOT implemented this pipeline yet - we are hoping to agree on the desired output with this MVP and get an owner for this before working backwards on the pipeline.

Endpoints

Endpoints want to offer:

  • GET /connectors
    • /connectors - the smallest payload of connectors, minus internal connectors
      • listing ['name', 'dateAdded', 'displayName', 'description', 'websiteUrl', 'documentationUrl', 'iconUrl', 'releaseStage', 'connectorType']
    • /connectors?internal - small payload of all connectors including internal connectors
    • /connectors?full - just dumping the full results.json, which includes extra stuff like docker name, version, sourceDefinitionId, etc
  • GET /connector/{connector_name}
    • return full info for a single connector
    • 404 if unrecognized name

Future

in future we maybe want to offer:

  • search
  • update (for now we choose to just update by updating the results.json file, with possibly a google spreadsheet in the loop)