koltyakov/cq-source-sharepoint

Publish to CloudQuery Hub

Closed this issue · 13 comments

candiduslynx commented

Announcement of CloudQuery Hub points out to a new centralized way for publishing plugins.

As #22 is complete, packaging & publishing to Hub becomes available.

To verify the plugin can be packaged, go run main.go package --docs-dir docsdir -m 'test' v1.0.0 . should run OK.

koltyakov commented

Thank you @candiduslynx! I've merged the PR, yet have minor questions.

go run main.go package --docs-dir docs -m @CHANGELOG.md $(git describe --tags --abbrev=0) .
Building for OS: linux, ARCH: amd64
Building for OS: linux, ARCH: arm64
Building for OS: windows, ARCH: amd64
Building for OS: darwin, ARCH: amd64
Building for OS: darwin, ARCH: arm64
Error: failed to copy docs: open docs: no such file or directory
Usage:
  sharepoint package -m <message> <version> <plugin_directory> [flags]

Flags:
  -D, --dist-dir string   dist directory to output the built plugin. (default: <plugin_directory>/dist)
      --docs-dir string   docs directory containing markdown files to copy to the dist directory. (default: <plugin_directory>/docs)
  -h, --help              help for package
  -m, --message string    message that summarizes what is new or changed in this version. Use @<file> to read from file. Supports markdown.

2023/10/31 13:32:07 failed to serve plugin: failed to copy docs: open docs: no such file or directory
exit status 1

What should be in docs? I mean, is there any requirements for .md files, naming convention, etc?

Should not the command be a part of releasing pipeline?

candiduslynx commented

@koltyakov Yes, the docs dis should be a dir with .md documentation for the plugin. We use the files available in our monorepo for this.

See publish_plugin_to_hub action in the CQ monorepo, namely this step.

You will need to create an account + team in the Hub & then add the API token to GitHub secrets to be used in the release/publish workflow.

  • 👍1
candiduslynx commented

You could create a docs dir & add the documentation regarding the plugin configuration, for example.

Useful baseline examples can be found in our monorepo.

koltyakov commented

@candiduslynx,

Which .md will be an entry point? In the CQ monorepo I can see docs/tables/README.md and many configuration moved to instructions.

Can I use images in the documentation? E.g. in current README.md, which hosts existing user documentation, I'm using animated images to highlight specific steps. If assets are supported, should the they be located within same docs directory?

erezrokah commented

Hi @koltyakov 👋 We'll try to get proper docs for publishing later on this week, but the conventions for the docs directory are:

  1. It should have markdown files (ending with .md)
  2. If there's an overview.md file it will show up first (I'm assuming that's what you mean by entry point)
  3. Any other non overview.md files will be sorted alphabetically.
  4. The name of the doc file will be title cased for display purposes (for example see https://hub.cloudquery.io/plugins/source/cloudquery/aws/v22.18.0/docs vs https://github.com/cloudquery/cloudquery/tree/151769e7c02028a04ef0ed280951c000ebb1f9c2/website/pages/docs/plugins/sources/aws)
    image
  5. Non markdown files will not be uploaded to the Hub, so in order to support assets you would need to use absolute links. For example replace ./assets/sp.svg with https://raw.githubusercontent.com/koltyakov/cq-source-sharepoint/main/assets/sp.svg
  6. I noticed you're using HTML tags in some of your docs. That's not supported at the moment so might require some modifications
  7. You might notice that in the AWS docs there are some files starting with _. Those are uploaded but used as placeholders to inject blocks into other files, for example:
    https://github.com/cloudquery/cloudquery/blob/151769e7c02028a04ef0ed280951c000ebb1f9c2/website/pages/docs/plugins/sources/aws/overview.md?plain=1#L16, will render the content of _authentication.md instead of the placeholder. This syntax is intended for internal usage while we support both docs sites, https://www.cloudquery.io/ and https://hub.cloudquery.io/

I'll raise the following issues internally:

  1. Support relative assets in markdown docs
  2. Support HTML tags in markdown docs (will need to be done in a secure way)
  • 👍2
koltyakov commented

Thank you @erezrokah! This is super helpful info.

I'll craft some docs accordingly. No problem for me to make sure there are no HTML fragments.

One more question, will the service receive updates? E.g. could I send docs update for the same version of the plugin.

erezrokah commented

One more question, will the service receive updates? E.g. could I send docs update for the same version of the plugin.

As long as the version is in draft everything is mutable (docs, assets and tables). Once the version is finalized it becomes immutable so you won't be able to update docs, assets or tables. This is similar to other registries like https://www.npmjs.com/. Finalized versions can't be deleted as well, and can only be redacted (similar to deprecated in npm). See reasoning in https://blog.npmjs.org/post/141905368000/changes-to-npms-unpublish-policy

So to update the docs you would need to publish a new version.
Once the plugin is packaged via go run main.go package ... you can run the CLI publish command to publish it to the Hub (you should be logged in first via cloudquery login). By default cloudquery publish creates a draft version, and you can pass -f to finalize it.

erezrokah commented

Hi @koltyakov, we created some docs on this topic in https://www.cloudquery.io/docs/developers/publishing-a-plugin-to-the-hub.

Please let us know if you have any questions or something is missing

  • 👍1
candiduslynx commented

Hi @koltyakov!

I can see https://hub.cloudquery.io/plugins/source/koltyakov/sharepoint/v2.0.0/docs?authors=community but with no docs. Do you need any assistance to fix that?

koltyakov commented

Hi @candiduslynx,

I have not published the plugin to the new hub yet.

I will need assistance at least in the part how to get hub's account. Not sure I have one.

candiduslynx commented

Could you provide me with an email to add you to the koltyakov team at https://cloud.cloudquery.io (& transfer the ownership there)? You can send it to me via either official Discord server (user: alex_cq) or to alex at cloudquery.io.

We already created the team for you and published an initial plugin version so that the users will be able to use either Cloudquery Hub or GitHub after Hub announcement. Hope it's OK.

  • 👍1
candiduslynx commented

@koltyakov I've replied via email & also the invitation to the proper team was sent. Ping me if any additional assistance is required.