Support repositories hosted outside of GitHub

[modocache]

It appears clib_package.c:clib_package_install only supports repositories hosted on GitHub. Users should have the option to install packages hosted on any Git server. What's the core team's opinion on the following syntax for package.json?

{
  "name": "my-package",
  "version": "0.1.0",
  "url": "https://github.com/me/my-package.git",
  "dependencies": {
    "my-friends-package": {
        "url": "https://my-friend@bitbucket.org/my-friend/my-friends-package.git",
        "version": "0.1.0"
    }
  },
  "install": "make install"
}

If a package declares both a url and a repo key, the url key is preferred and a warning should be displayed. If only a repo key is declared, then it's business as usual and the repo is assumed to be hosted on GitHub.

If my-friends-package uses a hash, as it does above, the url and version keys are required. If it is declared using the current syntax of "my-friends-package": "0.1.0", then it behaves as it does now. Therefore these syntax changes are backwards compatible.

If there's interest in such a feature I'd love to work on it.

[jwerle]

I think that would be an awesome feature. It appears we would need to make clib_package_url a bit more customizable.

See: https://github.com/stephenmathieson/clib-package/blob/master/src/clib-package.c#L473

cc @stephenmathieson

[stephenmathieson]

If the goal is BitBucket support (which I'm in support of, btw) I'd go about it a bit differently. Rather than providing "url" and "version", maybe something like:

  "dependencies": {
    "my-friend/my-friends-package": {
        "host": "bitbucket.org", // host defaults to raw.github.com
        "version": "0.1.0"
    },
  }

Then we could get file URLs with https://{host}/{owner}/{name}/{version}/{source file}.

[jwerle]

+1 I like that idea

[stephenmathieson]

err, actually, BitBucket raw URLs are https://{host}/{owner}/{name}/raw/{version}/{source file}.. we'll have to do this a bit differently :/

[modocache]

Beyond BitBucket, parsing arbitrary URLs would allow development teams to split up large, private libraries into smaller repositories. Since enterprise solutions from GitHub and BitBucket cost a lot of money compared to self-hosting, many of these teams use internal Git servers.

Supporting arbitrary URLs would also "future-proof" clib. Using the host approach, we'd have to add more code every time a new code hosting service opens its doors.

It might be a good idea to look at how go get is implemented--it allows for arbitrary URLs as well as a shorthand when using popular hosting services like GitHub and BitBucket.

[stephenmathieson]

This is true. We'd then either need libgit2 or to fetch a tar ball and extract certain files (like installing executables works currently).

I'll have to look at go get; I haven't investigated Go much.

[stephenmathieson]

What about:

"dependencies": {
  "my-friend/my-friends-package": {
      "url_format": "https://{owner}@bitbucket.org/{owner}/{name}/raw/0.1.0/{file}"
  }
}

And let url_format default to https://raw.github.com/{owner}/{name}/{version}/{file}?

This will allow any arbitrary URL format to be specified, thus future-proofing and removing the need for libgit2.

[modocache]

That seems fine to me. And since it's all JSON anyway we're free to change the structure in the future if need be.

For future reference, though, why not use libgit2? It seems better than rewriting git clone on our own. If we develop a way to specify a package.json for libraries outside of our control it'd also be a great opportunity to eat our own dogfood.

[stephenmathieson]

Well, we're not really cloning; we're just fetching the files listed in your package.json's src. If we were cloning, we may as well just use git submodules :p

[kellydunn]

What about:

"dependencies": {
"my-friend/my-friends-package": {
"url_format": "https://{owner}@bitbucket.org/{owner}/{name}/raw/0.1.0/{file}"
}
}
And let url_format default to https://raw.github.com/{owner}/{name}/{version}/{file}?

This will allow any arbitrary URL format to be specified, thus future-proofing and >removing the need for libgit2.

Specifying a format instead of just an arbitrary URL seems like an undesirable UX imo. When would a user want to specify a format over just copy and pasting the URL from their browser?

I'll also 👍 the comment for implementing a go get variant; while it's not the best for specifying versions, it is pretty handy in most use cases.