Support private repos

[fregante]

1. token instructions don't include private repos
2. if enabled, files are still fetched without token, so they 404

[loilo]

Side note: I've taken a look on this, but this is currently technically impossible due to raw.githubusercontent.com returning 403 answers for OPTIONS requests. However, those requests are inevitably made by the browser as preflight requests when making authorized calls with a bearer token attached.

This problem has been reported to GitHub relatively recently and at least they are aware.

However, until this changes, it will not be possible to do a purely client-side XHR request to private resources.

[loilo]

That said, file contents could still be fetched via the GitHub API.

@fregante What would be your shot at this? Would you wait for GitHub to change their CORS behavior or make requests to the GitHub API? I'd happily implement anything. 😄

[fregante]

That said, file contents could still be fetched via the GitHub API.

The GitHub API is limited (filesize) so this change would break downloading some larger files.

impossible due to raw.githubusercontent.com returning 403 answers for OPTIONS requests

When does this happen? How are you passing the authorization? Pre-flight doesn't happen if we use simple requests

[loilo]

Pre-flight does happen though if we're trying to authenticate with a bearer token (Authorization header → not a simple request). This seems to be the only working authentication mechanism for our case.

When private files are shown in the browser, GitHub appends a token GET parameter, however this does not seem to use regular access tokens — at least I'm getting 404 errors when I'm trying to use my access token in that place. (The token itself is fine, I can use it to e.g. grab the file when using it as a bearer token.)

[fregante]

This seems to be the only working authentication mechanism for our case.

Yeah, HTTP authentication via URL and XMLHttpRequest.open doesn't seem to be supported anymore. curl https://token@raw.etc/file.ext works but not in the browser.

[loilo]

Yep, those are the approaches I've tried as well. Seems that we're stuck with waiting for GitHub on this one.

[fregante]

What do you think about implementing the API as a fallback for private repos? I think it'd have to check whether all the files are under 1MB as well; thankfully this info is available in the list API call.

[loilo]

Sounds good to me. Going to take a look soon — got another PR regarding error handling to land first (i.e. in a few seconds, be prepared 😁).

[loilo]

We can use GitHub's blob API to fetch files up to 100 MB. However, we need the files' sha hashes for that. list-github-dir-content actually fetches that data but does only return the file names.

To avoid redundant requests: Would you copy the list functionality over to this project or extend the list package with an option to include file details in the returned array?

[fregante]

list-github-dir-content exists only because of download-directory.github.io. Since it doesn't have tests, download-directory.github.io serves as a "live test."

In short, I'd extend it rather than copy it here.

---

If you want to send a PR there, we could add a flag to get the whole file object instead of just the path

- files.push(item.path);
+ if (getWholeData) {
+ 	files.push(item);
+ } else {
+ 	files.push(item.path);
+ }

Thoughts?

---

With a new flag however I'd suggest changing its API(s) to:

viaTreesApi({
	user: '',
	repository: '',
	ref: '',
	directory: '',
	token: '',

	getWholeData: false // new, name TBD
})

[loilo]

Exactly how I'd have done it, nice. Going to submit a PR. Suggested names: detailed, withDetails, includeDetails, withMetadata, includeMetadata?

[loilo]

I don't like either of my suggestions. detailed is not self-explaining enough IMO and all of the others indicate that metadata is sent alongside the file names, not in their place.