Support git+http

[rndmh3ro]

Is your feature request related to a problem? Please describe.
I have a private gitlab-instance that has no ssh-access enabled. I have to clone/push/etc via https.

When using includes in my gitlab-ci.yaml, gitlab-ci-local tries to download these includes. The function downloadIncludeProjectFile uses git archive under the hood which does not work with https but only with ssh.

Describe the solution you'd like
I'd like to use includes with my instance that has no ssh enabled.

Describe alternatives you've considered

Additional context
I thought about doing it with curl or wget via the gitlab API like this:

    static async downloadIncludeProjectFile (cwd: string, stateDir: string, project: string, ref: string, file: string, gitData: GitData, fetchIncludes: boolean): Promise<void> {
        const remote = gitData.remote;
        const normalizedFile = file.replace(/^\/+/, "");
        try {
            const target = `${stateDir}/includes/${remote.host}/${project}/${ref}/`;
            if (await fs.pathExists(`${cwd}/${target}/${normalizedFile}`) && !fetchIncludes) return;
            await fs.mkdirp(`${cwd}/${target}`);
            await Utils.bash(`git archive --remote=ssh://git@${remote.host}:${remote.port}/${project}.git ${ref} ${normalizedFile} | tar -f - -xC ${target}`, cwd);
        } catch (e) {
            try {
              const target = `${stateDir}/includes/${remote.host}/${project}/${ref}/`;
              if (await fs.pathExists(`${cwd}/${target}/${normalizedFile}`) && !fetchIncludes) return;
              await fs.mkdirp(`${cwd}/${target}`);
              await Utils.bash(`curl --header "Private-Token: ${token}" -O ${target}/${normalizedFile} https://${fomain}/api/v4/projects/${project}/repository/files/${normalizedFile}?ref=${ref}`, cwd);
            } catch (e) {
            throw new ExitError(`Project include could not be fetched { project: ${project}, ref: ${ref}, file: ${normalizedFile} }`)
;
          }
        }
    }

Or maybe with axios? However I'm not proficient with javascript to include a working solution.

[firecow]

We need a way to specify the ACCESS token...

Is it only fetching includes that fail?

I'm asking because I've never tried gitlab-ci-local without ssh access.

Possible workaround below.
https://gist.github.com/taoyuan/bfa3ff87e4b5611b5cbe

[rndmh3ro]

I couldn't really test it further since it fails pretty early. Checking the code, remote variables will probably also fail:

> grep -r "git archive" src/
src/variables-from-files.ts:            const res = await Utils.bash(`git archive --remote=${url} ${ref} ${file} | tar -xO ${file}`, cwd);
src/parser-includes.ts:            await Utils.bash(`git archive --remote=ssh://git@${remote.host}:${remote.port}/${project}.git ${ref} ${normalizedFile} | tar -f - -xC ${target}`, cwd);
src/variables-from-files.js:            const res = await utils_1.Utils.bash(`git archive --remote=${url} ${ref} ${file} | tar -xO ${file}`, cwd);
src/parser-includes.js:            await utils_1.Utils.bash(`git archive --remote=ssh://git@${remote.host}:${remote.port}/${project}.git ${ref} ${normalizedFile} | tar -f - -xC ${target}`, cwd);

The workaround doesn't work for me..

[firecow]

@rndmh3ro I need a little more info about the workaround.

What exactly did you put your global git config?

[rndmh3ro]

I put this in my config. example.com replaced with my internal gitlab server.

git config --global url."git@git.example.com:".insteadOf https://git.example.com/
git config --global url."git://".insteadOf https://

This doesn't work, because the problem is that git archive does not work with https in my gitlab instance (or in general with all gitlab instances?). So replacing ssh-access with https-access locally is not useful.

[firecow]

Oh, I thought git archive would use those git instructions.
Well, in that case, we need a proper feature implemented to deal with the http case.

[iutipikin]

Looks like Gitlab is not support git archive over https at all. @firecow can you please update readme's --fetch-includes section to mention explicitly what we need to use SSH only authentication?
I have a use case similar to @rndmh3ro and I spent a lot of time digging root case of non-fetched includes, as there is no error message. For some reason git output of fatal: operation not supported by protocol was not detected by try-catch.

[firecow]

This is a dublicate

I shall explicitly warn users about git+http in the README.

Edit: it doesnt seem to be a dublicate, reopening.

Edit: I'm an idiot 🤣

[firecow]

0aecd8b