git-issue import from gitlab fails when it shouldn't

Question

git-issue import from gitlab fails when it shouldn't

Closed this issue 4 years ago · 10 comments

The problem

I ran

git issue import gitlab {username} {reponame}

and gets

gitlab API communication failure
URL: https://gitlab.com/api/v4/projects/{usernmae}%2F{reponame}/issues
Operation aborted

I tested for 2 repos and the problem was occurring consistently

Investigation so far

I looked at the source code and seems like the condition in this line

https://github.com/dspinellis/git-issue/blob/master/lib/git-issue/import-export.sh#L96

is evaluated to false and hence displays an error message. If I comment out the conditional block, the git-issue works as expected.

I'm suspecting return values from gitlab's api have changed? e.g., $prefix-header does not have the assumed format anymore? https://github.com/dspinellis/git-issue/blob/master/lib/git-issue/import-export.sh#L90-L94

I'm not proficient with bash scripts and any help would be appreciated.

Answer 1 · 2020-03-30T15:45:49.000Z

After some more investigation, I learned that the same command worked on my ubuntu 14.04 computer, which had

$ bash --version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

$ curl --version
curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP
silverarm  ~

$ grep --version
grep (GNU grep) 2.16
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others, see <http://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.

Originally, I ran the desired command on Mac OSX 10.14.16 with the following setup

$ bash --version
GNU bash, version 5.0.16(1)-release (x86_64-apple-darwin18.7.0)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

$ curl --version
curl 7.54.0 (x86_64-apple-darwin18.0) libcurl/7.54.0 LibreSSL/2.6.5 zlib/1.2.11 nghttp2/1.24.1
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy

$ grep --version
grep (GNU grep) 3.4
Packaged by Homebrew
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others; see
<https://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.

The main difference between the two machine was that the condition expression ! grep -q '^$Status: 200\|HTTP/[[:digit:]].[[:digit:]] 200 OK$' "$prefix-header" in https://github.com/dspinellis/git-issue/blob/master/lib/git-issue/import-export.sh#L96 was evaluated to true on Mac OSX and hence throws an error and evaluated to false on my ubuntu machine so it was evaluated to true.

However, I wasn't sure what grep -q '^$Status: 200\|HTTP/[[:digit:]].[[:digit:]] 200 OK$' "$prefix-header" was for since $prefix-header takes a value like issue-header or comment-header that does not exist in .issues folder (tried on both OSX and ubuntu). Is https://github.com/dspinellis/git-issue/blob/master/lib/git-issue/import-export.sh#L96-L103 outdated code?

Answer 2 · 2020-03-30T16:05:30.000Z

You're right $issue-header doesn't appear to get set. I wonder how it ever worked, or how it worked on your other setup. Can you set -e to see what's going on? @vyrondrosos any ideas?

Answer 3 · 2020-03-30T17:24:09.000Z

So it seems like $prefix-header is a filename so I used cat to see what's in it on Mac OSX

set-cookie: __cfduid={cfduid}; expires={date} link: <https://gitlab.com/api/v4/projects/{usernmae}%2F{reponame}/issues?id={usernmae}%2F{reponame}&order_by=created_at&page=1&per_page=20&sort=desc&state=all&with_labels_details=false>; rel="first", <https://gitlab.com/api/v4/projects/{usernmae}%2F{reponame}/issues?id={usernmae}%2F{reponame}&order_by=created_at&page=1&per_page=20&sort=desc&stat expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon cf-ray: {cf-ray}

and on Ubuntu

Set-Cookie: __cfduid={cfduid}; expires={date} Link: <https://gitlab.com/api/v4/projects/{username}%2F{reponame}/issues?id={username}%2F{reponame}&order_by=created_at&page=2&per_page=20&sort=desc&state=all&with_labels_details=false>; rel="next", <https://gitlab.com/api/v4/projects/{username}%2F{reponame}/issues?id={username}%2F{reponame}&order_by=created_at&page=1&per_page=20&sort=desc&state=all&with_labels_details=false>; rel="first", <https://gitlab.com/api/v4/projects/{username}%2F{reponame}/issues?id={username}%2F{reponame}&order_by=created_at&page=2&per_page=20&sort=desc&state=all&with_ Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon CF-RAY: {cf-ray}

I expect ! grep -q '^$Status: 200\|HTTP/[[:digit:]].[[:digit:]] 200 OK$' "$prefix-header" to return false regardless of OS. Now I'm a bit lost why it works on Ubuntu...

Answer 4 · 2020-03-30T17:41:26.000Z

Created a PR for fixing the above mentioned bug based on the difference between $prefix-header contents of successful REST api call vs. failed REST api call.

The content of $prefix-header with a failed call does not have certain fields like Link, e.g.

Set-Cookie: __cfduid={cfduid}; expires={date} Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon CF-RAY: {cf-ray}

Answer 5 · 2020-03-30T17:53:37.000Z

Many perplexing things.

Where is $prefix-header set?
This is an extended regex, so it shouldn't work with grep.

Can you try grep -E or egrep?

Answer 6 · 2020-03-30T20:38:06.000Z

I believe curl creates a file named $prefix-header at https://github.com/dspinellis/git-issue/blob/master/lib/git-issue/import-export.sh#L90-L91
both grep -E egrep failed with the same output gitlab API communication failure .... I believe this is because the $prefix-header file does not have any line that matches with the regex rule (even if it is working as expected) - see the outputs I posted above

I believe the logic for checking rest call outputs is outdated..

Answer 7 · 2020-03-30T21:28:08.000Z

OK, I got it. It's ${prefix}-header.

The grep seems to be checking for the HTTP header, including the HTTP response, while the file your're showing contains the header without the response.

What does header contain after running curl -D header https://gitlab.com/api/v4/projects/user/proj/issues?
On my system it's

HTTP/2 404
date: Mon, 30 Mar 2020 21:21:16 GMT
content-type: application/json
content-length: 25
set-cookie: __cfduid=d8110dff41baa2cc583d64cb273be1d5f1585603276; expires=Wed, 29-Apr-20 21:21:16 GMT; path=/; domain=.gitlab.com; HttpOnly; SameSite=Lax; Secure
...

Then for curl -D header https://gitlab.com/api/v4/projects/frite%2Fgitlab-ce/issues, it contains

HTTP/2 200
date: Mon, 30 Mar 2020 21:26:19 GMT
content-type: application/json
content-length: 2
set-cookie: __cfduid=def19bab9f8691e77dd4f6ab3d2ee8ff11585603579; expires=Wed, 29-Apr-20 21:26:19 GMT; path=/; domain=.gitlab.com; HttpOnly; SameSite=Lax; Secure
cache-control: max-age=0, private, must-revalidate
...

Answer 8 · 2020-03-31T17:24:57.000Z

When using an incorrect end-point, cat header shows

HTTP/2 404
date: Tue, 31 Mar 2020 17:23:03 GMT
content-type: application/json
content-length: 25
set-cookie: __cfduid={cfduid}; path=/; domain=.gitlab.com; HttpOnly; SameSite=Lax; Secure
...

When using a correct end-point, cat header shows

HTTP/2 200
date: Tue, 31 Mar 2020 17:22:00 GMT
content-type: application/json
set-cookie: __cfduid={cfduid}; path=/; domain=.gitlab.com; HttpOnly; SameSite=Lax; Secure
vary: Accept-Encoding
...

Answer 9 · 2020-03-31T17:28:40.000Z

Excellent, very helpful! So it does contain the HTTP result line. And what is grep's exit code on these headers? Any idea what's going on? If it's still a mystery, please also run sed -n l on each header file to see if it contains any strange characters.

Answer 10 · 2020-04-01T04:22:25.000Z

So it turns out the original regex '^$Status: 200\|HTTP/[[:digit:]].[[:digit:]] 200 OK$' did not matched the first line HTTP/2 404. Update the PR to include a better fix. Thanks for the pointers.