whiteinge/ok.sh

upload_asset with Upload URL error

hannesa2 opened this issue · 11 comments

hannesa2 commented

I can create a release but when I try to upload a asset to release run

./scripts/ok.sh -v upload_asset $USER $REPO 13452549 javadoc.jar application/x-tar < ./build/libs/javadoc.jar

in run into

parse error: Invalid numeric literal at line 1, column 9
jq parse error; invalid JSON.
ok.sh INFO: Remaining next link follows: 50
./scripts/ok.sh: line 1652: upload_url: Upload URL could not be retrieved.

I use jq version jq-1.5

Any ideas ?

whiteinge commented

This is #46. The HTTP headers are getting output along with the JSON and causing the parse error. I've been neglecting debugging it because I think this needs to be split into two separate functions (0066212). Sorry for the trouble; I'll take another look.

  • ❤️1
hannesa2 commented

This script exceed my shell skills, I tried to investigate with switch -v but it doesn't print out JSON.
This would be a improvement too and would help me to find a fix or workaround.

anyway, thank you for the script !

whiteinge commented

This may be a different error than I thought. I couldn't reproduce it, but I found a few other problems with that function that may be contributing. Could you please give the script in #65 a shot? (Direct link to script here.)

If you see an error again will you please run it with -vvv and paste the output here?
Redact the two(!) Authorization: Basic lines.

./scripts/ok.sh -vvv upload_asset $USER $REPO 13452549 javadoc.jar application/x-tar < ./build/libs/javadoc.jar 2>&1
hannesa2 commented

./scripts/ok.sh -vvv upload_asset hannesa2 travisTest 13452541 javadoc.jar application/x-tar < ./build/libs/javadoc.jar

0000: {. "url": "https://api.github.com/repos/hannesa2/travisTest/
0040: releases/13452541",. "assets_url": "https://api.github.com/repo
0080: s/hannesa2/travisTest/releases/13452541/assets",. "upload_ur
00c0: l": "https://uploads.github.com/repos/hannesa2/travisTest/rel
0100: eases/13452541/assets
<= Recv data, 1370 bytes (0x55a)
0000: {?name,label}",. "html_url": "https://github.com/hannesa2/tr
0040: avisTest/releases/tag/v10.1784",. "id": 13452541,. "node_id":
0080: "MDc6UmVsZWFzZTEzNDUyNTQx",. "tag_name": "v10.1784",. "target_
00c0: commitish": "master",. "name": "v10.1784",. "draft": false,.
0100: "author": {. "login": "Payback-CI-User",. "id": 38203578,.
0140: "node_id": "MDQ6VXNlcjM4MjAzNTc4",. "avatar_url": "https:
0180: //avatars1.githubusercontent.com/u/38203578?v=4",. "gravatar_
01c0: id": "",. "url": "https://api.github.com/users/Payback-CI-Use
0200: r",. "html_url": "https://github.com/Payback-CI-User",. "f
0240: ollowers_url": "https://api.github.com/users/Payback-CI-User/fol
0280: lowers",. "following_url": "https://api.github.com/users/Payb
02c0: ack-CI-User/following{/other_user}",. "gists_url": "https://a
0300: pi.github.com/users/Payback-CI-User/gists{/gist_id}",. "starr
0340: ed_url": "https://api.github.com/users/Payback-CI-User/starred{/
0380: owner}{/repo}",. "subscriptions_url": "https://api.github.com
03c0: /users/Payback-CI-User/subscriptions",. "organizations_url":
0400: "https://api.github.com/users/Payback-CI-User/orgs",. "repos_
0440: url": "https://api.github.com/users/Payback-CI-User/repos",.
0480: "events_url": "https://api.github.com/users/Payback-CI-User/even
04c0: ts{/privacy}",. "received_events_url": "https://api.github.co
0500: m/users/Payback-CI-User/received_events",. "type": "User",.
0540: "site_admin": false. },
<= Recv data, 321 bytes (0x141)
0000: . "prerelease": false,. "created_at": "2018-10-16T14:11:27Z",.
0040: "published_at": "2018-10-16T14:28:59Z",. "assets": [.. ],.
0080: "tarball_url": "https://api.github.com/repos/hannesa2/travisT
00c0: est/tarball/v10.1784",. "zipball_url": "https://api.github.com/
0100: repos/hannesa2/travisTest/zipball/v10.1784",. "body": null.}
0140: .
ok.sh DEBUG: Response status is: 200 Connection established
ok.sh DEBUG: Outputting requested header 'status_code'.
ok.sh DEBUG: Outputting requested header 'status_text'.
ok.sh DEBUG: Outputting requested header 'Link_next'.
parse error: Invalid numeric literal at line 1, column 9
jq parse error; invalid JSON.
ok.sh INFO: Remaining next link follows: 50
./scripts/ok.sh: line 1653: upload_url: Upload URL could not be retrieved.

  • 👍1
whiteinge commented

Thanks for that. Unfortunately this one has me stumped. I must have a bug somewhere where the headers and body aren't getting separated properly because the response you pasted seems to be valid JSON and the requisite upload_url field is present. I just haven't been able to reproduce it. 😕

GitHub makes uploading assets pretty cumbersome since we need to fetch, parse, & derive info from multiple API calls. I'm not confident that I can make this workflow reliable so I may just pull this function out and replace it with docs.

All the steps are below. Hopefully they'll work better for you while I continue to noodle on a better solution.

# Create release:
USER="whiteinge"
REPO="testrepo"
TAG="0.1.2"
FILE_NAME="foo.zip"
FILE_PATH="./tests/foo.zip"

# Create new release and upload:
ok.sh create_release "$USER" "$REPO" "$TAG" _filter='.upload_url' \
    | sed 's/{.*$/?name='"$FILE_NAME"'/' \
    | xargs -I@ ok.sh _post @ filename="$FILE_PATH"

# Or find existing release and upload:
ok.sh list_releases "$USER" "$REPO" \
    | awk -v "tag=$TAG" -F'\t' '$2 == tag { print $3; exit }' \
    | xargs -I@ ok.sh release "$USER" "$REPO" @ _filter='.upload_url' \
    | sed 's/{.*$/?name='"$FILE_NAME"'/' \
    | xargs -I@ ok.sh _post @ filename="$FILE_PATH"
hannesa2 commented

With current release 0.3.0

./internalDevelopment/scripts/ok.sh list_releases "$USER" "$REPO"

parse error: Invalid numeric literal at line 1, column 9
jq parse error; invalid JSON.
curl: (23) Failed writing body (1205 != 1370)

any ideas ?

hannesa2 commented

I see as a huge advantage, when command would pipes json out to extra file.
Then it's much easier to debug and see on which step it fails

  • 👍1
whiteinge commented

That's unexpected that list_releases would fail when used alone and other GET queries would not. But I'm glad we're narrowing the problem down!

Will you please run that again with the -j flag to avoid invoking jq? I'm interested to see the top few lines. What I'm expecting is that we'll see a few HTTP response headers output above the JSON response body.

If that's the case, few Qs:

  • What OS and curl version are you running?
  • What shell is /usr/bin/env sh on your machine?
  • Are you hitting vanilla https://api.github.com or something else like GH enterprise?

If my above guess is correct then I'd be interested to see the following output for where the response headers transition into the response body. (Note, I think the -A flag here is a GNU-ism.)

curl -nsSig "https://api.github.com/repos/${USER}/${REPO}/releases" | cat -A
hannesa2 commented
  • It's happen on Linux and macOS
  • shell
$ ps -p $$
  PID TTY           TIME CMD
75208 ttys005    0:00.01 sh
  • I'm using a private repos (is this the question ?)

and ./internalDevelopment/scripts/ok.sh list_releases "$USER" "$REPO" > output.txt delivers

output.txt

whiteinge commented

When I send that output through the list_releases pipeline it outputs all the releases via jq as expected, so I'm not sure yet how to reproduce this ☹️. My guess is something is preventing this match and jq is getting passed headers instead of just the body.

Two more requests, please.

  • Does this version of the script have the same problem?
  • Please run ok.sh -x list_releases "$USER" "$REPO" 2> ok.sh.log and upload that file as well.
whiteinge commented

Closing since we switched to matching on CRLF. Feel free to reopen if the problem persists.