FLIR/conservator-cli

cvc add new image to dataset and upload error

patrickpritchett opened this issue · 5 comments

patrickpritchett commented

(conservator-cli) patrick@patrick-ThinkPad-T430:~/git/flir-venv/conservator-cli/conservator-cli$ cvc clone A6MiBszaAb7LEFyQs
Cloning into './dsCli'...
remote: Counting objects: 14, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 14 (delta 3), reused 0 (delta 0)
Unpacking objects: 100% (14/14), done.

(conservator-cli) patrick@patrick-ThinkPad-T430:~/git/flir-venv/conservator-cli/conservator-cli$ cd dsCli/

(conservator-cli) patrick@patrick-ThinkPad-T430:~/git/flir-venv/conservator-cli/conservator-cli/dsCli$ cvc download
Downloading 534a4eb6f12be7fb4112e5e0210001 (0.00 MB): 100%|█████████████████████████████████████████████████████████████████| 110144/110144 [00:00<00:00, 52141920.97it/s]
Downloading 28d6b7fef34651c2019f6cafec5e35 (2.00 MB): 100%|██████████████████████████████████████████████████████████████| 2611292/2611292 [00:00<00:00, 156273042.84it/s]

(conservator-cli) patrick@patrick-ThinkPad-T430:~/git/flir-venv/conservator-cli/conservator-cli/dsCli$ cvc add ~/Pictures/1.jpg

(conservator-cli) patrick@patrick-ThinkPad-T430:~/git/flir-venv/conservator-cli/conservator-cli/dsCli$ cvc upload

1.jpg
{'Content-type': 'image/jpeg', 'x-amz-meta-originalfilename': '1.jpg'}
Traceback (most recent call last):
  File "/home/patrick/git/flir-venv/conservator-cli/bin/cvc", line 11, in <module>
    load_entry_point('conservator-cli', 'console_scripts', 'cvc')()
  File "/home/patrick/git/flir-venv/conservator-cli/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/patrick/git/flir-venv/conservator-cli/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/patrick/git/flir-venv/conservator-cli/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/patrick/git/flir-venv/conservator-cli/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/patrick/git/flir-venv/conservator-cli/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/patrick/git/flir-venv/conservator-cli/conservator-cli/FLIR/conservator/cli/cvc.py", line 25, in wrapper
    return func(ops, *args, **kwargs)
  File "/home/patrick/git/flir-venv/conservator-cli/conservator-cli/FLIR/conservator/cli/cvc.py", line 189, in upload
    local_dataset.push_staged_images()
  File "/home/patrick/git/flir-venv/conservator-cli/conservator-cli/FLIR/conservator/local_dataset.py", line 119, in push_staged_images
    self.upload_image(path, md5)
  File "/home/patrick/git/flir-venv/conservator-cli/conservator-cli/FLIR/conservator/local_dataset.py", line 178, in upload_image
    assert r.status_code == 200
AssertionError

*** conservator log:

[0] {"level":"audit","msg":"","operation":"query","queryName":"undefined","resolver":"user","vars":{},"queryTime":1,"userId":"YgLJxyRfJNj4kESuE","sessionId":"9584cf04-a297-4f6e-a73a-36250c22e2d1","filename":"index.js","pid":112}
[0] {"level":"debug","msg":["HEAD request for 9a96a8b9def442cd0f002b3571bfce91"],"filename":"index.js","pid":112}
[0] {"level":"error","msg":[{"message":null,"code":"NotFound","region":null,"time":"2020-12-31T15:02:52.553Z","requestId":"55A462EA7643E9FD","extendedRequestId":"njy6J4BOJIOwJ+vuVMhAHwQi8muPMvVVl3ZBVzYMHjkKjqT2lWmw+T03eanpwlloIY9rbECP0FU=","statusCode":404,"retryable":false,"retryDelay":69.3546123449731}],"filename":"index.js","pid":112}
[0] {"level":"debug","msg":["md5 9a96a8b9def442cd0f002b3571bfce91 not found"],"filename":"index.js","pid":112}
[0] {"level":"debug","msg":["PUT request for 9a96a8b9def442cd0f002b3571bfce91"],"filename":"index.js","pid":112}
[0] {"level":"debug","msg":["Generating url for Key dvc/9a/96a8b9def442cd0f002b3571bfce91 with filename 1.jpg"],"filename":"index.js","pid":112}
[0] {"level":"debug","msg":["Application mimetype: string","image/jpeg"],"filename":"index.js","pid":112}

Notes

not using s3rver
able to add with old python cvc.py add ~/Pictures/1.jpg then use cvc upload and cvc publish "msg"

senormeow commented

@n-wach Look at def cvc_push(args, subparsers): in the original CVC. The Conservator knows this is a loose frame (frame dethatched from a video) when the videoMetadata.frameId is the same as the datasetFrameId and the videoMetadata.videoId matches the datasetId

n-wach commented

This seems to be an issue when following the redirect from dvc to s3. The current dvc-handler sends HTTP redirect 302, which changes the operation from PUT to GET. I believe the original cvc manually performed the redirect to avoid this conversion (keeping it PUT for s3). Ideally, we should change the dvc redirect to use status 307, which retains the original op.

I believe this would be a one line change on line 47 of src/server/route-handlers/dvc-handler.js:

- return res.redirect(putUrl.signedUrl);
+ return res.redirect(307, putUrl.signedUrl);

In the meantime, I can use the original cvc's manual redirection preserving PUT?

senormeow commented

@n-wach I am curious about if 307 will just work, I can try that on conservator tomorrow.

  • 👍1
n-wach commented

PRs merged, closing this