opacity/storage-node

[Need Tests]Integration changes: Add an `/init-upload` endpoint

rfornea opened this issue · 7 comments

As parameters it should accept this:

{
  "address": "a 42-char eth address with 0x prefix",
  "signature": "a 130 character string created when you signed the request with your private key or account handle",
  "requestBody": "the upload body stringified",
  "metadata": "an array of metadata bytes"
}

For an explanation of why the we need to send a stringified requestBody, see this issue: #121 .

requestBody, when unmarshalled, should look like this:

type InitUploadFileObj struct {
	FileHandle string `form:"fileHandle" binding:"required,len=64" minLength:"64" maxLength:"64" example:"a deterministically created file handle"`
	FileSizeInByte int64     `form:"fileSizeInByte" binding:"required" example:"200000000000006"`
	EndIndex   int    `form:"endIndex" binding:"required,gtefield=PartIndex" example:"2"`
}

The responsibilities of the init-upload endpoint:
-verify the request and get the associated account (use returnAccountIfVerified method).
-check the FileSizeInByte and compare to the account's remaining storage and send an error response if they don't have enough space.
-create the file in the SQL table
-upload the data in metadata to s3URL/fileHandle/metadata.
-create the multipart upload for the file and call UpdateKeyAndUploadID on the file object. No need to worry about uploading any chunks in this request--they will be uploaded to s3URL/fileHandle/file in the /upload endpoint. We just need to create the multipart upload.
-send a response back to the frontend so they know they can proceed with calling the /upload endpoint to upload the actual chunks.

The part "-upload the data in metadata to s3URL/fileHandle/metadata."
I am not follow. Why we need to upload the metadata to s3, does it store in SQL?

@pzhao5 It's different data than the metadata that lives in badger. The opaque library needs that metadata on S3 because it uses it during the download. It's needed to reassemble and decrypt the file.

The metadata that we deal with in routes/metadata.go is the metadata for the whole account, whereas the metadata in this request is just for the file.

When we're finished with this, the struct for the request will look like this:

type InitFileUploadReq struct {
	verification
	RequestBody string
        Metadata []byte     // totally different type of metadata than the account metadata
}

And we'll need to update the swagger comments for this endpoint and any others we might have changed.

@funkydrummer Almost completely done. Outstanding:
-unit tests
-update swagger comments
-change json to expect form instead
-change data type of Metadada to []byte which is what the frontend will be sending

working on the few outstanding things now

This needs tests but is otherwise done.