Adapt download_acoustic_dataset() for use by OpenCPU
PietrH opened this issue · 3 comments
download_acoustic_dataset()
is one of only two functions within etn
that doesn't (exclusively) return an object, the other being write_dwc()
This issue is meant to track the decision process and progress in adapting this function for use by OpenCPU, specifically the division of subtasks between the client and the API service.
In short:
download_acoustic_dataset() creates 8 objects:
- if it doesn't exist, a new directory to write to
animals
objecttags
object (+tag_serial_numbers
)detections
objectdeployment
object (+acoustic_project_codes
)receivers
object (+receiver_ids
)datapackage.json
which is static and can be included in the client package- messaging: summary statistics, warning and progress messaging
of these, animals
, tags
, detections
, deployments
, and receivers
get written out as csv. datapackage.json is copied over, it makes sense to just include this in the client.
Now, animals
, tags
, detections
, deployments
, and receivers
can either all be generated by the API, and returned as a single list object via RDA. Or the client can make them itself via calls to get_animals()
, get_tags()
, get_acoustic_detections()
, get_acoustic_deployments()
and get_acoustic_receivers()
passing messaging over the API is possible by retreiving STDOUT, but some formatting is likely to get lost, and it'll get returned as a single blob of text. So it will not properly report progress. From this point of view it makes more sense to have the client make multiple requests and report on progress in between.
Options:
A. Make a lot of requests, maintain messaging and progress reporting
B. Make single request, but possibly tie up API for a while, probably lose progress reporting
Dependencies:
The implementation of this function is dependent on the previous implementation of the following functions:
- get_animals()
- get_tags()
- get_acoustic_detections()
- get_acoustic_deployments()
- get_acoustic_receivers()
And their respective dependencies.
general progress is tracked in inbo/etn#281
I've slept over it, and decided I want to initially develop this function client side. To preserve progress reporting, and to free up the API as quickly as possible (more small calls rather than single bigger ones). But either option could be transitioned into the other with some effort.
Implemented on the client side