inbo/etnservice

Adapt download_acoustic_dataset() for use by OpenCPU

PietrH opened this issue · 3 comments

PietrH commented

download_acoustic_dataset() is one of only two functions within etn that doesn't (exclusively) return an object, the other being write_dwc()

This issue is meant to track the decision process and progress in adapting this function for use by OpenCPU, specifically the division of subtasks between the client and the API service.

In short:

download_acoustic_dataset() creates 8 objects:

  1. if it doesn't exist, a new directory to write to
  2. animals object
  3. tags object (+ tag_serial_numbers)
  4. detections object
  5. deployment object (+ acoustic_project_codes)
  6. receivers object (+ receiver_ids)
  7. datapackage.json which is static and can be included in the client package
  8. messaging: summary statistics, warning and progress messaging

of these, animals, tags, detections, deployments, and receivers get written out as csv. datapackage.json is copied over, it makes sense to just include this in the client.

Now, animals, tags, detections, deployments, and receivers can either all be generated by the API, and returned as a single list object via RDA. Or the client can make them itself via calls to get_animals(), get_tags(), get_acoustic_detections(), get_acoustic_deployments() and get_acoustic_receivers()

passing messaging over the API is possible by retreiving STDOUT, but some formatting is likely to get lost, and it'll get returned as a single blob of text. So it will not properly report progress. From this point of view it makes more sense to have the client make multiple requests and report on progress in between.

Options:

A. Make a lot of requests, maintain messaging and progress reporting
B. Make single request, but possibly tie up API for a while, probably lose progress reporting

Dependencies:

The implementation of this function is dependent on the previous implementation of the following functions:

  • get_animals()
  • get_tags()
  • get_acoustic_detections()
  • get_acoustic_deployments()
  • get_acoustic_receivers()

And their respective dependencies.

PietrH commented

general progress is tracked in inbo/etn#281

PietrH commented

I've slept over it, and decided I want to initially develop this function client side. To preserve progress reporting, and to free up the API as quickly as possible (more small calls rather than single bigger ones). But either option could be transitioned into the other with some effort.

PietrH commented

Implemented on the client side