Cannot use results filestructure outside of `set_dataset`

[bodokaiser]

ARTIQ Feature Request

Problem this request addresses

We use set_dataset to persist the experimental parameters, e.g., frequencies, with each experimental run.

However, for

a) very large numpy arrages, e.g., images, or
b) saves we want to perform in the background, e.g., inside a Network Device Support Package (NDSP),

we want to save files ourself but reuse the existing filestructure created by artiq_master.

Describe the solution you'd like

If artiq_master would pass rid and the (absolute) result path as variables when creating the experimental instance,
we can directly save additional files together with the hdf5 files created by write_result.

For example, for a camera NDSP:

# camera/driver.py

class Camera:

  # ...

  def save_image(self, path):
    image = self.retrieve_image()
    np.save(path, image)

# repository/exp.py

class Experiment(EnvExperiment):

  def run():
    # trigger camera
    self.save_image("foo")

    @rpc(flags={"async"})
  def save_image(self, name):
    path = os.path.join(self._result_path, f"{self._rid:09d}-{name}.npy"")
    self.camera.save_image(path)

Additional context

At the moment we are doing an educated guess of the rid by looking up the latest number prefix in the result path:

def build(self):
        start_time = time.time()
        start_local_time = time.localtime(start_time)
        dirname = os.path.join(
            "/scratch-local/exp-ryd/sequences/results",
            time.strftime("%Y-%m-%d", start_local_time),
            time.strftime("%H", start_local_time),
        )
        files = [file for file in os.listdir(dirname) if file.endswith(".h5")]
        self.dirname = dirname
        self.rid = max([0] + [int(file.split("-")[0]) for file in files]) + 1

We also tried to use a relative path, e.g.,

dirname = os.path.join(
            "results",
            time.strftime("%Y-%m-%d", start_local_time),
            time.strftime("%H", start_local_time),
        )

but apparently, the working directory is not the project root.

[sbourdeauducq]

The RID is available as self.scheduler.rid.

[bodokaiser]

and the result path?

[bodokaiser]

Makes sense! self.scheduler.rid inside EnvExperiment to get the rid and working directory is equal to /results/%Y-%m-%d/%H, got it!