Allow `input_paths` to be absolute

[jritsema]

Hi there. I'm just starting to play with this and it's a super interesting project, so thank you for this!!!

I was trying to run the aws cli in a task and would like to copy in my ~/.aws/credentials.

Any reason not to allow input_paths to be absolute? similar to #268

[stepchowfun]

Hi @jritsema, so sorry for the slow response. Thank you for creating this issue!

Toast tasks are intended to be "self-contained" in the sense that all the files needed to run the task could be found in the directory containing the toastfile (I will call that directory the "project" from now on).

We let the cat out of the bag when we allowed mount_paths to be absolute and outside of the project. However, there is one safeguard that keeps this sane: any task that uses mount_paths must also set cache: false. That way, it's impossible to accidentally cache files outside the project, such as credentials, the Docker IPC socket, and large resources that shouldn't be copied/cached for performance reasons. (NB Toast has a feature called "remote caching", so it's doubly important to not cache things like system files, credentials, and other files outside the project that are specific to one machine.)

It seems to me that mount_paths is also a good fit for your use case. That will prevent the task from being cached, which also seems like the right behavior for AWS credentials. Is there any reason not to use that?

I will leave this issue open for a short while to give you a chance to let me know what you think.

[stepchowfun]

(cc @rpurdon-nf since you thumbed-up the issue)

[jritsema]

HI @stepchowfun thanks for the explanation. That does make sense, however, not sure how I would use mount_paths for this use case since the paths are different inside/outside the container (different users) and I don't want to hard code a user name. It works with the docker socket since they're the same. Any suggestions?

    mount_paths:
      - /var/run/docker.sock
      - /root/.aws/credentials

seems like i would want

    mount_paths:
      - /var/run/docker.sock
      - $HOME/aws/credentials:/root/aws/credentials

[stepchowfun]

Oh, that is a good call-out. There doesn't seem to be a way to do this cleanly right now.

If we allowed input_paths to be absolute, it seems we'd have the same problem.

This is a hack but it might unblock you for now: you could make ~/.aws/credentials a symlink that points to some hardcoded location (like /etc/.aws/credentials) and then mount the hardcoded path with mount_paths. Everyone who uses your toastfile would have to do the same, unfortunately.

I like your suggestion of allowing mount_paths/input_paths to be different inside vs. outside of the container with the colon syntax (and maybe keep the current policy that only mount_paths can be absolute). We'd also have to allow paths to contain environment variables which are expanded for this feature to work for your use case. This is within the realm of possibility, but I'd have to think about the implications (and find time to do it—PRs welcome!).

[stepchowfun]

I am going to close this issue now, since from this discussion I think the original feature request (absolute input_paths) is not actually the right path forward. Instead, it seems like there are two other features that would be more suitable:

1. Environment variable interpolation in paths
2. Allowing the path outside the container to be different from the path inside the container

If anyone is interested in pursuing those, feel free to open dedicated issues for them. I think both of these would require some discussion about the design, but hopefully they aren't too complicated.