insitro/redun

How to isolate tasks in their own directory?

Opened this issue · 0 comments

Asutu commented

I'm trying to isolate a task in its own directory because it can generate a large number of files. I saw tempdir and thought that it could serve this purpose, but when I run the task with tempdir=True it returns an empty list, whereas by omitting this switch it does return all files.

The task:

@task(tempdir=True)
def splitter(data: File) -> List[File]:
    return script(
        f'''
        split -d -l 2 mydata.txt part.
        ''',
        inputs = [data.stage('mydata.txt')],
        outputs = [File(f).stage(str(f)) for f in glob_file('part.*')],
    )

Maybe tempdir=True has a different purpose? In any case, what is the recommended approach to isolate tasks in their own directory. (I'm running redun locally in my own laptop)

Thanks.