[BUG] Backup Error in Egress when Using Subdirectories in `filePath`

[cruizba]

Describe the bug
Egress allows recordings to be stored in any S3 provider and has a backup_storage configuration option for saving files locally if the S3 upload fails.

When you make an egress request with the filePath containing a subdirectory, and the S3 upload fails, I expect Egress to create the necessary subdirectory in backup_storage and store the recording file there. But it is failing with the following error:

+-----------------+-----------------+--------------------------------+------------------------------------------------------------+
|    EGRESSID     |     STATUS      |           STARTED AT           |                           ERROR                            |
+-----------------+-----------------+--------------------------------+------------------------------------------------------------+
| EG_GAu9Hau6stad | EGRESS_FAILED   | 2023-12-01 15:03:25.863900265  | rename /home/egress/tmp/EG_GAu9Hau6stad/my-room-test.mp4   |
|                 |                 | +0100 CET                      | /home/egress/backup_storage/livekit-demo/my-room-test.mp4: |
|                 |                 |                                | no such file or directory                                  |
+-----------------+-----------------+--------------------------------+------------------------------------------------------------+

I think this is a bug, mainly because when the egress service is able to upload the file, the directory is created in S3 if the directory does not exist.
I suppose the behavior should be the same when backing it up.

Egress Version
v1.8.0

Egress Request

1. I am using this egress configuration:

backup_storage: /home/egress/backup_storage

s3:
  access_key: minioadmin
  secret: minioadmin
  region: us-east-1
  endpoint: http://dummy:9100
  bucket: dummy
  force_path_style: true

Note1: I am using a dummy endpoint and bucket to force an error with S3 not being reachable
Note2: Directory /home/egress/backup_storage exists before running egress

2. Create a room:

livekit-cli create-room --name my-first-room

3. Simulate a publisher (just to record something):

livekit-cli join-room --room my-first-room --identity bot-user --publish-demo

4. Start an egress. I am using room composite:

request.json

{
  "room_name": "my-first-room",
  "layout": "speaker",
  "file_outputs": [
    {
      "filepath": "livekit-demo/my-room-test.mp4"
    }
  ]
}

livekit-cli start-room-composite-egress --request request.json

5. Wait for the egress to be active.
6. Stop it:

livekit-cli stop-egress --id <egress-id>

7. The egress should appear FAILED with the error:

no such file or directory

This occurs because the livekit-demo directory does not exist in /home/egress/backup_storage

Additional context
If the filePath does not contain subdirectories, the backup works well. I've also noticed that the Egress appears as EGRESS_COMPLETE when the S3 fails and the recording is moved to the backup_storage directory. Is this normal? Shouldn't it appear as EGRESS_FAILED? I mean, the recording is in the backup, but not in the S3 directory.

Logs
No logs needed, the error appears in the Egress object, nothing relevant in the logs as far as I know.