setting times of 'file.txt': Permission denied Error

[raviprakash007]

The issue
I am trying to create files from running application and even manually to the mounted GCS Bucket. It works with vi editor but not with touch command. The file is created but the error is always coming as permission denied.

$ touch a.txt
setting times of 'a.txt': Permission denied
$ ls
. .. a.txt 

System & Version

• OS: Rocky Linux
• Platform [GCE VM]
• Version [Gcsfuse version: gcsfuse version 2.4.0 (Go version go1.22.4)]

Steps to reproduce the behavior with following information:

1. mount command:
   $ sudo gcsfuse --implicit-dirs --uid=1001 --gid=1002 --dir-mode 777 --file-mode 777 -o rw -o allow_other -o sync --foreground --debug_fuse --debug_fs --debug_gcs --debug_http bucket_number_10121 /mnt/cloud-storage

2. In second console
   $ cd /mnt/cloud-storage/audio <-- audio folder was already there
   $ sudo touch file.txt
   touch: setting times of 'file.txt': Permission denied

Additional context
Fuse Logs:

{"timestamp":{"seconds":1723450672,"nanos":295073887},"severity":"TRACE","message":"fuse_debug: Op 0x00000056        connection.go:420] <- CreateFile (parent 2, name \"a.txt\", PID 320074)"}
{"timestamp":{"seconds":1723450672,"nanos":295664395},"severity":"TRACE","message":"fuse_debug: Op 0x00000056        connection.go:513] -> OK (inode 3)"}
{"timestamp":{"seconds":1723450672,"nanos":295812818},"severity":"TRACE","message":"fuse_debug: Op 0x00000058        connection.go:420] <- FlushFile (inode 3, PID 320074)"}
{"timestamp":{"seconds":1723450672,"nanos":295919365},"severity":"TRACE","message":"gcs: Req              0x5: <- StatObject(\"audio/file.txt\")"}
{"timestamp":{"seconds":1723450672,"nanos":321834347},"severity":"TRACE","message":"gcs: Req              0x5: -> StatObject(\"audio/file.txt\") (25.937307ms): gcs.NotFoundError: storage: object doesn't exist"}
{"timestamp":{"seconds":1723450672,"nanos":325409711},"severity":"TRACE","message":"gcs: Req              0x6: <- CreateObject(\"audio/file.txt\")"}
{"timestamp":{"seconds":1723450672,"nanos":445545730},"severity":"TRACE","message":"gcs: Req              0x6: -> CreateObject(\"audio/file.txt\") (120.126235ms): OK"}
{"timestamp":{"seconds":1723450672,"nanos":445683643},"severity":"TRACE","message":"fuse_debug: Op 0x00000058        connection.go:513] -> OK ()"}
{"timestamp":{"seconds":1723450672,"nanos":445901949},"severity":"TRACE","message":"fuse_debug: Op 0x0000005a        connection.go:420] <- SetInodeAttributes (inode 3, PID 320074, atime 2024-08-12 08:17:52.44445889 +0000 UTC, mtime 2024-08-12 08:17:52.44445889 +0000 UTC)"}
{"timestamp":{"seconds":1723450672,"nanos":445999377},"severity":"TRACE","message":"gcs: Req              0x7: <- UpdateObject(\"audio/file.txt\")"}
{"timestamp":{"seconds":1723450672,"nanos":453054712},"severity":"TRACE","message":"gcs: Req              0x7: -> UpdateObject(\"audio/file.txt\") (7.000931ms): googleapi: Error 403: Access denied., forbidden"}
{"timestamp":{"seconds":1723450672,"nanos":453120388},"severity":"ERROR","message":"SetInodeAttributes: permission denied, SetMtime: UpdateObject: googleapi: Error 403: Access denied., forbidden"}
{"timestamp":{"seconds":1723450672,"nanos":453189636},"severity":"TRACE","message":"fuse_debug: Op 0x0000005a        connection.go:515] -> Error: \"permission denied\""}
{"timestamp":{"seconds":1723450672,"nanos":453206628},"severity":"ERROR","message":"fuse: *fuseops.SetInodeAttributesOp error: permission denied"}
{"timestamp":{"seconds":1723450672,"nanos":453365379},"severity":"TRACE","message":"fuse_debug: Op 0x0000005c        connection.go:420] <- FlushFile (inode 3, PID 320074)"}
{"timestamp":{"seconds":1723450672,"nanos":453499432},"severity":"TRACE","message":"fuse_debug: Op 0x0000005c        connection.go:513] -> OK ()"}
{"timestamp":{"seconds":1723450672,"nanos":453605179},"severity":"TRACE","message":"fuse_debug: Op 0x0000005e        connection.go:420] <- ReleaseFileHandle (PID 0, handle 2)"}
{"timestamp":{"seconds":1723450672,"nanos":453686328},"severity":"TRACE","message":"fuse_debug: Op 0x0000005e        connection.go:513] -> OK ()"}

The default Service Account is already having Storage Admin Role.

[sethiay]

Thanks for reaching out @raviprakash007 !

This error comes when the service account/user used in GCSFuse for authentication with GCS doesn't have storgage.object.Update permissions. However, as you mentioned you already have set storage admin role which should have update permissions (is it possible that the role was changed by any other application/process in your project when the file was being created ?).

Unfortunately, we were not able to reproduce the issue. Is this issue always reproducible on your side ? If yes, then we would need more information in order to debug this further, so we request you to reach out via support channel where we can request more information from you.

[raviprakash007]

No other applications use the Bucket. This is reproducible on all VM instances we have in GCP. For re-verification, I can launch a new instance and check it again with a new GCS bucket and new service account mounted with VM.

However, for other existing VMs, I have tested the scenario with the root user ( mouting with uid=0, gid=0) and even root user was also facing the same permission denied issue, while the file was being created.

In all cases, Vi editor works without any issues and no permission issue is there

[sethiay]

For re-verification, I can launch a new instance and check it again with a new GCS bucket and new service account mounted with VM.

Sure, if you are using default service account, then we suggest to check this option to allow propagation of the storage admin permission granted to the default service account:

In all cases, Vi editor works without any issues and no permission issue is ther

This happens because in case of Vi, the mtime is updated before the newly created file is sync'ed (Sync/Flush file call) to GCS, hence the mtime is updated locally. However in case of touch, the mtime is updated after Sync/Flush file call which requires updating the backing object in GCS and if the service account/user doesn't have storage object update permission, then this update fails. In short, there is a different in working of Vi and touch applications but the permission error in touch suggests that the service account/user being used doesn't have storage.object.Update permissions.

[raviprakash007]

Let me check this on a new VM

[webmaster@instance-20240814-141453 storage]$ sudo touch a.txt
[webmaster@instance-20240814-141453 storage]$ touch b.txt
[webmaster@instance-20240814-141453 storage]$ ls -ltr
. .. a.txt b.txt

Its working with Full access given to default sevice account.

[sethiay]

Thank you @raviprakash007 for confirming. Closing this issue. Please re-open or create one if you face any other issues.