permission denied removing old Terraform working directories in dev-container/start.sh

[jrandall]

For some reason the dev-container start script cannot remove the terraform working directories despite them easily being removed by rm -rf:

(venv) mercury@hgs4a:/nfs/humgen01/teams/hgi/users/mercury/checkouts/hgi-systems$ ./scripts/dev-container/start.sh
Updating taos-dev docker image...
New line characters in variable with key "SSH_PRIVATE_KEY" have been escaped to \\n
Updating apt cache in the background
Remove old Terraform working directories
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/environment’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/plugins/linux_amd64/terraform-provider-consul_v1.0.0_x4’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/plugins/linux_amd64/lock.json’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/plugins/linux_amd64/terraform-provider-openstack_v1.2.0_x4’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/plugins/linux_amd64/terraform-provider-null_v1.0.0_x4’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/plugins/linux_amd64/terraform-provider-template_v1.0.0_x4’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/terraform.tfstate’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/modules/d31655a351c4a15a5e90412abebcf761’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/modules/91a4a2084146ca683aff02aa003c77c2’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/modules/fa87ff018954cd2946d1d35db79b3b41’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/modules/353f3aef95de3e8193843b9c6afb6010’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/modules/922fea872a0f7420c6eb4be917d3adf4’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/modules/69f1a3bf0e81000968194b324d987bf6’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/modules/a5309f63d2c5a95653cb1e25f9009c4d’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/modules/48b76e8d51a85a4f4e71f6d16d1e397e’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/modules/dd68dae409642117f68ecbc68f3d113d’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/modules/fc28813bd29ff8990ee9d4a0dd2c8a20’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/modules/modules.json’: Permission denied
rm: cannot remove ‘/mnt/host/hgi-systems/terraform/delta-hgiarvados/.terraform/modules/e25f23ddf0381022a5057466f8a69642’: Permission denied
(venv) mercury@hgs4a:/nfs/humgen01/teams/hgi/users/mercury/checkouts/hgi-systems$ rm -rf terraform/delta-hgiarvados/.terraform
(venv) mercury@hgs4a:/nfs/humgen01/teams/hgi/users/mercury/checkouts/hgi-systems$ ./scripts/dev-container/start.sh
Updating taos-dev docker image...
New line characters in variable with key "SSH_PRIVATE_KEY" have been escaped to \\n
Updating apt cache in the background
Remove old Terraform working directories
Setting up copy of the host user
Change to user mercury
Setting Ansible Vault password
Setting OpenStack environment variables
Setting GitLab specific variables
Sourcing before scripts
Setup Git
Setting SSH keys
Starting shell...

[colin-nolan]

Unfortunately, I couldn't replicate the problem. The rm -rf is ran as root inside the container so there shouldn't be a problem removing files.

[colin-nolan]

Discussed with Josh: the problem is that root squash is enabled on nfs (which is why I couldn't replicate the problem when I tested on Lusture).