GDAL fails to find newly created files in Google Cloud Store

[jzxu]

What is the bug?

When opening rasters on Google Cloud Store using /vsigs/, GDAL is not able to find a newly created raster if it had opened another raster in the same folder prior to the creation of the new raster.

I suspect that under the hood, GDAL caches a listing of the parent GCS folder whenever it opens a file in that folder. When trying to access a new file created after the folder listing was cached, I guess that GDAL checks the cached folder listing, doesn't find the file there, and claims the file does not exist.

Steps to reproduce the issue

1. Open a raster on GCS using vsigs. For example, /vsigs/some-bucket/test/image1.tif.
2. Create a new raster in the same folder (e.g. /vsigs/some-bucket/test/image2.tif) using any method (e.g. copying the raster to the folder using gsutil).
3. Open the new raster. gdal.Open will return None.

If the new raster is created in a different folder from the previously accessed raster, GDAL is able to open it.

Test case using GDAL python bindings:

import os
from osgeo import gdal
import subprocess

local_path = '~/image.tif'
gcs_path1 = 'gs://some-bucket/test/image1.tif'
gcs_path2 = 'gs://some-bucket/test/image2.tif'
gcs_path3 = 'gs://some-bucket/another-folder/image3.tif'

# Delete the file if it exists.
subprocess.run(f'gsutil rm -f {gcs_path1}', shell=True)

# Access a different file in the same folder.
# If gcs_path2 is changed to gcs_path3, the error doesn't occur.
r2 = gdal.Open(gcs_path2.replace('gs://', '/vsigs/'))
print(r2.GetGeoTransform())

# Create the new file.
subprocess.check_call(f'gsutil cp {local_path} {gcs_path1}', shell=True)

r1 = gdal.Open(gcs_path1.replace('gs://', '/vsigs/'))
if r1 is not None:
  print(r1.GetGeoTransform())
else:
  print('Image not found')

Versions and provenance

Linux Debian Rodete

GDAL 3.8.5, released 2024/04/02

GDAL installed using apt-get.

GDAL Python bindings installed using pip install gdal==3.8.5

Additional context

Only tested with Python API.

[rouault]

yes, this is expected.

Use gdal.VSICurlClearCache()