gee-community/geemap

Some bugs about the extract_values_to_points function

zwy1502 opened this issue · 9 comments

zwy1502 commented

Environment Information

Please run the following code on your computer and share the output with us so that we can better debug your issue:

Sat Mar 09 11:45:21 2024 **标准时间
OS | Windows | CPU(s) | 20 | Machine | AMD64
Architecture | 64bit | RAM | 15.7 GiB | Environment | Jupyter
Python 3.12.1 \| packaged by conda-forge \| (main, Dec 23 2023, 07:53:56) [MSC v.1937 64 bit (AMD64)]
geemap | 0.32.0 | ee | 0.1.385 | ipyleaflet | 0.18.1
folium | 0.15.1 | jupyterlab | 4.0.10 | notebook | Module not found
ipyevents | 2.0.2 |   |   |  

<br class="Apple-interchange-newline"><!--EndFragment-->
</body>
</html>

Description

Two problems occurred when I used the extract_values_to_points function to extract the pixel values of multi-band images. The first problem was that the order of the extracted pixels was correct before extraction, but after this function it became 16 system, the second more serious problem is that the values of the extracted multi-band images are all the same, and this is impossible

What I Did

hhmc = ee.FeatureCollection("projects/ee-my-zwy/assets/yb2020hhmc")
def maskS2clouds(image):
    cloudProb = image.select('MSK_CLDPRB')
    snowProb = image.select('MSK_SNWPRB')
    cloud = cloudProb.lt(10) # 云覆盖量小于10%
    scl = image.select('SCL')
    shadow = scl.eq(3) # 3 = 云阴影
    cirrus = scl.eq(10) # 10 = 卷云
    # 云概率小于10%或云阴影分类
    mask = cloud.And(cirrus.Not()).And(shadow.Not())
    return image.updateMask(mask).divide(10000).select("B.*").set(image.toDictionary(image.propertyNames()))

# 创建影像集合
startDate = ee.Date('2020-01-01')
endDate = ee.Date('2021-01-01')
s2 = ee.ImageCollection('COPERNICUS/S2_SR_HARMONIZED').filterDate(startDate, endDate).filterBounds(studyArea).map(maskS2clouds)

# 计算指数
def addIndices(image):
    ndvi = image.normalizedDifference(['B8', 'B4']).rename('NDVI')
    lswi = image.normalizedDifference(['B8', 'B12']).rename('LSWI')
    rvi = image.select('B8').divide(image.select('B4')).rename('RVI')
    mndwi = image.normalizedDifference(['B3', 'B11']).rename('mNDWI')
    return image.addBands([ndvi, lswi, rvi, mndwi])
bandarray=['B2','B3','B4','B5','B6','B7', 'B8','B8A','B11','B12','NDVI','LSWI','RVI','mNDWI']

s2 = s2.map(addIndices).select(bandarray)
def monthly_median_composite(collection,year):
    composites = []
    for month in range(1, 13):
        # 定义每月的开始和结束日期
        startDate = ee.Date.fromYMD(year, month, 1)
        endDate = startDate.advance(1, 'month')
        # 筛选当月的影像并应用去云函数
        month_s2 = collection.filterDate(startDate, endDate)
        # 计算中值合成
        medianComposite = month_s2.median().clipToCollection(studyArea)
        composites.append(medianComposite.set('month', month))
    return composites

# 应用函数进行2020年的月度合成
monthly_composites_2020 = monthly_median_composite(s2,2020)
monthly_median_collection2020 = ee.ImageCollection.fromImages(monthly_composites_2020)
# 首先，只选取每个影像的NDVI波段
ndvi_collection = monthly_median_collection2020.select(['NDVI'])

# 然后，使用.toBands()方法将这个集合转换成一个多波段影像
ndvi_multi_band_image = ndvi_collection.toBands()

# 由于.toBands()会将波段命名为影像的ID，可能需要重命名波段以反映月份信息
# 创建一个包含月份名称的列表
months = ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12']

# 构建新的波段名称列表
new_band_names = [f'NDVI_{month}' for month in months]

# 重命名波段
ndvi_multi_band_image = ndvi_multi_band_image.rename(new_band_names)

geemap.extract_values_to_points(hhmc, ndvi_multi_band_image, r'E:\download\ndvi_multi_band_imagehhmc.csv'
zwy1502 commented
<style> </style>
NDVI_1 NDVI_2 NDVI_3 NDVI_4 NDVI_5 NDVI_6 NDVI_7 NDVI_8 NDVI_9 NDVI_10 NDVI_11 NDVI_12 class system:index label
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 0 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 1 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 2 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 3 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 4 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 5 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 6 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 7 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 8 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 9 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 0000000000000000000a hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 0000000000000000000b hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 0000000000000000000c hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 0000000000000000000d hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 0000000000000000000e hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 0000000000000000000f hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 10 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 11 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 12 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 13 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 14 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 15 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 16 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 17 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 18 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 19 hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 0000000000000000001a hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 0000000000000000001b hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 0000000000000000001c hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 0000000000000000001d hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 0000000000000000001e hhmc
0.015651135 0.606169939 0.749406159 0.747073472 0.60399425 0.120363973 0.021767115 0.663969815 0.703009725 0.680603921 0.435596526 0.275629222 0 0000000000000000001f hhmc

The above is my output result. The first problem is that the pixel values extracted by this function are exactly the same. This must be a problem. I have verified that the image is normal. The second problem is that there will always be something similar to 0000000000000000000a.
0000000000000000000b,
0000000000000000000c,
0000000000000000000d,
0000000000000000000e,
0000000000000000000f and other abnormal indexes, why is this happening?

giswqs commented

Please provide a complete script that can reproduce the issue. Make sure you make the asset publicly accessible. And the variable studyArea is not defined.

zwy1502 commented

#This is my complete script

import geemap
import ee
geemap.set_proxy(port='4780')
Map = geemap.Map()

Map.add_tile_layer(
    url="https://mt1.google.com/vt/lyrs=y&x={x}&y={y}&z={z}&scale=2",
    name="HD Google Basemap",
)
studyArea = ee.FeatureCollection("projects/ee-my-zwy/assets/sampleShp")
hhmc = ee.FeatureCollection("projects/ee-my-zwy/assets/yb2020hhmc")
def maskS2clouds(image):
    cloudProb = image.select('MSK_CLDPRB')
    snowProb = image.select('MSK_SNWPRB')
    cloud = cloudProb.lt(10) # 云覆盖量小于10%
    scl = image.select('SCL')
    shadow = scl.eq(3) # 3 = 云阴影
    cirrus = scl.eq(10) # 10 = 卷云
    # 云概率小于10%或云阴影分类
    mask = cloud.And(cirrus.Not()).And(shadow.Not())
    return image.updateMask(mask).divide(10000).select("B.*").set(image.toDictionary(image.propertyNames()))


startDate = ee.Date('2020-01-01')
endDate = ee.Date('2021-01-01')
s2 = ee.ImageCollection('COPERNICUS/S2_SR_HARMONIZED').filterDate(startDate, endDate).filterBounds(studyArea).map(maskS2clouds)


def addIndices(image):
    ndvi = image.normalizedDifference(['B8', 'B4']).rename('NDVI')
    lswi = image.normalizedDifference(['B8', 'B12']).rename('LSWI')
    rvi = image.select('B8').divide(image.select('B4')).rename('RVI')
    mndwi = image.normalizedDifference(['B3', 'B11']).rename('mNDWI')
    return image.addBands([ndvi, lswi, rvi, mndwi])
bandarray=['B2','B3','B4','B5','B6','B7', 'B8','B8A','B11','B12','NDVI','LSWI','RVI','mNDWI']

s2 = s2.map(addIndices).select(bandarray)
def monthly_median_composite(collection,year):
    composites = []
    for month in range(1, 13):

        startDate = ee.Date.fromYMD(year, month, 1)
        endDate = startDate.advance(1, 'month')

        month_s2 = collection.filterDate(startDate, endDate)

        medianComposite = month_s2.median().clipToCollection(studyArea)
        composites.append(medianComposite.set('month', month))
    return composites


monthly_composites_2020 = monthly_median_composite(s2,2020)
monthly_median_collection2020 = ee.ImageCollection.fromImages(monthly_composites_2020)

ndvi_collection = monthly_median_collection2020.select(['NDVI'])

ndvi_multi_band_image = ndvi_collection.toBands()


months = ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12']


new_band_names = [f'NDVI_{month}' for month in months]


ndvi_multi_band_image = ndvi_multi_band_image.rename(new_band_names)

geemap.extract_values_to_points(hhmc, ndvi_multi_band_image, r'E:\download\ndvi_multi_band_imagehhmc.csv'

Now I find that I can correctly export pixel values using this method without using the geemap function. I am wondering why the extract_values_to_points function cannot do it. Here are the pixel values I extracted using other methods, and the result is correct. code show as below:

def addIndex(fc):
     # Use ee.List.sequence to create a sequence with the same size as FeatureCollection
     indices = ee.List.sequence(0, fc.size().subtract(1))
    
     # Create a new FeatureCollection that will contain the index corresponding to the original Feature
     indexed_fc = fc.toList(fc.size()).zip(indices).map(lambda el: ee.Feature(ee.List(el).get(0)).set('custom_index', ee.List( el).get(1)))
    
     #Convert the list back to FeatureCollection
     return ee.FeatureCollection(indexed_fc)

#Apply this function to the original FeatureCollection
hhmc= addIndex(hhmc)

geemap.ee_to_df(hhmc)
# Use the indexed sample collection to reduceRegions
def extract_and_export_samples(indexedSampleCollection, image, description):
     # Use the reduceRegions method to extract pixel values
     extracted = image.reduceRegions(collection=indexedSampleCollection, reducer=ee.Reducer.first(), scale=10)

     # At this stage, each feature in extracted should have a 'custom_index' attribute
     #Keep this 'custom_index' when exporting
     task = ee.batch.Export.table.toDrive(collection=extracted,
                                          description=description,
                                          selectors=['custom_index','class','label','NDVI_1', 'NDVI_2', 'NDVI_3', 'NDVI_4', 'NDVI_5', 'NDVI_6', 'NDVI_7', 'NDVI_8', 'NDVI_9' , 'NDVI_10', 'NDVI_11', 'NDVI_12'], #Add other required attribute names here
                                          fileFormat='CSV')
     task.start()

# Call the function and pass in the indexed sample collection
extract_and_export_samples(lw, ndvi_multi_band_image, 'LW_Sample_Values_2020')
zwy1502 commented

In this way, I avoid using the extract_values_to_points function to get the correct result. At least the extracted pixel values will not be completely consistent, but this is not what I want because Professor Wu's function is simpler. Unfortunately I can't use it and I don't know what the problem is. Can you please teach me, Mr. Wu? My studyarea and sample hhmc were drawn manually on GEE and then uploaded to my own asset for use. There will be no problem of non-disclosure of assets.

I still have a small question. Is it now impossible to view the task progress on the GEE code editor when exporting GEE data? I also recently tried to export GEE data to Google Drive but I did not see the gear in the code editor. After a while, I saw the exported data in my Google Drive. How is this going?

giswqs commented

Make the assets publicly accessible:

  • ee.FeatureCollection("projects/ee-my-zwy/assets/sampleShp")
  • ee.FeatureCollection("projects/ee-my-zwy/assets/yb2020hhmc")
giswqs commented

Try setting the scale parameter for the extract_values_to_points function. It is likely that your workflow is computationally intensive. Earth Engine may use a coarser resolution for the computation.

zwy1502 commented

Sorry, teacher, I didn't see the message before. Now I have tried to make these two assets public. Teacher Wu, can you help me take a look? I don't want to change the resolution because my requirement is 10m resolution. The links for these two assets are https://code.earthengine.google.com/?asset=projects/ee-my-zwy/assets/yb2020hhmc and https://code.earthengine.google.com/?asset= projects/ee-my-zwy/assets/sampleShp

zwy1502 commented

Try setting the scale parameter for the extract_values_to_points function. It is likely that your workflow is computationally intensive. Earth Engine may use a coarser resolution for the computation.尝试为 extract_values_to_points 函数设置 scale 参数。您的工作流程可能属于计算密集型。 Earth Engine 可以使用较粗的分辨率进行计算。

Sorry, teacher, I didn't see the message before. Now I have tried to make these two assets public. Teacher Wu, can you help me take a look? I don't want to change the resolution because my requirement is 10m resolution. The links for these two assets are https://code.earthengine.google.com/?asset=projects/ee-my-zwy/assets/yb2020hhmc and https://code.earthengine.google.com/?asset= projects/ee-my-zwy/assets/sampleShp

giswqs commented 
geemap.extract_values_to_points(hhmc, ndvi_multi_band_image, 'dvi_multi_band_imagehhmc.csv', scale=10)
image