Google Earth Engine Batch Asset Manager with Addons is an extension of the one developed by Lukasz here and additional tools were added to include functionality for moving assets, conversion of objects to fusion table, cleaning folders, querying tasks. The ambition is apart from helping user with batch actions on assets along with interacting and extending capabilities of existing GEE CLI. It is developed case by case basis to include more features in the future as it becomes available or as need arises.
If you use this tool to download data for your research, and find this tool useful, star and cite it as below
Samapriya Roy. (2021, June 7). samapriya/gee_asset_manager_addon: GEE Asset Manager with Addons (Version 0.5.5).
Zenodo. http://doi.org/10.5281/zenodo.4905604
We assume Earth Engine Python API is installed and EE authorised as desribed here. From v0.3.4 onwards geeadd will only run on Python 3. Also with the new changes to the Earth Engine API library, the tool was completely modified to work with earthengine-api v0.1.127 and higher. Authenticate your earth engine client by using the following in your command line or terminal setup.
earthengine authenticate
Quick installation pip install geeadd or pip install geeadd --user
To install using github:
git clone https://github.com/samapriya/gee_asset_manager_addon
cd gee_asset_manager_addon && pip install -r requirements.txt
python setup.py install
The advantage of having it installed is being able to execute geeadd as any command line tool. I recommend installation within virtual environment. To install run
python setup.py develop or python setup.py install
In a linux distribution
sudo python setup.py develop or sudo python setup.py install
As usual, to print help:
To obtain help for a specific functionality, simply call it with help switch, e.g.: geeadd copy -h.
Now open the readme webpage in your browser using
geeadd readme
This tool is the very basic of tools in the toolbox which gives you your current quota. This gives you the total used and remaining quota in all your legacy folders or user root folders. It requires no additional arguments just that your earthengine api is enabled. Quota can now also handle Google Earth Engine with Google Cloud Projects enabled so you can pass project path as an argument.
> geeadd quota -h
usage: geeadd quota [-h] [--project PROJECT]
optional arguments:
-h, --help show this help message and exit
Optional named arguments:
--project PROJECT Project Name usually in format projects/project-
name/assets/
The search tool was added since v0.3.4 to enable users to search inside the Google Earth Engine catalog for images matching specific keywords and looks for matching images using names, ids , tags and so on. Try for example
geeadd search --keywords "fire"
> geeadd search -h
usage: geeadd search [-h] --keywords KEYWORDS
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--keywords KEYWORDS Keywords to search for can be id, provider, tag and so
on
The search tool also includes the capability to search within the community datasets catalog since v0.5.4.
geeadd search --keywords ph --source community
This tool writes out or prints the underlying earthengine code for any public earthengine app. The tool has an option to export the code into a javascript file that you can then paste into Google Earth Engine code editor.
> geeadd app2script -h
usage: geeadd app2script [-h] --url URL [--outfile OUTFILE]
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--url URL Earthengine app url
Optional named arguments:
--outfile OUTFILE Write the script out to a .js file: Open in any text
editor
Simple setup can be
geeadd app2script --url "https://gena.users.earthengine.app/view/urban-lights"
or write to a javascript file which you can then open with any text editor and paste in earthengine code editor
geeadd app2script --url "https://gena.users.earthengine.app/view/urban-lights" --outfile "Full path to javascript.js"
This tool recursively goes through all your assets(Includes Images, ImageCollection,Table,) and generates a report containing the following fields [Type,Asset Type, Path,Number of Assets,size(MB),unit,owner,readers,writers]. This tool creates a detailed report and may take sometime to complete.
> geeadd ee_report -h
usage: geeadd ee_report [-h] --outfile OUTFILE
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--outfile OUTFILE This it the location of your report csv file
A simple setup is the following
geeadd --outfile "C:\johndoe\report.csv"
This tool allows you to query the size of any Earth Engine asset[Images, Image Collections, Tables and Folders] and prints out the number of assets and total asset size in non-byte encoding meaning KB, MB, GB, TB depending on size.
> geeadd assetsize -h
usage: geeadd assetsize [-h] --asset ASSET
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--asset ASSET Earth Engine Asset for which to get size properties
This script counts all currently running, cancelled, pending and failed tasks and requires no arguments.
> geeadd tasks -h
usage: geeadd tasks [-h]
optional arguments:
-h, --help show this help message and exit
This is a simple tool to cancel tasks with specific controls. This allows you to cancel all tasks, all running tasks, all ready tasks or just a single task with a task id.
> geeadd cancel -h
usage: geeadd cancel [-h] --tasks TASKS
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--tasks TASKS You can provide tasks as running or pending or all or even a
single task id
This script allows us to recursively copy entire folders, collections, images or tables. If you have read acess to assets from another user this will also allow you to copy assets from their assets.
geeadd copy -h
usage: geeadd copy [-h] [--initial INITIAL] [--final FINAL]
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--initial INITIAL Existing path of assets
--final FINAL New path for assets
This script allows us to recursively move entire folders, collections, images or table from one location to the other.
> geeadd move -h
usage: geeadd move [-h] [--initial INITIAL] [--final FINAL]
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--initial INITIAL Existing path of assets
--final FINAL New path for assets
This tool allows you to set asset access for either folder , collection or image recursively meaning you can add collection access properties for multiple assets at the same time.
> geeadd access -h
usage: geeadd access [-h] --asset ASSET --user USER --role ROLE
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--asset ASSET This is the path to the earth engine asset whose permission
you are changing folder/collection/image
--user USER "user:person@example.com" or "group:team@example.com" or
"serviceAccount:account@gserviceaccount.com", try using
"allUsers" to make it public
--role ROLE Choose between reader, writer or delete
The delete is recursive, meaning it will delete also all children assets: images, collections and folders. Use with caution!
> geeadd delete -h
usage: geeadd delete [-h] --id ID
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--id ID Full path to asset for deletion. Recursively removes all
folders, collections and images.
This tool allows you to delete a specific property across a metadata. This is useful to reset any property for an ingested collection.
> geeadd delete_metadata -h
usage: geeadd delete_metadata [-h] --asset ASSET --property PROPERTY
optional arguments:
-h, --help show this help message and exit
Required named arguments.:
--asset ASSET This is the path to the earth engine asset whose
permission you are changing collection/image
--property PROPERTY Metadata name that you want to delete
- added folder migration tools
- improved recursive folder search & copy, move and permissions tools
- major improvements to copy and move tools for better migration of nested Folders
- Minor improvements and cleanup
- Updated search tool to use updated endpoint
- Search tool no longer downloads zip or parses CSV
- Minor improvements
- Updated app2script tool to handle change in element
- Updated copy tool to allow for non mirrored copy
- Updated task and task cancel tools to account for states Pending and Cancelling
- Updated quota tool to handle GCP projects inside GEE
- Updated Folder size reporting
- Updated to use earthengine-api>= 0.1.222
- Copy and move tool improvements to facilitate cloud alpha support.
- Updated task check tool to account for operations based handling.
- Fixed issue 11.
- Updated to recent API calls based on Issue and general improvements
- Added auto version check from pypi.
- Fixed issue with delete tool and shell call.
- Fixed issue with copy and move function for single collections
- Now includes asset_url and thumbnail_url for search.
- Formatting and general improvements.
- Now inclues license in sdist
- Fixed issue with app2script tool and string and text parsing.
- Added readme and version tools.
- Added readme docs and deployed environment.
- Removed git dependency and used urllib instead based on feedback
- Created conda forge release based on Issue 10
- Fixed relative import issue for earthengine.
- Fixed image collection move tool to parse ee object type correctly as image_collection.
- Made enhancement Issue 9.
- Search tool now return earth engine asset snippet and start and end dates as JSON object.
- Removed pretty table dependency.
- Improved quota tools to get all quota and asset counts.
- Added a search tool to search GEE catalog using keywords.
- Improved parsing for app to script tool.
- Detailed asset root for all root folders and recursively
- Cancel tasks now allows you to choose, running, ready or specific task ids.
- Assets copy and move now allows you to copy entire folders, collectiona and assets recursively
- Updated assets access tool
- Delete metadata allows you to delete metadata for existing collection.
- Overall general improvements and optimization.
- General improvements
- Added tool to get underlying code from earthengine app
- Updated list and asset size functions
- Updated function to generate earthengine asset report
- General optimization and improvements to distribution
- Better error handling
- Removed upload function
- Upload handles by geeup
- General optimization and improvements to distribution
- Better error handling
- Uses poster for streaming upload more stable with memory issues and large files
- Poster dependency limits use to Py 2.7 will fix in the new version
- Major improvement to move, batch copy, and task reporting
- Major improvements to access tool to allow users read/write permission to entire Folder/collection.
- Handles bandnames during upload thanks to Lukasz for original upload code
- Removed manifest option, that can be handled by seperate tool (ppipe)
- Removing the initialization loop error
- Added improvement to earthengine authorization
- Added capability to handle PlanetScope 4Band Surface Reflectance Metadata Type
- General Improvements
- Tool improvements and enhancements
- New tool EE_Report was added
- Fixed issues with install
- Dependencies now part of setup.py
- Updated Parser and general improvements