Go package for working with Who's On First data and S3 buckets
- This package has been superseded by whosonfirst/go-whosonfirst-blob and is no longer maintained.
- There is only local -> remote (S3) synchronization at the moment. See above.
- There is no retry (for failed uploads) code yet.
You will need to have both Go (specifically version 1.12 or higher) and the make programs installed on your computer. Assuming you do just type:
make tools
All of this package's dependencies are bundled with the code in the vendor directory.
Please write me
Given an ID (say 1159324849) this will recursively delete everything in PREFIX/115/932/484/9.
./bin/wof-s3-delete -h
Usage of ./bin/wof-s3-delete:
-dryrun
Go through the motions but don't actually delete anything.
-lambda-clients int
The number of concurrent Lambda functions to invoke. (default 10)
-lambda-dsn string
A valid go-whosonfirst-aws DSN string for talking to Lambda.
-lambda-func string
The name of the Lambda function to invoke.
-lambda-invoke
Invoke this code as a Lambda function.
-lambda-type string
A valid go-aws-sdk lambda.InvocationType string (default "RequestResponse")
-s3-dsn string
A valid go-whosonfirst-aws DSN string for talking to S3.
-stdin
Read IDs to delete from STDIN.
For example:
$> cat /usr/local/data/to-delete.csv | ./bin/wof-s3-delete -lambda-invoke -lambda-dsn 'region=us-west-2 credentials=session' -lambda-func DeleteMedia -dryrun -stdin
./bin/wof-s3-sync -h
Usage of ./bin/wof-s3-sync:
-acl string
A valid AWS S3 ACL string for permissions. (default "public-read")
-bucket string
The name of your S3 bucket. (default "data.whosonfirst.org")
-credentials string
What kind of AWS credentials to use for syncing data. (default "iam:")
-dryrun
Go through the motions but don't actually sync anything.
-dsn string
A valid go-whosonfirst-aws DSN string.
-force
Sync local files even if they haven't changed remotely.
-mode string
The mode to use for reading local data. Valid modes are: directory,feature,feature-collection,files,geojson-ls,meta,path,repo,sqlite. (default "repo")
-prefix string
The prefix (or subdirectory) for syncing data (default "data")
-rate-limit int
The maximum number or concurrent processes. (default 100000)
-region string
The region your S3 bucket lives in. (default "us-east-1")
-verbose
Be chatty.
For example:
./bin/wof-s3-sync -rate-limit 100000 -dsn 'bucket=data.whosonfirst.org region=us-east-1 prefix=data credentials=iam:' -mode repo /usr/local/data/whosonfirst-data
2017/12/12 14:12:02 109820 indexed
2017/12/12 14:13:02 209831 indexed
2017/12/12 14:14:02 309822 indexed
2017/12/12 14:15:02 409789 indexed
2017/12/12 14:16:02 509838 indexed
2017/12/12 14:17:02 609817 indexed
2017/12/12 14:18:02 709817 indexed
2017/12/12 14:19:02 809843 indexed
2017/12/12 14:20:02 909810 indexed
2017/12/12 14:20:23 time to index 9m20.532420899s
2017/12/12 14:20:23 time to index 936153 documents : 9m20.532461673s