This node script is intended to be used in order to translate Google takeout locations history to geojson.
- What is GeoJson? --> read here
- What is Google location history? --> read here
- Where can I get my Google account location history? --> look here
What follows is the content structure of a Google location history archive:
.zip
archive.html
info file- locations history folder
.json
locations history file
A basic example of an input file
{
"locations": [
{
"timestampMs": "1507330772000",
"latitudeE7": 419058658,
"longitudeE7": 125218684,
"accuracy": 16,
"velocity": 0,
"altitude": 66,
"activity": [
{
"timestampMs": "1507049587082",
"activity": [
{
"type": "TILTING",
"confidence": 100
}
]
},
{
"timestampMs": "1507049736368",
"activity": [
{
"type": "IN_VEHICLE",
"confidence": 33
},
{
"type": "STILL",
"confidence": 33
},
{
"type": "UNKNOWN",
"confidence": 17
},
{
"type": "ON_FOOT",
"confidence": 12
},
{
"type": "WALKING",
"confidence": 12
},
{
"type": "ON_BICYCLE",
"confidence": 6
}
]
}
]
}
]
}
A basic example of an output file
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
12.5218684,
41.9058658
]
},
"properties": {
"timestamp": "2017-10-06T22:59:32.000Z",
"accuracy": 16,
"velocity": null,
"altitude": 66
}
}
]
}
Even if the transformation is straightforward it is not possible to achieve it using a simple array map because of the huge amount of data to manage.
- File: 2.json
- Size: 1.16 GB
- Error message: Cannot create a string longer than 0x3fffffe7 characters
- Error code: ERR_STRING_TOO_LONG
Node cannot buffer the file for us because:
- the size of the file is bigger than the maximum one Node.js itself is capable of creating a string for.
- the size of the file is bigger than the maximum one Node.js is able to store in memory at once.
Defined in the streamProcessing file.
- [Read] --- fileToStream --> Input file to stream.
- [Transform] --- streamParser --> Consumes text, and produces a stream of data items corresponding to high-level tokens.
- [Transform] --- streamPicker --> Is a token item filter, it selects objects from a stream ignoring the rest and produces a stream of objects (locations in our case).
- [Transform] --- streamArrayer --> It assumes that an input token stream represents an array of objects and streams out assembled JavaScript objects (locations in our case).
- [Transform] --- streamGeoJsoner --> It transforms google takeout locations into GeoJson locations.
- [Transform] --- streamStringer --> It stringifies GeoJson locations.
- [Write] --- streamToFile --> Stream to Output file.
node v13.7.0
npm i
npm run test
- Copy one or more Google location history json files inside the
./input
folder. npm run start [-- fromTimestampMs toTimestampMs]
- Access the GeoJson files results of the translation inside the
./output
folder.
You can optionally set a time range window in order to filter locations out of the output.
Timestamp parameters must be defined as milliseconds epoch timestamps.
- fromTimestampMs: lower bound timestamp.
- toTimestampMs: upper bound timestamp.
Note that you can as well set only one of the two bounds either the lower or the upper one.
- Translate files without applying time filtering:
npm run start
- Maintain only locations with timestamp between Tuesday, 1 January 2019 00:00:00 and Tuesday, 31 December 2019 23:59:59:
npm run start -- 1546300800000 1577836799000
- Maintain only locations with timestamp subsequent to Tuesday, 1 January 2019 00:00:00:
npm run start -- 1546300800000
- Maintain only locations with timestamp prior to Tuesday, 31 December 2019 23:59:59:
npm run start -- x 1577836799000
- 1_locations.json
- 4_locations.json
- x_empty.json --> this is going to fail!
Please note the journey of the 2.json file:
- 1167.11297 Mb --> 1.16 GB
- 3688763 --> almost 4 milion locations processed
- 551.443 Secs --> almost 10 minutes processing
Please note how the number of GeoJson locations in the output decreases.