Deepparse Address Dataset

This repository contains the address data used to train the models of our address parsing package deepparse.

The data present in this repository has been generated using data from the lipbostal project.

Content of the Project

The script to merge all the countries' datasets into a single one (make_dataset).
A script to decompress LZMA archive into normal pickle format for Python (lzma_decompress).

Download the Data

Our dataset is hosted on the address below in a zip format. You can manually download it by clicking here or you can use wget as follow:

wget https://graal.ift.ulaval.ca/public/deepparse/dataset/data.zip

About the Data

The data is split into two main categories:

clean data: addresses containing elements from at least four categories, namely a street name, a municipality, a province and a postal code
incomplete data: addresses missing at least one category amongst the aforementioned ones.

Clean data

The clean_data directory contains three subdirectory train, test and zero_shot. The number of data samples per country for the train & test dataset. Each directory contains data for 20 countries, as described in Figure 1.

Country	Number of samples in `train`	Number of samples in `test`	Country	Number of samples in `train`	Number of samples in `test`
Italy	100,000	178,848	United States	100,000	8,000,000
Germany	100,000	1,576,059	Austria	100,000	335,800
South Korea	100,000	6,048,106	Canada	100,000	910,891
Mexico	100,000	4,853,349	Australia	100,000	5,428,043
Finland	100,000	280,219	Netherlands	100,000	1,202,173
France	100,000	20,050	United Kingdom	100,000	14,338
Russia	100,000	8,115	Norway	100,000	405,649
Switzerland	100,000	474,240	Poland	100,000	459,522
Brazil	100,000	8,000,000	Denmark	100,000	199,694
Spain	100,000	1,395,758	Czechia	100,000	195,269

Figure 1. The number of data samples per country in the train and test datasets for clean data.

The zero_shot dataset contains data for an additional 41 countries. The number of data samples per country is described in Figure 2.

Country	Number of samples	Country	Number of samples
Latvia	1,325	Faroe Islands	2,982
Colombia	569	Singapore	968
Réunion	2,514	Indonesia	2,259
Japan	14,089	Portugal	4,637
Algeria	601	Belgium	66,182
Malaysia	2,043	Ukraine	9,554
Estonia	1,024	Bangladesh	888
Slovenia	9,773	Hungary	17,460
Bermuda	2,065	Romania	19,420
Philippines	10,471	Belarus	7,590
Bosnia and Herzegovina	681	Moldova, Republic of	2,376
Lithuania	3,126	Paraguay	839
Croatia	5,671	Argentina	27,692
Ireland	638	Kazakhstan	1,087
Greece	4,974	Bulgaria	3,715
Serbia	6,792	New Caledonia	1,036
Sweden	32,291	Venezuela, Bolivarian Republic of	10,696
New Zealand	4,678	Iceland	13,617
India	26,075	Uzbekistan	505
Cyprus	836	Slovakia	18,975
South Africa	1,388

Figure 2. The number of data samples per country in the zero shot dataset for clean data.

About the Incomplete Data

The incomplete directory contains two subdirectories, each containing address data for the 20 countries listed in Figure 1. The train subdirectory contains 50,000 addresses per country, and the test subdirectory contains 25,000 per country.

About the `hyphen_unit_street_number.p` Data

Some countries use a hyphen to split the unit from the street number (e.g. 3-305 a street name, where 3 is the unit and 305 is the street number), and none were properly parsed in the training data. We have created a new dataset with such cases. The dataset consists of 3 012 automatically parsed addresses using regex where the hyphen has been removed, and tags have been adjusted. Namely, before the address '3-305 a street name' tags would have been [StreetNumber, StreetName, StreetName, StreetName]. In this dataset, the address is '3 305 a street name', and the tags are [Unit, StreetNumber, StreetName, StreetName, StreetName].

About the `make_dataset` Script

The make_dataset script merges all the individual files in a particular folder into one dataset. This script merges all the data so as to allow for balanced batches during training (when the batch size is greater than the number of files).

Run the following

python3 make_dataset.py -h

to get information about the arguments needed to run the script.

About the `lzma_decompress` Script

Since the dataset is pretty large (especially for Github hosting), we have used the LZMA compressing algorithm using Python to make our files as small as possible. To do so, we used the convenient compress_pickle wrapper library to compress the data. Since LZMA is really slow to decompress, we provide a script to decompress it into standard pickle format.

The script will decompress the data files into the path_to_save directory provided as an argument, and will create the same subdirectory as described earlier (clean_data and incomplete_data plus the subdirectories train, test and zero_shot).

This script takes a couple of minutes to execute.

Cite the Dataset

If you use the data provided with this repository, please cite us using the following:

@misc{deepparse-address-data,
    author = {Marouane Yassine and David Beauchemin},
    title  = {{Structured Multinational Address Data}},
    year   = {2020},
    note   = {\url{https://github.com/GRAAL-Research/deepparse-address-data}}
}

License

This dataset is under MIT License

Dataset Metadata

The following table is necessary for this dataset to be indexed by search engines such as Google Dataset Search.

property value

name Multinational Structured Address Dataset

alternateName Address Parser Dataset

alternateName deepparse-dataset

url https://github.com/GRAAL-Research/deepparse-address-data

description The Multinational Structured Address Dataset is a collection of addresses of 61 different countries. The addresses can either be "complete" (all the usual address components) or "incomplete" (missing some usual address components). \n Example of addresses ![preview](https://rawcdn.githack.com/GRAAL-Research/deepparse-address-data/master/address_parsing_example.png)\n

creator

property	value
name	`Marouane Yassine`
sameAs	`https://scholar.google.com/citations?user=EjZyhCAAAAAJ&hl=fr&oi=sra`
name	`David Beauchemin`
sameAs	`https://scholar.google.com/citations?hl=fr&user=ntoPgSUAAAAJ`

provider

property	value
name	`GRAIL`
sameAs	`https://grail.ift.ulaval.ca/`

license

property	value
name	`MIT`
url	`https://mit-license.org/`

citation https://arxiv.org/abs/2006.16152

GRAAL-Research/deepparse-address-data