elyase/geotext

Country names with three words or more are not detected

bbo2adwuff opened this issue · 3 comments

Here in countryInfo.txt you can see several country names with three words, i.e. United Arab Emirates, Antigua and Barbuda, Bosnia and Herzegovina, Central African Republic, ...

But due to the [ \-]? of the following regex, only countries names with a maximum of one space are detected.

city_regex = r"[A-ZÀ-Ú]+[a-zà-ú]+[ \-]?(?:d[a-u].)?(?:[A-ZÀ-Ú]+[a-zà-ú]+)*"

I just realized that prior to e9204a2 at least country names with three capitalized words (i.e. United Arab Emirates, Central African Republic, ...) were detected.

What about:
city_regex = r"[A-ZÀ-Ú]+[a-zà-ú]+[ \-]?(?:and)?(?:d[a-u].)?(?:[ \-]?[A-ZÀ-Ú]+[a-zà-ú]+)*"

Still have to test it, though.

In a first attempt to test all 252 countries in countryInfo.txt using proposed regex vs the current regex in geotext I find the following.

Current regex: 21 countries are not found and 5 times not the correct country is extracted.
Proposed regex: 9 countries are not found and 2 times not the correct country is extracted.

Still not found:

 "('Bonaire, Saint Eustatius and Saba ', 'BQ') - 0",
 "('Democratic Republic of the Congo', 'CD') - 0",
 "('Republic of the Congo', 'CG') - 0",
 "('South Georgia and the South Sandwich Islands', 'GS') - 0",
 "('Heard Island and McDonald Islands', 'HM') - 0",
 "('Saint Kitts and Nevis', 'KN') - 0",
 "('Sao Tome and Principe', 'ST') - 0",
 "('Saint Vincent and the Grenadines', 'VC') - 0",
 "('U.S. Virgin Islands', 'VI') - 0"

Still not correct:

 "('Isle of Man', 'IM') != CI"
 "('Saint Pierre and Miquelon', 'PM') != MU"

And here the code how I tested it (I just copied read_table and removed the .lower()):

from geotext import GeoText
import io
from pprint import pprint


def read_table(filename, usecols=(0, 1), sep='\t', comment='#', encoding='utf-8', skip=0):
    with io.open(filename, 'r', encoding=encoding) as f:
        # skip initial lines
        for _ in range(skip):
            next(f)

        # filter comment lines
        lines = (line for line in f if not line.startswith(comment))

        d = dict()
        for line in lines:
            columns = line.split(sep)
            key = columns[usecols[0]]  # .lower()
            value = columns[usecols[1]].rstrip('\n')
            d[key] = value
    return d


countries = read_table('./countryInfo.txt', usecols=[4, 0], skip=1)

missing = []
error = []
for i in countries.items():
    country_mentions = GeoText(i[0]).country_mentions
    len_country_mentions = len(country_mentions)
    if len_country_mentions != 1:
        missing.append(str(i) + ' - ' + str(len_country_mentions))
    else:
        if list(i)[1] != list(country_mentions)[0]:
            error.append(str(i) + ' != ' + list(country_mentions)[0])

print(len(missing))
pprint(missing)

print(len(error))
pprint(error)