Can download successfully, but error trying to run `addressr-loader`

Question

Can download successfully, but error trying to run `addressr-loader`

musiwei opened this issue 2 years ago · 20 comments

It happened on both local and ec2. It could download and unzip the files successfully. Then the error happens in the end. Could you please shed some lights?

How to repeat?

Run addressr-loader
See below errors:

[ec2-user@ip-172-31-1-204 addressr]$ addressr-loader
  api trying to reach elastic search on 127.0.0.1:9200... +0ms
Waiting for 127.0.0.1:9200.
Connected!
  api ...127.0.0.1:9200 is reachable +23ms
  api connecting elastic search client on 127.0.0.1:9200... +4ms
  api ...connected to 127.0.0.1:9200 +26ms
  api es client connected +0ms
======================
Addressr - Data Loader
======================
Version: 1.0.265
NODE_ENV: development|(set NODE_ENV to 'production' in production environments)
PORT: 8080
  api cached gnaf package data {
  body: `{"help": "https://data.gov.au/data/api/3/action/help_show?name=package_show", "success": true, "result": {"author": "Department of Industry, Science, Energy and Resources", "author_email": null, "contact_info": false, "contact_point": "data@industry.gov.au", "creator_user_id": "b8515871-b872-425b-9f5f-d62159f7534f", "data_model": "", "data_state": "active", "field_of_research": [], "geospatial_topic": [], "id": "19432f89-dc3a-4ef3-b943-5326ef1dbecc", "isopen": true, "jurisdiction": "Commonwealth of Australia", "language": "English", "license_id": "other-open", "license_title": "Other (Open)", "maintainer": null, "maintainer_email": null, "metadata_created": "2016-02-22T03:29:52.853307", "metadata_modified": "2023-02-23T01:30:50.556443", "name": "geocoded-national-address-file-g-naf", "notes": "Geoscape G-NAF is the geocoded address database for Australian businesses and governments. It\\u2019s the trusted source of geocoded address data for Australia with over 50 million contributed addresses distilled into 15.4 million G-NAF addresses. It is built and maintained by [Geoscape Australia](https://geoscape.com.au/) using independently examined and validated government data. \\r\\n\\r\\nFrom 22 August 2022, Geoscape Australia is making G-NAF available in an additional simplified table format. [G-NAF Core](https://geoscape.com.au/data/g-naf-core/?utm_source=DISER&utm_medium=referral&utm_term=data.gov.au&utm_content=download+data&utm_campaign=G-NAF+core) makes accessing geocoded addresses easier by utilising less technical effort.\\r\\n\\r\\nG-NAF Core is available [here](https://geoscape.com.au/data/g-naf-core/?utm_source=DISER&utm_medium=referral&utm_term=data.gov.au&utm_content=download+data&utm_campaign=G-NAF+core) and will be updated on a quarterly basis along with G-NAF.\\r\\n\\r\\nFurther information about contributors to G-NAF is available [here](https://geoscape.com.au/legal/data-copyright-and-disclaimer/).\\r\\n\\r\\nWith more than 15 million Australian physical address record, G-NAF is one of the most ubiquitous and powerful spatial datasets. The records include geocodes, which are latitude and longitude map coordinates. G-NAF does not contain personal information or details relating to individuals.\\r\\n\\r\\nUpdated versions of G-NAF are published on a quarterly basis. Previous versions are available [here](https://data.gov.au/dataset/ds-dga-e1a365fc-52f5-4798-8f0c-ed1d33d43b6d/details?q=previous%20gnaf)\\r\\n\\r\\nUsers have the option to download datasets with feature coordinates referencing either GDA94 or GDA2020 datums.\\r\\n\\r\\n**Changes in the February 2023 release**\\r\\n\\r\\n* Nationally, the February 2023 update of G-NAF shows an increase of 40,921 addresses overall (+0.27%). The total number of addresses in G-NAF now stands at 15,440,500 of which 14,623,727 or 94.71% are principal.\\r\\n\\r\\n* Ongoing changes include: identifying and creating principal\\u2013alias address relationships including address de-duplication, ongoing validation and removal of sensitive information in the 'building_name', 'address_site_name' and 'geocode_site_name' attributes, targeted address improvements and corrections as required.\\r\\n\\r\\n**Changes in the November 2022 release**\\r\\n\\r\\n* Nationally, the November 2022 update of G-NAF shows an increase of 42,093 addresses \\r\\noverall (+0.27%). The total number of addresses in G-NAF now stands at 15,399,579 of \\r\\nwhich 14,585,377 or 94.71% are principal.\\r\\n\\r\\n* Ongoing changes. These include: identifying and creating principal-alias address relationships including address de-duplication; ongoing validation and removal of sensitive information in selected attributes; and targeted address improvements and corrections as required.\\r\\n\\r\\nFurther information on G-NAF, including FAQs on the data, is available [here](https://geoscape.com.au/data/g-naf) or through Geoscape Australia\\u2019s network of [partners](https://geoscape.com.au/partners/). They provide a range of commercial products based on G-NAF, including software solutions, consultancy and support.\\r\\n\\r\\nAdditional information: On 1 October 2020, PSMA Australia Limited began trading as Geoscape Australia.\\r\\n\\r\\n##License Information\\r\\n\\r\\n[Use of the G-NAF downloaded from data.gov.au is subject to the End User Licence Agreement (EULA)](https://data.gov.au/dataset/ds-dga-e1a365fc-52f5-4798-8f0c-ed1d33d43b6d/distribution/dist-dga-0102be65-3781-42d9-9458-fdaf7170efed/details?q=previous%20gnaf)\\r\\n\\r\\nThe EULA terms are based on the Creative Commons Attribution 4.0 International license (CC BY 4.0).  However, an important restriction relating to the use of the open G-NAF for the sending of mail has been added. \\r\\n\\r\\nThe open G-NAF data must not be used for the generation of an address or the compilation of an address for the sending of mail unless the user has verified that each address to be used for the sending of mail is capable of receiving mail by reference to a secondary source of information.  Further information on this use restriction [is available here](https://data.gov.au/dataset/ds-dga-19432f89-dc3a-4ef3-b943-5326ef1dbecc/distribution/dist-dga-9a8f6baa-f790-49a0-84b1-3cb39a6a1b88/details?q=gnaf).\\r\\n\\r\\nEnd users must only use the data in ways that are consistent with the [Australian Privacy Principles](https://www.oaic.gov.au/privacy-law/privacy-act/australian-privacy-principles) issued under the Privacy Act 1988 (Cth).\\r\\n\\r\\n**Users must also note the following attribution requirements:**\\r\\n\\r\\n_Preferred attribution for the Licensed Material:_\\r\\n\\r\\n> _G-NAF \\u00a9 [Geoscape Australia](https://geoscape.com.au/legal/data-copyright-and-disclaimer/) licensed by the Commonwealth of Australia under the _[_Open Geo-coded National Address File (G-NAF) End User Licence Agreement_](https://data.gov.au/dataset/ds-dga-19432f89-dc3a-4ef3-b943-5326ef1dbecc/distribution/dist-dga-09f74802-08b1-4214-a6ea-3591b2753d30/details?q=gnaf).\\r\\n\\r\\n_Preferred attribution for Adapted Material:_\\r\\n\\r\\n> _Incorporates or developed using G-NAF \\u00a9 [Geoscape Australia](https://geoscape.com.au/legal/data-copyright-and-disclaimer/) licensed by the Commonwealth of Australia under the_ [_Open Geo-coded National Address File (G-NAF) End User Licence Agreement_](https://data.gov.au/dataset/ds-dga-19432f89-dc3a-4ef3-b943-5326ef1dbecc/distribution/dist-dga-09f74802-08b1-4214-a6ea-3591b2753d30/details?q=gnaf).\\r\\n\\r\\n## What to Expect When You Download G-NAF\\r\\n\\r\\nG-NAF is a complex and large dataset (approximately 5GB unpacked), consisting of multiple tables that will need to be joined prior to use. The dataset is primarily designed for application developers and large-scale spatial integration. Users are advised to read the [technical documentation](https://geoscape.com.au/documentation/), including product change notices and the individual product descriptions before downloading and using the product. [A quick reference guide on unpacking the G-NAF is also available.](https://geoscape.com.au/wp-content/uploads/2020/09/Unpacking-GNAF-.pdf)\\r\\n\\r\\n", "num_resources": 5, "num_tags": 11, "organization": {"id": "90153a8c-6a29-4068-a97d-7cf06dbef700", "name": "department-of-industry", "title": "Department of Industry, Science and Resources (DISR)", "type": "organization", "description": "The department drives growth and job creation for a more prosperous Australia, by facilitating economic transformation and boosting business competitiveness.", "image_url": "https://www.industry.gov.au/themes/custom/industry_theme/logo.png", "created": "2013-04-18T15:36:47.465027", "is_organization": true, "approval_status": "approved", "state": "active"}, "owner_org": "90153a8c-6a29-4068-a97d-7cf06dbef700", "private": false, "spatial": "{\\"type\\": \\"Polygon\\", \\"coordinates\\": [[[112.0, -44.0], [154.0, -44.0], [154.0, -9.0], [112.0, -9.0], [112.0, -44.0]]]}", "spatial_coverage": "{\\"type\\": \\"Polygon\\", \\"coordinates\\": [[[112.0, -44.0], [154.0, -44.0], [154.0, -9.0], [112.0, -9.0], [112.0, -44.0]]]}", "state": "active", "temporal_coverage_from": "2021-11-22 00:00:00", "title": "Geoscape Geocoded National Address File (G-NAF)", "type": "dataset", "unpublished": false, "update_freq": "quarterly", "url": null, "version": null, "resources": [{"cache_last_updated": null, "cache_url": null, "created": "2022-11-21T00:01:09.029111", "datastore_active": false, "datastore_contains_all_records_of_source_file": false, "description": "", "format": "PDF", "hash": "", "id": "7cff62e0-79c8-4c3f-8f18-ef77d9a619eb", "last_modified": "2023-02-20", "metadata_modified": "2023-02-21T00:30:58.911811", "mimetype": "application/pdf", "mimetype_inner": null, "name": "G-NAF Release Report February 2023", "package_id": "19432f89-dc3a-4ef3-b943-5326ef1dbecc", "position": 0, "resource_type": null, "size": 296972, "state": "active", "url": "https://data.gov.au/data/dataset/19432f89-dc3a-4ef3-b943-5326ef1dbecc/resource/7cff62e0-79c8-4c3f-8f18-ef77d9a619eb/download/g-naf-release-report-february-2023.pdf", "url_type": "upload", "zip_extract": false}, {"cache_last_updated": null, "cache_url": null, "created": "2022-11-20T23:59:45.837863", "datastore_active": false, "datastore_contains_all_records_of_source_file": false, "description": "GDA94", "format": "ZIP", "hash": "", "id": "f06900d8-69c7-45d3-bc1d-f0bdc98d6bcf", "last_modified": "2023-02-20", "metadata_modified": "2023-02-21T00:30:58.911931", "mimetype": "application/zip", "mimetype_inner": null, "name": "FEB 23 - Geoscape G-NAF - GDA94", "package_id": "19432f89-dc3a-4ef3-b943-5326ef1dbecc", "position": 1, "resource_type": null, "size": 1615461846, "state": "active", "url": "https://data.gov.au/data/dataset/19432f89-dc3a-4ef3-b943-5326ef1dbecc/resource/f06900d8-69c7-45d3-bc1d-f0bdc98d6bcf/download/g-naf_feb23_allstates_gda94_psv_1010.zip", "url_type": "upload", "zip_extract": false}, {"cache_last_updated": null, "cache_url": null, "created": "2022-11-21T00:00:32.091571", "datastore_active": false, "datastore_contains_all_records_of_source_file": false, `... 4656 more characters,
  headers: {
    'content-type': 'application/json;charset=utf-8',
    'transfer-encoding': 'chunked',
    connection: 'close',
    vary: 'Accept-Encoding, Origin',
    date: 'Thu, 16 Mar 2023 01:34:15 GMT',
    server: 'nginx',
    'cache-control': 'public, max-age=0, must-revalidate',
    'content-encoding': 'gzip',
    'x-cache': 'Miss from cloudfront',
    via: '1.1 bc177ce25ddc555a7d303bc4d290a6ec.cloudfront.net (CloudFront)',
    'x-amz-cf-pop': 'SYD62-P2',
    'x-amz-cf-id': '6gt_aFuv_8FFtz9hozDKulr3oyR00HGeuQHfBlBQpb0U4kh3h158NA==',
    'x-xss-protection': '1; mode=block',
    'x-frame-options': 'SAMEORIGIN',
    'referrer-policy': 'strict-origin-when-cross-origin',
    'x-content-type-options': 'nosniff',
    'strict-transport-security': 'max-age=31536000'
  }
} +0ms
  api created 2023-03-16T01:34:15.000Z +4ms
  api dataResource {
  "cache_last_updated": null,
  "cache_url": null,
  "created": "2022-11-20T23:59:45.837863",
  "datastore_active": false,
  "datastore_contains_all_records_of_source_file": false,
  "description": "GDA94",
  "format": "ZIP",
  "hash": "",
  "id": "f06900d8-69c7-45d3-bc1d-f0bdc98d6bcf",
  "last_modified": "2023-02-20",
  "metadata_modified": "2023-02-21T00:30:58.911931",
  "mimetype": "application/zip",
  "mimetype_inner": null,
  "name": "FEB 23 - Geoscape G-NAF - GDA94",
  "package_id": "19432f89-dc3a-4ef3-b943-5326ef1dbecc",
  "position": 1,
  "resource_type": null,
  "size": 1615461846,
  "state": "active",
  "url": "https://data.gov.au/data/dataset/19432f89-dc3a-4ef3-b943-5326ef1dbecc/resource/f06900d8-69c7-45d3-bc1d-f0bdc98d6bcf/download/g-naf_feb23_allstates_gda94_psv_1010.zip",
  "url_type": "upload",
  "zip_extract": false
} +1ms
  api url https://data.gov.au/data/dataset/19432f89-dc3a-4ef3-b943-5326ef1dbecc/resource/f06900d8-69c7-45d3-bc1d-f0bdc98d6bcf/download/g-naf_feb23_allstates_gda94_psv_1010.zip +0ms
  api headers {
  "content-type": "application/json;charset=utf-8",
  "transfer-encoding": "chunked",
  "connection": "close",
  "vary": "Accept-Encoding, Origin",
  "date": "Thu, 16 Mar 2023 01:34:15 GMT",
  "server": "nginx",
  "cache-control": "public, max-age=0, must-revalidate",
  "content-encoding": "gzip",
  "x-cache": "HIT",
  "via": "1.1 bc177ce25ddc555a7d303bc4d290a6ec.cloudfront.net (CloudFront)",
  "x-amz-cf-pop": "SYD62-P2",
  "x-amz-cf-id": "6gt_aFuv_8FFtz9hozDKulr3oyR00HGeuQHfBlBQpb0U4kh3h158NA==",
  "x-xss-protection": "1; mode=block",
  "x-frame-options": "SAMEORIGIN",
  "referrer-policy": "strict-origin-when-cross-origin",
  "x-content-type-options": "nosniff",
  "strict-transport-security": "max-age=31536000"
} +0ms
  api basename g-naf_feb23_allstates_gda94_psv_1010.zip +0ms
  api directory exits. Skipping extract target/gnaf/g-naf_feb23_allstates_gda94_psv_1010 +7ms
  api Data dir target/gnaf/g-naf_feb23_allstates_gda94_psv_1010 +0ms
  api Data dir contents [ 'Contents.txt', 'G-NAF' ] +0ms
[ 'G-NAF/' ]
  api Main Data dir target/gnaf/g-naf_feb23_allstates_gda94_psv_1010 +55ms
  error Error: ENOENT: no such file or directory, access 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/Counts.csv' +0ms
  api reading /home/ec2-user/addressr/target/gnaf/g-naf_feb23_allstates_gda94_psv_1010 (. in target/gnaf/g-naf_feb23_allstates_gda94_psv_1010) +2ms
  api reading /home/ec2-user/addressr/target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/G-NAF (./G-NAF in target/gnaf/g-naf_feb23_allstates_gda94_psv_1010) +1ms
  api reading /home/ec2-user/addressr/target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/G-NAF/Documents (./G-NAF/Documents in target/gnaf/g-naf_feb23_allstates_gda94_psv_1010) +10ms
  api reading /home/ec2-user/addressr/target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/G-NAF/Extras (./G-NAF/Extras in target/gnaf/g-naf_feb23_allstates_gda94_psv_1010) +0ms
  api reading /home/ec2-user/addressr/target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/G-NAF/G-NAF FEBRUARY 2023 (./G-NAF/G-NAF FEBRUARY 2023 in target/gnaf/g-naf_feb23_allstates_gda94_psv_1010) +3ms
  api reading /home/ec2-user/addressr/target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/G-NAF/Extras/GNAF_TableCreation_Scripts (./G-NAF/Extras/GNAF_TableCreation_Scripts in target/gnaf/g-naf_feb23_allstates_gda94_psv_1010) +0ms
  api reading /home/ec2-user/addressr/target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/G-NAF/Extras/GNAF_View_Scripts (./G-NAF/Extras/GNAF_View_Scripts in target/gnaf/g-naf_feb23_allstates_gda94_psv_1010) +0ms
  api reading /home/ec2-user/addressr/target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/G-NAF/G-NAF FEBRUARY 2023/Authority Code (./G-NAF/G-NAF FEBRUARY 2023/Authority Code in target/gnaf/g-naf_feb23_allstates_gda94_psv_1010) +1ms
  api reading /home/ec2-user/addressr/target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/G-NAF/G-NAF FEBRUARY 2023/Standard (./G-NAF/G-NAF FEBRUARY 2023/Standard in target/gnaf/g-naf_feb23_allstates_gda94_psv_1010) +0ms
  api authCodeFiles [
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_ADDRESS_ALIAS_TYPE_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_ADDRESS_CHANGE_TYPE_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_ADDRESS_TYPE_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_FLAT_TYPE_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_GEOCODED_LEVEL_TYPE_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_GEOCODE_RELIABILITY_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_GEOCODE_TYPE_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_LEVEL_TYPE_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_LOCALITY_ALIAS_TYPE_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_LOCALITY_CLASS_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_MB_MATCH_CODE_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_PS_JOIN_TYPE_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_STREET_CLASS_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_STREET_LOCALITY_ALIAS_TYPE_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_STREET_SUFFIX_AUT_psv.psv',
  './G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_STREET_TYPE_AUT_psv.psv'
] +55s
  api loaded '8' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_ADDRESS_ALIAS_TYPE_AUT_psv.psv' into key 'Authority_Code_ADDRESS_ALIAS_TYPE_AUT_psv' +12ms
  api loaded '511' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_ADDRESS_CHANGE_TYPE_AUT_psv.psv' into key 'Authority_Code_ADDRESS_CHANGE_TYPE_AUT_psv' +3ms
  api loaded '56' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_ADDRESS_TYPE_AUT_psv.psv' into key 'Authority_Code_ADDRESS_TYPE_AUT_psv' +2ms
  api loaded '53' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_FLAT_TYPE_AUT_psv.psv' into key 'Authority_Code_FLAT_TYPE_AUT_psv' +1ms
  api loaded '8' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_GEOCODED_LEVEL_TYPE_AUT_psv.psv' into key 'Authority_Code_GEOCODED_LEVEL_TYPE_AUT_psv' +1ms
  api loaded '6' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_GEOCODE_RELIABILITY_AUT_psv.psv' into key 'Authority_Code_GEOCODE_RELIABILITY_AUT_psv' +1ms
  api loaded '29' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_GEOCODE_TYPE_AUT_psv.psv' into key 'Authority_Code_GEOCODE_TYPE_AUT_psv' +1ms
  api loaded '15' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_LEVEL_TYPE_AUT_psv.psv' into key 'Authority_Code_LEVEL_TYPE_AUT_psv' +1ms
  api loaded '2' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_LOCALITY_ALIAS_TYPE_AUT_psv.psv' into key 'Authority_Code_LOCALITY_ALIAS_TYPE_AUT_psv' +2ms
  api loaded '9' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_LOCALITY_CLASS_AUT_psv.psv' into key 'Authority_Code_LOCALITY_CLASS_AUT_psv' +1ms
  api loaded '5' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_MB_MATCH_CODE_AUT_psv.psv' into key 'Authority_Code_MB_MATCH_CODE_AUT_psv' +1ms
  api loaded '2' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_PS_JOIN_TYPE_AUT_psv.psv' into key 'Authority_Code_PS_JOIN_TYPE_AUT_psv' +2ms
  api loaded '2' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_STREET_CLASS_AUT_psv.psv' into key 'Authority_Code_STREET_CLASS_AUT_psv' +1ms
  api loaded '2' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_STREET_LOCALITY_ALIAS_TYPE_AUT_psv.psv' into key 'Authority_Code_STREET_LOCALITY_ALIAS_TYPE_AUT_psv' +1ms
  api loaded '19' rows from 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_STREET_SUFFIX_AUT_psv.psv' into key 'Authority_Code_STREET_SUFFIX_AUT_psv' +1ms
  error Error loading 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_STREET_TYPE_AUT_psv.psv'. Expected '270' rows, got '271' +55s
  error error loading data Error loading 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_STREET_TYPE_AUT_psv.psv'. Expected '270' rows, got '271' +0ms
node:internal/process/promises:279
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "Error loading 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Authority Code/Authority_Code_STREET_TYPE_AUT_psv.psv'. Expected '270' rows, got '271'".] {
  code: 'ERR_UNHANDLED_REJECTION'
}

And if I send a request:

Waiting for 127.0.0.1:9200.
Connected!
=======================
Addressr - API Server 2
=======================
Version: 1.0.265
NODE_ENV: development|(set NODE_ENV to 'production' in production environments)
PORT: 8080
ResponseError: index_not_found_exception: [index_not_found_exception] Reason: no such index [addressr]
    at onBody (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/node_modules/@opensearch-project/opensearch/lib/Transport.js:425:23)
    at IncomingMessage.onEnd (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/node_modules/@opensearch-project/opensearch/lib/Transport.js:340:11)
    at IncomingMessage.emit (node:events:525:35)
    at endReadableNT (node:internal/streams/readable:1358:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  meta: {
    body: { error: [Object], status: 404 },
    statusCode: 404,
    headers: {
      'content-type': 'application/json; charset=UTF-8',
      'content-length': '377'
    },
    meta: {
      context: null,
      request: [Object],
      name: 'opensearch-js',
      connection: [Object],
      attempts: 0,
      aborted: false
    }
  }
}

Answer 1 · 2023-03-16T06:56:10.000Z

Hi, that looks like an error with the latest data file from data.gov.au. I'll have a look.

Answer 2 · 2023-03-16T07:38:04.000Z

Thank-you Tom, I tried with Nov 2022 data too, same failure.

Answer 3 · 2023-03-16T12:28:42.000Z

Sorry, Tom, unfortunately, the error remains the same, I tried:

run npm update and checked the file had been updated
reinstall plugin
redownload the whole dataset

As a temporary solution, I manually added a new line to some files.

However, there is a new error seems from invalid data:

...
  api Loading streets ACT +3ms
  api Loading suburbs ACT +91ms
  api Skipping geos. set 'ADDRESSR_ENABLE_GEO' env var to enable +3ms
  api addr {
  "structured": {
    "number": {
      "number": 5
    },
    "street": {
      "name": "CUTLACK",
      "type": {
        "code": "STREET",
        "name": "ST"
      },
      "class": {
        "code": "C",
        "name": "CONFIRMED"
      }
    },
    "confidence": 2,
    "locality": {
      "name": "EVATT",
      "class": {
        "code": "G",
        "name": "GAZETTED LOCALITY"
      }
    },
    "postcode": "2617",
    "state": {
      "name": "AUSTRALIAN CAPITAL TERRITORY",
      "abbreviation": "ACT"
    }
  },
  "pid": "GAACT714958541",
  "mla": [
    "5 CUTLACK ST",
    "EVATT ACT 2617"
  ],
  "sla": "5 CUTLACK ST, EVATT ACT 2617"
} +34ms
  api 0% +0ms
  error Unknown Level Type Code: '_9' +56s
  error Unknown Flat Type Code: '_5' +0ms
  error Errors reading 'target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/./G-NAF/G-NAF FEBRUARY 2023/Standard/ACT_ADDRESS_DETAIL_psv.psv': [object Object] +332ms

Tried with NSW data only, looks like they introduced some invalid data too.

  api Loading streets NSW +3ms
  api Loading suburbs NSW +1s
  api Skipping geos. set 'ADDRESSR_ENABLE_GEO' env var to enable +59ms
  api addr {
  "structured": {
    "number": {
      "number": 42,
      "last": {
        "number": 48
      }
    },
    "flat": {
      "type": {
        "code": "UNIT",
        "name": "UNIT"
      },
      "number": 12
    },
    "street": {
      "name": "MERTON",
      "type": {
        "code": "STREET",
        "name": "ST"
      },
      "class": {
        "code": "C",
        "name": "CONFIRMED"
      }
    },
    "confidence": 2,
    "locality": {
      "name": "SUTHERLAND",
      "class": {
        "code": "G",
        "name": "GAZETTED LOCALITY"
      }
    },
    "postcode": "2232",
    "state": {
      "name": "NEW SOUTH WALES",
      "abbreviation": "NSW"
    }
  },
  "precedence": "secondary",
  "pid": "GANSW717907810",
  "mla": [
    "UNIT 12",
    "42-48 MERTON ST",
    "SUTHERLAND NSW 2232"
  ],
  "sla": "UNIT 12, 42-48 MERTON ST, SUTHERLAND NSW 2232",
  "smla": [
    "12/42-48 MERTON ST",
    "SUTHERLAND NSW 2232"
  ],
  "ssla": "12/42-48 MERTON ST, SUTHERLAND NSW 2232"
} +17ms
  api 0% +0ms
  error Unknown Level Type Code: '_6' +57s
  error error sending index request TypeError: Cannot read properties of undefined (reading 'STREET_NAME')
    at mapStreetLocality (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/lib/service/address-service.js:480:18)
    at mapAddressDetails (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/lib/service/address-service.js:756:15)
    at /home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/lib/service/address-service.js:827:26
    at Array.forEach (<anonymous>)
    at Object.chunk (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/lib/service/address-service.js:826:22)
    at ReadableStreamStreamer.ChunkStreamer.parseChunk (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/node_modules/papaparse/papaparse.js:554:18)
    at ReadableStreamStreamer._nextChunk (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/node_modules/papaparse/papaparse.js:866:10)
    at ReadableStreamStreamer.ChunkStreamer.parseChunk (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/node_modules/papaparse/papaparse.js:575:10)
    at ParserHandle.resume (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/node_modules/papaparse/papaparse.js:1129:19)
    at /home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/lib/service/address-service.js:853:22 +12ms
/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/lib/service/address-service.js:480
  return { ...(l.STREET_NAME !== '' && {
                 ^

TypeError: Cannot read properties of undefined (reading 'STREET_NAME')
    at mapStreetLocality (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/lib/service/address-service.js:480:18)
    at mapAddressDetails (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/lib/service/address-service.js:756:15)
    at /home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/lib/service/address-service.js:827:26
    at Array.forEach (<anonymous>)
    at Object.chunk (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/lib/service/address-service.js:826:22)
    at ReadableStreamStreamer.ChunkStreamer.parseChunk (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/node_modules/papaparse/papaparse.js:554:18)
    at ReadableStreamStreamer._nextChunk (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/node_modules/papaparse/papaparse.js:866:10)
    at ReadableStreamStreamer.ChunkStreamer.parseChunk (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/node_modules/papaparse/papaparse.js:575:10)
    at ParserHandle.resume (/home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/node_modules/papaparse/papaparse.js:1129:19)
    at /home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/lib/service/address-service.js:853:22

Answer 4 · 2023-03-16T20:58:27.000Z

The stack trace line numbers don't line up with the current version of addressr. Please try removing it and explicitly installing the latest version. e.g. npm i @mountainpass/addressr@1.0.266 -g

Answer 5 · 2023-03-16T22:23:34.000Z

The version showed correctly in package.json, but the source file was old.

vim /home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/package.json

vim /home/ec2-user/.nvm/versions/node/v16.19.1/lib/node_modules/@mountainpass/addressr/lib/service/address-service.js

On github

Answer 6 · 2023-03-16T23:18:34.000Z

Quick update: just tried on my local and the same issue repeated. Looks like npm is not aware of the change.

Answer 7 · 2023-03-16T23:23:56.000Z

npm's Code feature does not contain this update neither: https://www.npmjs.com/package/@mountainpass/addressr/v/1.0.268?activeTab=explore

Note the line number is messed up so I copied the code across to my IDE:

Answer 8 · 2023-03-19T12:08:18.000Z

Hey Tom, I hope you have had a great weekend :) Could you please take a look at the above when you have time? Thanks in advance!

Answer 9 · 2023-03-20T10:18:12.000Z

I'm looking into it...

Answer 10 · 2023-03-20T12:39:33.000Z

I've reproduced the error and there's definitely something weird going on. Here's some of the output and I have no idea where all those '_*' values are coming from. They're not in the GNAF files.

  error Unknown Level Type Code: '_9' +12s
  error {
  error   address: {
  error     ADDRESS_DETAIL_PID: 'GAVIC420457363',
  error     DATE_CREATED: '2004-04-29',
  error     DATE_LAST_MODIFIED: '2021-07-07',
  error     DATE_RETIRED: '',
  error     BUILDING_NAME: '_1',
  error     LOT_NUMBER_PREFIX: '_2',
  error     LOT_NUMBER: '_3',
  error     LOT_NUMBER_SUFFIX: '_4',
  error     FLAT_TYPE_CODE: '_5',
  error     FLAT_NUMBER_PREFIX: '_6',
  error     FLAT_NUMBER: '_7',
  error     FLAT_NUMBER_SUFFIX: '_8',
  error     LEVEL_TYPE_CODE: '_9',
  error     LEVEL_NUMBER_PREFIX: '_10',
  error     LEVEL_NUMBER: '_11',
  error     LEVEL_NUMBER_SUFFIX: '_12',
  error     NUMBER_FIRST_PREFIX: '_13',
  error     NUMBER_FIRST: '25',
  error     NUMBER_FIRST_SUFFIX: '_14',
  error     NUMBER_LAST_PREFIX: '_15',
  error     NUMBER_LAST: '_16',
  error     NUMBER_LAST_SUFFIX: '_17',
  error     STREET_LOCALITY_PID: 'VIC1982259',
  error     LOCATION_DESCRIPTION: '_18',
  error     LOCALITY_PID: 'locb3584a0e11e8',
  error     ALIAS_PRINCIPAL: 'P',
  error     POSTCODE: '3338',
  error     PRIVATE_STREET: '_19',
  error     LEGAL_PARCEL_ID: '200\\LP52514',
  error     CONFIDENCE: '2',
  error     ADDRESS_SITE_PID: '420595172',
  error     LEVEL_GEOCODED_CODE: '7',
  error     PROPERTY_PID: '_20',
  error     GNAF_PROPERTY_PID: '1415611',
  error     PRIMARY_SECONDARY: '_21'
  error   }
  error } +1ms

Answer 11 · 2023-03-20T20:48:13.000Z

Thanks for looking into this Tom.

I also looked into GNAF and couldn't see the _*.

As a last resort, we can still filter the string if it matches the pattern _{number}.

But hold your beer! Did you notice that they may be placeholders? For the fields that are empty on each row, a number is given, and they follow a sequential order.

Answer 12 · 2023-03-20T21:08:53.000Z

I found the record you were looking into in target/gnaf/g-naf_feb23_allstates_gda94_psv_1010/G-NAF/G-NAF FEBRUARY 2023/Standard/VIC_ADDRESS_DETAIL_psv.psv:

GAVIC420457363|2004-04-29|2021-07-07|||||||||||||||25|||||VIC1982259||locb3584a0e11e8|P|3338||200\LP52514|2|420595172|7||1415611|

Reformat a bit and it matched with the above guess except DATE_RETIRED field, but it's probably because fields that do not require indexing will be replaced by an empty string automatically.

All the empty lines with only a bar is a placeholder _{number} in your output.

GAVIC420457363|
2004-04-29|
2021-07-07|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
25|
|
|
|
|
VIC1982259|
|
locb3584a0e11e8|
P|
3338|
|
200\LP52514|
2|
420595172|
7|
|
1415611|

I hope this could be helpful, let me know what you think.

Answer 13 · 2023-03-20T21:29:29.000Z

Safe, yes. Hacky, also yes.

Answer 14 · 2023-03-20T21:41:23.000Z

Haha I agree, but I can't think of a better/quicker solution...

Do you think it's a bad idea to use regular expression to replace the '_{number}' pattern _([1-9][0-9]*) with ''?

Answer 15 · 2023-03-21T07:57:02.000Z

Thank you! Please let me know when the latest version is published on npm.

Answer 16 · 2023-03-21T22:01:26.000Z

Yep, working on it

Answer 17 · 2023-03-26T12:34:59.000Z

Hi Tom, any updates for the above? Thanks :)

Answer 18 · 2023-03-27T20:58:49.000Z

@musiwei 1.0.269 has been released. https://www.npmjs.com/package/@mountainpass/addressr/v/1.0.269

enjoy

Answer 19 · 2023-03-27T21:52:30.000Z

Thank you sir!

Answer 20 · 2023-03-29T00:13:36.000Z

Much appreciate your quick fix, Tom! 💯

Just to give a little something back: the new dataset requires more than 2G memory or the import will FAIL. I tested with both 2G and 4G memory EC2 instances, while 2G crashed OpenSearch. I reckon this could be added to README for those who need it.