smnorris/bcdata

misc failures

smnorris opened this issue · 4 comments

smnorris commented

A peculiar feature count error happened, but I can't replicate. Running the command again on the same system works fine.

bcdata bc2pg --db_url $DATABASE_URL whse_admin_boundaries.fadm_designated_areas --schema bcdata -k feature_id
2023-07-10 11:58:18,999:INFO:bcdata.wfs: Getting feature geometry type
2023-07-10 11:58:19,722:INFO:bcdata.wfs: Total features requested: 10
2023-07-10 11:58:21,251:INFO:bcdata.wfs: https://openmaps.gov.bc.ca/geo/pubwfs?service=WFS&version=2.0.0&request=GetFeature&typeName=WHSE_ADMIN_BOUNDARIES.FADM_DESIGNATED_AREAS&outputFormat=json&SRSNAME=epsg%3A4326&count=10
2023-07-10 11:58:21,258:INFO:bcdata.wfs: {'start_time': 402216.564385979, 'attempt_number': 1, 'idle_for': 0}
2023-07-10 11:58:21,259:WARNING:bcdata.wfs: Dataset WHSE_ADMIN_BOUNDARIES.FADM_DESIGNATED_AREAS has multiple geometry types: MULTIPOLYGON,POLYGON
11:58:21,368:INFO:bcdata.database: Creating table bcdata.fadm_designated_areas
2023-07-10 11:58:28,447:INFO:bcdata.wfs: {'start_time': 402218.083993468, 'attempt_number': 1, 'idle_for': 0}
2023-07-10 11:58:28,448:INFO:bcdata.wfs: Total features requested: None
 Traceback (most recent call last):
   File "/home/snorris/.local/bin/bcdata", line 8, in <module>
     sys.exit(cli())
   File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
     return self.main(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
     rv = self.invoke(ctx)
   File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke
     return _process_result(sub_ctx.command.invoke(sub_ctx))
   File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
     return ctx.invoke(self.callback, **ctx.params)
   File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
     return __callback(*args, **kwargs)
   File "/home/snorris/.local/lib/python3.10/site-packages/bcdata/cli.py", line 371, in bc2pg
     out_table = bcdata.bc2pg(
   File "/home/snorris/.local/lib/python3.10/site-packages/bcdata/bc2pg.py", line 120, in bc2pg
     urls = bcdata.define_requests(
   File "/home/snorris/.local/lib/python3.10/site-packages/bcdata/wfs.py", line 181, in define_requests
     chunks = math.ceil(count / pagesize)
 TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'
smnorris commented

Again, can't replicate. Retry on owslib exceptions may be required.

bcdata bc2pg whse_tantalis.ta_crown_tenures_svw --schema bcdata -k objectid
 2023-07-13 03:22:24,424:INFO:bcdata.wfs: Getting feature geometry type
 2023-07-13 03:22:25,159:INFO:bcdata.wfs: Total features requested: 10
 Traceback (most recent call last):
   File "/home/snorris/.local/bin/bcdata", line 8, in <module>
     sys.exit(cli())
   File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
     return self.main(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
     rv = self.invoke(ctx)
   File "/usr/lib/python3/dist-packages/click/core.py", line 1659, in invoke
     return _process_result(sub_ctx.command.invoke(sub_ctx))
   File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
     return ctx.invoke(self.callback, **ctx.params)
   File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
     return __callback(*args, **kwargs)
   File "/home/snorris/.local/lib/python3.10/site-packages/bcdata/cli.py", line 371, in bc2pg
     out_table = bcdata.bc2pg(
   File "/home/snorris/.local/lib/python3.10/site-packages/bcdata/bc2pg.py", line 97, in bc2pg
     geometry_type = bcdata.get_types(dataset, 10)[0]
   File "/home/snorris/.local/lib/python3.10/site-packages/bcdata/wfs.py", line 334, in get_types
     for f in get_features(
   File "/home/snorris/.local/lib/python3.10/site-packages/bcdata/wfs.py", line 306, in get_features
     urls = define_requests(
   File "/home/snorris/.local/lib/python3.10/site-packages/bcdata/wfs.py", line 177, in define_requests
     schema = wfs.get_schema("pub:" + table)
   File "/home/snorris/.local/lib/python3.10/site-packages/owslib/feature/__init__.py", line 380, in get_schema
     return get_schema(self.url, typename, self.version, auth=self.auth)
   File "/home/snorris/.local/lib/python3.10/site-packages/owslib/feature/schema.py", line 49, in get_schema
     root = _get_remote_describefeaturetype(url, timeout=timeout,
   File "/home/snorris/.local/lib/python3.10/site-packages/owslib/feature/schema.py", line 176, in _get_remote_describefeaturetype
     res = openURL(url, timeout=timeout, headers=headers, auth=auth)
   File "/home/snorris/.local/lib/python3.10/site-packages/owslib/util.py", line 211, in openURL
     raise ServiceException(req.text)
 owslib.util.ServiceException: <?xml version="1.0" encoding="UTF-8"?><ows:ExceptionReport xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="2.0.0" xsi:schemaLocation="http://www.opengis.net/ows/1.1 http://openmaps.gov.bc.ca/geo/schemas/ows/1.1.0/owsAll.xsd">
   <ows:Exception exceptionCode="InvalidParameterValue" locator="DescribeFeatureType">
     <ows:ExceptionText>Could not find type: {http://delivery.openmaps.gov.bc.ca/geo/}WHSE_TANTALIS.TA_CROWN_TENURES_SVW</ows:ExceptionText>
   </ows:Exception>
 </ows:ExceptionReport>
smnorris commented

to minimize:

  • wrap owslib.wfs.WebfeatureService.get_schema in retry
  • cache the get_schema responses (or just the name of the geometry column) to minimize requests
smnorris commented

consider also using the catalogue for schema requests - are all geometry columns of type SDO_GEOMETRY ?

smnorris commented

not all resources are available via BCDC, it is more reliable to use the WFS schema.