Error 500 when calling us_census()

[juancalvof]

• Commit SHA:
  commit c4af4d7 (HEAD -> master, origin/master, origin/HEAD)
  Merge: f23824d 3de7d90
  Author: Manuel Alvarez Campo manuel@pythiac.com
  Date: Sat Apr 4 13:48:16 2020 +0200

  Merge pull request #35 from shaikh-raj/master

  Adding metadata for CDS datasource

• Python version: 3.7

• Operating System: Windows

• Trello card: https://trello.com/c/AtA819Yd

Description

I got the error 500 when calling us_census()

Error

data_us_census = ds.us_census.us_census()
Traceback (most recent call last):
File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\site-packages\IPython\core\interactiveshell.py", line 3331, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
data_us_census = ds.us_census.us_census()
File "c:\users\juanc\desktop\01 juan calvo\01 big data\01 proyectos\covid-19\code\repositories\task-geo\task_geo\data_sources\demographics\us_census\us_census.py", line 31, in us_census
df = us_census_connector()
File "c:\users\juanc\desktop\01 juan calvo\01 big data\01 proyectos\covid-19\code\repositories\task-geo\task_geo\data_sources\demographics\us_census\us_census.py", line 47, in us_census_connector
urllib.request.urlretrieve(url, "uscensus.zip")
File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 247, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 531, in open
response = meth(req, response)
File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 569, in error
return self._call_chain(*args)
File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 503, in _call_chain
result = func(*args)
File "C:\Users\juanc\Anaconda3\envs\coronawhy\lib\urllib\request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Internal Server Error

[ManuelAlvarezC]

Hi @JuanCalvoFerrandiz, and thanks for reporting.

I have been able to replicate it and will start fixing it soon. I will keep you updated.

[DanielRobertNicoud]

The .zip file that the program wants to download is this one:
https://data.census.gov/cedsci/table?q=demographic%20data%20by%20county&hidePreview=false&tid=ACSCP5Y2018.CP05&vintage=2018&g=0100000US.050000

[cgomez9]

What is causing the problem is that the dataset URL was obtained getting the link address for the downloaded file in the web browser. This URL has the following form:

https://data.census.gov/api/access/table/download?download_id=TOKEN

The problem is that this TOKEN is changed in each download, making it useless when trying to download the file multiple times. We need to find a URL using the API directly.