Downloading the HMDA data

Question

Downloading the HMDA data

SantoshSrinivas79 opened this issue 11 years ago · 14 comments

SantoshSrinivas79 commented 11 years ago

Is there a way to create a torrent to allow downloading the data more effectively for offline analysis. I am trying to download the entire data available at The Home Mortgage Disclosure Act - Explore

The data is about 72.3 GB. There is absolutely no way of downloading it. The download times out and fails through a download manager too.

Please advise.

Answer 1 · 2014-02-13T15:01:30.000Z

@fountainhead Thanks for your question. I can't speak for the @cfpb as to what their current plans definitely are, but I think that putting up the entire data set as static files on S3 or something like that is planned. I will check in and see what I can do to make that happen.

Answer 2 · 2014-02-13T15:03:40.000Z

Thanks. That would help.

Answer 3 · 2014-02-21T06:42:37.000Z

Any luck with this request?

Answer 4 · 2014-02-21T14:43:19.000Z

We've found an issue with our deployment that is preventing large downloads that should be fixed soon.

Answer 5 · 2014-02-22T13:18:24.000Z

The fix for this has been deployed, so feel free to give large downloads another try.

Answer 6 · 2014-02-22T16:30:59.000Z

Doesn't work! The download stopped after 2GB for the 2012 data (about 12GB estimated!)

Can you guys please move the data to S3?

Answer 7 · 2014-03-01T02:05:45.000Z

@cndreisbach Could you please reopen the issue? The download is still timing out.
Can you help host the data on a service that allows resumable downloads?

Answer 8 · 2014-03-07T21:41:55.000Z

@fountainhead I just wanted to let you know we haven't forgotten about you. I'm preparing an export of the HMDA data by year to flat files, which we'll put on S3. I'll let you know when that is done.

For future issues directly with api.consumerfinance.gov, the cfpb/api repo is probably the best place to file issues.

Answer 9 · 2014-03-10T09:25:24.000Z

Thanks @cndreisbach . Look forward to working the data. Please try to include zipped files in the download section and allow resuming.

Thank you.

Answer 10 · 2014-03-12T20:56:15.000Z

@fountainhead You can find the last five years of data now, served through S3, so resuming will work fine. Here are the links:

http://files.consumerfinance.gov/hmda/hmda_lar-2012.csv.gz
http://files.consumerfinance.gov/hmda/hmda_lar-2011.csv.gz
http://files.consumerfinance.gov/hmda/hmda_lar-2010.csv.gz
http://files.consumerfinance.gov/hmda/hmda_lar-2009.csv.gz
http://files.consumerfinance.gov/hmda/hmda_lar-2008.csv.gz

And an MD5 checksum: http://files.consumerfinance.gov/hmda/hmda_lar.md5

Answer 11 · 2016-12-07T03:57:59.000Z

@cndreisbach Would it be possible to upload the archives through 2013-2015 as above?

Answer 12 · 2016-12-07T13:54:38.000Z

@SantoshSrinivas79 You can download a zip file for each individual year by going to http://www.consumerfinance.gov/data-research/hmda/explore, and in the "Filter" box, select the single year you wish to download. Then, toward the bottom of the screen, click the Download button. Repeat for all years in question.

There's logic under the hood that routes single-year, full-dataset requests directly to a zip file without querying from the API

Answer 13 · 2016-12-08T03:47:10.000Z

@marcesher Thank you sir!

Answer 14 · 2016-12-08T03:52:09.000Z

@marcesher Is there way to do the same for this dataset too? Consumer Complaint Database > Consumer Financial Protection Bureau

An s3 zip would be great!