Download all 990's from one year

[ethantenison]

How would I download all the 990 data from one year? Is that possible?

The examples is very specific, but what I'd like to do is pull all the data and then throw out the parts I don't need.

[jsfenfen]

All of the efiled 990's are available in an S3 bucket. There are lotsa tools to download 'em, here's how you could "sync" a local directory to the contents of the bucket using AWS' command line tool.
aws s3 sync s3://irs-form-990/ ./

If you just want one year you can probably achieve that with --include / --exclude flags operating on file names. Finally there is a csv index file for each year at YYYY_index.csv I believe, you could figure out which files are in a year from that. Although there's no guarantee that the file is correct, of course.

You also might look at this repo, which is not as well documented but more aimed at dealing with many files in bulk: https://github.com/jsfenfen/990-xml-database . There are also more resources available here: https://registry.opendata.aws/irs990/

[ethantenison]

When you say there's no guarantee the files are correct, do you mean that some of the records are in the wrong year? Or that it's just missing a lot of data?

The end goal of my project is to assess the impact of revenue streams on nonprofit sustainability. If there's a particular group that's been removed from the data, that would definitely not be good. 😫

Thank you for your wisdom!

[jsfenfen]

When you say there's no guarantee the files are correct, do you mean that

some of the records are in the wrong year? Or that it's just missing a lot of data? I haven't looked at this recently, it should be easy to download all the filings and compare it to the index file. I think there's been slight errors in the past--some missing, some listed but not present--so lotsa folks just ignore the index files altogether.

If there's a particular group that's been removed from the data

It's really important to remember that electronic filings are just a subset of all 990 filings representing the groups that want to efile; something like 40% of private foundations file on paper and can continue to do so until 2022ish. Here's a mediocre summary of the bill, they seem to omit the fact that the efiling deadline for private foundations is a year later than for "regular" c3's. https://www.hklaw.com/en/insights/publications/2019/07/efiling-now-mandatory-for-all-exempt-organizations <- they seem to skip over a carve-out delaying thsi for a year form some EZ filers, see here: https://www.irs.gov/newsroom/irs-recent-legislation-requires-tax-exempt-organizations-to-e-file-forms

…

On Thu, Jul 2, 2020 at 7:36 AM Ethan Tenison ***@***.***> wrote: When you say there's no guarantee the files are correct, do you mean that some of the records are in the wrong year? Or that it's just missing a lot of data? The end goal of my project is to assess the impact of revenue streams on nonprofit sustainability. If there's a particular group that's been removed from the data, that would definitely not be good. 😫 — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE4AJFVY2T6AB7HMFUBYLDRZSLPFANCNFSM4OODVQUQ> .