/mattharrisondatasets-1

Datasets for ML, Analysis, etc

Ames Housing Data

http://jse.amstat.org/v19n3/decock.pdf

Data at ./data/ames-housing-dataset.zip

Weather data for those years at ../data/asos-ames-2007-2010.txt

Stack Overflow 2019 Survey (developer_survey_2019.zip 18M)

https://insights.stackoverflow.com/survey/2019

License: ODbL

Automobile Fuel Economy 1984-2020 (vehicles.csv.zip)

https://www.fueleconomy.gov/feg/download.shtml

Presidents

From https://qrc.depaul.edu/Excel_File_Listing_Pages/Excellist.asp

Pokemon

From https://www.kaggle.com/rounakbanik/pokemon (CC0: Public Domain)

Alta

1980-2019

https://www.ncdc.noaa.gov/cdo-web/datasets/GHCND/stations/GHCND:USC00420072/detail

Data at ../data/snow-alta-1990-2017.csv

Documentation - https://www1.ncdc.noaa.gov/pub/data/cdo/documentation/GHCND_documentation.pdf

  • STATION_NAME (max 50 characters) is the (usually city/airport name). Optional output field.
  • STATION - 17 characters) is the station identification code. Please see http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt
  • NAME - name of the station
  • LATITUDE
  • LONGITUDE
  • ELEVATION - meters
  • DATE - YYYY-MM-DD
  • DAPR - Number of days included in the multiday precipitation total (MDPR)
  • DAPR_ATTRIBUTES
  • DASF - Number of days included in the multiday snowfall total (MDSF)
  • DASF_ATTRIBUTES
  • MDPR - Multiday precipitation total (mm or inches as per user preference; use with DAPR and DWPR, if available)
  • MDPR_ATTRIBUTES
  • MDSF - Multiday snowfall total (mm or inches as per user preference)
  • MDSF_ATTRIBUTES
  • PRCP - Precipitation (mm or inches as per user preference, inches to hundredths on Daily Form pdf file)
  • PRCP_ATTRIBUTES
  • SNOW - Snowfall (mm or inches as per user preference, inches to tenths on Daily Form pdf file)
  • SNOW_ATTRIBUTES
  • SNWD - Snow depth (mm or inches as per user preference, inches on Daily Form pdf file)
  • SNWD_ATTRIBUTES
  • TMAX - Maximum temperature (Fahrenheit or Celsius as per user preference, Fahrenheit to tenths on Daily Form pdf file
  • TMAX_ATTRIBUTES
  • TMIN - Minimum temperature (Fahrenheit or Celsius as per user preference, Fahrenheit to tenths on Daily Form pdf file
  • TMIN_ATTRIBUTES
  • TOBS - Temperature at the time of observation (Fahrenheit or Celsius as per user preference)
  • TOBS_ATTRIBUTES
  • WT01 - Fog, ice fog, or freezing fog (may include heavy fog)
  • WT01_ATTRIBUTES
  • WT03 - Thunder
  • WT03_ATTRIBUTES
  • WT04 - Ice pellets, sleet, snow pellets, or small hail
  • WT04_ATTRIBUTES
  • WT05 - Hail (may include small hail)
  • WT05_ATTRIBUTES
  • WT06 - Glaze or rime
  • WT06_ATTRIBUTES
  • WT11 - High or damaging winds
  • WT11_ATTRIBUTES

El Nino (tao-all2.dat.gz)

https://archive.ics.uci.edu/ml/datasets/El+Nino

zonal winds (west<0, east>0), meridional winds (south<0, north>0),

# Data transformation from previous notebook
# col names in tao-all2.col from website
names = '''obs
year
month
day
date
latitude
longitude
zon.winds
mer.winds
humidity
air temp.
s.s.temp.'''.split('\n')

nino = pd.read_csv('data/tao-all2.dat.gz', sep=' ', names=names, na_values='.', 
                   parse_dates=[[1,2,3]])

def clean_cols(val):
    return val.replace('.', '_').replace(' ', '_')

nino = (nino
  .rename(columns=clean_cols)
  .assign(air_temp_F=lambda df_: df_.air_temp_ * 9/5 + 32,
        zon_winds_mph=lambda df_: df_.zon_winds*2.237,
        mer_winds_mph=lambda df_: df_.mer_winds*2.237)
  .drop(columns='obs')
)