parser-rosstat-kep: A Python repository from BupycHuk

Parser	KEP
Data source	"Short-term Economic Indicators" (KEP) by Rosstat
Parsing result	Annual, quarterly and monthly time series in CSV files
Schedule	2018

Concept

In this repo we publish a dataset of Russian macroeconomic time series as machine-readable CSV files. We keep track of monthly macroeconomic data releases (vintages) since April 2009. Original files by Rosstat are in MS Word format.

Interface

manage.py does the following job:

download and unpack MS Word files from Rosstat
extract tables from Word files and assigns variable names
create pandas dataframes with time series (at annual, quarterly and monthly frequency)
save dataframes as CSV files at stable URL

Access to parsing result

Stable URL: https://github.com/mini-kep/parser-rosstat-kep/tree/master/data/processed/latest

import pandas as pd

def get_dataframe_from_web(freq):
    url_base = ('https://raw.githubusercontent.com/mini-kep/parser-rosstat-kep/'
                'master/data/processed/latest/{}')
    filename = "df{}.csv".format(freq)
    url = url_base.format(filename)
    return pd.read_csv1(url, converters={0: pd.to_datetime}, index_col=0)

dfa, dfq, dfm = (get_dataframe_from_web(freq) for freq in 'aqm')

Repo management

Around this schedule on a Windows machine I run:

invoke add <year> <month>

and commit changes to this repo.

This command:

downloads a rar file from Rosstat,
unpacks MS Word files,
dumps all tables from MS Word files to an interim CSV file,
parses interim CSV file to three dataframes by frequency
transforms some variables (eg. deaccumulates government expenditures)
validates parsing result
saves dataframes as processed CSV files
saves csv for latest date (todo)
saves an Excel file for latest date (todo).

Same job can be done by manage.py

Parcer summary

Parcer	mini-kep
Job	Parse sections of Short-term Economic Indicators (KEP) monthly Rosstat publication
Source URL	Rosstat KEP page
Source type	MS Word
Frequency	Monthly
When released	Start of month as in schedule
Code	https://github.com/epogrebnyak/mini-kep/tree/master/src/
Test health
Test coverage
Documentation
CSV endpoint	https://github.com/epogrebnyak/mini-kep/tree/master/data/processed/latest
Transformation	Government revenue/expenses deaccumaulated to monthly values
Validation	Hardcoded checkpoints and consistency checks

All historic raw data available on internet?

Yes
No (data prior to 2016-12 is in this repo only)

Is scrapper automated (can download required_labels information from internet without manual operations)?

Notes