Parser refactoring
Closed this issue · 2 comments
kazhuravlev commented
I think, that parser have some problems like:
- magic constants: https://github.com/tadata-ru/dtp-stat/blob/0b6cf3804f9753aec122aa66a6a534de0cf6a65d/dtp_parser/update_dtp.py#L85
- not optimal upsert/get_or_create and other db operations https://github.com/tadata-ru/dtp-stat/blob/0b6cf3804f9753aec122aa66a6a534de0cf6a65d/dtp_parser/update_dtp.py#L109-L115
https://github.com/tadata-ru/dtp-stat/blob/0b6cf3804f9753aec122aa66a6a534de0cf6a65d/dtp_parser/update_dtp.py#L156 - unsafe JSON construction https://github.com/tadata-ru/dtp-stat/blob/0b6cf3804f9753aec122aa66a6a534de0cf6a65d/dtp_parser/update_dtp.py#L244 and url construction https://github.com/tadata-ru/dtp-stat/blob/0b6cf3804f9753aec122aa66a6a534de0cf6a65d/dtp_parser/update_regions.py#L29
- work without timeouts https://github.com/tadata-ru/dtp-stat/blob/0b6cf3804f9753aec122aa66a6a534de0cf6a65d/dtp_parser/update_regions.py#L79
- and some other problems (code style, untestable code)
Before rewriting parser we must do refactoring and may be write tests.
kazhuravlev commented
I can make some changes - assign issue to me
ustnv commented
Замечания по делу, спасибо. Можно их также на новый парсер перекинуть. Давайте их тут обсуждать - https://github.com/tadata-ru/dtp-stat/issues/6