--describe your project at a high level--
The project follows the follow steps:
- Step 1: Scope the Project and Gather Data
- Step 2: Explore and Assess the Data
- Step 3: Define the Data Model
- Step 4: Run ETL to Model the Data
- Step 5: Complete Project Write Up
Explain what you plan to do in the project in more detail. What data do you use? What is your end solution look like? What tools did you use? etc>
Describe the data sets you're using. Where did it come from? What type of information is included?
Identify data quality issues, like missing values, duplicate data, etc.
Document steps necessary to clean the data
Map out the conceptual data model and explain why you chose that model
List the steps necessary to pipeline the data into the chosen data model
Build the data pipelines to create the data model.
Explain the data quality checks you'll perform to ensure the pipeline ran as expected. These could include:
- Integrity constraints on the relational database (e.g., unique key, data type, etc.)
- Unit tests for the scripts to ensure they are doing the right thing
- Source/Count checks to ensure completeness
Run Quality Checks
Create a data dictionary for your data model. For each field, provide a brief description of what the data is and where it came from. You can include the data dictionary in the notebook or in a separate file.
- Clearly state the rationale for the choice of tools and technologies for the project.
- Propose how often the data should be updated and why.
- Write a description of how you would approach the problem differently under the following scenarios:
- The data was increased by 100x.
- The data populates a dashboard that must be updated on a daily basis by 7am every day.
- The database needed to be accessed by 100+ people.
https://www.datos.gov.co/Ambiente-y-Desarrollo-Sostenible/Datos-Hidrometeorol-gicos-Crudos-Red-de-Estaciones/sbwg-7ju4 https://www.datos.gov.co/Ambiente-y-Desarrollo-Sostenible/Humedad-del-Aire-2-metros/uext-mhny
https://www.upra.gov.co/web/guest/evaluaciones-agropecuarias-municipales-eva (2019-2021) https://www.datos.gov.co/Agricultura-y-Desarrollo-Rural/Evaluaciones-Agropecuarias-Municipales-EVA/2pnw-mmge (2007-2018)
https://www.dane.gov.co/index.php/estadisticas-por-tema/agropecuario/sistema-de-informacion-de-precios-sipsa (2013-2022) https://www.dane.gov.co/index.php/estadisticas-por-tema/agropecuario/sistema-de-informacion-de-precios-sipsa/componente-precios-mayoristas (daily)
https://cloudbytes.dev/snippets/run-selenium-and-chrome-on-wsl2