This project uses fake data related to the field of communication (such as IMSI / IMEI / TAC / SNR).
The idea for this project is inspired by a series on Garage Education, in which Eng. Mostafa Alaa shows the process of building an ETL project using Scala.
- Read, process and transfer data directly. Simple ETL package.
- Dealing with data that do not meet the quality requirements specified in the project file bad data and error handling.
- Handle files stored in a specific folder "looping through files".
- Archive files that are moving processed.
- Auditing of processed data.
Control Flow:
A control flow is a graphical model that sequences data flows and mining flows, integrates external commands, programs, and stored procedures, and provides conditional processing logic for a warehousing application.
Data Flow:
The Data Flow task encapsulates the data flow engine that moves data between sources and destinations, and lets the user transform, clean, and modify data as it is moved. Addition of a Data Flow task to a package control flow makes it possible for the package to extract, transform, and load data.
