design

Sources

  • File - local or HDFS
  • Stream - spark streaming
  • Kafka Topic
  • Flume
  • Kinesys

Segregator and parser

Remedies

  • AWS
  • Azure
  • GCP
  • OS
  • new fixes