agile-lab-dev/wasp

When topic avro/json deserialization fails for a single record, the whole pipegraph crashes

Opened this issue · 0 comments

Sometimes reading from a source with serialized data in avro/json, can happen that a record with some typo or serialized with an unknown avro schema came in, with the actual readers behaviour, when this happens, an exceptions will be raised and pipegraph will crash.
The decision on how to handle parsing errors should be left to downstream implementations, especially for streaming jobs in which is often desirable that the application keep running.

The available options when a parsing error occurs should be:

  1. raise an exception (as it does now)
  2. filter out records that cannot be parsed and continue the job execution
  3. handle the errors by returning to the user, which can then choose how to operate