bpolaszek/bentools-etl

Illegal offset type error could occur during transformations

Closed this issue · 0 comments

Contract of a Transformer is to return a \Generator.

When a transformer returns a Generator with either:

  • Duplicate keys
  • Objects as keys

Loader doesn't receive the accurate items to load.
In the 2nd case, a Illegal offset type error warning is raised.

This is due to the ETL process which requires to traverse generators and build them back again before sending them to loaders, so as to allow hooking on transformation errors and safely dismiss an item which failed to be transformed.