AbsaOSS/pramen

Do not run `count()` before running a transformation

Opened this issue · 0 comments

Background

Running count() on a big transformation is too expensive. It can be avoided but just always executing the transformation, and then reading the record count from the metstore.

Feature

Do not run count() before running a transformation.

Example

--

Proposed Solution

  • Add an option to control the behavior
  • Use the old behavior by default