Support sampling rate instead of target sampling size

Question

Support sampling rate instead of target sampling size

domoritz opened this issue 7 years ago · 2 comments

E.g. keep 20% of the tuples.

{"type": "sample", "rate": 0.2}

Answer 1 · 2017-07-13T01:08:10.000Z

I guess that's just

{"type": "filter", "expr": "random() < 0.2"}

Answer 2 · 2017-07-13T04:25:54.000Z

Careful! A filter with a randomized predicate can cause your sample to change if re-run (e.g., due to upstream operator changes). A safer approach might be to calculate the desired sample window size in a signal (e.g., 0.2 * length(data('source'))) and use that to parameterize a sample transform.