JuliaData/Missings.jl

Add convenience skip-missing methods for Base aggregation functions

Opened this issue · 8 comments

CameronBieganek commented

xref: https://discourse.julialang.org/t/why-are-missing-values-not-ignored-by-default/106756

There are some users who dislike mean(skipmissing(x)) because they view it as too verbose or too many keystrokes. Aggregation functions that skip missing values are very commonly needed, so I think it would be worth adding convenience skip-missing methods for Base aggregation functions to Missings.jl, e.g.

  • smean
  • ssum
  • svar
  • scor

The svar and scor functions could have a keyword argument to control exactly how missing values are skipped, similar to cor in R.

nalimilan commented

I'm not too much of a fan of these ad-hoc solutions, as the list of functions to support is quite arbitrary. If we go that way we should probably provide functions for all Statistics and StatsBase functions at least, and be prepared to get request to add more functions.

Also the problem is more general than these functions. The annoyance when working with missing values is also to be able to propagate them when transforming values or vectors.

  • 👍1
CameronBieganek commented

The annoyance when working with missing values is also to be able to propagate them when transforming values or vectors.

Isn't that already handled by passmissing? (I write Python for my day job, so I'm not up to speed on the nuances of missing handling in Julia and DataFrames.)

If we go that way we should probably provide functions for all Statistics and StatsBase functions at least, and be prepared to get request to add more functions.

I would probably specify in the documentation that only skip-missing versions of Base/stdlib functions are provided (and maybe StatsBase??).

Adding smean and friends is not elegant, but it is very practical. We're just providing a higher level function for something that people do everyday.

nalimilan commented

Isn't that already handled by passmissing? (I write Python for my day job, so I'm not up to speed on the nuances of missing handling in Julia and DataFrames.)

Yes, but skipping missing values is already handled by skipmissing, so why change anything? :-p My point is that adding lots of functions that skip missing values will only alleviate one part of the problem. A macro in DataFramesMeta would potentially cover most use cases with a single syntax.

bkamins commented

My thinking is that it is OK if someone makes a package with smean, ssum, etc. functions (or whatever naming convention the package author chooses).
Then the package author can decide what to include in it and users can just start using it if they prefer to use such a functionality instead of what we have now.

Such a package would be small and simple and that is why I think it would be a reasonable solution (as opposed to discussing the creation of a complex package that would take many days of work, which indeed requires significant coordination effort of the community upfront).

pdeffebach commented

I think #122 will handle this case... we just need much simpler names. like skip or something.

knuesel commented

Having a generic solution like @pdeffebach proposes would be nice but I think users would still appreciate simple "direct" functions like smean and ssum (though maybe skmean, sksum would be more distinctive while still quite brief).

I think abstract operations like spreadmissings add mental load and can be a bit overwhelming for new users who "just want to calculate a mean or a z score". Especially when you have to understand and remember the differences between skipmissing, passmissing, skipmissings, spreadmissings...