Use relative difference in reach for Awario growth
njordhov opened this issue · 9 comments
What is the problem you are seeing? Please describe.
Awario growth score is calculated as relative change, which has problematic properties including that a low past reach score disproportionately benefits a high growth score:
IF(GrowthIneligible, "", IF(PastReach+Reach=0, 0, (Reach-PastReach)/PastReach))
For example, a Past Reach of 1000 and a Reach of 11000 gives a Growth score of 10, while a Past Reach of 2000 and a Reach of 12000 results in half the Growth score (5) even if the practical difference between the the two is minor and may come down to luck.
Plot showing Growth as relative change in Reach for a Past Reach of 1000 vs 2000.
With a Past Reach of 1000 and a Reach of 129.000 (the highest reach in November) the Growth Score would be a high 128; Also, a tiny difference in Past Score from 999 to 1000 would make a significant difference in the ranking.
How is this problem misaligned with goals of app mining?
It encourages careful calibration of Past Reach to the lowest still eligible for Growth, with the goal of getting an extraordinary high growth score in the next month. Apps that manage to get the disproportionate growth scores (or got lucky) stand to gain considerably in ranking.
What is the explicit recommendation you’re looking to propose?
Change the calculation of the Growth Score to relative difference:
IF(Reach=PastReach, 0, (Reach-PastReach)/MAX(Reach, PastReach))
The change is only effective when Reach is larger than Past Reach, making it represent the percentage of the Reach that is higher than previously. The growth calculation is the same as before when Reach is less than Past Reach.
Plot showing Growth as relative difference in Reach with Past Reach = {1000, 2000}
The formula bounds the Growth to be between -1 and 1. Its behavior is symmetric for increases vs decreases in Reach. It also behaves reasonably for lower values of Reach, avoiding the huge growth percentage outliers, potentially making it possible to have a Growth score for all values of Reach rather than making growth ineligible for lower Reach scores.
What is the dry run period (if any)
Test with data from November and December, deploy asap.
Describe your long term considerations in proposing this change. Please include the ways you can predict this recommendation could go wrong and possible ways mitigate.
The highest Growth score will be 1, leaving a lingering advantage for those that already have an outlier high Growth score (due to the "memory function".)
There will be marginal gains in Growth as the increase in Reach get high. This can be considered a feature; The function rewards some level of growth but making reach go over the moon makes only a small difference in the growth score. Compare with LOG(Reach / PastReach) and LOG(Reach - PastReach) which don't limit the value of Growth as the difference in Reach get higher.
Additional context
Hm, this is an interesting proposal. Although I'd add doubts that it's beneficial to 'game' your reach to 1000 one month, just to get a big growth score the next month. You'd be hurting your 'reach' score and would still have to increase growth in the following months. If you kept switching between 1000 reach and 100,000 reach, you'd end up "cancelling" the gains you got.
However, this -1 to 1 for growth does feel very clean, and I like that it's symmetric. I'm open to doing a dry run in December's scores, to possibly include in January.
If you kept switching between 1000 reach and 100,000 reach, you'd end up "cancelling" the gains you got.
See issue #129 for a discussion of why the scoring incentivizes this tactic
We haven't really seen anyone intentionally game this, but this proposal makes the score stronger, so I'm generally in favor. Hank will provide dry-run data this month.
Here are two hypothetical apps:
one app increasing reach each month by 1000 (on the left)
one app changing reach variously (on the right)
Apps with 0 in the current and previous month should get -1.5 or so.
This looks like a really challenging score to increase over time.
This looks like a really challenging score to increase over time.
Consider the Reach score and this Growth score together. When Reach is low, it is easier to get a high Growth score; As the Reach gets higher, the Growth score will typically be more moderate, but the higher Reach score makes up for it.
The Reach has to be earned each month and I wouldn't expect apps to increase reach continuously but have higher and lower (even 0) reach. Some big news/updates will create peeks.
Apps with 0 in the current and previous month should get -1.5 or so.
The formula can be tuned to give these apps a lower Growth score, e.g. by incrementing the Past Reach score by one:
(Reach-(PastReach+1))/MAX(Reach, PastReach+1)
As a consequence, apps with zero Reach in consecutive months will get the lowest possible Growth score (-1). The impact on the Growth score of apps with higher Reach is ignorable, although it will affect their z-scores due to changes in the score distribution.
Note that the IF test is no longer required in the formula as division by zero is impossible when PastReach always is above zero.
Awario growth score is calculated as relative change, which has problematic properties including that a low past reach score disproportionately benefits a high growth score
For December 2019, the top app had a growth of 2136% and a Growth Z-Score of 5.074, massively outperforming all the others, due to getting a Reach of 81341 from a Past Reach of 3637...
The formula can be tuned to give these apps a lower Growth score, e.g. by incrementing the Past Reach score by one
A better scoring is to increment Past Reach with a higher tuning factor such as 1000 or more for n:
(Reach-(PastReach+n))/MAX(Reach, PastReach+n)
This makes it a bit more challenging to go from zero to top score, as it requires you to have a reach above n.