discrepant ease factors due to learning steps

Question

discrepant ease factors due to learning steps

galantra opened this issue 4 years ago · 7 comments

I have rather medium length learning steps and I've noticed a big discrepancy between some cards.

Here is an example:

Both cards are quite difficult, compared to other cards of the same kind (meaning, Spanish vocabulary). I know this because they have not passed the learning stage without fail.

For such a difficulty, 347 % seems to much (1st card). This is the first surprise.

The second surprise is that the 2nd card's review history doesn't seem to differ much from the first card's, but the calculated ease is much lower.

Note also that in the first case, the ease is higher than the starting ease, whereas in the second case it is lower.

This seems quite odd to me. I'd like to inquire if this phenomenon is intended functioning of the add-on.

Answer 1 · 2020-09-20T01:45:39.000Z

The algorithm is heavily responsive to streaks, and nine in a row does look like a stronger recovery to me than the intermittent failures of the other card.

I'd like to run a few alternative scenarios to more thoroughly test though. I'm traveling right now, so bear with me, but I'll look into it.

Answer 2 · 2020-09-20T04:26:16.000Z

I totally think it might be a plausible pattern. I just wanted to check back – mostly to understand the add-on better.

BTW, I just had another thought. My settings right now are:

{
"leash": 90,
"max_ease": 5000,
"min_ease": 1400,
"moving_average_weight": 0.15,
"stats_duration": 20000,
"stats_enabled": false,
"target_ratio": 0.85,
"two_button_mode": false
}

Perhaps the moving average weight is more impactful than I had thought. Therefore, the streak makes a bigger difference than I thought.

Answer 3 · 2020-09-20T08:38:32.000Z

I've also thought of the fact that my retention rate with the default Anki mechanism is higher than 85%, with most cards. This thought came up when seeing this steep boost of ease:

2020-09-20 @ 10:28	Review	3	⁨3.9⁩ years	2024-07-31	500%	⁨16.73⁩ seconds
2019-12-15 @ 07:29	Review	3	⁨9.3⁩ months	2020-09-20	265%	⁨10.93⁩ seconds
2019-09-02 @ 09:00	Review	3	⁨3.4⁩ months	2019-12-14	265%	⁨8.1⁩ seconds
2019-07-24 @ 10:27	Review	3	⁨1.3⁩ months	2019-09-02	250%	⁨26.85⁩ seconds
2019-07-07 @ 21:06	Review	3	⁨17⁩ days	2019-07-24	250%	⁨1⁩ minute
2019-06-28 @ 20:08	Review	3	⁨8⁩ days	2019-07-06	250%	⁨12.31⁩ seconds
2019-06-23 @ 21:43	Review	3	⁨2⁩ days	2019-06-25	250%	⁨4.77⁩ seconds
2019-06-22 @ 22:35	Learn	3	⁨1⁩ day	2019-06-23	250%	⁨11.41⁩ seconds
2019-06-22 @ 21:29	Learn	3	⁨10⁩ minutes	na	250%	⁨9.96⁩ seconds
2019-05-10 @ 23:09	Review	3	⁨23⁩ days	2019-06-02	130%	⁨3.91⁩ seconds
2019-04-23 @ 21:09	Review	3	⁨16⁩ days	2019-05-09	130%	⁨9.7⁩ seconds
2019-04-11 @ 19:08	Review	3	⁨12⁩ days	2019-04-23	130%	⁨7.53⁩ seconds
2019-04-03 @ 20:36	Review	3	⁨8⁩ days	2019-04-11	130%	⁨12.51⁩ seconds
2019-03-26 @ 19:47	Review	3	⁨6⁩ days	2019-04-01	130%	⁨31.18⁩ seconds
2019-03-23 @ 20:06	Relearn	3	⁨3⁩ days	2019-03-26	130%	⁨1⁩ minute
2019-03-23 @ 19:54	Review	1	⁨10⁩ minutes	na	130%	⁨1.3⁩ seconds
2019-02-13 @ 19:00	Review	3	⁨1.2⁩ months	2019-03-22	130%	⁨18.88⁩ seconds
2019-01-15 @ 20:50	Review	2	⁨28⁩ days	2019-02-12	130%	⁨22.49⁩ seconds
2018-12-23 @ 16:38	Review	3	⁨23⁩ days	2019-01-15	130%	⁨8.75⁩ seconds
2018-12-06 @ 11:30	Review	3	⁨17⁩ days	2018-12-23	130%	⁨9.96⁩ seconds
2018-11-20 @ 15:03	Review	3	⁨16⁩ days	2018-12-06	130%	⁨5.96⁩ seconds
2018-11-09 @ 14:26	Review	3	⁨11⁩ days	2018-11-20	130%	⁨16.68⁩ seconds
2018-11-01 @ 13:47	Review	3	⁨8⁩ days	2018-11-09	130%	⁨7.28⁩ seconds
2018-10-25 @ 13:19	Review	3	⁨7⁩ days	2018-11-01	130%	⁨10.35⁩ seconds
2018-10-19 @ 10:28	Review	3	⁨5⁩ days	2018-10-24	130%	⁨27.31⁩ seconds
2018-10-16 @ 14:41	Review	3	⁨3⁩ days	2018-10-19	130%	⁨17.36⁩ seconds
2018-10-15 @ 12:26	Relearn	3	⁨1⁩ day	2018-10-16	130%	⁨10.97⁩ seconds
2018-10-15 @ 12:13	Review	1	⁨10⁩ minutes	na	130%	⁨12.43⁩ seconds
2018-10-11 @ 12:10	Review	3	⁨4⁩ days	2018-10-15	130%	⁨9.08⁩ seconds
2018-10-09 @ 19:07	Relearn	3	⁨2⁩ days	2018-10-11	130%	⁨2.73⁩ seconds
2018-10-09 @ 18:52	Review	1	⁨10⁩ minutes	na	130%	⁨34.89⁩ seconds
2018-09-07 @ 16:31	Review	3	⁨28⁩ days	2018-10-05	150%	⁨24.77⁩ seconds
2018-08-21 @ 23:39	Review	2	⁨17⁩ days	2018-09-07	150%	⁨12.62⁩ seconds
2018-08-07 @ 14:00	Review	3	⁨14⁩ days	2018-08-21	165%	⁨29.94⁩ seconds
2018-07-28 @ 22:48	Review	2	⁨10⁩ days	2018-08-07	165%	⁨24.98⁩ seconds
2018-07-16 @ 17:06	Review	3	⁨6⁩ days	2018-07-22	180%	⁨6.72⁩ seconds
2018-07-12 @ 16:45	Review	3	⁨4⁩ days	2018-07-16	180%	⁨12.57⁩ seconds
2018-07-11 @ 14:01	Relearn	3	⁨1⁩ day	2018-07-12	180%	⁨3.54⁩ seconds
2018-07-11 @ 13:52	Review	1	⁨10⁩ minutes	na	180%	⁨22.26⁩ seconds
2018-06-29 @ 07:31	Review	3	⁨12⁩ days	2018-07-11	200%	⁨1⁩ minute
2018-06-23 @ 10:32	Review	3	⁨6⁩ days	2018-06-29	200%	⁨4.66⁩ seconds
2018-06-19 @ 21:09	Review	2	⁨4⁩ days	2018-06-23	200%	⁨1⁩ minute
2018-06-17 @ 10:53	Review	3	⁨2⁩ days	2018-06-19	215%	⁨14.35⁩ seconds
2018-06-16 @ 08:23	Relearn	3	⁨1⁩ day	2018-06-17	215%	⁨10.01⁩ seconds
2018-06-16 @ 08:13	Review	1	⁨10⁩ minutes	na	215%	⁨1⁩ minute
2018-06-13 @ 08:30	Review	2	⁨3⁩ days	2018-06-16	235%	⁨12.07⁩ seconds
2018-06-04 @ 11:57	Learn	3	⁨1⁩ day	2018-06-05	250%	⁨9.33⁩ seconds
2018-06-04 @ 11:34	Learn	3	⁨10⁩ minutes	na		⁨9.84⁩ seconds
2018-06-04 @ 11:33	Learn	1	⁨1⁩ minute	na		⁨8.52⁩ seconds
2018-06-04 @ 11:21	Learn	3	⁨10⁩ minutes	na		⁨7.81⁩ seconds

Answer 4 · 2020-09-22T14:36:51.000Z

Hmm, yeah, if you are already outperforming your target, the algorithm will aggressively space those cards out until they are at more risk. I built a simulator you can download as simulator.py, but I'm still testing it, I'm not sure it's perfectly duplicating the results of the algorithm yet. (It doesn't produce the same results that you got, for example, still needs more work.)

EDIT: To run it, you'll need to first install pysimpleguiqt:
pip install --upgrade PySimpleGUIQt
(or pip3... on Linux)

Answer 5 · 2020-09-24T16:19:35.000Z

The simulator is a good idea. It will come in handy.

For now, I have changed the maximum ease from 5000 to 4000. I guess that the added review time over a lifetime (as well as over the time to exams) is negligible and that the security benefit will outweigh it.

BTW, thank you for all the work and time you put in this add-on! Not only do you constantly improve the add-on, you also respond to the reviews on ankiweb and issues on github, and in an especially very friendly way at that.

Answer 6 · 2020-09-25T02:19:52.000Z

Just remembered this quote from the original author that seems relevant --

I think limiting the adjustments to no more than 20% up or down is a reasonable way to keep the adjustments from getting too extreme. As the success rate gets closer and closer to the desired 85%, the adjustments become smaller and smaller. - ejs

EJS had great instincts here, I had scrapped this for the "leash" approach.

If I can combine both, I might be able to make the algorithm progressively more aggressive as it gets more and more data, but only up to a point, and never tooo aggressive... I'll take a look at bringing something like this back in.

EDIT: Ah, nope. The challenge with that original approach is the risk of ease hell. If you tie the amount of movement to the current ease values, it's easier for ease values to go down than up. 20% of a 5000 ease factor is a much bigger move than 20% of an ease factor of 1000. This should still be tied to reps somehow, but maybe the relationship just shouldn't be strictly linear... hmm...

BTW, thank you for all the work and time you put in this add-on!

Of course! Still a lot of room for improvement, happy to try to make this a little better each time.

Answer 7 · 2020-10-18T14:03:57.000Z

Simulator seems to be in pretty good shape now. I'll close this for now, but will keep thinking about ways to improve the balance of calibration and certainty, in line with #8.