mitodl/micromasters

MM legacy: validate program letters, disable incorrect ones

pdpinch opened this issue · 5 comments

pdpinch commented

A user alerted us that they were seeing a program letter for a program that they had not completed. Normally, these letters are only offered on program completion. See https://odl.zendesk.com/agent/tickets/140048

After investigating, I confirmed a large number of such letters had been generated, almost all of them on the same day (on a date before the program had even started)

We need a clean, safe and reliable way to validate existing program letters (MicromastersProgramCommendation) and disable (or delete) any that were mistakenly created.

Acceptance Criteria:

  • Take a look a the state of the code and the production data, and propose a way to clean this up
arslanashraf7 commented

I did some initial analysis on this and here are the details:

MicromastersProgramCommendation creation attempt paths:

  1. Create through a command generate_program_letter.py. I believe we didn't run this.
  2. In signals, when a MM Program certificate is created. Whenever a program certificate is created for a user we would attempt to create a Program Letter.
  3. In signals, When a FinalGrade entry is created and the program associated with that course doesn't have financial Aid enabled.

Preconditions when a Program Letter creation is attempted:

  1. If the program has FinancialAid enabled and also the user has a MMProgramCertificate it would create the ProgramLetter.
  2. If condition in point#1 doesn't meet we just check if the user has completed the program and we generate the letter.

I suspect this might have started happening when we switched off the FinancialAid for DEDP.

Now, I think the Program letters that we are considering invalid, don't have the ProgramCertificate? which means they might have been created through point#2 mentioned above.

As for the solution:

  1. We don't have an is_active field in these models and we might not be able to deactivate unless we add a new field and change the logic of how we show Program letters so that we only show active ones.
  2. Another solution is to change the above logic mentioned in preconditions for Program letters so that we don't generate new ones based on that logic, and then delete the invalid Program letters.

What we might need here is to specify what are the conditions for invalid Program letters so that we can change the logic in letter generation.

pdpinch commented

In my review, I found ~1000 program letters created for program id=5 on 12/19/2019, even though there were no certificates granted in any of that program's courses until 2020.

  1. If condition in point#1 doesn't meet we just check if the user has completed the program and we generate the letter.

If we test these program letters to see if the user has completed the program, what do we get?

arslanashraf7 commented

If we test these program letters to see if the user has completed the program, what do we get?

You are correct, Testing these letters against the program completion shows many that don't have the program completed.

I was able to get the stats that might help us conclude the analysis. I've shared those with you in Slack.

pdpinch commented

Ok. I think we need to split this into a few separate issues:

  • #5250
  • #5251
  • RCA #5252
  • RCA erroneous program letters created after 12/19/2019
pdpinch commented

I neglected to mention that the FIN team (program id=5) has removed their program letter from the CMS for now, which blocks all access to letters while we resolve this.

There are still 3 letters that are not program id=5 that need to be reviewed and revoked.