Course/Program platform should not be nullable

Question

Course/Program platform should not be nullable

mbertrand opened this issue a year ago · 15 comments

Every course and program should have a Platform assigned to it. The schema should be changed to reflect this:

    platform = models.ForeignKey(
        Platform, on_delete=models.PROTECT, null=False, blank=False
    )

Answer 1 · 2023-10-02T08:31:15.000Z

Some clarifications are needed before doing this.

Context:

Initially, the plan was to add Partner instead of Platform. But for the sake of consistency within multiple apps in mit-open we decided to add the Platform field instead. This was done as part of #2698.
The field was left nullable/blankable because we did not have partners for all the courses and thought the platform would only be set for external courses and not internal.

Problem: If we are to make it non nullable/blankable we need to decide how we will populate the existing data for courses and programs. Possible problems are:

How many Platform entries are to be created in the system before we do this change?
How do we decide which Course/Program to associate with which Platform?

A possible Solution: We can create a default platform in xPRO named MITxPRO and in the migration we set all the courses and programs to it by default. Later on, we can change the platform manually where needed. I believe manual intervention will be required for external courses only.

Answer 2 · 2023-10-02T15:50:25.000Z

@arslanashraf7 how many courses/program do we have where platform is not set ?

Answer 3 · 2023-10-03T08:15:43.000Z

@Ferdi I looked at the data and here are the stats:

On Production:

Total Course	Courses Without Platform	Total Programs	Courses Without Program
154	77	26	19

Details (readable_id, live, is_external) about courses and programs with missing platforms can be seen below:

Courses list without platfrom

[
   {'readable_id': 'course-v1:xPRO+SysEngxB2', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERxE4B', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERxE3B', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERxE2B', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERxE1B', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-ADGM+YDL2', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-ADGM+YDL3', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-ADGM-FDL1', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-ADGM+FDL2', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-ADGM+FDL3', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-LASER-SL-Course2', 'live': True, 'is_external': True},
   {'readable_id': 'course-v1:xPRO-LASER-SL-Course3', 'live': True, 'is_external': True},
   {'readable_id': 'course-v1:xPRO-LASER-SL-Course4', 'live': True, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+CDAT2_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+CDAT3_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+CDAT1_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+AMxB', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+AMx_CROWN', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+AMxF+R1', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+ADETxB', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+ADSTxB', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+ADVTxB', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+DECA_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+DSCA_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+DVCA_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+SysEngx1_Raytheon', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+PCCx3+R1', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+QCFx1TouchEdu', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+PCCYx2+R1', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+PCCYx1+R1', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+DSx', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+AMxDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+MLx2DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+SysEng1xDEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASERE4xDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+DSxDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASERx4DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+ENxDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+HNxDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+QCFx1DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASERx3DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASERE3xDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+MLx1DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+SysEng3xDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+SysEng2xDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASERx2DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+GNxDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+QCRx1DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+SysEng4xDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+QCFx2DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+QCRx2DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:MITxPRO+LASERxE1DEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASERE2xDEMO+DEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+LASER1xDEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+PCCx2+R1', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+AMx_CROWN_FUNDAMENTALS', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+NHx', 'live': True, 'is_external': False},
   {'readable_id': 'v1-bootcamp-innovation-leadership', 'live': True, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+MBDTx', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:DEMO+EngLeadxDEMO', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:MITxPRO+LASERMC+R1', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+MLx1+R0', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+GNx', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+SPOC_MIT_PE', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+PCCx+R1+Course2_DO_NOT_USE', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+PCCx+R1+Course3_DO_NOT_USE', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+PCCYx-DONOTUSE', 'live': False, 'is_external': True},
   {'readable_id': 'program-v1:xPRO+PCDEx+Course2', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+PCDGx+Course2', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+PCTPMx+Course3', 'live': False, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+ENxSE', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERxE1_FreddieMac', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-WHU+TTLS', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-WHU+TTLS+R0', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+SysEngxNAV', 'live': False, 'is_external': False},
   {'readable_id': 'v1-bootcamp-venture', 'live': True, 'is_external': True},
   {'readable_id': 'course-v1:xPRO+PCCx1+R1', 'live': False, 'is_external': True}
]

Programs list without platfrom

[
   {'readable_id': 'program-v1:xPRO-SL+LASERx-SL', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:xPRO-ADGM+FDL', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:xPRO-ADGM+YDL', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+CB_Boeing', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+LASERx', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+SysEngxB', 'live': False, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+LASERxB', 'live': False, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+LASERxEB',  'live': False, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+PCCx+R1', 'live': False,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+PCCxW+R1',  'live': False,  'is_external': True},
   {'readable_id': 'program-v1:xPRO+PCCYx', 'live': False, 'is_external': True},
   {'readable_id': 'program-v1:xPRO+TIAPx+R1',  'live': True,  'is_external': True},
   {'readable_id': 'program-v1:xPRO+PCDEx', 'live': False, 'is_external': True},
   {'readable_id': 'program-v1:xPRO+PCCFTx', 'live': False, 'is_external': True},
   {'readable_id': 'program-v1:xPRO+PCTPMx', 'live': False, 'is_external': True},
   {'readable_id': 'program-v1:xPRO+LASER-SL',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+PCGDx', 'live': True, 'is_external': True},
   {'readable_id': 'program-v1:xPRO+DONOTUSE',  'live': False,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+CDAT_Boeing',  'live': True,  'is_external': False}
]

On RC:

Total Course	Courses Without Platform	Total Programs	Courses Without Program
55	65	16	15

Courses list without platfrom

[
   {'readable_id': 'course-v1:xPRO+SysEngxB1',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+SysEngxB3',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+SysEngxB2',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+SysEngxB4',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+DgtlLearn1',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+DgtlLearn2',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+DgtlLearn3',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+AnlgLearn1',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+AnlgLearn2',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+AnlgLearn3',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERx4', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERx3', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERx2', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+LASERx1', 'live': True, 'is_external': False},
   {'readable_id': 'SysEngxB3', 'live': False, 'is_external': False},
   {'readable_id': 'SysEngBx2', 'live': False, 'is_external': False},
   {'readable_id': 'SysEngBx4', 'live': False, 'is_external': False},
   {'readable_id': 'SysEngBx3', 'live': False, 'is_external': False},
   {'readable_id': 'TestX1', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:TESTx+hidden2x',  'live': False,  'is_external': False},
   {'readable_id': 'course-v1:TESTx+hidden1x',  'live': False,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+TestCourse1',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+TestCourse2',  'live': True,  'is_external': False},
   {'readable_id': 'ATCx', 'live': False, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+Bonx', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+Bonx+SPOC',  'live': False,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+external1+R1',  'live': True,  'is_external': True},
   {'readable_id': 'course-v1:xPRO+external2+R1',  'live': True,  'is_external': True},
   {'readable_id': 'course-v1:xPRO+external3+R1',  'live': True,  'is_external': True},
   {'readable_id': 'course-v1:arslan_external_1',  'live': True,  'is_external': True},
   {'readable_id': 'ATP_external_2', 'live': True, 'is_external': True},
   {'readable_id': 'arslan_test_internal_externl_1',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+BONx2', 'live': True, 'is_external': False},
   {'readable_id': 'course_without_cms_page',  'live': True,  'is_external': False},
   {'readable_id': 'AMxB', 'live': True, 'is_external': False},
   {'readable_id': 'AMxBIntro', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:MITx+Ali-Test-Course',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:MITx+101', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO-ARSLAN2', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:edx+BlockchainBasics',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+ExtCourse',  'live': True,  'is_external': True},
   {'readable_id': 'brian_test_external_page_2',  'live': False,  'is_external': True},
   {'readable_id': 'DSx', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+ENx', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+MLx1+R0', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:MAPLE+MAPLE', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:xPRO+EngWeb', 'live': True, 'is_external': True},
   {'readable_id': 'QCFx2', 'live': True, 'is_external': False},
   {'readable_id': 'RCHNTC', 'live': True, 'is_external': False},
   {'readable_id': 'course-v1:edX+DemoX+Demo_Course',  'live': True,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+NotLive1',  'live': False,  'is_external': False},
   {'readable_id': 'course-v1:xPRO+QuantCompx1',  'live': True,  'is_external': False},
   {'readable_id': 'external-readable-id', 'live': True, 'is_external': True},
   {'readable_id': 'TC3', 'live': True, 'is_external': False}
]

Programs list without platfrom

[
   {'readable_id': 'program-v1:ARSLAN+TestProgram',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+LASERx', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:MITxPro+SysEngxB',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+SysEngx',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+DgtlLearn',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+AnlgLearn',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+SysEngBx',  'live': True,  'is_external': False},
   {'readable_id': 'TestX', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:TESTx+SysEngBx',  'live': False,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+TestProgram',  'live': True,  'is_external': False},
   {'readable_id': 'program-v1:xPRO+Bonx', 'live': True, 'is_external': False},
   {'readable_id': 'program-v1:xPRO+externalprogram',  'live': True,  'is_external': True},
   {'readable_id': 'ATP_external_1', 'live': True, 'is_external': True},
   {'readable_id': 'test_internal_program_external',  'live': True,  'is_external': False},
   {'readable_id': 'progam_without_cms_page',  'live': True,  'is_external': False}
]

Let me know if you require further details.

Answer 4 · 2023-10-03T13:06:43.000Z

Thanks Arslan, why don't I see these courses here https://xpro.mit.edu/api/courses/?

Answer 5 · 2023-10-03T13:37:55.000Z

Thanks Arslan, why don't I see these courses here https://xpro.mit.edu/api/courses/?

That would be because the /api/courses only returns those courses that have:

live=True (This is from Django)
coursepage.live=True (This is CMS page for the course)
externalcoursepage.live=True (Again, from CMS but for external courses)

*Edit
In the list that I provided above, I didn't filter the data based on any of these conditions because we will need a platform for all of those irrespective of the above conditions.

Answer 6 · 2023-10-03T13:51:51.000Z

ok, should we fix the data before making the field non-nullable or after. Most of them seem to be external = false, so we can set the platform to mitxpro.

Answer 7 · 2023-10-03T14:00:39.000Z

ok, should we fix the data before making the field non-nullable or after. Most of them seem to be external = false, so we can set the platform to mitxpro.

We can do both ways.

If we know what platform we want for all the remaining courses and programs, we can automatically fix it in the Django migration that we will generate to make this field non-nullable.
If not, Then we need to fix the data before we make it non-nullable.

Answer 8 · 2023-10-03T14:14:02.000Z

it seems like external courses/programs need to be fixed before. Internal ones (platform = mitxpro) can be done during migration

Answer 9 · 2023-10-03T14:24:49.000Z

it seems like external courses/programs need to be fixed before. Internal ones (platform = mitxpro) can be done during migration

I agree. I suppose the course team would fix that data? Or I can but I will require details about the external courses. In any case, we can put up the PR as soon as the data is fixed.

Answer 10 · 2023-10-04T02:05:51.000Z

Can we get a list of the external courses/programs for @cachob to look at ?

Answer 11 · 2023-10-04T10:42:22.000Z

Can we get a list of the external courses/programs for @cachob to look at ?

Sure.

@cachob Please take a look below for the list of external courses and programs without the platform. For ease, I've convertedthe lists to easily clickable links.

Production:

Here is the list of external courses without a platform

Here is the list of external programs without a platform

https://xpro.mit.edu/admin/courses/program/20/change/
https://xpro.mit.edu/admin/courses/program/21/change/
https://xpro.mit.edu/admin/courses/program/22/change/
https://xpro.mit.edu/admin/courses/program/23/change/
https://xpro.mit.edu/admin/courses/program/24/change/
https://xpro.mit.edu/admin/courses/program/25/change/
https://xpro.mit.edu/admin/courses/program/27/change/

RC: (Changing this doesn't matter much since this is just test data but in case you would like to take a look)

Here is the list of external courses without a platform

Here is the list of external programs without a platform

https://rc.xpro.mit.edu/admin/courses/program/13/change/
https://rc.xpro.mit.edu/admin/courses/program/14/change/

Answer 12 · 2023-10-04T13:29:45.000Z

@arslanashraf7 - I noticed that a lot of the courses are the dummy courses that we had to create to correctly display the amount of courses for a program in the catalog. The others are also not set as live. I will clean up and update both courses and programs on the lists as needed, and set the default value as xPRO if applicable

Answer 13 · 2023-10-04T13:31:23.000Z

@arslanashraf7 - I noticed that a lot of the courses are the dummy courses that we had to create to correctly display the amount of courses for a program in the catalog. The others are also not set as live. I will clean up and update both courses and programs on the lists as needed, and set the default value as xPRO if applicable

@cachob Thanks. Let us know when you are done.

Answer 14 · 2023-10-04T13:57:54.000Z

@arslanashraf7 - production data is all set

Answer 15 · 2023-10-18T08:38:15.000Z

Done via #2786. This has been deployed on production. The data migration for this was successful and there are no more courses or programs without a platform.