Переезд ШАД
Closed this issue · 3 comments
ctrltz commented
- создать проект для ШАДа, перенести settings и запустить
- создать отделение для ШАДа
- проверить, что от ШАДа ничего не появилось на сайтах центра/клуба
- добавить состояние черновика для прочтения курса
- дальше идти по вьюхам и смотреть, что могло протечь
ctrltz commented
Поэтапное тестирование сайтов CS центра/клуба при добавлении контента на сайте ШАДа:
- Просто создал пустой сайт - могли быть какие-то лишные табы, но ничего не появилось.
- Добавил прочтение курса только для ШАДа - в Обучении всё ОК, в Курировании на сайте центра доступна ведомость по курсу, также курс в списке для поиска пересечений. Как должно работать Преподавание между сайтами?
MAJOR:
- когда захожу на сайте ШАД в раздел Курсы с центрового аккаунта, получаю TOO_MANY_REDIRECTS
MINOR:
- добавить в Полезное какой-то текст на случай, когда в базе пусто.
- Пофиксить переводы для LMS
SITE SPECIFIC
- Обучение > Библиотека
- Набор
- Проекты
- Курирование (например, в списке отображаются ведомости всех курсов, но открыть можно только ведомость курсов с текущего сайта)
- Админка?
- Статистика
SITE CONFIGURATION
- лого
- год основания
- вынести
DEFAULT_BRANCH_CODEиз core в настройки сайта
TEMPLATES
- Computer Science Center -> request.site.name в шаблонах (<title>, img[alt], footer)
- Computer Science Center в переводах/тексте полей моделек
- "Кураторы CS центра" в e-mail уведомлении по проектам
- убрать надписи "CS центр" из шаблонов
to be updated
ctrltz commented
Скрипт для импорта данных из энитаска.
"""
Import Anytask data to CSC LMS
Minimal set of attributes:
* MetaCourse: name_ru, slug, description
* Course: meta_course, grading_type, semester, main_branch, language, material_visibility, teachers, group_mode
* StudentProfile: type, branch, user, year_of_admission, year_of_curriculum (if present), status
* Enrollment: user, profile, course, grade
* StudentGroup: type, name, course
Required data:
- exports from Anytask
- info about courses
- metacourse slugs mapping
- user mapping to resolve special cases
Plan:
+ extract course teachers and get/create users for them
+ create meta_courses
+ create course and link them to corresponding teachers and meta_courses
+ create student groups for courses
+ create users for students
+ create student profiles and enrollments with filled grades and student groups
TODO:
- carefully process users with matching first name and last name
- student profiles for course teachers (?)
- setup additional branches for courses (?)
- distinguish Academic 1 & 2 (?)
- place students into student groups (resolve multiple groups for student in one course)
- fix skipping of CSC students
- multiple branches in logs
- Подчистить инфу об отделениях из фамилий (?)
FIX:
- Глубокое обучение и _Глубокое обучение объединили - из-за этого IntegrityError('duplicate key value violates
unique constraint "learning_enrollment_student_id_course_offering_id"\nDETAIL: Key
(student_id, course_id)=(7891, 777) already exists.\n')
TEST:
- check slugs for Основы стохастики and Продукт: создание и управление
- check that everything is OK with imported 10-point grades (no students that have passed the course were imported)
"""
import json
import logging
from collections import Counter
from datetime import datetime
import pandas as pd
from django.db import IntegrityError
from django.db.models import Q
from core.models import Branch
from core.utils import ru_en_mapping
from courses.constants import MaterialVisibilityTypes
from courses.models import MetaCourse, Course, Semester, StudentGroupTypes
from courses.services import CourseService
from learning.models import Enrollment, StudentGroup
from learning.settings import GradingSystems, GradeTypes, StudentStatuses
from study_programs.models import AcademicDiscipline
from users.constants import GenderTypes, Roles
from users.models import User, StudentProfile, StudentTypes, StudentStatusLog
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
SHAD_COURSES = '/home/kapralovn/work/cscenter/misc/anytask/shad_courses.csv'
SHAD_DATA = '/home/kapralovn/work/cscenter/misc/anytask/shad_data.json'
SHAD_MIGRATOR_USERNAME = 'robot_shad_migrator'
MARKS = {
'отлично': GradeTypes.EXCELLENT,
'хорошо': GradeTypes.GOOD,
'зачет': GradeTypes.CREDIT,
'незачет': GradeTypes.UNSATISFACTORY,
'10': GradeTypes.TEN,
'9': GradeTypes.NINE,
'8': GradeTypes.EIGHT,
'7': GradeTypes.SEVEN,
'6': GradeTypes.SIX,
'5': GradeTypes.FIVE,
'4': GradeTypes.FOUR,
'3': GradeTypes.THREE,
'2': GradeTypes.TWO,
'1': GradeTypes.ONE,
None: GradeTypes.NOT_GRADED
}
USER_STATUS_MAPPING = {
1: {'status': StudentStatuses.ACADEMIC_LEAVE},
2: {'status': StudentStatuses.EXPELLED},
3: {'year_of_curriculum': 2016},
4: {'status': StudentStatuses.ACADEMIC_LEAVE},
5: {'branch': 'Заочное отделение'},
6: {'branch': 'Нижний Новгород'},
7: {'branch': 'Екатеринбург'},
8: {'branch': 'Минск'},
9: {'branch': 'Москва'},
10: {'type': StudentTypes.VOLUNTEER},
11: {'academic_disciplines': 'ds'},
12: {'academic_disciplines': 'cs'},
13: {'academic_disciplines': 'bd'},
14: {'year_of_curriculum': 2017},
15: {'status': StudentStatuses.GRADUATE},
16: {'year_of_curriculum': 2016},
17: {'cs_center': True},
18: {'year_of_curriculum': 2018},
20: {'academic_disciplines': 'ds'},
21: {'academic_disciplines': 'ml'},
22: {'academic_disciplines': 'bd'},
23: {'year_of_curriculum': 2019},
24: {'year_of_curriculum': 2019},
25: {'year_of_curriculum': 2019},
}
COURSE_WITH_TEN_POINT_MARK_SYSTEM = 575
SKIP_COURSE = 133
USER_MAPPING = {
'pritykovskaya': 586,
'sobrietate69': 6684,
}
SPECIAL_CHARACTERS_MAPPING = {
' ': '-',
'(': '',
')': '',
'.': '',
':': '',
',': '',
'+': 'p',
}
ACADEMIC_DISCIPLINES = [
dict(code='bd', name='Инфраструктура больших данных'),
dict(code='ml', name='Разработка машинного обучения')
]
ANYTASK_PROFILE_TEMPLATE = 'https://anytask.org/accounts/profile/{username}'
ADMIN_USER = User.objects.get(pk=1)
# YDS
SITE_ID = 3
DEFAULT_BRANCH_CODE = 'msk'
def anytask_profile_url(user):
return ANYTASK_PROFILE_TEMPLATE.format(username=user['username'])
def parse_user_status(user_status):
data = {}
for status in user_status:
data.update(USER_STATUS_MAPPING[status])
return data
def get_or_create_user(user_data, user_mapping=None, check=False):
"""
Gets or creates user of the CSC website from anytask data.
Use `user_mapping` to resolve special cases when 2 or more users with the similar credentials exist.
@param user_mapping: dictionary {anytask_username, csc_user.pk}
@return user, created
"""
user_profile = user_data['profile']
username = user_data['username']
email = user_data['email']
# parsed_user_status = parse_user_status(user_profile['user_status'])
# if 'cs_center' in parsed_user_status:
# return None
if user_mapping and username in user_mapping:
csc_pk = user_mapping[username]
return User.objects.get(pk=csc_pk), False
csc_users = User.objects.filter(Q(username=username) | Q(email=email))
if csc_users.count() > 1:
logger.error(f'Cannot create user with username {username} and email {email} - '
f'more than one CSC user with the same data already exists')
return None
return csc_users.get_or_create(defaults=dict(
username=username, email=email,
first_name=user_data['first_name'], last_name=user_data['last_name'],
patronymic=user_profile['middle_name'] if user_profile['middle_name'] else '',
gender=GenderTypes.OTHER, anytask_url=anytask_profile_url(user_data),
time_zone=user_profile['time_zone']
))
def get_or_create_meta_course(meta_course_name, metacourse_slug_mapping=None):
try:
meta_course = MetaCourse.objects.get(name_ru=meta_course_name)
return meta_course
except MetaCourse.DoesNotExist:
if not metacourse_slug_mapping or meta_course_name not in metacourse_slug_mapping:
logger.info(f'Cannot create a meta course {meta_course_name}')
return None
meta_course_slug = metacourse_slug_mapping[meta_course_name]
meta_course, _ = MetaCourse.objects.get_or_create(name_ru=meta_course_name, defaults={
'slug': meta_course_slug, 'description': 'TBD'
})
return meta_course
def get_or_create_course(course_data, courses_df, csc_meta_courses, csc_teachers):
course_id = course_data['id']
course_info = courses_df.loc[course_id, :]
course_teachers = course_data['teachers']
course_groups = course_data['groups']
meta_course_name = course_info['LMS Course']
meta_course = csc_meta_courses[meta_course_name]
semester_type = course_info['Semester Type']
semester, _ = Semester.objects.get_or_create(year=course_info['Year'], type=semester_type)
branch = Branch.objects.get(name_ru=course_info['Main Branch'], site_id=SITE_ID)
grading_type = GradingSystems.BASE if course_id != COURSE_WITH_TEN_POINT_MARK_SYSTEM else GradingSystems.TEN_POINT
group_mode = StudentGroupTypes.MANUAL if len(course_groups) > 1 else StudentGroupTypes.NO_GROUPS
course, _ = Course.objects.update_or_create(meta_course=meta_course, semester=semester, main_branch=branch,
defaults=dict(
grading_type=grading_type, language='ru',
materials_visibility=MaterialVisibilityTypes.VISIBLE,
group_mode=group_mode
))
# Add `distance` branch to all courses
distance_branch = Branch.objects.get(code='distance', site_id=SITE_ID)
course.additional_branches.add(distance_branch)
# Sync branches
CourseService.sync_branches(course)
# Add course teachers
for course_teacher in course_teachers:
teacher_username = course_teacher['username']
# Robot user for shad migration was added as a teacher to all shad courses, skip it
if teacher_username == SHAD_MIGRATOR_USERNAME:
continue
teacher = csc_teachers[teacher_username]
course.teachers.add(teacher)
course.save()
# Add student groups if necessary
if len(course_groups) > 1:
for group in course_groups:
StudentGroup.objects.update_or_create(type=StudentGroupTypes.MANUAL,
name=group['name'],
course=course)
return course
def get_or_create_student_profile(user, user_data, csc_courses, group_course_mapping):
user_profile = user_data['profile']
user_profile_logs = user_data['profiles_logs_by_user']
user_groups = user_data['group_set']
# Check whether we need to create student profile - user should be in one of the groups that have access to courses
user_group_ids = set(group['id'] for group in user_groups)
course_groups = user_group_ids.intersection(group_course_mapping.keys())
if not course_groups:
logger.info(f'{user} - no student profile needed')
return
# Derive courses for user from groups
user_courses = set()
for group in course_groups:
user_courses |= set(group_course_mapping[group])
# Gather marks for courses
user_marks = {}
for course_mark in user_data['studentcoursemark_set']:
course_id = course_mark['course']
if course_id in user_courses:
mark = course_mark['mark']
grade = MARKS[mark['name']] if mark else GradeTypes.NOT_GRADED
user_marks[course_id] = grade
for course in user_courses:
if course not in user_marks:
user_marks[course] = GradeTypes.NOT_GRADED
# Gather data for student profile
profile_data = {}
# Year of admission = year of the earliest group for the student
earliest_group = sorted(user_groups, key=lambda g: g['year']['start_year'])[0]
profile_data['year_of_admission'] = earliest_group['year']['start_year']
# Parse user status - retrieve branch, year_of_curriculum, academic_disciplines, status, type
profile_data['type'] = StudentTypes.REGULAR
parsed_user_status = parse_user_status(user_profile['user_status'])
profile_data.update(parsed_user_status)
# Skip CSC users
if 'cs_center' in profile_data:
logger.info(f'{user} - CSC student: {user.get_absolute_url()}')
return
# Retrieve academic discipline from DB if status was found
# OR Use the most frequent course branch if no status was found, fallback to DEFAULT_BRANCH_CODE
if 'branch' in profile_data:
profile_data['branch'] = Branch.objects.get(name=profile_data['branch'], site_id=SITE_ID)
else:
course_branches = []
for course in user_courses:
course_branches.append(csc_courses[course].main_branch)
branch_counts = Counter(course_branches)
max_count = max(branch_counts.values())
max_count_branches = []
for branch, count in branch_counts.items():
if count == max_count:
max_count_branches.append(branch)
if len(max_count_branches) == 1:
profile_data['branch'] = max_count_branches[0]
else:
profile_data['branch'] = Branch.objects.get(code=DEFAULT_BRANCH_CODE, site_id=SITE_ID)
# Update user branch accordingly
user.branch = profile_data['branch']
user.save()
# Retrieve academic discipline from DB
academic_discipline = None
if 'academic_disciplines' in profile_data:
academic_discipline = AcademicDiscipline.objects.get(code=profile_data['academic_disciplines'])
del profile_data['academic_disciplines']
# Create student profile
student_profile, _ = StudentProfile.objects.update_or_create(user=user, **profile_data)
if academic_discipline:
student_profile.academic_disciplines.add(academic_discipline)
student_profile.save()
# Create status logs for all status changes
user_profile_logs.append(user_profile)
last_status_added = None
for log in user_profile_logs:
parsed_status = parse_user_status(log['user_status']).get('status', None)
timestamp = log['update_time']
date = datetime.fromisoformat(timestamp[:-1]).date()
if parsed_status != last_status_added:
StudentStatusLog.objects.get_or_create(status_changed_at=date, status=parsed_status,
student_profile=student_profile, entry_author=ADMIN_USER)
last_status_added = parsed_status
# Add all enrollments with grades
for course_id, grade in user_marks.items():
try:
course = csc_courses[course_id]
enrollment = Enrollment.objects.update_or_create(student=user, student_profile=student_profile,
course=course, grade=grade)
logger.debug(f'{user} - {student_profile} - {enrollment}')
except IntegrityError as e:
logger.error(f'{user} - {student_profile} - {repr(e)}')
return student_profile
def skip_bad_course(courses):
logger.info(f'Skipping course with id={SKIP_COURSE}')
return [course for course in courses if course['id'] != SKIP_COURSE]
def extract_course_teachers(courses):
course_teachers = {}
for course in courses:
teachers = course['teachers']
for teacher in teachers:
username = teacher['username']
# Robot user for shad migration was added as a teacher to all shad courses, skip it
if username == SHAD_MIGRATOR_USERNAME:
continue
course_teachers[username] = teacher
return course_teachers
def check_duplicates(users):
for user in users:
found = False
username = user['username']
csc_users = User.objects.filter(username=username)
if csc_users.count() > 0:
print(f'Users with username {username} already exist!')
found = True
email = user['email']
csc_users = User.objects.filter(email=email)
if csc_users.count() > 0:
print(f'Users with email {email} already exist!')
found = True
first_name = user['first_name']
last_name = user['last_name']
if first_name and last_name:
patronymic = user['profile']['middle_name']
if patronymic:
csc_users = User.objects.filter(first_name=first_name, last_name=last_name, patronymic=patronymic)
else:
csc_users = User.objects.filter(first_name=first_name, last_name=last_name)
if csc_users.count() > 0:
print(f"User '{first_name} {last_name} {patronymic or ''}' already exists!")
found = True
if found:
print(f'Anytask: {anytask_profile_url(user)} ')
for csc_user in csc_users:
print(f'CSC: {csc_user} - {csc_user.get_absolute_url()}')
def create_student_profiles(users, csc_users, csc_courses, group_course_mapping):
num_created = 0
for user in users:
username = user['username']
csc_user = csc_users[username]
student_profile = get_or_create_student_profile(csc_user, user, csc_courses, group_course_mapping)
if student_profile:
num_created += 1
def create_users(users, user_mapping):
csc_users = {}
num_created = 0
for user in users:
username = user['username']
csc_users[username], created = get_or_create_user(user, user_mapping)
if created:
num_created += 1
bad_users = [u for u in csc_users.values() if not u]
print(f'Done. Got {len(csc_users)} users total. '
f'Created {num_created} new users, could not create {len(bad_users)} users.')
return csc_users
def create_meta_courses(meta_courses, metacourse_slug_mapping=None):
return {meta_course: get_or_create_meta_course(meta_course, metacourse_slug_mapping)
for meta_course in meta_courses}
def create_courses(courses, courses_df, csc_meta_courses, csc_teachers):
csc_courses = {}
for course in courses:
course_id = course['id']
csc_courses[course_id] = get_or_create_course(course, courses_df, csc_meta_courses, csc_teachers)
return csc_courses
def replace_special_characters(slug):
for key, value in SPECIAL_CHARACTERS_MAPPING.items():
slug = slug.replace(key, value)
return slug
def generate_metacourse_slugs(meta_courses):
return [replace_special_characters(meta_course.lower()[:50].translate(ru_en_mapping))
for meta_course in meta_courses]
def get_courses_without_mark_system(courses):
"""
Some old courses do not have a mark system attached.
"""
courses_without_mark_system = []
for course in courses:
if not course['mark_system']:
courses_without_mark_system.append(course['id'])
return courses_without_mark_system
def check_group_intersections(courses):
from itertools import combinations
for course in courses:
course_groups = course['groups']
if len(course_groups) > 1:
for g1, g2 in combinations(course_groups, 2):
intersection = set(g1['students']).intersection(set(g2['students']))
if intersection:
print(course['id'], g1['id'], g2['id'], len(intersection), intersection, sep='\t')
def create_academic_disciplines(disciplines):
for discipline in disciplines:
AcademicDiscipline.objects.update_or_create(code=discipline['code'], name=discipline['name'])
def get_group_course_mapping(courses):
group_course_mapping = {}
for course in courses:
course_id = course['id']
for group in course['groups']:
group_id = group['id']
if group_id not in group_course_mapping:
group_course_mapping[group_id] = []
group_course_mapping[group_id].append(course_id)
return group_course_mapping
def done():
logger.info('Done.')
def load_anytask_data():
# Load CSV with course info
logger.info('Loading CSV with hand-curated info about courses...')
courses_df = pd.read_csv(SHAD_COURSES, delimiter=',', skiprows=1,
names=['Anytask Course', 'Anytask Year', 'LMS Course', 'Org',
'Main Branch', 'Semester', 'Link', 'ID']).set_index('ID')
courses_df['Semester'] = courses_df['Semester'].str.split(' ')
courses_df['Semester Type'] = courses_df['Semester'].apply(
lambda chunks: 'spring' if chunks[0] == 'весна' else 'autumn')
courses_df['Year'] = courses_df['Semester'].apply(lambda chunks: int(chunks[1]))
done()
# Load data from Anytask
logger.info('Loading JSON with Anytask data...')
with open(SHAD_DATA, 'r') as f:
shad_data = json.load(f)
users = shad_data['users']
courses = shad_data['courses']
courses = skip_bad_course(courses)
done()
# Create users for course teachers
logger.info('Creating LMS users for course teachers...')
course_teachers = extract_course_teachers(courses)
csc_teachers = create_users(users=course_teachers.values(), user_mapping=USER_MAPPING)
assert len(csc_teachers) == len(course_teachers), 'Failed to create course teachers'
for csc_teacher in csc_teachers.values():
csc_teacher.add_group(Roles.TEACHER)
done()
# Generate slugs for new meta courses
logger.info('Generating slugs for new meta courses...')
meta_courses = set(courses_df['LMS Course'].values)
existing_meta_courses = {mc.name: mc for mc in MetaCourse.objects.filter(name__in=meta_courses)}
new_meta_courses = [mc for mc in meta_courses if mc not in existing_meta_courses.keys()]
metacourse_slugs = generate_metacourse_slugs(new_meta_courses)
metacourse_slug_mapping = dict(zip(new_meta_courses, metacourse_slugs))
done()
# Create metacourses
logger.info('Creating new meta courses...')
meta_courses = set(courses_df['LMS Course'].values)
csc_meta_courses = create_meta_courses(meta_courses, metacourse_slug_mapping)
assert len(meta_courses) == len(csc_meta_courses), 'Failed to create meta courses'
done()
# Create academic disciplines
logger.info('Creating academic disciplines...')
create_academic_disciplines(ACADEMIC_DISCIPLINES)
done()
# Create courses
logger.info('Creating courses...')
csc_courses = create_courses(courses, courses_df, csc_meta_courses, csc_teachers)
assert len(courses) == len(csc_courses), 'Failed to create courses'
done()
# Create users for students
logger.info('Creating LMS users for students...')
csc_users = create_users(users, user_mapping=USER_MAPPING)
assert len(users) == len(csc_users), 'Failed to create users'
done()
# Create student profiles and fill grades
logger.info('Creating student profiles and filling grades...')
group_course_mapping = get_group_course_mapping(courses)
create_student_profiles(users, csc_users, csc_courses, group_course_mapping)
done()
ctrltz commented
Скрипт для выгрузки данных из энитаска.
from django.contrib.auth.models import User
from rest_framework import serializers
from rest_framework.renderers import JSONRenderer
from courses.models import StudentCourseMark, MarkField, Course, \
CourseMarkSystem, DefaultTeacher, FilenameExtension
from groups.models import Group
from issues.model_issue_field import IssueField
from issues.model_issue_status import IssueStatus, IssueStatusSystem
from issues.models import Issue, Event, File
from tasks.models import Task, TaskGroupRelations
from users.model_user_status import UserStatus
from users.models import UserProfile, UserProfileLog
from years.models import Year
class YearSerializer(serializers.ModelSerializer):
class Meta:
model = Year
fields = ['start_year']
class GroupSerializer(serializers.ModelSerializer):
year = YearSerializer()
class Meta:
model = Group
fields = '__all__'
class MarkFieldSerializer(serializers.ModelSerializer):
class Meta:
model = MarkField
fields = '__all__'
class StudentCourseMarkSerializer(serializers.ModelSerializer):
mark = MarkFieldSerializer()
class Meta:
model = StudentCourseMark
fields = ['course', 'mark']
class UserStatusSerializer(serializers.ModelSerializer):
class Meta:
model = UserStatus
fields = '__all__'
class UserProfileSerializer(serializers.ModelSerializer):
user_status = UserStatusSerializer(many=True)
class Meta:
model = UserProfile
fields = '__all__'
class UserProfileLogSerializer(serializers.ModelSerializer):
user_status = UserStatusSerializer(many=True)
class Meta:
model = UserProfileLog
fields = '__all__'
class UserSerializer(serializers.ModelSerializer):
group_set = GroupSerializer(many=True)
profile = UserProfileSerializer()
profiles_logs_by_user = UserProfileLogSerializer(many=True)
studentcoursemark_set = StudentCourseMarkSerializer(many=True)
class Meta:
model = User
fields = ['username', 'first_name', 'last_name', 'email', 'is_staff',
'is_active', 'date_joined',
'group_set', 'profile', 'profiles_logs_by_user',
'studentcoursemark_set']
class FilenameExtensionSerializer(serializers.ModelSerializer):
class Meta:
model = FilenameExtension
fields = '__all__'
class CourseMarkSystemSerializer(serializers.ModelSerializer):
marks = MarkFieldSerializer(many=True)
class Meta:
model = CourseMarkSystem
fields = '__all__'
class IssueStatusSerializer(serializers.ModelSerializer):
class Meta:
model = IssueStatus
fields = '__all__'
class IssueStatusSystemSerializer(serializers.ModelSerializer):
statuses = IssueStatusSerializer(many=True)
class Meta:
model = IssueStatusSystem
fields = '__all__'
class FileSerializer(serializers.ModelSerializer):
class Meta:
model = File
fields = '__all__'
class IssueFieldSerializer(serializers.ModelSerializer):
class Meta:
model = IssueField
fields = '__all__'
class EventSerializer(serializers.ModelSerializer):
file_set = FileSerializer(many=True)
class Meta:
model = Event
fields = '__all__'
class IssueSerializer(serializers.ModelSerializer):
event_set = EventSerializer(many=True)
class Meta:
model = Issue
fields = '__all__'
class TaskGroupRelationsSerializer(serializers.ModelSerializer):
class Meta:
model = TaskGroupRelations
fields = '__all__'
class TaskSerializer(serializers.ModelSerializer):
issue_set = IssueSerializer(many=True)
taskgrouprelations_set = TaskGroupRelationsSerializer(many=True)
class Meta:
model = Task
fields = '__all__'
class DefaultTeacherSerializer(serializers.ModelSerializer):
class Meta:
model = DefaultTeacher
fields = '__all__'
class CourseSerializer(serializers.ModelSerializer):
defaultteacher_set = DefaultTeacherSerializer(many=True)
filename_extensions = FilenameExtensionSerializer(many=True)
groups = GroupSerializer(many=True)
issue_status_system = IssueStatusSystemSerializer()
issue_fields = IssueFieldSerializer(many=True)
mark_system = CourseMarkSystemSerializer()
task_set = TaskSerializer(many=True)
teachers = UserSerializer(many=True)
year = YearSerializer()
class Meta:
model = Course
fields = '__all__'
def serialize(serializer_class, queryset):
return serializer_class(queryset, many=True, read_only=True).data
# Constants
OUTPUT_PATH = 'shad_data_3.json'
PROFILE_LINKS = [
'https://anytask.org/accounts/profile/bolyarich',
'https://anytask.org/accounts/profile/noath',
'https://anytask.org/accounts/profile/corwin',
'https://anytask.org/accounts/profile/pew-pew',
'https://anytask.org/accounts/profile/elevely',
'https://anytask.org/accounts/profile/rvg77',
'https://anytask.org/accounts/profile/gostkin',
'https://anytask.org/accounts/profile/pitamakan',
'https://anytask.org/accounts/profile/Demidov_Artem',
'https://anytask.org/accounts/profile/zhog96',
'https://anytask.org/accounts/profile/zubkovmaxim',
'https://anytask.org/accounts/profile/kvk1920',
'https://anytask.org/accounts/profile/alex-kozinov',
'https://anytask.org/accounts/profile/komendart',
'https://anytask.org/accounts/profile/tardis.forever',
'https://anytask.org/accounts/profile/Vemmy124',
'https://anytask.org/accounts/profile/KMAMONOV',
'https://anytask.org/accounts/profile/mhuman',
'https://anytask.org/accounts/profile/AleksIks',
'https://anytask.org/accounts/profile/lexolordan',
'https://anytask.org/accounts/profile/cromtus',
'https://anytask.org/accounts/profile/%2B',
'https://anytask.org/accounts/profile/SYury',
'https://anytask.org/accounts/profile/sysak-ma',
'https://anytask.org/accounts/profile/bulatuseinov',
'https://anytask.org/accounts/profile/a_chubcheva',
'https://anytask.org/accounts/profile/yamalutdinovav',
'https://anytask.org/accounts/profile/Mogbymo'
]
USERNAMES = [link.split('/')[-1] for link in PROFILE_LINKS]
TARGET_COURSE_IDS = []
def dump_users(user_queryset):
return serialize(UserSerializer, queryset=user_queryset)
def dump_courses(course_queryset):
return serialize(CourseSerializer, queryset=course_queryset)
def dump():
# Users
user_queryset = User.objects.filter(username__in=USERNAMES)
user_data = dump_users(user_queryset)
print 'Gathered data for {} user(s)'.format(len(user_data))
# Courses
course_queryset = Course.objects.filter(pk__in=TARGET_COURSE_IDS)
course_data = dump_course(course_queryset)
print 'Targeted {} course(s)'.format(len(TARGET_COURSE_IDS))
print 'Gathered data for {} course(s)'.format(len(course_data))
# Combine to JSON
renderer = JSONRenderer()
data = {
# 'courses': course_data,
'users': user_data
}
with open(OUTPUT_PATH, 'w') as f:
f.write(renderer.render(data))
print 'Finished. Output file: {}'.format(OUTPUT_PATH)