/techpath-diploma

Information retrieval web application of MOOCs (mass open online courses) for IT specialties

Primary LanguageTypeScript

TechPath - Diploma Project

Thesis: Information retrieval web application of MOOCs (mass open online courses) for IT specialties

Table of Contents

About the project

TechPath is a web platform developed to simplify searching for educational materials to study IT disciplines. Every person may find himself drowning in a vast variery of topics, paths and learning resources.

That is why this app may come in handy, helping to clear one's mind by suggesting hand-picked materials and roadmaps to learn a topic or even a whole profession.

Screenshots

Architecture Overview

The repo consists of 5 packages (apps):

  • Client - react web app
  • Server - primary endpoint for a client app
  • Parser - web crawler that parses data from different resources (e.g., Coursera, Stepik) and stores the unprocessed data in a RabbitMQ message queue
  • DataProcessor - python script that processes courses data from the message queue - performs data cleanup, determines languages and tags
  • Shared - components (e.g., utils, types, constants)
System Architecture Diagram GraphQL Schema Diagram

Blue color on architecture diagram describes a component that is to be added, red - a components that is not planned yet.

Tech Stack

Frontend

  • TypeScript
  • React
  • Mantine
  • GraphQL
  • Apollo
  • Vite

Backend

  • TypeScript
  • Nest.js
  • Apollo
  • Crawlee
  • Neo4j
  • RabbitMQ

Data Processor

  • Python
  • Pika (RabbitMQ adapter)
  • Neo4j
  • Fasttext
  • Transformers

Misc

  • Nx (monorepo tool)
  • Docker
  • Docker compose

Getting started

Prerequisites

A list of required software:

  • npm/yarn
  • docker
  • python3

Installation

  1. Clone the repo
git clone git@github.com:FloydanTheBeast/diploma.git
  1. Install dependencies
npm i
## OR
yarn
  1. Copy env.example config file and fill it with desired values
  2. Create and start docker containers
docker-compose up -d

Usage

  1. Serve NX package - server, client or parser
nx serve <server|client|parser>
  1. Generate GraphQL types based on a schema introspection for client or server
yarn gql-codegen:<client|server>

Roadmap

Server

  • Role-based JWT auth
  • Parse and process courses data from different vendors
  • Courses languages and tags classification
  • GraphQL API with CRUD operations for every entity
  • Data filtering, sorting, pagination and aggregation
  • Global fuzzy search for courses and roadmaps
  • Basic recommendation system
  • Roadmaps serialization and deserealization to the format supported by the client
  • Train own model to improve classification accuracy
  • User lists for saved courses/roadmaps
  • Personalized courses/roadmaps compilations
  • Tag clustering by specializations
  • Personalized collaborative recommendation system, user data collection
  • Testing system for users to determine their preferences
  • CI/CD deployment pipeline

Client

  • Basic JWT auth
  • Role-based routing
  • View courses, roadmaps and course vendors
  • Table pagination, search, sort and filter
  • Specific course page
  • Global search
  • Bookmarks for courses and roadmaps
  • In-app reviews for courses and roadmaps
  • View courses recommendations
  • Visual roadmap building tool
  • Improve roadmaps visually
  • Combine several roadmaps into a series
  • Use roadmaps as template
  • Stats dashboard for admins
  • View user profiles
  • Guest access to the app
  • Internationalization
  • Adaptation for mobile devices
  • CI/CD deployment pipeline

2022-2023 Graduate Work (Diploma) by Shadi Abdelsalam
Higher School of Economics, Faculty of Computer Science, Software Engineering Department