Project available at: ttr.guide
Recorded demo available at: https://youtu.be/5jQCix0P_fE
Sorry to everyone whose been using it, but as of October 2024 this is now sunset.
Anyone who wants to use this is free to set it up on their own!
I am sunsetting this due to high costs of DynamoDb on AWS, I'd love to make another iteration of this project in the future, but for now it's not feasible. It definitely needs lots of changes, first of all getting rid of DynamoDb, architecture redesign and drastically improving the code quality.
Thank you for understanding and sorry for the inconvenience!
As this project is sunset I made a dump of all collected data from the DynamoDB database.
It is available in the backend/dynamo-db-dump folder.
The Tools and Technologies Research guide (TTR.guide) project aimed to provide a comprehensive guide to job market analysis and provide valuable insights for both end-users and developers by leveraging the power of natural language processing (NLP) techniques. This open-source project utilised the GPT-3.5 Turbo OpenAI API to extract tools and technologies from job postings. The project followed an Agile methodology, which allowed for continuous iteration and improvement, while careful consideration of ethical, legal, and social aspects related to data handling and user privacy were also prioritised.
The TTR platform encompasses a backend with an API and a frontend with dashboard visualisations and API documentation. The data collection component has so far amassed over 60,000 job posts, primarily focusing on programming and engineering jobs sourced from reed.co.uk API. The GPT model was used for efficient data processing, capable of processing diverse and unstructured data. The system successfully processed three bursts of data, each containing about 10,000 job posts per second.
The TTR system's adaptability and modularity are key strengths, facilitating scalability and potential expansion. The frontend features a landing page with SEO optimisations, metadata tags, and branding, an interactive API documentation page, and a search and statistics page that delivers valuable insights to users. The TTR project's open-source future is secured by licensing it under the GPL-3.0 licence, encouraging contributions from other developers and researchers while fostering growth and contributions within the broader open-source community.
The TTR project's success was due to the effectiveness of GPT-based NLP for data processing, the Agile methodology's adaptability, effective data handling and processing, scalability, open-source nature and best practices, and careful consideration of ethical, legal, and social aspects. The project provided valuable learning experiences, including NLP and GPT techniques, software development, architecture of systems and project management, and ethical, legal, and social considerations.
Future improvements to the TTR platform could include dataset expansion, enhancing NLP and GPT techniques, additional features, routine maintenance and optimization, and collaborations with job posting websites. In conclusion, the TTR project represents a significant achievement in providing a valuable and adaptable tool for job market analysis, with the experiences gained and lessons learned throughout its development serving as a strong foundation for future projects and endeavours.