anuk909/novel_problem_generator

Novel Problem Generator

Description

The Novel Problem Generator is an innovative project developed as part of the Natural Language Processing (NLP) course at Tel Aviv University. It leverages Large Language Models (LLMs), specifically GPT-4, to generate unique and challenging programming problems, evaluating them and improving them.

Table of Contents

• Features
• Installation
• Security Notice
• Usage
• Project Structure
• Configuration
• Contributing

Features

• Generates novel programming problems using GPT-4
• Customizable problem generation with various options
• Supports problem validation and improvement
• Exports problems in both JSON and Markdown formats

Installation

Ensure you have Python 3.7 or later installed on your system.

1. Clone the repository:

   git clone https://github.com/anuk909/novel_problem_generator.git
   cd novel_problem_generator

2. Install the package:

   pip install -e .

Security Notice

The human-eval package is used to run untrusted model-generated code, specifically solutions generated by the LLM. You are strongly advised against running this code outside of a robust security sandbox.

Usage

Here's an example of generating a problem with all features enabled:

# Set up Azure OpenAI key and endpoint (see Configuration section)

cd novel_problem_generator
python main.py --attempts 1 \
               --output-base-name all_features \
               --check-gpt-feedback \
               --improve-after-first-try \
               --use-cover-story \
               --use-topics \
               --use-field \
               --require-30-lines

python problems_to_markdown.py problems/all_features.jsonl md

After execution, you can find the generated problem in:

• JSON format: problems/all_features.jsonl
• Markdown format: md/ directory

Run python main.py --help for a full list of available options and their descriptions. The files under resources are arbitrary and it's possible to upgrade or change them as you want.

Project Structure

.
├── baseline.py             # Basic standalone code for generating questions in leet code style 
├── config.py               # Configuration handling
└── experiment_results      # Problems and evaluations from the experiment that was done in the project.
   ├── problems
   ├── evaluations.csv
├── file_handler.py         # File I/O operations
├── __init__.py
├── main.py                 # Main execution script
├── problem_generator.py    # Core problem generation logic
├── problems_to_markdown.py # Converts problems to Markdown
├── problem_validator.py    # Validates generated problems
└── resources/              # Static resources
    ├── cover_story_words.json
    ├── example_hard_problem.json
    ├── fields.json
    └── topics.json

Configuration

The project uses environment variables for sensitive configuration:

• AZURE_OPENAI_API_KEY: Your Azure OpenAI API key
• AZURE_OPENAI_ENDPOINT: Your Azure OpenAI endpoint

You can set these variables in your shell or use a .env file (remember to add .env to your .gitignore).

Contributing

This project was developed and researched by:

• Shmulik Cohen
• Koren Sanger
• Hanan Gevir

Under the mentorship of Ofir Press.