/Softmax-classifier-implementation

This project demonstrates the implementation of a Softmax classifier from scratch, featuring both naive (loop-based) and vectorized versions for educational and performance comparison purposes. The implementation is based on CIFAR 10 dataset.

Primary LanguageJupyter NotebookMIT LicenseMIT

Softmax Classifier Implementation

A complete implementation of a Softmax classifier with cross-entropy loss for multi-class classification, applied to the CIFAR-10 dataset.

Overview

This project implements both naive and vectorized versions of the Softmax classifier with the following features:

  • Numerical Stability: Proper handling of overflow/underflow issues
  • Vectorized Implementation: Efficient matrix operations for faster training
  • Cross-entropy Loss: Standard loss function for multi-class classification
  • L2 Regularization: Prevents overfitting
  • CIFAR-10 Dataset: Real-world image classification task

Features

  • ✅ Naive implementation with explicit loops
  • ✅ Vectorized implementation for efficiency
  • ✅ Numerical stability improvements
  • ✅ L2 regularization support
  • ✅ Training visualization
  • ✅ Model saving/loading
  • ✅ Performance comparison between implementations

Installation

  1. Clone the repository:
git clone https://github.com/HoomKh/softmax-classifier.git
cd softmax-classifier
  1. Install dependencies:
pip install -r requirements.txt
  1. Download the CIFAR-10 dataset:
    • Place the cifar-10-batches-py folder in the dataset/ directory
    • Or update the path in the notebook to point to your dataset location

Project Structure

softmax-classifier/
├── README.md
├── requirements.txt
├── softmax.py                 # Core softmax implementations
├── linear_classifier.py       # Linear classifier base class
├── data_utils.py             # Data loading utilities
├── softmax_classifier.ipynb  # Main training notebook
└── dataset/                  # CIFAR-10 dataset (not included)
    └── cifar-10-batches-py/

Usage

Running the Notebook

  1. Start Jupyter:
jupyter notebook

  1. Open softmax_classifier.ipynb

  2. Run all cells to:

    • Load and preprocess CIFAR-10 data
    • Train the softmax classifier
    • Visualize training progress
    • Evaluate model performance

Key Parameters

  • Learning Rate: 1e-7 (default)
  • Regularization: 2.5e4 (default)
  • Training Iterations: 1500 (default)
  • Batch Size: 200 (default)

Implementation Details

Softmax Loss Function

The softmax loss combines:

  • Softmax activation: P(y=k|x) = exp(s_k) / Σ exp(s_i)
  • Cross-entropy loss: L = -log(P(y_true|x))
  • L2 regularization: R(W) = λ * Σ W²

Numerical Stability

The implementation includes several numerical stability improvements:

  • Subtracting maximum score before exponential
  • Proper gradient normalization
  • Correct regularization loss calculation

Results

On CIFAR-10 dataset:

  • Training Accuracy: ~32.6%
  • Validation Accuracy: ~34%
  • Training Time: ~0.9 seconds for 1500 iterations

Performance Comparison

Implementation Training Time Loss
Naive ~2.5s 2.34
Vectorized ~0.07s 2.34

Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contact

Acknowledgments

  • CIFAR-10 dataset from CIFAR
  • Implementation based on CS231n course materials

Troubleshooting

Common Issues

  1. Module not found errors: Ensure all dependencies are installed
  2. Dataset path errors: Update the cifar10_dir path in the notebook
  3. Memory issues: Reduce batch size or use smaller dataset subset
  4. Numerical instability: The implementation includes fixes for this

Getting Help

If you encounter any issues:

  1. Check the troubleshooting section above
  2. Search existing issues on GitHub
  3. Create a new issue with detailed error information