Step-by-step implementation of basic machine learning algorithms with performance comparisons against the sklearn implementation and solving classification and prediction problems.
- ๐ Implemented both analytical and numerical solutions:
- Closed-form linear regression
- Powellโs optimization method (scipy)
- ๐งช Tested on synthetic 1D datasets (
make_regression, noise=16) - โฑ๏ธ Compared in terms of accuracy and runtime
- ๐งพ Analytical implementation with Tikhonov regularization
- ๐งช Benchmarked against
RidgeClassifierfrom sklearn - โค๏ธ Applied to Cleveland Heart Disease dataset:
- Data preprocessing, outlier removal
- Statistical profiling and feature selection (manual vs automatic)
- โ๏ธ Custom implementation using:
- Gradient Descent (with learning rate tuning)
- โจ Mini-Batch optimization
- ๐ Squared Gradients Sum (AdaGrad-style)
- ๐งช Tested on synthetic datasets:
- Binary classification
- Multi-class scenarios
- ๐ง Three kernel implementations:
- โ Linear Kernel (hard/soft margin)
- ๐ถ Polynomial Kernel (degree customization)
- ๐ RBF Kernel (gamma parameter tuning)
- โ๏ธ Benchmarking against sklearn's SVM:
- Accuracy comparison
- Training time analysis
- ๐ Visualization of decision boundaries
- ๐ก๏ธ Entropy Minimization approach:
- Recursive binary splitting
- Pre-pruning parameters
- ๐งช Test scenarios:
- Single-feature datasets
- High-dimensional multi-class problems
- ๐ Comparison with sklearn's DecisionTreeClassifier
- ๐๏ธ Custom ensemble features:
- Bagging with replacement
- Feature subspace selection
- Comparison with sklearn.ensemble.RandomForest
- ๐ง Manually built multi-layer feedforward neural network
- ๐ฆ Trained on MNIST with adjustable activation functions (e.g., ReLU, Sigmoid)
- ๐ Reports metrics after each epoch: accuracy, sensitivity, specificity
Paweล Marchel
If you find this project helpful, feel free to star โญ the repo or get in touch for collaboration.