Rational Networks in Supervized Learning such as MNIST, CIFAR and Imagenet Classification Tasks.
Rational functions outperformes every non-learnable one (cf. Padé Activation Units: ...).
Rational are also here used for lesioning. They replace Residual Blocks in a ResNet101:
| Eval | Lesion | L2.B3 | L3.B13 | L3.B19 | L4.B2 |
|---|---|---|---|---|---|
| training | Standard | 100.9 | 120.2 | 90.5 | 58.9 |
| Rational | 101.1 | 120.3 | 104.0 | 91.1 | |
| testing | Standard | 93.1 | 102.0 | 97.1 | 81.7 |
| Rational | 90.5 | 102.6 | 97.6 | 85.3 | |
| % dropped params | 0.63 | 2.51 | 2.51 | 10.0 |
- Python 3.6+
- PyTorch
- rational-activations
If you want to use (Recurrent) Rational Networks, an additional README is provided in each folder to explain how to train the networks, make score tables, plots, ...etc
Rational RL - for (Recurrent) Rational Networks on Reinforcement Learning