Forked from: https://github.com/alecwangcq/KFAC-Pytorch. Also using code from: https://github.com/ntselepidis/KFAC-Pytorch