Support for invertible FC neural networks with inputs of shape (1,)

[sahandrez]

Hi,

I have not been able to create an invertible fully-connected NN that has an input with the shape (1,). A minimal example ro recreate the error is:

import torch.nn as nn
import FrEIA.framework as Ff
import FrEIA.modules as Fm

input_dim = 1
hidden_dim = 256

def subnet_initialization(m):
    if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):
        nn.init.kaiming_uniform_(m.weight.data)
        m.bias.data *= 0.1

def subnet_fc(c_in, c_out):
    net = nn.Sequential(
        nn.Linear(c_in, hidden_dim),
        nn.ReLU(),
        nn.Linear(hidden_dim, c_out)
    )
    net.apply(subnet_initialization)
    return net

inn = Ff.SequenceINN(input_dim)
for k in range(3):
    inn.append(Fm.AllInOneBlock, subnet_constructor=subnet_fc, permute_soft=True)

Which results in the followin traceback, seems like the issue is in special_ortho_group method of scipy.

Traceback (most recent call last):
  File "/home/sahand/workspace/hpg/test.py", line 29, in <module>
    inn.append(Fm.AllInOneBlock, subnet_constructor=subnet_fc, permute_soft=True)
  File "/home/sahand/workspace/FrEIA/FrEIA/framework/sequence_inn.py", line 53, in append
    module = module_class(dims_in, **kwargs)
  File "/home/sahand/workspace/FrEIA/FrEIA/modules/all_in_one_block.py", line 144, in __init__
    w = special_ortho_group.rvs(channels)
  File "/home/sahand/miniconda3/envs/dmc_3.8/lib/python3.8/site-packages/scipy/stats/_multivariate.py", line 3371, in rvs
    dim = self._process_parameters(dim)
  File "/home/sahand/miniconda3/envs/dmc_3.8/lib/python3.8/site-packages/scipy/stats/_multivariate.py", line 3343, in _process_parameters
    raise ValueError("""Dimension of rotation must be specified,
ValueError: Dimension of rotation must be specified,
                                and must be a scalar greater than 1.

Is there a workaround for this issue? A naive way would be to repeat the input or concat a column of zeros so that the neural net would have inputs of shape (2,).

Thanks!

[krishnarld]

Hi,

Did you try increasing the input size? In one of the issues, it is mentioned the code has issues with batch size =1.

[psorrenson]

Unfortunately, flows with 1-dimensional inputs and outputs are not currently implemented in FrEIA. You might be able to use the NLSQ flow from this repo: https://github.com/harvardnlp/TextFlow. Alternatively you can add a noise dimension to your input, as in the Augmented Flow: https://arxiv.org/abs/2002.07101