CNN Architecture

Hello,

I should have written this issue when we noticed it a while ago but the architecture of the CNN does not match the Nature CNN one (I assume that was the goal), the last layer should have 64 channels too.
This repo:

pytorch-a2c-ppo-acktr-gail/a2c_ppo_acktr/model.py

Lines 176 to 180 in f60ac80

    
           self.main = nn.Sequential( 
        
               init_(nn.Conv2d(num_inputs, 32, 8, stride=4)), nn.ReLU(), 
        
               init_(nn.Conv2d(32, 64, 4, stride=2)), nn.ReLU(), 
        
               init_(nn.Conv2d(64, 32, 3, stride=1)), nn.ReLU(), Flatten(), 
        
               init_(nn.Linear(32 * 7 * 7, hidden_size)), nn.ReLU())

SB2 repo (following OpenAI Baselines repo):
https://github.com/hill-a/stable-baselines/blob/a4efff01ca678bcceee3eb21801c410612df209f/stable_baselines/common/policies.py#L16-L29

or in the SB3 repo:
https://github.com/DLR-RM/stable-baselines3/blob/88e1be9ff5e04b7688efa44951f845b7daf5717f/stable_baselines3/common/torch_layers.py#L76-L84

	self.main = nn.Sequential(
	init_(nn.Conv2d(num_inputs, 32, 8, stride=4)), nn.ReLU(),
	init_(nn.Conv2d(32, 64, 4, stride=2)), nn.ReLU(),
	init_(nn.Conv2d(64, 32, 3, stride=1)), nn.ReLU(), Flatten(),
	init_(nn.Linear(32 * 7 * 7, hidden_size)), nn.ReLU())