OpenAI Gym Custom Environment Observation Space returns "None"

Question

OpenAI Gym Custom Environment Observation Space returns "None"

Closed this issue 2 years ago · 4 comments

After setting up a custom environment, I was testing whether my observation_space and action_space were properly defined. I was able to call:

env.observation_space and get the properly defined observation_space
env.observation_space.sample() and get a well-working sample

Though when calling env.observation_space.shape, I got "None" as a return which doesn't make sense considering I could sample it properly.

When I run:
obs_space = env.observation_space
action_space = env.action_space
print("The observation space: {}".format(obs_space))
print("The action space: {}".format(action_space))

I got:
The observation space: None
The action space: None

Because of this error, I could not use these codes for example:

state_shape = env.observation_space.shape
or
state_shape= env.observation_space.n

I got en error:
AttributeError: 'NoneType' object has no attribute 'shape'

In order to set my DQN, I need this.

What am I doing wrong?

Regards,
Serap

Answer 1 · 2022-06-24T08:02:43.000Z

You would like to call a env.reset() first before getting these attributes because they, in fact, are initiated as None. If you are planning to use stable_baselines3, for example, to choose the type of RL among the ones they have implemented, you can use check_env() function also. It can help you regarding these problems (sometimes you can have a wrong shape also, the check helps with it).

You can check in init() function that they are None by default. They will have a value only after the first communication between the simulation and the python process running the Env. That is because it needs to know what you want as actions and observations, and that comes from gymConnection module in omnet++ scenario.

When reset() is called, that first communication happens, and it fills observation and action as intended.

Please check if you are trying to print after or before a reset().

Answer 2 · 2022-06-27T09:02:44.000Z

Please update VeinsGym to 0.3.3+ and seee if the error persists. Also see #8.

Answer 3 · 2022-06-27T14:39:39.000Z

@dbuse Thank you I updated.

@lionyouko Thanks for detailed help.
I tried to use check_env() function but because of tensorflow.contrib error I could't make it. I reinstall tensorflow=1.14.0 (because of stable_baseline accepts), then my other codes broke down :(

But I could started to calculate and implement DQN. But, I think I got an error for the same reason. Here are my codes and error at the end:

`#!/usr/bin/env python3
"""
Trivial example agent for the dcc environment.
"""

import logging
import os
import random
import time

import gym
import veins_gym
from veins_gym import veinsgym_pb2

import torch, numpy as np
from torch import nn

import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten
from tensorflow.keras.optimizers import Adam
import tensorflow

from rl.agents import DQNAgent
from rl.policy import BoltzmannQPolicy
from rl.memory import SequentialMemory

def serialize_action(actions):
"""Searialize a list of floats into an action."""
reply = veinsgym_pb2.Reply()
reply.action.box.values.extend(actions)
return reply.SerializeToString()

gym.register(
id="veins-v1",
entry_point="veins_gym:VeinsEnv",
kwargs={
"scenario_dir": os.path.relpath(
os.path.join(
os.path.dirname(os.path.abspath(file)), "..", "scenario"
)
),
"timeout": 150.0,
"print_veins_stdout": False,
#"action_serializer": serialize_action,
# to run in a GUI, use:
# "user_interface": "Qtenv",
# "timeout": -1,
},
)

def build_model(states, actions):
model = Sequential()
#model.add(Flatten(input_shape=(1,states)))
model.add(Dense(24, activation='relu', input_shape=(1,)))
model.add(Dense(24, activation='relu'))
model.add(Dense(actions, activation='linear'))
#model.add(Flatten(input_shape=(states,1)))
return model

#del model

def build_agent(model, actions):
policy = BoltzmannQPolicy()
memory = SequentialMemory(limit=50000, window_length=1)
dqn = DQNAgent(model=model, memory=memory, policy=policy,
nb_actions=actions, nb_steps_warmup=10, target_model_update=1e-2)
return dqn

def main():
"""
Run the trivial agent.
"""
logging.basicConfig(level=logging.DEBUG)

env = gym.make("veins-v1")
logging.info("Env created")

env.reset()
logging.info("Env reset")
done = False
   
episodes = 5
rewards = []
    
states = env.observation_space.shape
print(states)
actions = env.action_space.n
print(actions)


#del model 

model = build_model(states, actions)
model.summary()

print(model.input_shape)


dqn = build_agent(model, actions)
dqn.compile(Adam(lr=1e-3), metrics=['mae'])
dqn.fit(env, nb_steps=50000, visualize=False, verbose=1)

#agent.train(10000)


for episode in range(1, episodes+1):

    print("Env reset for episode:", episode)
    observation, reward, done, info = env.step(random.randint(0, 1))
    
    while not done:
        r_action = random.randint(0, 1)
        observation, reward, done, info = env.step(r_action)
        rewards.append(reward)
        
    # 1604022> the last action sent by omnetpp to veinsgym will be a shutdown
    # and that will make the done set to true. In the veinsgym code that this happens, it will
    # also generates a random of the observation (_parse request function)
    # that will be sent in step function
    # and must then be discarded (as it is an undesired step + 1 anyway)
    #if not done:
        logging.debug(
            "Last action: %s, Reward: %.3f, Observation: %s",
            r_action,
            reward,
            observation,
            
        )
        
     
    model = build_model(observation, r_action)
    model.summary()
    
    dqn = build_agent(model, r_action)
    dqn.compile(Adam(lr=1e-3), metrics=['mae'])
    dqn.fit(env, nb_steps=50000, visualize=False, verbose=1)   
        
    print(type(env.observation_space))
    print(type(env.action_space))
        
    print('Episode:{} Reward:{}'.format(episode, reward))
        
   
    print("Number of steps taken:", len(rewards))
    print("Mean reward:", sum(rewards) / len(rewards))
    rewards = []
    time.sleep(0.059)
    #episodes=episodes+1;

if name == "main":
main()
`

An the errors of Console as:
2022-06-27 16:31:21.593068: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/seergun/anaconda3/lib/python3.8/site-packages/cv2/../../lib64::/home/seergun/.mujoco/mujoco210/bin 2022-06-27 16:31:21.593098: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. INFO:root:Env created INFO:root:Closing VeinsEnv. DEBUG:root:Listening on random port 37635 DEBUG:root:Launching veins experiment using command ['./run', '-uCmdenv', '-cGeneral', '--seed-set=0', '--.manager.seed=0', '--.gym_connection.port=37635']`
DEBUG:root:Veins process launched with pid 985818
INFO:root:Launched veins experiment, waiting for request.
INFO:root:Received first request from Veins, ready to run.
(1,)
2
INFO:root:Env reset
Model: "sequential"

Layer (type) Output Shape Param #

dense (Dense) (None, 24) 48

dense_1 (Dense) (None, 24) 600

dense_2 (Dense) (None, 2) 50

=================================================================
Total params: 698
Trainable params: 698
Non-trainable params: 0

(None, 1)
/home/seergun/anaconda3/lib/python3.8/site-packages/keras/optimizers/optimizer_v2/adam.py:110: UserWarning: The lr argument is deprecated, use learning_rate instead.
super(Adam, self).init(name, **kwargs)
2022-06-27 16:31:25.093644: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/seergun/anaconda3/lib/python3.8/site-packages/cv2/../../lib64::/home/seergun/.mujoco/mujoco210/bin
2022-06-27 16:31:25.093683: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-06-27 16:31:25.093704: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (seergun-OptiPlex-9010): /proc/driver/nvidia/version does not exist
2022-06-27 16:31:25.101442: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
Training for 50000 steps ...
INFO:root:Closing VeinsEnv.
DEBUG:root:Closing VeinsEnv server socket.
DEBUG:root:Listening on random port 39673
DEBUG:root:Launching veins experiment using command ['./run', '-uCmdenv', '-cGeneral', '--seed-set=0', '--*.manager.seed=0', '--*.gym_connection.port=39673']
DEBUG:root:Veins process launched with pid 985839
INFO:root:Launched veins experiment, waiting for request.
INFO:root:Received first request from Veins, ready to run.
Interval 1 (0 steps performed)
Traceback (most recent call last):
File "/home/seergun/Downloads/dcc-env-master/agents/dqnlearning.py", line 200, in
main()
File "/home/seergun/Downloads/dcc-env-master/agents/dqnlearning.py", line 127, in main
dqn.fit(env, nb_steps=50000, visualize=False, verbose=1)
File "/home/seergun/anaconda3/lib/python3.8/site-packages/rl/core.py", line 168, in fit
action = self.forward(observation)
File "/home/seergun/anaconda3/lib/python3.8/site-packages/rl/agents/dqn.py", line 224, in forward
q_values = self.compute_q_values(state)
File "/home/seergun/anaconda3/lib/python3.8/site-packages/rl/agents/dqn.py", line 68, in compute_q_values
q_values = self.compute_batch_q_values([state]).flatten()
File "/home/seergun/anaconda3/lib/python3.8/site-packages/rl/agents/dqn.py", line 63, in compute_batch_q_values
q_values = self.model.predict_on_batch(batch)
File "/home/seergun/anaconda3/lib/python3.8/site-packages/keras/engine/training_v1.py", line 1186, in predict_on_batch
inputs, _, _ = self._standardize_user_data(
File "/home/seergun/anaconda3/lib/python3.8/site-packages/keras/engine/training_v1.py", line 2323, in _standardize_user_data
return self._standardize_tensors(
File "/home/seergun/anaconda3/lib/python3.8/site-packages/keras/engine/training_v1.py", line 2351, in _standardize_tensors
x = training_utils_v1.standardize_input_data(
File "/home/seergun/anaconda3/lib/python3.8/site-packages/keras/engine/training_utils_v1.py", line 633, in standardize_input_data
raise ValueError('Error when checking ' + exception_prefix +
ValueError: Error when checking input: expected dense_input to have 2 dimensions, but got array with shape (1, 1, 1)`

Dimension and shape issues make trouble.

As different from these:
Today, I tried totally different issue as:
I tried to connect my project to veins-gym like you commented previous

But again :( unfortunately, got an error.

Maybe because of run file or snakemake I don't know :(

I really need help from you as experts. I'm badly stuck I am confused.
What should I do?
I tried to change serpentine-env and dcc-env to my project with different application layer files. Could not.
I tried to connect my project to gym. Could not.

So sorry for the long issue.
Regards,

Answer 4 · 2022-07-04T10:03:23.000Z

@serapergun I think was you that was waiting for the example scenario, right?
By this time your work should be much more advanced than it, but I going to let here a link for you:
https://github.com/lionyouko/veinsgym_lowerloss.git

Please, @dbuse , could you see if it is suitable as an example for this repo? Please, see if you can run it as it is on a linux machine (maybe at maximum rebuild it in omnet). I don't know how to make snakemake files, so I still use omnet to build before to use, and the run script is solely to run the executable directly. Thank you