FileNotFoundError: [Errno 2] No such file or directory: '/data1/maoyuning/NetShare-master/results/test_cidds/generated_data/best_syn_dfs/syn.csv'

[MUNeng]

I'm running on a single machine(Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-148-generic x86_64) ) and turn Ray on.
The driver.py looks:

import netshare.ray as ray

from netshare import Generator

if __name__ == '__main__':
    # Change to False if you would not like to use Ray
    ray.config.enabled = True
    ray.init(address="auto")

    # configuration file
    generator = Generator(config="netflow/config_example_netflow_nodp_cidds.json")

    # `work_folder` should not exist o/w an overwrite error will be thrown.
    # Please set the `worker_folder` as *absolute path*
    # if you are using Ray with multi-machine setup
    # since Ray has bugs when dealing with relative paths.
    generator.train_and_generate(work_folder='/data1/maoyuning/NetShare-master/results/test_cidds')

    ray.shutdown()

The netflow/config_example_netflow_nodp_cidds.json is as follows:

{
    "global_config": {
        "original_data_file": "../traces/cidds/raw.csv",
        "dataset_type": "netflow",
        "n_chunks": 10,
        "dp": false
    },
    "pre_post_processor": {
        "class": "NetsharePrePostProcessor",
        "config": {
            "max_flow_len": null
        }
    },
    "model": {
        "class": "DoppelGANgerTFModel",
        "config": {
            "iteration": 20,
            "extra_checkpoint_freq": 10,
            "epoch_checkpoint_freq": 5
        }
    },
    "default": "netflow.json"
}

Here is the error message:

Traceback (most recent call last):
  File "driver.py", line 17, in <module>
    generator.train_and_generate(work_folder='/data1/maoyuning/NetShare-master/results/test_cidds')
  File "/data1/maoyuning/NetShare-master/netshare/generators/generator.py", line 205, in train_and_generate
    if not self.generate(work_folder):
  File "/data1/maoyuning/NetShare-master/netshare/generators/generator.py", line 176, in generate
    work_folder)):
  File "/data1/maoyuning/NetShare-master/netshare/generators/generator.py", line 110, in _post_process
    log_folder=log_folder)
  File "/data1/maoyuning/NetShare-master/netshare/pre_post_processors/pre_post_processor.py", line 38, in post_process
    log_folder=log_folder)
  File "/data1/maoyuning/NetShare-master/netshare/pre_post_processors/netshare/netshare_pre_post_processor.py", line 532, in _post_process
    os.path.join(output_folder, "syn.csv")
  File "/data1/maoyuning/.conda/envs/tf-1.15-py36/lib/python3.6/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/data1/maoyuning/NetShare-master/results/test_cidds/generated_data/best_syn_dfs/syn.csv'

[sniperyyc]

Thanks for bringing up the problem. We will take a look and get back to you with your previous issue together :)

[hallavar]

Hello, same issue here, with a slightly different config file.

I think this is raised when the visualization tries to save some data, I guess.
I can try to run it with vis_freq > iteration, in order to see if the error is raised.

[sniperyyc]

Feel free to try the latest refactored PyTorch version which includes the updated examples and reopen this issue if the bug persists.