addLink error at 113th vnf created in same datacenter
Closed this issue · 3 comments
Hello,
I am using the virtual image SONATA Emulator Demo VM 2017 Download (~4-5GB) from https://github.com/sonata-nfv/son-tutorials/tree/master/upb-containernet-emulator-summerschool-demo, and I had encountered the following issue:
After creating one datacenter, I was trying to start multiple vnfs (around 150 in the same datacenter), and at the 113th vnf the emulator gave me the following error:
ERROR:root:API error.
Traceback (most recent call last):
File "/home/sonata/son-emu/src/emuvim/api/rest/compute.py", line 74, in put
compute_name, image=image, command=command, network=nw_list)
File "/home/sonata/son-emu/src/emuvim/dcemulator/node.py", line 276, in startCompute
self.net.addLink(d, self.switch, params1=nw, cls=Link, intfName1=nw.get('id'))
File "/home/sonata/son-emu/src/emuvim/dcemulator/net.py", line 185, in addLink
link = Containernet.addLink(self, node1, node2, **params)
File "/home/sonata/containernet/mininet/net.py", line 402, in addLink
link = cls( node1, node2, **options )
File "/home/sonata/containernet/mininet/link.py", line 440, in init
link=self, mac=addr1, **params1 )
File "/home/sonata/containernet/mininet/link.py", line 61, in init
self.config( **params )
File "/home/sonata/containernet/mininet/link.py", line 194, in config
self.setParam( r, 'setIP', ip=ip )
File "/home/sonata/containernet/mininet/link.py", line 177, in setParam
result = f( value )
File "/home/sonata/containernet/mininet/link.py", line 77, in setIP
return self.ifconfig( ipstr, 'up' )
File "/home/sonata/containernet/mininet/link.py", line 69, in ifconfig
return self.cmd( 'ifconfig', self.name, *args )
File "/home/sonata/containernet/mininet/link.py", line 65, in cmd
return self.node.cmd( *args, **kwargs )
File "/home/sonata/containernet/mininet/node.py", line 959, in cmd
out = self.dcli.exec_start(exec_dict)
File "/usr/local/lib/python2.7/dist-packages/docker/utils/decorators.py", line 35, in wrapper
return f(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/docker/api/exec_api.py", line 144, in exec_start
return self._read_from_socket(res, stream)
File "/usr/local/lib/python2.7/dist-packages/docker/api/client.py", line 357, in _read_from_socket
return six.binary_type().join(frames_iter(socket))
File "/usr/local/lib/python2.7/dist-packages/docker/utils/socket.py", line 73, in frames_iter
n = next_frame_size(socket)
File "/usr/local/lib/python2.7/dist-packages/docker/utils/socket.py", line 60, in next_frame_size
data = read_exactly(socket, 8)
File "/usr/local/lib/python2.7/dist-packages/docker/utils/socket.py", line 45, in read_exactly
next_data = read(socket, n - len(data))
File "/usr/local/lib/python2.7/dist-packages/docker/utils/socket.py", line 27, in read
select.select([socket], [], [])
ValueError: filedescriptor out of range in select()
After this error, if I try to create other vnfs in the same datacenter, the following error appears:
ERROR:root:API error.
Traceback (most recent call last):
File "/home/sonata/son-emu/src/emuvim/api/rest/compute.py", line 74, in put
compute_name, image=image, command=command, network=nw_list)
File "/home/sonata/son-emu/src/emuvim/dcemulator/node.py", line 276, in startCompute
self.net.addLink(d, self.switch, params1=nw, cls=Link, intfName1=nw.get('id'))
File "/home/sonata/son-emu/src/emuvim/dcemulator/net.py", line 185, in addLink
link = Containernet.addLink(self, node1, node2, **params)
File "/home/sonata/containernet/mininet/net.py", line 402, in addLink
link = cls( node1, node2, **options )
File "/home/sonata/containernet/mininet/link.py", line 430, in init
node1, node2, deleteIntfs=False )
File "/home/sonata/containernet/mininet/link.py", line 474, in makeIntfPair
deleteIntfs=deleteIntfs )
File "/home/sonata/containernet/mininet/util.py", line 202, in makeIntfPair
( intf1, intf2, cmdOutput ) )
Exception: Error creating interface pair (vnf227-eth0,dc1.s1-eth113): RTNETLINK answers: File exists
This problem appeared each time I had done this test (using other datacenter or restarting the virtual machine)
Could you please provide a feedback and/or a solution regarding this issue?
Thank you!
Interesting. Looks like docker is not able connect to the 113th container and configure it properly. The issue might be related to the particular version of the emulator and the actual setup inside the demo VM.
I would suggest to use the latest version of the emulator (which is now called vim-emu) and part of the OSM project as mentioned in the README.md of this repository:
- https://osm.etsi.org/gitweb/?p=osm/vim-emu.git;a=summary
- Wiki: https://osm.etsi.org/wikipub/index.php/VIM_emulator
You should install it in a fresh Ubuntu 16:04 machine with at least 8 GB memory if you want to run so many emulated VNFs. Pretty sure that it should work with the latest version of the emulator since we used it recently to do some experiments with up to 256 VNFs.
Hello Mr. Peuster,
Thank you for your response.
I tried to install vim-emu by following the installation steps:
sudo apt-get install lxd lxd-client
$ wget -O install_osm.sh "https://osm.etsi.org/gitweb/?p=osm/devops.git;a=blob_plain;f=installers/install_osm.sh;hb=HEAD"
$ chmod +x install_osm.sh
$ ./install_osm.sh --lxdimages --vimemu
After running the last command, the following error appears
luni 26 februarie 2018, 22:28:47 +0200 container_exec_stderr: exec in RO "curl -sL -w %{http_code}\n http://www.google.com/ -o /dev/null
luni 26 februarie 2018, 22:28:47 +0200 wait_container_up: waiting for container RO to start
luni 26 februarie 2018, 22:28:47 +0200 wait_container_up: expected '200' in Permission denied, are you in the lxd group?
Could you please tell me the appropriate configuration of lxd and if any other prerequisites are needed?
I also tried the manual installation having the same result.
Thank you!
I guess you do not plan to use OSM, right? So you should go with the manual bare metal installation that installs vim-emu without OSM:
see Option 1: Bare-metal installation (https://osm.etsi.org/wikipub/index.php/VIM_emulator#Option_1:_Bare-metal_installation)
Option 1: Bare-metal installation
- Requires: Ubuntu 16.04 LTS
$ sudo apt-get install ansible git aptitude
Step 1: Containernet installation
$ cd
$ git clone https://github.com/containernet/containernet.git
$ cd ~/containernet/ansible
$ sudo ansible-playbook -i "localhost," -c local install.yml
Step 2: vim-emu installation
$ cd
$ git clone https://osm.etsi.org/gerrit/osm/vim-emu.git
$ cd ~/vim-emu/ansible
$ sudo ansible-playbook -i "localhost," -c local install.yml
$ cd ~/vim-emu
$ sudo python setup.py install
This is the nightly tested way of installing vim-emu which should work in any case.
Important: This should be performed on a FRESH Ubuntu 16.04 LTS installation.