Example 1 does not work: JSON parsing error
knodir opened this issue · 11 comments
I installed using vagrant image and the basic example did work. However, I get simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0) error when I try to chain "client" interface with "server" interface (see below).
ubuntu@ubuntu-xenial:~/son-emu$ son-emu-cli compute list
+--------------+-------------+----------------------------+------------------+-------------------------+
| Datacenter | Container | Image | Interface list | Datacenter interfaces |
+==============+=============+============================+==================+=========================+
| dc2 | server | sonatanfv/sonata-empty-vnf | server-eth0 | dc2.s1-eth2 |
+--------------+-------------+----------------------------+------------------+-------------------------+
| dc1 | client | sonatanfv/sonata-empty-vnf | client-eth0 | dc1.s1-eth2 |
+--------------+-------------+----------------------------+------------------+-------------------------+
ubuntu@ubuntu-xenial:~/son-emu$
ubuntu@ubuntu-xenial:~/son-emu$
ubuntu@ubuntu-xenial:~/son-emu$ son-emu-cli network add -b -src client:client-eth0 -dst server:server-eth0
Traceback (most recent call last):
File "/usr/local/bin/son-emu-cli", line 11, in <module>
load_entry_point('emuvim', 'console_scripts', 'son-emu-cli')()
File "/home/ubuntu/son-emu/src/emuvim/cli/son_emu_cli.py", line 54, in main
restnetw.main(sys.argv[2:])
File "/home/ubuntu/son-emu/src/emuvim/cli/rest/network.py", line 138, in main
c.execute_command(args)
File "/home/ubuntu/son-emu/src/emuvim/cli/rest/network.py", line 42, in execute_command
getattr(self, args["command"])(args)
File "/home/ubuntu/son-emu/src/emuvim/cli/rest/network.py", line 64, in add
pp.pprint(response.json())
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 886, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/simplejson-3.10.0-py2.7-linux-x86_64.egg/simplejson/__init__.py", line 516, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python2.7/dist-packages/simplejson-3.10.0-py2.7-linux-x86_64.egg/simplejson/decoder.py", line 374, in decode
obj, end = self.raw_decode(s)
File "/usr/local/lib/python2.7/dist-packages/simplejson-3.10.0-py2.7-linux-x86_64.egg/simplejson/decoder.py", line 404, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
ubuntu@ubuntu-xenial:~/son-emu$
@stevenvanrossem can you help here? Seems to be a problem with the networking API.
There is indeed a bug in the son-emu-cli network command, it is not using the latest REST API syntax for the networking. I will update it.
The son-emu-cli network cli is a bit outdated, since it was looked over when new REST API updates were introduced...
I pushed a fix: #224 and #225
Meanwhilie I also updated our documentation on the wiki a bit:
wiki CLI overview
wiki REST API overview
Thanks @stevenvanrossem for quickly looking into this issue. However, now it is failing with different error NetworkXError: The node None is not in the digraph (see below).
Terminal 2
ubuntu@ubuntu-xenial:~$ son-emu-cli compute list
+--------------+-------------+----------------------------+------------------+-------------------------+
| Datacenter | Container | Image | Interface list | Datacenter interfaces |
+==============+=============+============================+==================+=========================+
| dc2 | server | sonatanfv/sonata-empty-vnf | server-eth0 | dc2.s1-eth2 |
+--------------+-------------+----------------------------+------------------+-------------------------+
| dc1 | client | sonatanfv/sonata-empty-vnf | client-eth0 | dc1.s1-eth2 |
+--------------+-------------+----------------------------+------------------+-------------------------+
ubuntu@ubuntu-xenial:~$
ubuntu@ubuntu-xenial:~$ son-emu-cli network add -b -src client:client-eth0 -dst server:server-eth0
u'"The node None is not in the digraph."\n'
ubuntu@ubuntu-xenial:~$
Terminal 1
containernet>
DEBUG:dcemulator.net:addLink: n1=server intf1=server-eth0 -- n2=dc2.s1 intf2=dc2.s1-eth2
DEBUG:dcemulator.net:call chainAddFlow vnf_src_name=None, vnf_src_interface=None, vnf_dst_name=None, vnf_dst_interface=None
ERROR:root:API error.
Traceback (most recent call last):
File "/home/ubuntu/son-emu/src/emuvim/api/rest/network.py", line 118, in _NetworkAction
monitor_placement=monitor_placement)
File "/home/ubuntu/son-emu/src/emuvim/dcemulator/net.py", line 567, in setChain
ret = self._chainAddFlow(vnf_src_name, vnf_dst_name, vnf_src_interface, vnf_dst_interface, **kwargs)
File "/home/ubuntu/son-emu/src/emuvim/dcemulator/net.py", line 594, in _chainAddFlow
connected_sw = self.DCNetwork_graph.neighbors(vnf_src_name)[0]
File "/usr/local/lib/python2.7/dist-packages/networkx/classes/digraph.py", line 765, in successors
return list(self.successors_iter(n))
File "/usr/local/lib/python2.7/dist-packages/networkx/classes/digraph.py", line 751, in successors_iter
raise NetworkXError("The node %s is not in the digraph."%(n,))
NetworkXError: The node None is not in the digraph.
containernet>
Hi @knodir,
This example is working on my setup.
Can you make sure you pulled the latest version? (I just merged the pull requests that should fix this)
Additionally, can you check the same command by using the REST API instead of the CLI:
curl -X PUT -i -H 'Content-Type: application/json' -d '{"vnf_src_name":"client","vnf_src_interface":"client-eth0","vnf_dst_name":"server", "vnf_dst_interface":"server-eth0","bidirectional":true}' http://localhost:5001/restapi/network
Thanks @stevenvanrossem, The node None is not in the digraph. error got resolved with your latest change. Now links are getting correctly attached. However, ping still does not work. It fails with Destination Host Unreachable.
To reduce the scope, I skipped snort vnf from the setup and just tried to connect client vnf to the server vnf. This one is also failing with Destination Host Unreachable. I also tried the same with the REST API call and faced the same Destination Host Unreachable, i.e., CLI and REST API are working the same (as expected).
Here are commands and their respective output. It is a bit verbose (and long), but I hope this will give you richer debugging information. Thanks for taking time to look at this!
Terminal 1
sudo python src/emuvim/examples/sonata_y1_demo_topology_1.py
[ ...truncated... ]
*** Starting controller
c0
*** Starting 3 switches
dc1.s1 (10ms delay) dc2.s1 (20ms delay) s1 (10ms delay) (20ms delay) ...(10ms delay) (20ms delay) (10ms delay) (20ms delay)
*** Starting CLI:
containernet>
Terminal 2 (add client vnf)
ubuntu@ubuntu-xenial:~$ son-emu-cli compute start -d dc2 -n client -i sonatanfv/sonata-iperf3-vnf
{ u'cpu_period': None,
u'cpu_quota': -1,
u'cpu_shares': None,
u'cpuset': None,
u'datacenter': u'dc2',
u'docker_network': u'172.17.0.2',
u'flavor_name': u'tiny',
u'hostname': u'client',
u'id': u'487ef4c629ab8e30fba97ccbe6f2e32eda42bac52c01a92da9c52cbef006c625',
u'image': u'sonatanfv/sonata-iperf3-vnf',
u'mem_limit': None,
u'memswap_limit': None,
u'name': u'client',
u'network': [ { u'dc_portname': u'dc2.s1-eth2',
u'intf_name': u'client-eth0',
u'ip': u'10.0.0.2/8',
u'mac': u'be:69:8c:09:7b:ab',
u'netmask': u'8',
u'status': u'OK',
u'up': True}],
u'short_id': u'487ef4c629ab',
u'state': { u'Dead': False,
u'Error': u'',
u'ExitCode': 0,
u'FinishedAt': u'0001-01-01T00:00:00Z',
u'OOMKilled': False,
u'Paused': False,
u'Pid': 2323,
u'Restarting': False,
u'Running': True,
u'StartedAt': u'2017-06-03T08:23:38.898096914Z',
u'Status': u'running'}}
Relevant debugging output from terminal 1
DEBUG:dcemulator.node:Starting compute instance u'client' in data center 'dc2'
client: update resources {'cpu_quota': -1}
DEBUG:dcemulator.net:addLink: n1=client intf1=client-eth0 -- n2=dc2.s1 intf2=dc2.s1-eth2
Terminal 2 (add server vnf)
ubuntu@ubuntu-xenial:~$ son-emu-cli compute start -d dc2 -n server -i sonatanfv/sonata-iperf3-vnf
{ u'cpu_period': None,
u'cpu_quota': -1,
u'cpu_shares': None,
u'cpuset': None,
u'datacenter': u'dc2',
u'docker_network': u'172.17.0.3',
u'flavor_name': u'tiny',
u'hostname': u'server',
u'id': u'01151f12c5c7994648afbd7b1f3277453626e7aeb62cd5790fbd3ae5fa1d0275',
u'image': u'sonatanfv/sonata-iperf3-vnf',
u'mem_limit': None,
u'memswap_limit': None,
u'name': u'server',
u'network': [ { u'dc_portname': u'dc2.s1-eth3',
u'intf_name': u'server-eth0',
u'ip': u'10.0.0.4/8',
u'mac': u'8e:25:77:68:cc:71',
u'netmask': u'8',
u'status': u'OK',
u'up': True}],
u'short_id': u'01151f12c5c7',
u'state': { u'Dead': False,
u'Error': u'',
u'ExitCode': 0,
u'FinishedAt': u'0001-01-01T00:00:00Z',
u'OOMKilled': False,
u'Paused': False,
u'Pid': 2585,
u'Restarting': False,
u'Running': True,
u'StartedAt': u'2017-06-03T08:23:52.295086019Z',
u'Status': u'running'}}
ubuntu@ubuntu-xenial:~$
Relevant debugging output from terminal 1
DEBUG:dcemulator.node:Starting compute instance u'server' in data center 'dc2'
server: update resources {'cpu_quota': -1}
DEBUG:dcemulator.net:addLink: n1=server intf1=server-eth0 -- n2=dc2.s1 intf2=dc2.s1-eth3
Terminal 2
ubuntu@ubuntu-xenial:~$ son-emu-cli compute list
+--------------+-------------+-----------------------------+------------------+-------------------------+
| Datacenter | Container | Image | Interface list | Datacenter interfaces |
+==============+=============+=============================+==================+=========================+
| dc2 | client | sonatanfv/sonata-iperf3-vnf | client-eth0 | dc2.s1-eth2 |
+--------------+-------------+-----------------------------+------------------+-------------------------+
| dc2 | server | sonatanfv/sonata-iperf3-vnf | server-eth0 | dc2.s1-eth3 |
+--------------+-------------+-----------------------------+------------------+-------------------------+
Terminal 2 (add network link from client to server)
ubuntu@ubuntu-xenial:~$ son-emu-cli network add -b -src client:client-eth0 -dst server:server-eth0
u'"path add-flow between client and server\\npath add-flow between server and client"\n'
ubuntu@ubuntu-xenial:~$
Relevant debugging output from terminal 1
INFO:root:data: ImmutableMultiDict([('priority', u'0'), ('bidirectional', u'True'), ('vnf_dst_name', u'server'), ('vnf_src_name', u'client'), ('vnf_src_interface', u'client-eth0'), ('vnf_dst_interface', u'server-eth0')])
DEBUG:dcemulator.net:call chainAddFlow vnf_src_name=u'client', vnf_src_interface=u'client-eth0', vnf_dst_name=u'server', vnf_dst_interface=u'server-eth0'
INFO:dcemulator.net:Path between client and server: ['dc2.s1']
INFO:dcemulator.net:end node reached: server
DEBUG:dcemulator.net:set vlan in switch: dc2.s1 in_port: dc2.s1-eth2 vlan tag: 0
DEBUG:dcemulator.net:call chainAddFlow vnf_src_name=u'server', vnf_src_interface=u'server-eth0', vnf_dst_name=u'client', vnf_dst_interface=u'client-eth0'
INFO:dcemulator.net:Path between server and client: ['dc2.s1']
INFO:dcemulator.net:end node reached: client
DEBUG:dcemulator.net:set vlan in switch: dc2.s1 in_port: dc2.s1-eth3 vlan tag: 1
Terminal 1
containernet> client ifconfig
client-eth0 Link encap:Ethernet HWaddr be:69:8c:09:7b:ab
inet addr:10.0.0.2 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::bc69:8cff:fe09:7bab/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:16 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1296 (1.2 KB) TX bytes:648 (648.0 B)
eth0 Link encap:Ethernet HWaddr 02:42:ac:11:00:02
inet addr:172.17.0.2 Bcast:0.0.0.0 Mask:255.255.0.0
inet6 addr: fe80::42:acff:fe11:2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:24 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1944 (1.9 KB) TX bytes:648 (648.0 B)
lo Linkencap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
containernet> server ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:ac:11:00:03
inet addr:172.17.0.3 Bcast:0.0.0.0 Mask:255.255.0.0
inet6 addr: fe80::42:acff:fe11:3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:648 (648.0 B) TX bytes:648 (648.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
server-eth0 Link encap:Ethernet HWaddr 8e:25:77:68:cc:71
inet addr:10.0.0.4 Bcast:10.255.255.255 Mask:255.0.0.0
inet6 addr: fe80::8c25:77ff:fe68:cc71/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:648 (648.0 B) TX bytes:648 (648.0 B)
Terminal 1 (ping server from client)
containernet> client ping -c2 server
PING 10.0.0.4 (10.0.0.4): 56 data bytes
92 bytes from 10.0.0.2: Destination Host Unreachable
92 bytes from 10.0.0.2: Destination Host Unreachable
--- 10.0.0.4 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
I got this working, but want to double check if my understanding of the problem is correct.
I think vnf placement was the problem. Wiki places all client, server, and snort_vnf on dc2, which did not work for me. Then I placed client and server on dc1 and snort_vnf got placed on dc2 by default (containernet debug info shows RoundRobinDcPlacementWithSAPs algorithm made such placement). Then client was able ping server.
Here are my changes:
- son-emu-cli compute start -d dc2 -n client -i sonatanfv/sonata-iperf3-vnf
+ son-emu-cli compute start -d dc1 -n client -i sonatanfv/sonata-iperf3-vnf
- son-emu-cli compute start -d dc2 -n server -i sonatanfv/sonata-iperf3-vnf
+ son-emu-cli compute start -d dc1 -n server -i sonatanfv/sonata-iperf3-vnf
I am suspicious about VLAN tagging/untagging when I do son-emu-cli network add. For the working version I see the path for snort_vnf -> server is ['dc2.s1', 's1', 'dc1.s1']. On this path VLAN tagging is done on dc2.s1-eth4 vlan tag: 3 and untagging on dc1.s1-eth3 vlan tag: 3.
The reverse server -> snort_vnf path is ['dc1.s1', 's1', 'dc2.s1'] with tagging dc1.s1-eth3 vlan tag: 4 and untagging dc2.s1-eth4 vlan tag: 4. See the full debug info from containernet below.
INFO:root:data: ImmutableMultiDict([('priority', u'0'), ('bidirectional', u'True'), ('vnf_dst_name', u'server'), ('vnf_src_name', u'snort_vnf'), ('vnf_src_interface', u'output'), ('vnf_dst_interface', u'server-eth0')])
DEBUG:dcemulator.net:call chainAddFlow vnf_src_name=u'snort_vnf', vnf_src_interface=u'output', vnf_dst_name=u'server', vnf_dst_interface=u'server-eth0'
INFO:dcemulator.net:Path between snort_vnf and server: ['dc2.s1', 's1', 'dc1.s1']
DEBUG:dcemulator.net:set vlan in switch: dc2.s1 in_port: dc2.s1-eth4 vlan tag: 3
INFO:dcemulator.net:end node reached: server
DEBUG:dcemulator.net:set vlan in switch: dc1.s1 in_port: dc1.s1-eth3 vlan tag: 3
DEBUG:dcemulator.net:call chainAddFlow vnf_src_name=u'server', vnf_src_interface=u'server-eth0', vnf_dst_name=u'snort_vnf', vnf_dst_interface=u'output'
INFO:dcemulator.net:Path between server and snort_vnf: ['dc1.s1', 's1', 'dc2.s1']
DEBUG:dcemulator.net:set vlan in switch: dc1.s1 in_port: dc1.s1-eth3 vlan tag: 4
INFO:dcemulator.net:end node reached: snort_vnf
DEBUG:dcemulator.net:set vlan in switch: dc2.s1 in_port: dc2.s1-eth4 vlan tag: 4
Similar tagging and untagging happens between client and snort_vnf.
However, for non-working case (all nodes on dc2) I see VLAN mentioned only for one port, which I am assuming to be for tagging only. The path for snort_vnf -> server is ['dc2.s1'] and we have tagging on dc2.s1 in_port: dc2.s1-eth4 vlan tag: 3 only. Similarly, for server -> snort_vnf we have dc2.s1 in_port: dc2.s1-eth6 vlan tag: 4. See the full debug info below. Are we missing untagging?
INFO:root:data: ImmutableMultiDict([('priority', u'0'), ('bidirectional', u'True'), ('vnf_dst_name', u'server'), ('vnf_src_name', u'snort_vnf'), ('vnf_src_interface', u'output'), ('vnf_dst_interface', u'server-eth0')])
DEBUG:dcemulator.net:call chainAddFlow vnf_src_name=u'snort_vnf', vnf_src_interface=u'output', vnf_dst_name=u'server', vnf_dst_interface=u'server-eth0'
INFO:dcemulator.net:Path between snort_vnf and server: ['dc2.s1']
INFO:dcemulator.net:end node reached: server
DEBUG:dcemulator.net:set vlan in switch: dc2.s1 in_port: dc2.s1-eth4 vlan tag: 3
DEBUG:dcemulator.net:call chainAddFlow vnf_src_name=u'server', vnf_src_interface=u'server-eth0', vnf_dst_name=u'snort_vnf', vnf_dst_interface=u'output'
INFO:dcemulator.net:Path between server and snort_vnf: ['dc2.s1']
INFO:dcemulator.net:end node reached: snort_vnf
DEBUG:dcemulator.net:set vlan in switch: dc2.s1 in_port: dc2.s1-eth6 vlan tag: 4
Two other relevant questions.
- does this example work if everything is placed on the same dc? If yes, perhaps we could remove switch and one data center to simplify the example.
- can we manually place
snort_vnf, i.e., avoidRoundRobinDcPlacementWithSAPs? Perhaps during VNF creation we could specify name of the data center we want VNF to be placed by adding an extra option to thecurlcall, such as
- curl -X POST http://127.0.0.1:5000/instantiations -d "{}"
+ curl -X POST http://127.0.0.1:5000/instantiations -d "{}" -placement "dc1"
This would be very helpful if one wants to use external tools for VNF placement.
Thanks!
If you do NOT execute the son-emu-cli network add command, the ping should work.
This works because the topology is started with enable_learning=True.
(check in: sudo python src/emuvim/examples/sonata_y1_demo_topology_1.py)
This configures the ovs switches/datacenters as standalone switches, and thus no explicit chaining is needed to forward the traffic.
After executing the network add command, the ping stops working.
If you add a priority setting > 0, it works again:
son-emu-cli network add -b -src client:client-eth0 -dst server:server-eth0 -p10
The priority was not set high enough, the chaining flowrules are not matched and traffic gets indeed blocked by the different VLAN tagging. (The VLAN tagging is needed to isolate emulated LAN segments, which is a feature currently only accessible by deploying VNFs via a SONATA service package, not yet via the cli/rest API ). If the chaining flowrules get a higher prioirity, they override the VLAN tagging.
I will add default higher values for the priority setting when adding new flowrules.
This option: enable_learning=True is currently not yet included in the unittests, I will try to make them cover also this case…
regarding the other question about VNF placement: it is possible to specify the datacenter where the VNF will be placed, I expanded a bit on this topic here: #227
as Steven explained in #227 manual placement is only possible through CLI/REST API. To modify the placement of the dummy gatekeeper (SONATA service package instantiation) you need to replace the default RoundRobin placement algorithm, e.g., by a algorithm that always places in the first DC in the list:
class FirstDcPlacement(object):
"""
Placement: Always use one and the same data center from the GK.dcs dict.
"""
def place(self, nsd, vnfds, saps, dcs):
for id, vnfd in vnfds.iteritems():
vnfd["dc"] = list(dcs.itervalues())[0](this example algorithm is also already in the code can can be used)
Unfortunately we don't have a nice configuration option for this yet and you have to activate your algorithm in this line: https://github.com/sonata-nfv/son-emu/blob/master/src/emuvim/api/sonata/dummygatekeeper.py#L200
self._calculate_placement(FirstDcPlacement)Thanks @stevenvanrossem and @mpeuster for working on these issues!
Commit #228 resolved the ping issue when all 3 nodes are placed on the dc2. I can also see the default priority 1000 applied to flow rules. Following verbose is very useful (thanks @stevenvanrossem)
$ son-emu-cli network add -b -src client:client-eth0 -dst snort_vnf:input
"success: add-flow between client and snort_vnf with options: {
"priority": "1000",
"path": [
"dc2.s1"
],
"vlan": 1,
"cookie": "10",
"match_input": null
}
success: add-flow between snort_vnf and client with options: {
"priority": "1000",
"path": [
"dc2.s1"
],
"vlan": 2,
"cookie": "10",
"match_input": null
}"
$ son-emu-cli network add -b -src snort_vnf:output -dst server:server-eth0
"success: add-flow between snort_vnf and server with options: {
"priority": "1000",
"path": [
"dc2.s1"
],
"vlan": 3,
"cookie": "10",
"match_input": null
}
success: add-flow between server and snort_vnf with options: {
"priority": "1000",
"path": [
"dc2.s1"
],
"vlan": 4,
"cookie": "10",
"match_input": null
}"