REQUEUEFAILURE

Question

REQUEUEFAILURE

Closed this issue 2 years ago · 16 comments

Hi PanduKonala

this time it shows different error, please advise what would be the fix?

root@ubuntu:~/Batfish-Network-Visualizer# python3 batfish_network_visualizer_v1.py

Batfish Network Visualizer v1.0 Created by Pandu Konala

$ Analysing Configurations...
Traceback (most recent call last):
File "/root/Batfish-Network-Visualizer/batfish_network_visualizer_v1.py", line 225, in
main()
File "/root/Batfish-Network-Visualizer/batfish_network_visualizer_v1.py", line 211, in main
analyse_network(report_dir)
File "/root/Batfish-Network-Visualizer/batfish_network_visualizer_v1.py", line 40, in analyse_network
l3edge = bfq.layer3Edges().answer().frame() #Extract L3 edges
File "/usr/local/lib/python3.10/dist-packages/pybatfish/question/question.py", line 192, in answer
return _bf_answer_obj(
File "/usr/local/lib/python3.10/dist-packages/pybatfish/client/internal.py", line 58, in _bf_answer_obj
workhelper.execute(work_item, session, background, extra_args)
File "/usr/local/lib/python3.10/dist-packages/pybatfish/client/workhelper.py", line 132, in execute
raise BatfishException(
pybatfish.exception.BatfishException: Work finished with status REQUEUEFAILURE
work_item: {"containerName": "network", "id": "c1e7f5a4-6be8-49f5-9e87-07f95c2a47a0", "requestParams": {"answer": "", "questionname": "__layer3Edges_0e0df8df-4d0b-40bb-93e0-4685b965ad71", "testrig": "batfish-candidate"}, "testrigName": "batfish-candidate"}
task_details: {'batches': [{'completed': 0, 'description': "Couldn't requeue after unblocking.\nCannot queue dataplane dependent work for 5d0aa216-3ea7-4c5c-b064-703d08967851: Status is DATAPLANING_FAIL but no incomplete dataplaning work exists", 'size': 0, 'startDate': '2022-04-27T04:43:24.301+00:00'}], 'obtained': '2022-04-27T04:43:24.301+00:00', 'status': 'RequeueFailure'}

Answer 1 · 2022-04-27T07:49:42.000Z

looks like it is related to batfish rather than your code.

Answer 2 · 2022-04-27T07:52:19.000Z

Hi,

I think something is going wrong with the latest docker image of batfish. I will post the version info of the docker image I was using for the project soon. I hope it solves the issue.

Answer 3 · 2022-04-27T08:28:51.000Z

yes I think so. all routing related query failed.

Answer 4 · 2022-04-28T09:39:11.000Z

Hi,

I think something is going wrong with the latest docker image of batfish. I will post the version info of the docker image I was using for the project soon. I hope it solves the issue.

do you have time to check the docker image version? I guess I have to try the docker one by one.

when I run it with 20+ config files, it works ok.

Issue happens when run with 200+ files.

Answer 5 · 2022-04-28T10:07:51.000Z

Hi,

Sorry for the delay. I have the docker image in a compressed format and you can download it from this link

Please let me know if it works.

P.S: The link expires after 24 hours.

Answer 6 · 2022-04-28T14:24:07.000Z

thanks, will try it tomorrow when back to office.

Answer 7 · 2022-04-29T02:51:07.000Z

docker load has below error

root@ubuntu:~/batfish# docker load batfish_docker_version.tar
"docker load" accepts no arguments.
See 'docker load --help'.

Usage: docker load [OPTIONS]

Load an image from a tar archive or STDIN
root@ubuntu:~/batfish# docker load --help

Usage: docker load [OPTIONS]

Load an image from a tar archive or STDIN

Options:
-i, --input string Read from tar archive file, instead of STDIN
-q, --quiet Suppress the load output

root@ubuntu:~/batfish# docker load -i batfish_docker_version.tar
open /var/lib/docker/tmp/docker-import-2914322182/bin/json: no such file or directory

after using "docker import"

I have this

root@ubuntu:~/batfish# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
batfish 20220429 228c8dbc19e4 2 minutes ago 1.06GB

but docker run commnd failed.

root@ubuntu:/batfish# docker run --name batfish -v /root/batfish/batfish-data:/data -p 8888:8888 -p 9997:9997 -p 9996:9996 batfish:20220429
docker: Error response from daemon: No command specified.
See 'docker run --help'.
root@ubuntu:/batfish#

what command did you use to export the docker image? docker save or docker export?

a simple "docker run" wont work too.
as you can see, docker run hello-world is ok.

root@ubuntu:/batfish# docker run batfish:20220429
docker: Error response from daemon: No command specified.
See 'docker run --help'.
root@ubuntu:/batfish#
root@ubuntu:/batfish#
root@ubuntu:/batfish# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete
Digest: sha256:10d7d58d5ebd2a652f4d93fdd86da8f265f5318c6a73cc5b6a9798ff6d2b2e67
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:

The Docker client contacted the Docker daemon.
The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/

For more examples and ideas, visit:
https://docs.docker.com/get-started/

root@ubuntu:~/batfish#

Answer 8 · 2022-04-29T09:43:16.000Z

Hi,

I have used docker export to export the docker image. To run the image I use the command "$sudo docker container start batfish". After running the above command and running "$sudo docker container ls" below is the output.

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2c3d355aba91 batfish/allinone "./wrapper.sh" 8 months ago Up 2 minutes 0.0.0.0:8888->8888/tcp, :::8888->8888/tcp, 0.0.0.0:9996-9997->9996-9997/tcp, :::9996-9997->9996-9997/tcp batfish

To import the image use the command "$sudo tar -c . | sudo docker import - exampleimagedir" for more information check "https://docs.docker.com/engine/reference/commandline/import/"

Answer 9 · 2022-04-29T09:56:59.000Z

If none of the above work could you please try this version and get back to me "$docker pull batfish/allinone:6668c2e_b60484b"

I tested the program using the batfish version "2021.04.12.882" and it works.

Answer 10 · 2022-05-03T07:57:26.000Z

can't use this exported images.

(batfishvenv) root@ubuntu:/Batfish-Network-Visualizer/Images# tar -c . | docker import batfish_docker_version.tar
sha256:532747d2b10f97dd8d8a4f2f762f96e5d1d7a50684fff0234887ac079471fa66
(batfishvenv) root@ubuntu:/Batfish-Network-Visualizer/Images# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
532747d2b10f 6 seconds ago 1.06GB
batfish/allinone latest 050a5fc47b5c 3 weeks ago 1.11GB

(batfishvenv) root@ubuntu:/Batfish-Network-Visualizer/Images# docker tag 532747d2b10f batfish:20220503
(batfishvenv) root@ubuntu:/Batfish-Network-Visualizer/Images#
(batfishvenv) root@ubuntu:~/Batfish-Network-Visualizer/Images# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
batfish 20220503 532747d2b10f 34 seconds ago 1.06GB
batfish/allinone latest 050a5fc47b5c 3 weeks ago 1.11GB

(batfishvenv) root@ubuntu:~/Batfish-Network-Visualizer# docker run --name batfish -v batfish-data:/data -p 8888:8888 -p 9997:9997 -p 9996:9996 -d batfish:20220503
docker: Error response from daemon: No command specified.
See 'docker run --help'.

Hi,

I have used docker export to export the docker image. To run the image I use the command "$sudo docker container start batfish". After running the above command and running "$sudo docker container ls" below is the output.

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2c3d355aba91 batfish/allinone "./wrapper.sh" 8 months ago Up 2 minutes 0.0.0.0:8888->8888/tcp, :::8888->8888/tcp, 0.0.0.0:9996-9997->9996-9997/tcp, :::9996-9997->9996-9997/tcp batfish

To import the image use the command "$sudo tar -c . | sudo docker import - exampleimagedir" for more information check "https://docs.docker.com/engine/reference/commandline/import/"

Answer 11 · 2022-05-03T07:59:24.000Z

If none of the above work could you please try this version and get back to me "$docker pull batfish/allinone:6668c2e_b60484b"

I tested the program using the batfish version "2021.04.12.882" and it works.

I can pull the image, however still has error.

(batfishvenv) root@ubuntu:/Batfish-Network-Visualizer# docker run -d --name batfish -v batfish-data:/data -p 8888:8888 -p 9997:9997 -p 9996:9996 batfish/allinone:6668c2e_b60484b
d558f5854855c36c2c08e7efb8381a4a873406e37eabf4fcb19fcbc5d8573729
(batfishvenv) root@ubuntu:/Batfish-Network-Visualizer#
(batfishvenv) root@ubuntu:/Batfish-Network-Visualizer#
(batfishvenv) root@ubuntu:/Batfish-Network-Visualizer# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d558f5854855 batfish/allinone:6668c2e_b60484b "./wrapper.sh" 4 seconds ago Up 2 seconds 0.0.0.0:8888->8888/tcp, :::8888->8888/tcp, 0.0.0.0:9996-9997->9996-9997/tcp, :::9996-9997->9996-9997/tcp batfish
(batfishvenv) root@ubuntu:/Batfish-Network-Visualizer#
(batfishvenv) root@ubuntu:/Batfish-Network-Visualizer#
(batfishvenv) root@ubuntu:/Batfish-Network-Visualizer# python3
batfish_network_visualizer_v1.py .git/ results/
batfishvenv/ Images/ snapshot/
(batfishvenv) root@ubuntu:/Batfish-Network-Visualizer# python3 batfish_network_visualizer_v1.py

Batfish Network Visualizer v1.0 Created by Pandu Konala

$ Analysing Configurations...
Traceback (most recent call last):
File "/root/Batfish-Network-Visualizer/batfish_network_visualizer_v1.py", line 225, in
main()
File "/root/Batfish-Network-Visualizer/batfish_network_visualizer_v1.py", line 211, in main
analyse_network(report_dir)
File "/root/Batfish-Network-Visualizer/batfish_network_visualizer_v1.py", line 40, in analyse_network
l3edge = bfq.layer3Edges().answer().frame() #Extract L3 edges
File "/root/Batfish-Network-Visualizer/batfishvenv/lib/python3.10/site-packages/pybatfish/question/question.py", line 192, in answer
return _bf_answer_obj(
File "/root/Batfish-Network-Visualizer/batfishvenv/lib/python3.10/site-packages/pybatfish/client/internal.py", line 58, in _bf_answer_obj
workhelper.execute(work_item, session, background, extra_args)
File "/root/Batfish-Network-Visualizer/batfishvenv/lib/python3.10/site-packages/pybatfish/client/workhelper.py", line 132, in execute
raise BatfishException(
pybatfish.exception.BatfishException: Work finished with status REQUEUEFAILURE
work_item: {"containerName": "network", "id": "6c1debe1-f44f-49cb-91c6-10adf99f09eb", "requestParams": {"answer": "", "questionname": "__layer3Edges_85eece89-998f-41ec-939c-9c0fafe69779", "testrig": "batfish-candidate"}, "testrigName": "batfish-candidate"}
task_details: {'batches': [{'completed': 0, 'description': "Couldn't requeue after unblocking.\nCannot queue dataplane dependent work for b3aba6b8-7391-42d9-b7df-4bb209517b48: Status is DATAPLANING_FAIL but no incomplete dataplaning work exists", 'size': 0, 'startDate': '2022-05-03T07:43:54.174+0000'}], 'obtained': '2022-05-03T07:43:54.174+0000', 'status': 'RequeueFailure'}
(batfishvenv) root@ubuntu:~/Batfish-Network-Visualizer#

Answer 12 · 2022-05-03T08:14:26.000Z

it works with 50 config files though. not with 200+

Answer 13 · 2022-05-03T08:32:03.000Z

Hi,

I think the older version of batfish docker has a limit on how many it can process with a single request.

Answer 14 · 2022-05-03T08:35:52.000Z

I am using ubuntu 22.04, going to try ubuntu 18.04 (which recommended by batfish), see if there is any difference.

Answer 15 · 2022-05-19T06:42:56.000Z

can close this ticket because it is confirmed related to batfish. thanks

Answer 16 · 2022-05-24T20:41:40.000Z

Awesome!

Have a great day!