Using bash kernel in notebook-http mode
3goats opened this issue · 5 comments
Hi I'm following the docs here: https://jupyter-kernel-gateway.readthedocs.io/en/latest/http-mode.html
It doesn't seem to mention how to select a different kernel when running in http mode. I was assuming it would take the kernel type from the .ipynb file that is used to provide the mappings for the http endpoints. Am I missing something here ?
I can see I have 2 kernels installed e.g.
Available kernelspecs: {
default: 'python3',
kernelspecs: {
python3: { name: 'python3', spec: [Object], resources: [Object] },
bash: { name: 'bash', spec: [Object], resources: {} }
}
}
The bash kernel is working perfectly running in Web Socket mode. However, its doesn't seem to work in http-mode.
I just saw that there was an option to force the kernel during startup. e.g. --KernelGatewayApp.force_kernel_name=bash however this does not appear to have any affect on the error.
A little more info:
I have a cell in a notebook file:
# GET /hello/bash
echo "fsdfdsadsds"
it gets registered perfectly:
[I 2023-02-07 17:07:36.781 KernelGatewayApp] Registering resource: /hello/bash, methods: (['GET'])
....
I 2023-02-07 17:07:36.781 KernelGatewayApp] Jupyter Kernel Gateway 2.5.2 is available at http://0.0.0.0:80
[D 230207 17:08:08 handlers:193] Request code for notebook cell is: REQUEST = "{\"body\": \"\", \"args\": {}, \"path\": {}, \"headers\": {\"Host\": \"127.0.0.1:5006\", \"User-Agent\": \"curl/7.85.0\", \"Accept\": \"*/*\"}}"
[D 2023-02-07 17:08:08.196 KernelGatewayApp] connecting shell channel to tcp://127.0.0.1:48567
[D 2023-02-07 17:08:08.196 KernelGatewayApp] Connecting to: tcp://127.0.0.1:48567
[D 2023-02-07 17:08:08.206 KernelGatewayApp] activity on 85950786-e434-48b0-bc66-fca0387cf859: status (busy)
[D 2023-02-07 17:08:08.218 KernelGatewayApp] activity on 85950786-e434-48b0-bc66-fca0387cf859: execute_input
[D 2023-02-07 17:08:08.259 KernelGatewayApp] activity on 85950786-e434-48b0-bc66-fca0387cf859: stream
[D 2023-02-07 17:08:08.321 KernelGatewayApp] activity on 85950786-e434-48b0-bc66-fca0387cf859: error
[D 2023-02-07 17:08:08.323 KernelGatewayApp] activity on 85950786-e434-48b0-bc66-fca0387cf859: status (idle)
[E 230207 17:08:08 web:2271] 500 GET /hello/bash (172.17.0.1) 129.24ms
The error corresponds to my curl command that looks like this:
curl http://127.0.0.1:5006/hello/bash
Error : 127
Just for clarity, here's a complete log - Looks like the "bash' kernel is being instantiated properly:
...
[D 2023-02-07 18:08:24.104 KernelGatewayApp] Instantiating kernel 'Bash' with kernel provisioner: local-provisioner
[D 2023-02-07 18:08:24.117 KernelGatewayApp] Starting kernel: ['/usr/local/bin/python3', '-m', 'bash_kernel', '-f',
...
Full output:
D 2023-02-07 18:08:23.966 KernelGatewayApp] Searching ['/root/.jupyter', '/root/.local/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2023-02-07 18:08:23.966 KernelGatewayApp] Looking for jupyter_config in /etc/jupyter
[D 2023-02-07 18:08:23.966 KernelGatewayApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2023-02-07 18:08:23.966 KernelGatewayApp] Looking for jupyter_config in /root/.local/etc/jupyter
[D 2023-02-07 18:08:23.966 KernelGatewayApp] Looking for jupyter_config in /root/.jupyter
[D 2023-02-07 18:08:23.966 KernelGatewayApp] Looking for jupyter_kernel_gateway_config in /etc/jupyter
[D 2023-02-07 18:08:23.966 KernelGatewayApp] Looking for jupyter_kernel_gateway_config in /usr/local/etc/jupyter
[D 2023-02-07 18:08:23.966 KernelGatewayApp] Looking for jupyter_kernel_gateway_config in /root/.local/etc/jupyter
[D 2023-02-07 18:08:23.966 KernelGatewayApp] Looking for jupyter_kernel_gateway_config in /root/.jupyter
[D 2023-02-07 18:08:24.104 KernelGatewayApp] Instantiating kernel 'Bash' with kernel provisioner: local-provisioner
[D 2023-02-07 18:08:24.117 KernelGatewayApp] Starting kernel: ['/usr/local/bin/python3', '-m', 'bash_kernel', '-f', '/root/.local/share/jupyter/runtime/kernel-1650fe8c-551a-44c3-b9cd-2d363e49a039.json']
[D 2023-02-07 18:08:24.118 KernelGatewayApp] Connecting to: tcp://127.0.0.1:39635
[D 2023-02-07 18:08:24.129 KernelGatewayApp] Connecting to: tcp://127.0.0.1:36543
[I 2023-02-07 18:08:24.129 KernelGatewayApp] Kernel started: 1650fe8c-551a-44c3-b9cd-2d363e49a039, name: bash
[D 2023-02-07 18:08:24.129 KernelGatewayApp] Kernel args: {'kernel_name': 'bash'}
[D 2023-02-07 18:08:24.130 KernelGatewayApp] connecting iopub channel to tcp://127.0.0.1:36543
[D 2023-02-07 18:08:24.130 KernelGatewayApp] Connecting to: tcp://127.0.0.1:36543
[D 2023-02-07 18:08:24.130 KernelGatewayApp] connecting shell channel to tcp://127.0.0.1:55331
[D 2023-02-07 18:08:24.130 KernelGatewayApp] Connecting to: tcp://127.0.0.1:55331
[D 2023-02-07 18:08:24.130 KernelGatewayApp] connecting stdin channel to tcp://127.0.0.1:33659
[D 2023-02-07 18:08:24.130 KernelGatewayApp] Connecting to: tcp://127.0.0.1:33659
[D 2023-02-07 18:08:24.130 KernelGatewayApp] connecting heartbeat channel to tcp://127.0.0.1:60573
[D 2023-02-07 18:08:24.131 KernelGatewayApp] connecting control channel to tcp://127.0.0.1:39635
[D 2023-02-07 18:08:24.131 KernelGatewayApp] Connecting to: tcp://127.0.0.1:39635
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[D 2023-02-07 18:08:27.027 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (starting)
[D 2023-02-07 18:08:27.029 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (busy)
[D 2023-02-07 18:08:27.031 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (idle)
[D 2023-02-07 18:08:27.031 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (busy)
[D 2023-02-07 18:08:27.032 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (idle)
[D 2023-02-07 18:08:27.032 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (busy)
[D 2023-02-07 18:08:27.033 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (idle)
[D 2023-02-07 18:08:27.242 KernelGatewayApp] Destroying zmq context for <jupyter_client.blocking.client.BlockingKernelClient object at 0xffffa6ec6fd0>
[D 2023-02-07 18:08:27.249 KernelGatewayApp] Connecting to: tcp://127.0.0.1:36543
[I 2023-02-07 18:08:27.250 KernelGatewayApp] Registering resource: /hello/bash, methods: (['GET'])
[I 2023-02-07 18:08:27.250 KernelGatewayApp] Registering resource: /_api/spec/swagger.json, methods: (GET)
[I 2023-02-07 18:08:27.250 KernelGatewayApp] Jupyter Kernel Gateway 2.5.2 is available at http://0.0.0.0:80
[D 230207 18:08:29 handlers:193] Request code for notebook cell is: REQUEST = "{\"body\": \"\", \"args\": {}, \"path\": {}, \"headers\": {\"Host\": \"127.0.0.1:5006\", \"User-Agent\": \"curl/7.85.0\", \"Accept\": \"*/*\"}}"
[D 2023-02-07 18:08:29.019 KernelGatewayApp] connecting shell channel to tcp://127.0.0.1:55331
[D 2023-02-07 18:08:29.019 KernelGatewayApp] Connecting to: tcp://127.0.0.1:55331
[D 2023-02-07 18:08:29.021 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (busy)
[D 2023-02-07 18:08:29.022 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: execute_input
[D 2023-02-07 18:08:29.074 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: stream
[D 2023-02-07 18:08:29.141 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: error
[D 2023-02-07 18:08:29.144 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (idle)
[E 230207 18:08:29 web:2271] 500 GET /hello/bash (172.17.0.1) 125.12ms
[D 230207 18:08:30 handlers:193] Request code for notebook cell is: REQUEST = "{\"body\": \"\", \"args\": {}, \"path\": {}, \"headers\": {\"Host\": \"127.0.0.1:5006\", \"User-Agent\": \"curl/7.85.0\", \"Accept\": \"*/*\"}}"
[D 2023-02-07 18:08:30.681 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (busy)
[D 2023-02-07 18:08:30.681 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: execute_input
[D 2023-02-07 18:08:30.737 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: stream
[D 2023-02-07 18:08:30.793 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: error
[D 2023-02-07 18:08:30.794 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (idle)
[E 230207 18:08:30 web:2271] 500 GET /hello/bash (172.17.0.1) 116.01ms
[D 230207 18:08:31 handlers:193] Request code for notebook cell is: REQUEST = "{\"body\": \"\", \"args\": {}, \"path\": {}, \"headers\": {\"Host\": \"127.0.0.1:5006\", \"User-Agent\": \"curl/7.85.0\", \"Accept\": \"*/*\"}}"
[D 2023-02-07 18:08:31.881 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (busy)
[D 2023-02-07 18:08:31.882 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: execute_input
[D 2023-02-07 18:08:31.938 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: stream
[D 2023-02-07 18:08:31.993 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: error
[D 2023-02-07 18:08:31.995 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (idle)
[E 230207 18:08:31 web:2271] 500 GET /hello/bash (172.17.0.1) 115.60ms
[D 230207 18:10:13 handlers:193] Request code for notebook cell is: REQUEST = "{\"body\": \"\", \"args\": {}, \"path\": {}, \"headers\": {\"Host\": \"127.0.0.1:5006\", \"User-Agent\": \"curl/7.85.0\", \"Accept\": \"*/*\"}}"
[D 2023-02-07 18:10:13.415 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (busy)
[D 2023-02-07 18:10:13.417 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: execute_input
[D 2023-02-07 18:10:13.469 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: stream
[D 2023-02-07 18:10:13.525 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: error
[D 2023-02-07 18:10:13.529 KernelGatewayApp] activity on 1650fe8c-551a-44c3-b9cd-2d363e49a039: status (idle)
[E 230207 18:10:13 web:2271] 500 GET /hello/bash (172.17.0.1) 120.62ms
requirements.txt:
-i https://pypi.org/simple
anyio==3.6.2 ; python_full_version >= '3.6.2'
appnope==0.1.3 ; platform_system == 'Darwin'
argon2-cffi==21.3.0 ; python_version >= '3.6'
argon2-cffi-bindings==21.2.0 ; python_version >= '3.6'
arrow==1.2.3 ; python_version >= '3.6'
asttokens==2.2.1
attrs==22.2.0 ; python_version >= '3.6'
backcall==0.2.0
bash-kernel==0.9.0
beautifulsoup4==4.11.2 ; python_full_version >= '3.6.0'
bleach==6.0.0 ; python_version >= '3.7'
certifi==2022.12.7 ; python_version >= '3.6'
cffi==1.15.1
charset-normalizer==3.0.1
click==8.1.3 ; python_version >= '3.7'
comm==0.1.2 ; python_version >= '3.6'
debugpy==1.6.6 ; python_version >= '3.7'
decorator==5.1.1 ; python_version >= '3.5'
defusedxml==0.7.1 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
entrypoints==0.4 ; python_version >= '3.6'
executing==1.2.0
fastjsonschema==2.16.2
flask==2.2.2
fqdn==1.5.1
idna==3.4 ; python_version >= '3.5'
ipykernel ; python_version >= '3.8'
ipython==8.9.0 ; python_version >= '3.8'
ipython-genutils==0.2.0
isoduration==20.11.0
itsdangerous==2.1.2 ; python_version >= '3.7'
jedi==0.18.2 ; python_version >= '3.6'
jinja2==3.1.2 ; python_version >= '3.7'
jsonpointer==2.3
jsonschema==4.17.3 ; python_version >= '3.7'
jupyter-client==7.4.4 ; python_version >= '3.7'
jupyter-core==5.2.0 ; python_version >= '3.8'
jupyter-events==0.6.3 ; python_version >= '3.7'
jupyter-kernel-gateway==2.5.2 ; python_version >= '3.8'
jupyter-server==2.2.1 ; python_version >= '3.8'
jupyter-server-terminals==0.4.4 ; python_version >= '3.8'
jupyterlab-pygments==0.2.2 ; python_version >= '3.7'
markupsafe==2.1.2 ; python_version >= '3.7'
matplotlib-inline==0.1.6 ; python_version >= '3.5'
mistune==2.0.5
nbclassic==0.5.1 ; python_version >= '3.7'
nbclient==0.7.2 ; python_full_version >= '3.7.0'
nbconvert==7.2.9 ; python_version >= '3.7'
nbformat==5.7.3 ; python_version >= '3.7'
nest-asyncio==1.5.6 ; python_version >= '3.5'
notebook==6.5.2 ; python_version >= '3.7'
notebook-shim==0.2.2 ; python_version >= '3.7'
packaging==23.0 ; python_version >= '3.7'
pandocfilters==1.5.0 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
parso==0.8.3 ; python_version >= '3.6'
pexpect==4.8.0 ; sys_platform != 'win32'
pickleshare==0.7.5
platformdirs==3.0.0 ; python_version >= '3.7'
prometheus-client==0.16.0 ; python_version >= '3.6'
prompt-toolkit==3.0.36 ; python_full_version >= '3.6.2'
psutil==5.9.4 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
ptyprocess==0.7.0 ; os_name != 'nt'
pure-eval==0.2.2
pycparser==2.21
pygments==2.14.0 ; python_version >= '3.6'
pyrsistent==0.19.3 ; python_version >= '3.7'
python-dateutil==2.8.2 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
python-json-logger==2.0.4 ; python_version >= '3.5'
pyyaml==6.0 ; python_version >= '3.6'
pyzmq==25.0.0 ; python_version >= '3.6'
requests==2.28.2 ; python_version >= '3.7' and python_version < '4'
rfc3339-validator==0.1.4 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
rfc3986-validator==0.1.1 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
send2trash==1.8.0
six==1.16.0 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
sniffio==1.3.0 ; python_version >= '3.7'
soupsieve==2.3.2.post1 ; python_version >= '3.6'
stack-data==0.6.2
terminado==0.17.1 ; python_version >= '3.7'
tinycss2==1.2.1 ; python_version >= '3.7'
tornado==6.2 ; python_version >= '3.7'
traitlets==5.9.0 ; python_version >= '3.7'
uri-template==1.2.0
urllib3==1.26.14 ; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
wcwidth==0.2.6
webcolors==1.12
webencodings==0.5.1
websocket-client==1.5.1 ; python_version >= '3.7'
werkzeug==2.2.2 ; python_version >= '3.7'
Well, this was an interesting exercise...
I just saw that there was an option to force the kernel during startup. e.g. --KernelGatewayApp.force_kernel_name=bash however this does not appear to have any affect on the error.
Yes, the default kernel used is the one pulled from the referenced notebook's metadata, unless force_kernel_name is provided: see here
In looking into this, error 127 relative to bash is "command not found". While debugging, I learned that prior to sending the command referenced in the cell, JKG sends the REQUEST information to allow the code in the cell to gain further context as documented here.
That REQUEST information is actually sent as an execution request and takes the form:
REQUEST = "{\\"body\\": \\"\\", \\"args\\": {}, \\"path\\": {}, \\"headers\\": {\\"Host\\": \\"127.0.0.1:10100\\", \\"User-Agent\\": \\"curl/7.85.0\\", \\"Accept\\": \\"*/*\\"}}"
If I enter that into the bash kernel using jupyter-websocket mode, I will receive:
Unfortunately, we only get back the : 127 (error) portion when handling this particular exception.
Had the "request_code" executed correctly, JKG would then send a request to execute the cell's contents, in this case the echo statement.
The "request_code" is formatted here as follows:
bundle = json.dumps(bundle)
if kernel_language.lower() == 'perl':
statement = "my $REQUEST = {}".format(bundle)
else:
statement = "REQUEST = {}".format(bundle)
return statementand is essentially initializing a variable named REQUEST.
Since bash initialization doesn't allow spaces around the '=', I removed those spaces - since that will also be tolerated in python and, I would imagine, most other languages.
Once that was changed, the bash-kernel appears to work in notebook-http mode as it does in jupyter-websocket mode.
So, it turns out you just picked a different kernel that doesn't allow space-separated initialization.
Do you have a particular need for using the bash-kernel in notebook-http mode? Or are you just experimenting?
Thank you. Yes my use case is based on a playground app for some of out customers. A lot of out tooling is CLI based and the plan is to build some tutorials that include pre-built commands that can be easily edited. Plan is to allow customers to have a containerised instance sort of like a sandbox. We will then build a custom front-end that can be used to demo/play with the commands in the form of a wizard/step by step. So getting this to work is going to be important for this project.
Should I change the code or can we get a permanent fix ?
I've submitted #382. If you could confirm and provide a review, that would be appreciated.
OK worked it out - just manually updated the file for now. Yes it fixed the problem all working now. Thanks again for sorting this. This capability is super cool. Just trying to work out pro's/cons for http-mode vs web socket mode. Think web socket mode is probably best for lots of command output.