'state' KeyError when calling `html_exporter.from_notebook_node`

[hussam-i-am]

Hi, with the most recent release v7.16.2 I noticed that notebooks that do not have the "state" field are causing exceptions to be thrown:

    (html, resources) = html_exporter.from_notebook_node(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/lib/python3.11/site-packages/nbconvert/exporters/html.py", line 268, in from_notebook_node
    html, resources = super().from_notebook_node(nb, resources, **kw)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/lib/python3.11/site-packages/nbconvert/exporters/templateexporter.py", line 424, in from_notebook_node
    output = self.template.render(nb=nb_copy, resources=resources)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/data/notebooks/.venv/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/data/notebooks/app/templates/nbconvert/safe_html/safe_html_index.html.j2", line 1, in top-level template code
    {%- extends 'index.html.j2' -%}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/lab/index.html.j2", line 4, in top-level template code
    {% from 'jupyter_widgets.html.j2' import jupyter_widgets %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/lab/base.html.j2", line 3, in top-level template code
    {% from 'cell_id_anchor.j2' import cell_id_anchor %}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/base/display_priority.j2", line 1, in top-level template code
    {%- extends 'base/null.j2' -%}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/base/null.j2", line 26, in top-level template code
    {%- block body -%}
  File "/data/notebooks/app/templates/nbconvert/safe_html/safe_html_index.html.j2", line 110, in block 'body'
    {% block body_loop %}
  File "/data/notebooks/app/templates/nbconvert/safe_html/safe_html_index.html.j2", line 112, in block 'body_loop'
    window.NOTEBOOK_DATA = {{ super() | process_html }}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/base/null.j2", line 31, in block 'body_loop'
    {%- block any_cell scoped -%}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/base/null.j2", line 34, in block 'any_cell'
    {%- block codecell scoped -%}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/lab/base.html.j2", line 13, in block 'codecell'
    {{ super() }}
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/base/null.j2", line 44, in block 'codecell'
    {%- block output_group -%}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/lab/base.html.j2", line 39, in block 'output_group'
    {{ super() }}
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/base/null.j2", line 48, in block 'output_group'
    {%- block outputs scoped -%}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/lab/base.html.j2", line 45, in block 'outputs'
    {{ super() }}
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/base/null.j2", line 50, in block 'outputs'
    {%- block output scoped -%}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/lab/base.html.j2", line 92, in block 'output'
    {{ super() }}
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/base/null.j2", line 67, in block 'output'
    {%- block display_data scoped -%}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/base/null.j2", line 68, in block 'display_data'
    {%- block data_priority scoped -%}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/lab/base.html.j2", line 131, in block 'data_priority'
    {{ super() }}
  File "/data/notebooks/.venv/share/jupyter/nbconvert/templates/base/display_priority.j2", line 7, in block 'data_priority'
    {%- for type in output.data | filter_data_type -%}
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/notebooks/.venv/lib/python3.11/site-packages/nbconvert/filters/widgetsdatatypefilter.py", line 58, in __call__
    metadata["widgets"][WIDGET_STATE_MIMETYPE]["state"]
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'state'

cc @yuvipanda

[Caceresenzo]

I had the same problem, see #1731 (comment).

In short, just delete notebook.metadata.widgets and it works.

Make sure that values exist before you can delete them, so that you do not get a KeyError depending on the notebook:

if "metadata" in notebook:
    notebook.metadata.pop("widgets", None)

(body, resources) = html_exporter.from_notebook_node(notebook)