In VS Code, KQLMagic is polluting stdout for jupyter notebook cells
JPvRiel opened this issue · 1 comments
JPvRiel commented
Code Sample, a copy-pastable example if possible
E.g. in a given cell
%env KQLMAGIC_CONFIGURATION='show_init_banner=False;show_what_new=False;timeout=900;auto_limit=1000000;display_limit=100;'
%reload_ext Kqlmagic
tenant_id = '<private>'
workspace_id = '<private>'
print(f'https://microsoft.com/devicelogin')
%kql loganalytics://code().tenant(tenant_id).workspace(workspace_id)
Problem description
Before updating KQLMagic, stdout did not have INFO logging pollution. After updating to version 0.1.114.post19, stdout for the cell is "polluted" with kqlmagic logs. E.g.
* Serving Flask app 'kqlmagic_temp_files_server' (lazy loading)
* Environment: development
* Debug mode: off
2022-06-29 20:15:56,625 - INFO - start parent_monitor for parent id: 1672253
2022-06-29 20:15:56,625 - INFO - * parent id: 1672253
2022-06-29 20:15:56,625 - INFO - * Base folder: /home/<private>
2022-06-29 20:15:56,625 - INFO - * Folder list: ['.kqlmagic/temp_files/v2-1403532lpLfNUvoVYHu']
2022-06-29 20:15:56,625 - INFO - *
2022-06-29 20:15:56,625 - INFO - **** Later on, other cells also have logging polluted stdout with messages like:
2022-06-29 20:16:57,685 - INFO - ParentProcessIdValueMonitor for ppid 1672399, parent is running.If query related, does it happen on other platforms (Kusto Web UI, Kusto Explorer, Log Analytics in Azure Monitor)?
N/A
Output of pip freeze and %kql --bugreport
[bugreport must be executed immediatly after bug noticed]
Kqlmagic bug report #4ebb5139-7d46-4d7f-b7af-aa61be804fdd
In summary, the bug report is inconvenient to include when run from within an .ipynb notebook context. It includes a lot of user/env/connection info I don't want to share on github. In vscode notebooks, its output exceeds cell size limit, and the default colored syntax highlighting makes the raw view of output "corrupted" with terminal colour escape codes. So the broken JSON is due to '...' when vscode noteboook display limits cell output length. I attempted a partial extract with the logs cut short as they starting spilling subscription/env info all over the place and I decided to omit all connection specific sections/identifiers. The kqlmagic_default_options section has the ... due to cell length limitations.
{ "Kqlmagic_last_execution": {
"args": {
"cell": null,
"line": "loganalytics://code().tenant(tenant_id).workspace(workspace_id)",
"override_connection": null,
"override_options": null,
"override_query_properties": null,
"override_result_set": null,
"override_vars": null
},
"connection": "<private>@loganalytics",
"log": [
"Kqlmagic_core::execute - input: \n\rline: loganalytics://code().tenant(tenant_id).workspace(workspace_id)\n\rcell:\n\rNone",
"Kqlmagic_core::execute - parsed_queries: [{'connection_string': 'loganalytics://code().tenant(tenant_id).workspace(workspace_id)', 'query': '', 'options': {'auto_dataframe': False, 'short_errors': True, 'feedback': True, 'show_conn_info': 'current', 'columns_to_local_vars': False, 'show_query_time': True, 'show_query': False, 'show_query_link': False, 'query_link_destination': 'Kusto.WebExplorer', 'enable_suppress_result': True, 'plotly_fs_includejs': False, 'popup_window': False, 'auto_limit': 1000000, 'display_limit': 100, 'timeout': 900, 'prettytable_style': 'DEFAULT', 'last_raw_result_var': '_kql_raw_result_', 'table_package': 'prettytable', 'plot_package': 'plotly', 'dsn_filename': 'odbc.ini', 'validate_connection_string': True, 'auto_popup_schema': True, 'json_display': 'formatted', 'schema_json_display': 'auto', 'palette_desaturation': 1.0, 'params_dict': None, 'palette_name': 'tab10', 'cache': None, 'use_cache': None, 'temp_folder_name': 'temp_files', 'cache_folder_name': 'cache_files', 'export_folder_name': 'exported_files', 'add_kql_ref_to_help': True, 'add_schema_to_help': True, 'notebook_app': 'visualstudiocode', 'debug': False, 'check_magic_version': True, 'show_what_new': False, 'show_init_banner': False, 'is_kernel_intializtion': False, 'warn_missing_dependencies': True, 'warn_missing_env_variables': True, 'allow_single_line_cell': True, 'allow_py_comments_before_cell': True, 'kqlmagic_kernel': False, 'extras_require': 'default', 'test_notebook_app': 'none', 'cloud': 'public', 'enable_sso': False, 'sso_db_gc_interval': 168, 'auth_use_http_client': False, 'try_azcli_login': False, 'try_azcli_login_by_profile': False, 'try_vscode_login': False, 'try_azcli_login_subscription': None, 'try_token': None, 'try_msi': None, 'request_id_tag': None, 'request_app_tag': None, 'request_user_tag': None, 'request_user_agent_tag': None, 'request_cache_max_age': 0, 'device_code_login_notification': 'auto', 'device_code_notification_email': '', 'save_as': None, 'save_to': None, 'query_properties': None, 'palette_colors': 10, 'palette_reverse': False, 'popup_schema': False, 'display_id': False, 'display_handlers': {}, 'popup_interaction': 'auto', 'temp_files_server': 'auto', 'temp_files_server_address': None, 'kernel_location': 'local', 'kernel_id': 'v2-1403532lpLfNUvoVYHu', 'notebook_service_address': None, 'dynamic_to_dataframe': 'object', 'temp_folder_location': 'user_dir', 'plotly_layout': None, 'auth_token_warnings': False, 'enable_curly_brackets_params': False, 'nop': False, 'assign_var': None, 'cursor_var': None, 'is_magic': True, 'code_auth_interactive_mode': 'device_code'}, 'command': {}, 'last_query': True}]",
"FilesServerManagement::startServer init: server_py_code_path: /home/<private>/.local/lib/python3.10/site-packages/Kqlmagic/my_files_server.py, server_url: http://127.0.0.1:58007, base_folder: /home/<private>, folders: .kqlmagic/temp_files/v2-1403532lpLfNUvoVYHu, _kernel_id: v2-1403532lpLfNUvoVYHu, liveness_mode: parent_process_id_value",
"FilesServerManagement::startServer",
"FilesServerManagement::startServer with show_window: False",
"FilesServerManagement::startServer start sub process command: /usr/local/bin/python3.10 \"/home/<private>/.local/lib/python3.10/site-packages/Kqlmagic/my_files_server.py\" -protocol=http -host=127.0.0.1 -port=58007 -base_folder=\"/home/<private>\" -folders=\".kqlmagic/temp_files/v2-1403532lpLfNUvoVYHu\" -parent_id=\"1672253\" -parent_kernel_id=\"v2-1403532lpLfNUvoVYHu\" -liveness_mode=\"parent_process_id_value\" -clean=\"folders\"",
"FilesServerManagement::startServer try_register_to_ipython_atexit: _abortServer(http://127.0.0.1:58007, v2-1403532lpLfNUvoVYHu)",
"FilesServerManagement::pingServer: url: http://127.0.0.1:58007/ping?kernelid=v2-1403532lpLfNUvoVYHu, result: False",
"FilesServerManagement::pingServer: url: http://127.0.0.1:58007/ping?kernelid=v2-1403532lpLfNUvoVYHu, result: True",
"kql_engine.py -_parse_common_connection_str - params: conn_str: loganalytics://code().tenant(tenant_id).workspace(workspace_id); current: None, uri_schema_name: loganalytics;mandatory_key: workspace, valid_keys_combinations: [['workspace', 'alias', 'datasourceurl', 'tenant', 'aadurl', 'clientid', 'clientsecret'], ['workspace', 'alias', 'datasourceurl', 'tenant', 'aadurl', 'clientid', 'code'], ['workspace', 'alias', 'datasourceurl', 'tenant', 'aadurl', 'code'], ['workspace', 'alias', 'datasourceurl', 'tenant', 'aadurl', 'clientid', 'username', 'password'], ['workspace', 'alias', 'datasourceurl', 'tenant', 'aadurl', 'username', 'password'], ['workspace', 'alias', 'datasourceurl', 'anonymous'], ['workspace', 'alias', 'datasourceurl', 'appkey']]",
"kql_engine.py - _parse_connection_str - params: conn_str: loganalytics://code().tenant(tenant_id).workspace(workspace_id)"
]
},
"kqlmagic": {
"version": "0.1.114.post19"
},
"kqlmagic_default_env": {
"KQLMAGIC_CONFIGURATION": "'show_init_banner=False;show_what_new=False;timeout=900;auto_limit=1000000;display_limit=100;'"
},
"kqlmagic_default_options": {
"add_kql_ref_to_help": true,
"add_schema_to_help": true,
"allow_py_comments_before_cell": true,
"allow_single_line_cell": true,
"assign_var": null,
"auth_token_warnings": false,
"auth_use_http_client": false,
"auto_dataframe": false,
"auto_limit": 1000000,
"auto_popup_schema": true,
"cache": null,
"cache_folder_name": "cache_files",
"check_magic_version": true,
"cloud": "public",
"code_auth_interactive_mode": "device_code",
"columns_to_local_vars": false,
"cursor_var": null,
"debug": false,
"device_code_login_notification": "auto",
"device_code_notification_email": "",
"display_limit": 100,
"dsn_filename": "odbc.ini",
"dynamic_to_dataframe": "object",
...
"branch": "",
"name": "CPython",
"version": "3.10.4"
}
}[paste the output of pip freeze here below this line]
adal==1.2.7
ansible-core==2.12.7
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asn1crypto==1.5.1
asttokens==2.0.5
attrs==21.4.0
azure-common==1.1.28
azure-core==1.23.0
azure-identity==1.8.0
backcall==0.2.0
beautifulsoup4==4.10.0
bleach==4.1.0
blessings==1.7
boto3==1.21.13
botocore==1.24.13
bpython==0.22.1
certifi==2021.10.8
certvalidator==0.11.1
cffi==1.15.0
charset-normalizer==2.0.12
click==8.0.4
colorful==0.5.4
cryptography==36.0.1
curtsies==0.3.10
cwcwidth==0.1.6
cycler==0.11.0
debugpy==1.5.1
decorator==5.1.1
defusedxml==0.7.1
entrypoints==0.4
executing==0.8.3
Flask==2.0.3
fonttools==4.29.1
future==0.18.2
greenlet==1.1.2
idna==3.3
install==1.3.5
ipykernel==6.9.1
ipython==8.1.1
ipython-genutils==0.2.0
ipywidgets==7.6.5
isodate==0.6.1
itsdangerous==2.1.1
jarowinkler==1.0.2
jedi==0.18.1
Jinja2==3.0.3
jmespath==1.0.1
joblib==1.1.0
jsonschema==4.4.0
jupyter==1.0.0
jupyter-client==7.1.2
jupyter-console==6.4.0
jupyter-core==4.9.2
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.2
kiwisolver==1.3.2
Kqlmagic==0.1.114.post19
lxml==4.8.0
Markdown==3.3.6
MarkupSafe==2.1.0
matplotlib==3.5.1
matplotlib-inline==0.1.3
mistune==0.8.4
msal==1.17.0
msal-extensions==0.3.1
msrest==0.6.21
msrestazure==0.6.4
nbclient==0.5.11
nbconvert==6.4.2
nbformat==5.1.3
nest-asyncio==1.5.4
notebook==6.4.8
numpy==1.22.2
oauthlib==3.2.0
oscrypto==1.3.0
packaging==21.3
pandas==1.4.2
pandocfilters==1.5.0
parso==0.8.3
passlib==1.7.4
password-strength==0.0.3.post2
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.0.1
plotly==5.6.0
polyfuzz==0.3.4
portalocker==2.4.0
prettyprinter==0.18.0
prettytable==3.2.0
prometheus-client==0.13.1
prompt-toolkit==3.0.28
psutil==5.9.0
ptyprocess==0.7.0
pure-eval==0.2.2
pycparser==2.21
Pygments==2.11.2
PyJWT==2.3.0
PyNaCl==1.5.0
pyparsing==3.0.7
pyperclip==1.8.2
pyrsistent==0.18.1
python-dateutil==2.8.2
python-whois==0.7.3
pytz==2021.3
pyxdg==0.27
PyYAML==6.0
pyzmq==22.3.0
qtconsole==5.2.2
QtPy==2.0.1
rapidfuzz==2.0.7
requests==2.27.1
requests-oauthlib==1.3.1
resolvelib==0.5.4
ruamel.yaml==0.17.20
ruamel.yaml.clib==0.2.6
s3transfer==0.5.2
scikit-learn==1.0.2
scipy==1.8.0
seaborn==0.11.2
Send2Trash==1.8.0
six==1.16.0
soupsieve==2.3.1
stack-data==0.2.0
tenacity==8.0.1
terminado==0.13.2
testpath==0.6.0
threadpoolctl==3.1.0
tornado==6.1
tqdm==4.64.0
traitlets==5.1.1
typing_extensions==4.1.1
urllib3==1.26.8
vcert==0.15.0
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==2.0.3
widgetsnbextension==3.5.2
mbnshtck commented
fixed in 0.114.post20