kyungeonchoi/ServiceXDataBinder

ServiceX error (I/O operation on closed file) from Uproot transformer

Closed this issue · 1 comments

kyungeonchoi commented
The following error occurs for some DIDs 
WARNING - Transform 3ff9ed3c-36fc-4e72-b72c-4a55365b60e5 had 3 errors:
WARNING -   Error transforming file: root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlaslocalgroupdisk/rucio/user/mgeyik/20/52/user.mgeyik.26575883._000002.out.root
WARNING -   -> error: Failed to transform input file root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlaslocalgroupdisk/rucio/user/mgeyik/20/52/user.mgeyik.26575883._000002.out.root: I/O operation on closed file
WARNING -   Error transforming file: root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlaslocalgroupdisk/rucio/user/mgeyik/e4/99/user.mgeyik.26575883._000001.out.root
WARNING -   -> error: Failed to transform input file root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlaslocalgroupdisk/rucio/user/mgeyik/20/52/user.mgeyik.26575883._000002.out.root: I/O operation on closed file
WARNING -   Error transforming file: root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlaslocalgroupdisk/rucio/user/mgeyik/6e/9c/user.mgeyik.26575883._000003.out.root
WARNING -   -> error: Failed to transform input file root://fax.mwt2.org:1094//pnfs/uchicago.edu/atlaslocalgroupdisk/rucio/user/mgeyik/20/52/user.mgeyik.26575883._000002.out.root: I/O operation on closed file
ERROR - Exception occured while gettting data via ServiceX
Traceback (most recent call last):
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex.py", line 654, in _get_files
    async for r in stream_local_files:
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex.py", line 767, in _get_files_from_servicex
    async for info in stream_downloaded:
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex.py", line 736, in _download_a_file
    async for f in stream:
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex.py", line 814, in _get_minio_bucket_files_from_servicex
    async for info in stream_new_object:
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/minio_adaptor.py", line 181, in find_new_bucket_files
    async for _ in update:
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/utils.py", line 209, in stream_unique_updates_only
    async for p in stream:
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex_adaptor.py", line 239, in trap_servicex_failures
    raise ServiceXFailedFileTransform(f'ServiceX failed to transform {did_fail} '
servicex.utils.ServiceXFailedFileTransform: (ServiceXFailedFileTransform(...), 'ServiceX failed to transform 1 files - data incomplete (remaining: 0, processed: 2).')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex_databinder/servicex_databinder.py", line 26, in deliver
    output_parquet_list = self._sx.get_servicex_data()
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex_databinder/frontend.py", line 67, in get_servicex_data
    data = newloop.run_until_complete(_get_my_data())
  File "/home/kyungeon/.local/lib/python3.6/site-packages/nest_asyncio.py", line 70, in run_until_complete
    return f.result()
  File "/usr/lib64/python3.6/asyncio/tasks.py", line 180, in _step
    result = coro.send(None)
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex_databinder/frontend.py", line 64, in _get_my_data
    return await asyncio.gather(*tasks)
  File "/usr/lib64/python3.6/asyncio/tasks.py", line 250, in _wakeup
    future.result()
  File "/usr/lib64/python3.6/asyncio/tasks.py", line 180, in _step
    result = coro.send(None)
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex_databinder/frontend.py", line 47, in bound_get_data
    return await sx_ds.get_data_parquet_async(query)
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex_utils.py", line 49, in cached_version_of_fn
    result = await fn(*args, **kwargs)
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex.py", line 302, in get_data_parquet_async
    return await self._file_return(selection_query, 'parquet', title)
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex.py", line 428, in _file_return
    return await self._data_return(selection_query, convert_to_file, title, data_format)
  File "/home/kyungeon/.local/lib/python3.6/site-packages/backoff/_async.py", line 133, in retry
    ret = await target(*args, **kwargs)
  File "/home/kyungeon/.local/lib/python3.6/site-packages/backoff/_async.py", line 133, in retry
    ret = await target(*args, **kwargs)
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex.py", line 528, in _data_return
    async for f in self._stream_return(selection_query, title, converter, data_format)
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex.py", line 526, in <dictcomp>
    all_data = {
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex.py", line 565, in _stream_return
    async for r in as_data:
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex.py", line 561, in <genexpr>
    as_data = (StreamInfoData(f.file, await asyncio.ensure_future(converter(f.path)))
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex.py", line 598, in _stream_local_files
    async for name, a_path in as_files:
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex.py", line 595, in <genexpr>
    (f async for f in
  File "/home/kyungeon/.local/lib/python3.6/site-packages/servicex/servicex.py", line 678, in _get_files
    raise ServiceXException(f'Failed to transform all files in {request_id}') from e
servicex.utils.ServiceXException: (ServiceXException(...), 'Failed to transform all files in 3ff9ed3c-36fc-4e72-b72c-4a55365b60e5')
  • Investigated messages in Kibana and localized the error
  • New Uproot transformer (PR) which includes the newer version of uproot crashes.
  • Tested with the older version of Uproot transformer image and just works fine.
kyungeonchoi commented

Issue is followed by ServiceX developers (ssl-hep/ServiceX_Uproot_Transformer#22) and fix is published to Docker hub on Oct 27.
ServiceX Binder 0.1.8 includes the fixed uproot transformer image.