bioconda-utils autobump --create-pr crashes, but adding --no-shuffle fixes it
niemasd opened this issue · 0 comments
niemasd commented
I'm trying to autobump my viral_consensus recipe using the "Updating recipes" guide, and when I ran the following:
bioconda-utils autobump recipes/ config.yml --packages viral_consensus --create-prit crashed as follows:
08:14:08 BIOCONDA INFO Hosters loaded: ['FTPHoster', 'GithubRelease', 'GithubTag', 'GithubReleaseAttachment', 'GithubRepoStore', 'Bioconductor', 'CargoPort', 'SourceForge', 'PyPi', 'Bioarchive', 'CPAN', 'CRAN', 'BitBucketTag', 'BitBucketDownload', 'GitlabTag']
08:14:08 BIOCONDA WARNING Selected 1 packages
08:14:08 BIOCONDA INFO Building Recipe DAG
Loading Recipes...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 9289/9289 [00:31<00:00, 298.97it/s]
08:14:39 BIOCONDA INFO Building Recipe DAG: done (8565 nodes, 28466 edges)
08:14:39 BIOCONDA WARNING Graph contains 4 packages (blacklist excluded)
08:14:40 BIOCONDA WARNING Excluding blacklisted recipes
08:14:40 BIOCONDA WARNING Checking out master
08:14:40 BIOCONDA INFO Updating master to latest project master
08:14:40 BIOCONDA INFO Updating and pruning remotes
08:15:19 BIOCONDA INFO Loading package lists for ['conda-forge']
defaults/noarch: 5.03MB [00:00, 31.2MB/s] | 0/9 [00:00<?, ?files/s]
defaults/osx: 44.4MB [00:01, 29.2MB/s] | 1/9 [00:00<00:03, 2.47files/s]
defaults/linux: 48.4MB [00:03, 15.2MB/s]█▎ | 2/9 [00:03<00:13, 1.97s/files]
bioconda/linux: 35.5MB [00:03, 10.7MB/s]█████████████ | 3/9 [00:03<00:07, 1.29s/files]
conda-forge/noarch: 82.8MB [00:04, 17.9MB/s]████████████████████▋ | 4/9 [00:04<00:04, 1.01files/s]
bioconda/osx: 28.4MB [00:02, 12.8MB/s]██████████████████████████████████████▎ | 5/9 [00:06<00:05, 1.48s/files]
bioconda/noarch: 36.4MB [00:00, 64.0MB/s]███████████████████████████████████████████████ | 6/9 [00:07<00:03, 1.16s/files]
conda-forge/osx: 198MB [00:10, 19.8MB/s]███████████████████████████████████████████████████████████▋ | 7/9 [00:09<00:03, 1.62s/files]
conda-forge/linux: 234MB [00:15, 15.5MB/s]█████████████████████████████████████████████████████████████████████▎ | 8/9 [00:16<00:03, 3.14s/files]Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:22<00:00, 2.49s/files]
08:15:43 BIOCONDA INFO Running pipeline with these steps:
08:15:43 BIOCONDA INFO 1. Exclude recipes disabled via config
08:15:43 BIOCONDA INFO 2. Exclude blacklisted recipes
08:15:43 BIOCONDA INFO 3. Exclude sub-recipes
08:15:43 BIOCONDA INFO 4. Exclude recipes depending on packages in need of update
08:15:43 BIOCONDA INFO 5. Load recipe from git (honoring active branches)
08:15:43 BIOCONDA INFO 6. Exclude recipes in conda-forge
08:15:43 BIOCONDA INFO 7. Bump recipes in need of rebuild after pinning changes
08:15:43 BIOCONDA INFO 8. Scan upstream for new releases and update recipe
08:15:43 BIOCONDA INFO 9. Update source checksums
08:15:43 BIOCONDA INFO 10. Write recipe to per-recipe git branch
08:15:43 BIOCONDA INFO 11. Create or Update PR on GitHub
08:15:45 BIOCONDA ERROR Dropping into debugger
Traceback (most recent call last):
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 78, in wrapper
func(*args, **kwargs)
File "<boltons.funcutils.FunctionBuilder-19>", line 2, in autobump
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 97, in wrapper
func(*args, **kwargs)
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 1021, in autobump
scanner.run()
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 204, in run
res = super().run()
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py", line 126, in run
self.loop.run_until_complete(task)
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 225, in _async_run
return await super()._async_run()
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py", line 163, in _async_run
await self.queue_items(source_q, return_q)
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 217, in queue_items
await self.recipe_source.queue_items(send_q, return_q)
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 147, in queue_items
random.shuffle(remaining_recipes)
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/random.py", line 307, in shuffle
x[i], x[j] = x[j], x[i]
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/networkx/classes/reportviews.py", line 194, in __getitem__
return self._nodes[n]
KeyError: 0
0%| | 0/4 [00:00<?, ?it/s]Traceback (most recent call last):
File "/home/niema/miniconda3/envs/bioconda/bin/bioconda-utils", line 10, in <module>
sys.exit(main())
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 1114, in main
argh.dispatch_commands([
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/argh/dispatching.py", line 328, in dispatch_commands
dispatch(parser, *args, **kwargs)
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/argh/dispatching.py", line 174, in dispatch
for line in lines:
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/argh/dispatching.py", line 277, in _execute_command
for line in result:
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/argh/dispatching.py", line 260, in _call
result = function(*positional, **keywords)
File "<boltons.funcutils.FunctionBuilder-22>", line 2, in autobump
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 137, in wrapper
func(*args, **kwargs)
File "<boltons.funcutils.FunctionBuilder-21>", line 2, in autobump
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 66, in wrapper
func(*args, **kwargs)
File "<boltons.funcutils.FunctionBuilder-20>", line 2, in autobump
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 78, in wrapper
func(*args, **kwargs)
File "<boltons.funcutils.FunctionBuilder-19>", line 2, in autobump
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 97, in wrapper
func(*args, **kwargs)
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 1021, in autobump
scanner.run()
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 204, in run
res = super().run()
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py", line 126, in run
self.loop.run_until_complete(task)
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 225, in _async_run
return await super()._async_run()
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py", line 163, in _async_run
await self.queue_items(source_q, return_q)
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 217, in queue_items
await self.recipe_source.queue_items(send_q, return_q)
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 147, in queue_items
random.shuffle(remaining_recipes)
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/random.py", line 307, in shuffle
x[i], x[j] = x[j], x[i]
File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/networkx/classes/reportviews.py", line 194, in __getitem__
return self._nodes[n]
KeyError: 0
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-54' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95eee0>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-53' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e7f0>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-52' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e0a0>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-51' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e9a0>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-50' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e970>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-49' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e7c0>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-48' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95eb80>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-47' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e820>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-46' coro=<AsyncPipeline.show_progress() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:180> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e1f0>()]>>
0%|
Surprisingly, the crash was occurring during random.shuffle (not sure why). I then added the --no-shuffle argument:
bioconda-utils autobump recipes/ config.yml --packages viral_consensus --create-pr --no-shuffleand it didn't crash:
08:23:20 BIOCONDA INFO Hosters loaded: ['FTPHoster', 'GithubRelease', 'GithubTag', 'GithubReleaseAttachment', 'GithubRepoStore', 'Bioconductor', 'CargoPort', 'SourceForge', 'PyPi', 'Bioarchive', 'CPAN', 'CRAN', 'BitBucketTag', 'BitBucketDownload', 'GitlabTag']
08:23:20 BIOCONDA WARNING Selected 1 packages
08:23:20 BIOCONDA INFO Building Recipe DAG
Loading Recipes...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 9289/9289 [00:31<00:00, 290.84it/s]
08:23:52 BIOCONDA INFO Building Recipe DAG: done (8565 nodes, 28466 edges)
08:23:52 BIOCONDA WARNING Graph contains 4 packages (blacklist excluded)
08:23:52 BIOCONDA WARNING Excluding blacklisted recipes
08:23:52 BIOCONDA WARNING Checking out master
08:23:53 BIOCONDA INFO Updating master to latest project master
08:23:53 BIOCONDA INFO Updating and pruning remotes
08:23:55 BIOCONDA INFO Loading package lists for ['conda-forge']
defaults/noarch: 5.03MB [00:00, 17.9MB/s] | 0/9 [00:00<?, ?files/s]
bioconda/linux: 35.5MB [00:01, 28.3MB/s] | 1/9 [00:00<00:06, 1.26files/s]
defaults/linux: 48.4MB [00:02, 25.4MB/s]█▎ | 2/9 [00:02<00:08, 1.27s/files]
defaults/osx: 44.4MB [00:04, 11.5MB/s]███████████████ | 3/9 [00:04<00:09, 1.60s/files]
bioconda/osx: 28.4MB [00:00, 35.5MB/s]██████████████████████████▋ | 4/9 [00:04<00:05, 1.19s/files]
conda-forge/noarch: 82.8MB [00:06, 14.1MB/s]████████████████████████████████▎ | 5/9 [00:05<00:04, 1.02s/files]
bioconda/noarch: 36.4MB [00:04, 9.43MB/s]███████████████████████████████████████████████ | 6/9 [00:09<00:06, 2.06s/files]conda-forge/osx: 198MB [00:10, 19.0MB/s]███████████████████████████████████████████████████████████▋ | 7/9 [00:10<00:03, 1.61s/files]
conda-forge/linux: 234MB [00:16, 14.5MB/s]█████████████████████████████████████████████████████████████████████▎ | 8/9 [00:16<00:03, 3.10s/files]
Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:23<00:00, 2.64s/files]
08:24:20 BIOCONDA INFO Running pipeline with these steps:
08:24:20 BIOCONDA INFO 1. Exclude recipes disabled via config
08:24:20 BIOCONDA INFO 2. Exclude blacklisted recipes
08:24:20 BIOCONDA INFO 3. Exclude sub-recipes
08:24:20 BIOCONDA INFO 4. Exclude recipes depending on packages in need of update
08:24:20 BIOCONDA INFO 5. Load recipe from git (honoring active branches)
08:24:20 BIOCONDA INFO 6. Exclude recipes in conda-forge
08:24:20 BIOCONDA INFO 7. Bump recipes in need of rebuild after pinning changes
08:24:20 BIOCONDA INFO 8. Scan upstream for new releases and update recipe
08:24:20 BIOCONDA INFO 9. Update source checksums
08:24:20 BIOCONDA INFO 10. Write recipe to per-recipe git branch
08:24:20 BIOCONDA INFO 11. Create or Update PR on GitHub
0%| | 0/4 [00:00<?, ?it/s]Updating build index: /home/niema/miniconda3/envs/bioconda/conda-bld
08:24:26 BIOCONDA INFO htslib had no changes
08:24:26 BIOCONDA INFO Recipe viral_consensus: updating from remote bump/viral_consensus
75%|██████████████████████████████████████████████████████████████████████████████████████████▊ | 3/4 [00:04<00:01, 1.32s/it]Updating build index: /home/niema/miniconda3/envs/bioconda/conda-bld
08:24:29 BIOCONDA INFO Updating checksum for viral_consensus 0.0.5
08:24:35 BIOCONDA INFO Checking out branch bump/viral_consensus
08:24:36 BIOCONDA INFO viral_consensus has update in progress
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:14<00:00, 3.51s/it]
08:24:36 BIOCONDA WARNING Finished update
08:24:36 BIOCONDA INFO Unrecognized URL stats:
08:24:36 BIOCONDA INFO
08:24:36 BIOCONDA INFO Recipe status statistics:
08:24:36 BIOCONDA INFO IsSubRecipe: 2
08:24:36 BIOCONDA INFO NoChanges: 1
08:24:36 BIOCONDA INFO UpdateInProgress: 1
08:24:36 BIOCONDA INFO SUM: 4
08:24:36 BIOCONDA WARNING Switching back to master
It still didn't seem to succeed in the autobump, but at least it didn't crash. It's surprising that the default command in the guideline crashed when shuffling, so I thought I should report this issue