encoutering error using Py 3.10, Ubuntu 22.04
EMCP opened this issue · 7 comments
EMCP commented
via meltano I've installed the target.. but suddenly past few weeks been seeing this
2023-05-29T18:21:35.665396Z [info ] File "/home/ubuntu/meltano_proj_repo/.meltano/loaders/target-parquet/venv/lib/python3.10/site-packages/target_parquet/__init__.py", line 289, in main [15/1872]
elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.665972Z [info ] state = persist_messages( cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.666328Z [info ] File "/home/ubuntu/meltano_proj_repo/.meltano/loaders/target-parquet/venv/lib/python3.10/site-packages/target_parquet/__init__.py", line 240, in persist_messag
es cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.666867Z [info ] t2.start() cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.667082Z [info ] File "/usr/lib/python3.10/multiprocessing/process.py", line 121, in start cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=
target-parquet
2023-05-29T18:21:35.667634Z [info ] self._popen = self._Popen(self) cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.667896Z [info ] File "/usr/lib/python3.10/multiprocessing/context.py", line 288, in _Popen cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id
=target-parquet
2023-05-29T18:21:35.668114Z [info ] return Popen(process_obj) cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.668258Z [info ] File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__ cmd_type=elb consumer=True name=target-parquet producer=False stdio=stder
r string_id=target-parquet
2023-05-29T18:21:35.668379Z [info ] super().__init__(process_obj) cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.668708Z [info ] File "/usr/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__ cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.668898Z [info ] self._launch(process_obj) cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.669244Z [info ] File "/usr/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.669470Z [info ] reduction.dump(process_obj, fp) cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.670124Z [info ] File "/usr/lib/python3.10/multiprocessing/reduction.py", line 60, in dump cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.670298Z [info ] ForkingPickler(file, protocol).dump(obj) cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.670696Z [info ] AttributeError: Can't pickle local object 'persist_messages.<locals>.consumer' cmd_type=elb consumer=True name=target-parquet producer=False stdio=stderr string_id=target-parquet
2023-05-29T18:21:35.707450Z [info ] Warning: In a future version of Poetry, PyPI will be disabled automatically if at least one custom source is configured with another priority than 'explicit'. In order to avoid a breaking change and make your pyproject.toml forward compatible, add PyPI explicitly via 'poetry source add pypi'. By the way, this has the advantage that you can set the priority of PyPI as with any other source. cmd_type=elb consumer=False name=tap-ibkr producer=True stdio=stderr string_id=tap-ibkr
2023-05-29T18:21:35.739704Z [error ] Loader failed
2023-05-29T18:21:35.740143Z [error ] Block run completed. block_type=ExtractLoadBlocks err=RunnerError('Loader failed') exit_codes={<PluginType.LOADERS: 'loaders'>: 1} set_number=0 success=False
auyer commented
Looks like the last PR broke compatibility in Linux. Can you pin to the previous version for now ?
Ref:
#14 (comment)
EMCP commented
ah I see #14 (comment) as well
I can look up and attempt to pin to.. a commit ?
Since my fix was released with the other https://github.com/estrategiahq/target-parquet/releases/tag/v0.2.7, I will need to pin to the commit
EMCP commented
you know.. it might be just as simple as this line area here getting patched
https://github.com/estrategiahq/target-parquet/blob/master/target_parquet/__init__.py#L87
where he says if == darwin .. you need to support linux with the same nomenclature I think... I will try this out and come back
Another idea is what is outlined here https://stackoverflow.com/questions/72584230/get-contextspawn-not-work-in-multiprocessing
EMCP commented
I can confirm that fixes it... writing a PR for linux
auyer commented
I just released your PR. Can you confirm it works for you now ?