fail to import brightway

Question

fail to import brightway

Closed this issue 3 years ago · 30 comments

Hello, I have recently re-installed brightway and I get an error when I trying to import bw2calc (v1.8.0) on a windows machine. It seems to be due to library used by the solver Pardiso (v 0.2.2 installed) ... I am not sure how to address this.

 FileNotFoundError                         Traceback (most recent call last)
<ipython-input-1-a3ddbb8a34e8> in <module>
----> 1 import brightway2 as bw

~\Anaconda3\envs\bw\lib\site-packages\brightway2\__init__.py in <module>
      1 # -*- coding: utf-8 -*
      2 from bw2data import *
----> 3 from bw2calc import *
      4 from bw2io import *
      5

~\Anaconda3\envs\bw\lib\site-packages\bw2calc\__init__.py in <module>
     24 __version__ = (1, 8, 0)
     25
---> 26 from .lca import LCA
     27 from .dense_lca import DenseLCA
     28 from .independent_lca import IndependentLCAMixin

~\Anaconda3\envs\bw\lib\site-packages\bw2calc\lca.py in <module>
     27
     28 try:
---> 29     from pypardiso import factorized, spsolve
     30 except ImportError:
     31     from scipy.sparse.linalg import factorized, spsolve

~\Anaconda3\envs\bw\lib\importlib\_bootstrap.py in _find_and_load(name, import_)

~\Anaconda3\envs\bw\lib\importlib\_bootstrap.py in _find_and_load_unlocked(name, import_)

~\Anaconda3\envs\bw\lib\importlib\_bootstrap.py in _load_unlocked(spec)

~\Anaconda3\envs\bw\lib\importlib\_bootstrap.py in _load_backward_compatible(spec)

<frozen zipimport> in load_module(self, fullname)

~\Anaconda3\envs\bw\lib\site-packages\pypardiso-0.2.2-py3.6.egg\pypardiso\__init__.py in <module>
      2
      3 from .pardiso_wrapper import PyPardisoSolver
----> 4 from .scipy_aliases import spsolve, factorized
      5 from .scipy_aliases import pypardiso_solver as ps
      6

~\Anaconda3\envs\bw\lib\importlib\_bootstrap.py in _find_and_load(name, import_)

~\Anaconda3\envs\bw\lib\importlib\_bootstrap.py in _find_and_load_unlocked(name, import_)

~\Anaconda3\envs\bw\lib\importlib\_bootstrap.py in _load_unlocked(spec)

~\Anaconda3\envs\bw\lib\importlib\_bootstrap.py in _load_backward_compatible(spec)

<frozen zipimport> in load_module(self, fullname)

~\Anaconda3\envs\bw\lib\site-packages\pypardiso-0.2.2-py3.6.egg\pypardiso\scipy_aliases.py in <module>
      7 # pypardsio_solver is used for the 'spsolve' and 'factorized' functions. Python crashes on windows if multiple
      8 # instances of PyPardisoSolver make calls to the Pardiso library
----> 9 pypardiso_solver = PyPardisoSolver()
     10
     11

~\Anaconda3\envs\bw\lib\site-packages\pypardiso-0.2.2-py3.6.egg\pypardiso\pardiso_wrapper.py in __init__(self, mtype, phase, size_limit_storage)
     64             self.libmkl = ctypes.CDLL('libmkl_rt.dylib')
     65         elif sys.platform == 'win32':
---> 66             self.libmkl = ctypes.CDLL('mkl_rt.dll')
     67         else:
     68             self.libmkl = ctypes.CDLL('libmkl_rt.so')

~\Anaconda3\envs\bw\lib\ctypes\__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error, winmode)
    379
    380         if handle is None:
--> 381             self._handle = _dlopen(self._name, mode)
    382         else:
    383             self._handle = handle

FileNotFoundError: Could not find module 'mkl_rt.dll' (or one of its dependencies). Try using the full path with constructor syntax.

Answer 1 · 2021-06-01T11:03:34.000Z

Tried installing brightway into a new environment today and getting the same error. This is the command I used:
conda create -n BW2 -y -c conda-forge -c cmutel -c haasad brightway2 jupyter

My machine runs Microsoft Windows [Version 10.0.19043.985].

Answer 2 · 2021-06-01T11:50:10.000Z

I had the same/similar issue after the last update.
I first tried to reinstall pyparadisio because it flagged an error there in my case but this did not resolve it.
What die resolve it was to simply remove and reinstall brightway (in the same environment, but setting up a new one should probably work equally fine).

Answer 3 · 2021-06-01T11:52:01.000Z

@m-y-reich What are the commands you used for deinstalling and reinstalling brightway?

Answer 4 · 2021-06-01T12:03:51.000Z

@m-y-reich What are the commands you used for deinstalling and reinstalling brightway?

in the respective environment I simply entered
conda remove brightway2
(see https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html)

and then I reinstalled it with the installation-command from the the brightway installation guide
conda install -y -q -c conda-forge -c cmutel -c haasad brightway2 jupyter

Answer 5 · 2021-06-01T13:08:44.000Z

@cmutel, please let me know if you need any input/changes from my side. I know I haven't contributed to brightway in quite some time, but I feel responsible for pypardiso

Answer 6 · 2021-06-01T13:20:40.000Z

@m-y-reich the problem persists. Can you redo the procedure on your computer in a new environment to see if it still works?

Answer 7 · 2021-06-01T13:42:45.000Z

@BenPortner @haasad I am trying to rebuild this using the conda-forge, which will produce specific versions for each OS and Py version (https://github.com/cmutel/PyPardisoProject). At some point I would also like to extend PyPardiso to support the newer versions which are supposed to be much faster.

@haasad I would love to chat, please get in touch :)

Answer 8 · 2021-06-01T16:32:57.000Z

I think this might help. Will take a day or two to get integrated into the conda-forge pipeline.

Answer 9 · 2021-06-01T19:28:32.000Z

FYI: the complex matrices weren't merged to pypardiso since the author of the PR found errors, see haasad/PyPardiso#9. I'd be careful to put this on conda-forge without further testing

Answer 10 · 2021-06-01T21:10:38.000Z

@haasad @BenPortner Using conda-forge packaging didn't help - same error.

The line loading the problematic linked library is innocuous:

self.libmkl = ctypes.CDLL('mkl_rt.dll')

This worked just fine for years, until it didn't. Something changed in the way MKL was packaged, and this isn't something that we have a lot of control over.

Here are some possible answers:

Install an older version of MKL.
Set an environment variable to change how Conda searches for libraries.
Installing Cython?

In the longer term, someone with a Win10 computer needs to install MKL in a clean environment, and find out why it isn't on the path. If it helps, we aren't the only ones struggling with this...

Answer 11 · 2021-06-02T12:44:36.000Z

I opened haasad/PyPardiso#12

I did some tests on windows, it seems the new mkl 2021.2.0 from conda-forge breaks something with the path discovery on windows. mkl 2020.2 from the defaults channel still worked in my tests.

As a quick fix you can create a new environment like this:

conda create -n bw38 -c conda-forge -c cmutel -c haasad brightway2 mkl=2020.2 python=3.8

I would be grateful if you could test this @BenPortner, @m-y-reich.

I'll look for a permanent fix asap.

Answer 12 · 2021-06-02T14:04:25.000Z

As a quick fix you can create a new environment like this:
conda create -n bw38 -c conda-forge -c cmutel -c haasad brightway2 mkl=2020.2 python=3.8

Work's like a charm.

Answer 13 · 2021-06-02T15:49:00.000Z

I was finally able to get a working recipe for conda-forge, so this problem should go away soon!

Answer 14 · 2021-06-03T09:40:57.000Z

It is looking for "mkl_rt.dll" while the DLL file is now named "mkl_rt.1.dll" for some reason unknown to me.

Answer 15 · 2021-06-03T11:46:30.000Z

After fiddling with the anaconda client for way too long (it's broken on 3.9 and fails silently) I released v0.3.0 of pypardiso which contains a fix for this issue.

Updating pypardiso will resolve the problems on windows and new environments should work again:

conda update -c haasad pypardiso

@cmutel, I'll prepare a PR for the conda-forge recipe

Answer 16 · 2021-06-03T12:03:29.000Z

@haasad Already done. Please let me know if you want to take over or be listed as a maintainer.

Answer 17 · 2021-06-03T12:06:58.000Z

Great! I'd appreciate if you add me as a maintainer 👍

Answer 18 · 2021-06-03T12:11:30.000Z

@haasad OK, you need to comment in the linked PR to show you are OK being listed as such.

Answer 19 · 2022-01-18T15:33:03.000Z

@m-y-reich What are the commands you used for deinstalling and reinstalling brightway?

in the respective environment I simply entered conda remove brightway2 (see https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html)

and then I reinstalled it with the installation-command from the the brightway installation guide conda install -y -q -c conda-forge -c cmutel -c haasad brightway2 jupyter

@haasad Hi! I reinstall and install it again but unfortunately, I still have this error:
I really appreciate your help

Traceback (most recent call last):
File "C:/Users/u0135744a/PycharmProjects/10.01.2022/import ecoinvent.py", line 1, in
import brightway2
File "C:\ProgramData\Anaconda3\envs\brightway2\lib\site-packages\brightway2_init_.py", line 2, in
from bw2data import *
File "C:\ProgramData\Anaconda3\envs\brightway2\lib\site-packages\bw2data_init_.py", line 35, in
from .meta import (
File "C:\ProgramData\Anaconda3\envs\brightway2\lib\site-packages\bw2data\meta.py", line 186, in
geomapping = GeoMapping()
File "C:\ProgramData\Anaconda3\envs\brightway2\lib\site-packages\bw2data\meta.py", line 66, in init
super(GeoMapping, self).init(*args, **kwargs)
File "C:\ProgramData\Anaconda3\envs\brightway2\lib\site-packages\bw2data\serialization.py", line 122, in init
self.load()
File "C:\ProgramData\Anaconda3\envs\brightway2\lib\site-packages\bw2data\serialization.py", line 127, in load
self.data = self.deserialize()
File "C:\ProgramData\Anaconda3\envs\brightway2\lib\site-packages\bw2data\serialization.py", line 240, in deserialize
return self.unpack(pickle.load(open(self.filepath, "rb")))
ValueError: unsupported pickle protocol: 5

Answer 20 · 2022-01-18T16:48:22.000Z

@Haniezarafshani As far as I know this is an error that shouldn't happen unless you use an older python version (below 3.8). You can check which version you're using by running:
python --version or conda list

I suggest to use the same environment that you created earlier for the activity-browser to run your brightway code. Otherwise you can also create a new separate environment for brightway. Something like this:

conda create -n bw_new -c conda-forge -c cmutel brightway2 intel-openmp=2021.4 python=3.9

I assume you then also have to select the environment in Pycharm.

See also: brightway-lca/brightway2-data#83

Answer 21 · 2022-01-19T07:34:57.000Z

@Haniezarafshani As far as I know this is an error that shouldn't happen unless you use an older python version (below 3.8). You can check which version you're using by running: python --version or conda list

I suggest to use the same environment that you created earlier for the activity-browser to run your brightway code. Otherwise you can also create a new separate environment for brightway. Something like this:
conda create -n bw_new -c conda-forge -c cmutel brightway2 intel-openmp=2021.4 python=3.9
I assume you then also have to select the environment in Pycharm.

See also: brightway-lca/brightway2-data#83

Thank you very much for your quick reply. it was so useful!
the problem was solved by [ValueError: unsupported pickle protocol: 5 - Numpy 1.13.3 or above is required for this version of scipy (detected version 1.11.3) #73]

Answer 22 · 2022-05-14T13:13:13.000Z

Hello everyone,

Recently I am struggling with a problem of brightway import. I am performing an LCA project for my master thesis and it is stuck. Since yesterday, python/jupyternotebook is giving the following problem. I need to find a solution asap. Is there anyone who can help me? Any advice or answer would be awesomely helpful.

FYI: I deleted and re-created brightway2 for my env. Also, I deleted all software and re-installed it as it is recommended previously. Unfortunately, it didn`t work.

INPUT

import brightway2 as bw
import os
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt

OUTPUT

ImportError Traceback (most recent call last)
File ~\anaconda3\envs\MG\lib\site-packages\bw2data\serialization.py:242, in PickledDict.deserialize(self)
241 try:
--> 242 return self.unpack(pickle.load(open(self.filepath, "rb")))
243 except ImportError:

File ~\anaconda3\envs\MG\lib\site-packages\bw2data\backends_init_.py:5, in
3 from eight import *
----> 5 from .base import LCIBackend
6 from .peewee import SQLiteBackend

File ~\anaconda3\envs\MG\lib\site-packages\bw2data\backends\base.py:5, in
3 from eight import *
----> 5 from .. import (
6 config,
7 databases,
8 geomapping,
9 mapping,
10 projects,
11 )
12 from ..data_store import ProcessedDataStore

ImportError: cannot import name 'databases' from partially initialized module 'bw2data' (most likely due to a circular import) (C:\Users\cansu\anaconda3\envs\MG\lib\site-packages\bw2data_init_.py)

During handling of the above exception, another exception occurred:

PickleError Traceback (most recent call last)
Input In [1], in <cell line: 1>()
----> 1 import brightway2 as bw
2 import os # to use "operating system dependent functionality"
3 import numpy as np # "the fundamental package for scientific computing with Python"

File ~\anaconda3\envs\MG\lib\site-packages\brightway2_init_.py:2, in
1 # -- coding: utf-8 -
----> 2 from bw2data import *
3 from bw2calc import *
4 from bw2io import *

File ~\anaconda3\envs\MG\lib\site-packages\bw2data_init_.py:35, in
33 from .project import projects
34 from .utils import set_data_dir
---> 35 from .meta import (
36 dynamic_calculation_setups,
37 calculation_setups,
38 databases,
39 geomapping,
40 mapping,
41 methods,
42 normalizations,
43 preferences,
44 weightings,
45 )
47 # Add metadata class instances to global list of serialized metadata
48 config.metadata.extend([
49 dynamic_calculation_setups,
50 calculation_setups,
(...)
57 weightings,
58 ])

File ~\anaconda3\envs\MG\lib\site-packages\bw2data\meta.py:192, in
190 preferences = Preferences()
191 weightings = WeightingMeta()
--> 192 calculation_setups = CalculationSetups()
193 dynamic_calculation_setups = DynamicCalculationSetups()

File ~\anaconda3\envs\MG\lib\site-packages\bw2data\serialization.py:123, in SerializedDict.init(self, dirpath)
118 raise NotImplemented("SerializedDict must be subclassed, and the filename must be set.")
119 self.filepath = os.path.join(
120 dirpath or projects.dir,
121 self.filename
122 )
--> 123 self.load()

File ~\anaconda3\envs\MG\lib\site-packages\bw2data\serialization.py:128, in SerializedDict.load(self)
126 """Load the serialized data. Creates the file if not yet present."""
127 try:
--> 128 self.data = self.deserialize()
129 except IOError:
130 # Create if not present
131 self.data = {}

File ~\anaconda3\envs\MG\lib\site-packages\bw2data\serialization.py:245, in PickledDict.deserialize(self)
243 except ImportError:
244 TEXT = "Pickle deserialization error in file '%s'" % self.filepath
--> 245 raise PickleError(TEXT)

PickleError: Pickle deserialization error in file 'C:\Users\cansu\AppData\Local\pylca\Brightway3\default.c21f969b5f03d33d43e04f8f136e7682\setups.pickle'

Answer 23 · 2022-09-05T10:15:57.000Z

Hi, this morning out of the blue brightway started giving me the same error as @bthnkrts . Did anyone find a solution to this? Thanks!

Answer 24 · 2022-09-05T10:19:33.000Z

Hi,
go in your folder C:\Users\cansu\AppData\Local\pylca\Brightway3\default.c21f969b5f03d33d43e04f8f136e7682
and

make a copy of setups.pickle (just in case)
delete setups.pickle file (don't worry, it'll be regenerated),
and restart your notebook.

Answer 25 · 2022-09-09T09:58:04.000Z

@morphiumdeus Does a thumbs up mean that your problem is fixed?

In general, this is a stupid error caused by a mistake on my fault. This is because we saved used pickle.HIGHEST_PROTOCOL, and a new, backwards-incompatible, protocol was introduced in 3.8. The problem comes when you have two environments on your computer - the one with Python >= 3.8 will save in the latest protocol, and then the <3.8 environment can't read it and raises an error.

The later releases of Brightway avoid this by forcing protocol 4 (<3.8 compatible) in the database and in separate pickle files.

Answer 26 · 2022-09-09T10:02:16.000Z

Sorry, @cmutel, yes this solves the problem. Thanks for the quick help!

Answer 27 · 2023-01-11T14:16:58.000Z

Hey there!

I'm a newbie to brightway2, so please forgive any "dumb" questions.

I've been trying to install brightway2 following the Instructions step-by-step, however I've been getting the following error:

Could someone help me solve this, please? I've been running in circles here :(

thanks!

Answer 28 · 2023-01-11T15:05:40.000Z

Welcome to the community!

This is not your fault, something needs to be updated in Brightway. Something changed in the library numpy. You can fix this on your side for now by doing the following in conda:

conda activate bw2 (if you hadn't already activated the environment)
conda install numpy=1.23.5

This will install the right version of numpy, which will solve your problem until the problem is properly solved in Brightway.

Good luck!

Answer 29 · 2023-01-11T15:06:15.000Z

Hi @joanaalbano,

Thanks for using brightway! Your error and the solution have been described here: brightway-lca/brightway2-data#117. It can be fixed by downgrading numpy to version 1.23.5: conda install numpy=1.23.5.

Ben

Answer 30 · 2023-01-11T21:07:49.000Z

Hello @marc-vdm and @BenPortner! For some reason, I was getting massive errors when using conda install numpy=1.23.5, but it worked when using pip install --upgrade numpy==1.23.5.

Thanks a lot for your help!!