nlpub/pymystem3

Hangs in Google Colab

ukoloff opened this issue · 15 comments

ukoloff commented

In Google Colab pymystem3 succesfuly installed, imported, but lemmatization itself hangs and never stops

See example.

alexanderpanchenko commented

Thanks. I never used Colab before and do not know specificities of this environment.

I've just tried on Ubuntu 16.04 server and it works (any suggestion and patch to support Colab is welcome!):

(root) panchenko@ltnode1a:~/sensegram/model$ python
Python 3.6.3 |Anaconda, Inc.| (default, Oct 13 2017, 12:02:49)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
(root) panchenko@ltnode1a:~/sensegram/model$ pip install pymystem3
Collecting pymystem3
  Downloading https://files.pythonhosted.org/packages/00/8c/98b43c5822620458704e187a1666616c1e21a846ede8ffda493aabe11207/pymystem3-0.2.0-py3-none-any.whl
Requirement already satisfied: requests in /home/panchenko/anaconda/lib/python3.6/site-packages (from pymystem3) (2.18.4)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/panchenko/anaconda/lib/python3.6/site-packages (from requests->pymystem3) (3.0.4)
Requirement already satisfied: idna<2.7,>=2.5 in /home/panchenko/anaconda/lib/python3.6/site-packages (from requests->pymystem3) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /home/panchenko/anaconda/lib/python3.6/site-packages (from requests->pymystem3) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in /home/panchenko/anaconda/lib/python3.6/site-packages (from requests->pymystem3) (2018.1.18)
Installing collected packages: pymystem3
Successfully installed pymystem3-0.2.0
You are using pip version 18.0, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(root) panchenko@ltnode1a:~/sensegram/model$ pip install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 1.1MB/s
Installing collected packages: pip
  Found existing installation: pip 18.0
    Uninstalling pip-18.0:
      Successfully uninstalled pip-18.0
Successfully installed pip-18.1
(root) panchenko@ltnode1a:~/sensegram/model$ ipython
Python 3.6.3 |Anaconda, Inc.| (default, Oct 13 2017, 12:02:49)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from pymystem3 import Mystem

In [2]: m = Mystem()
Installing mystem to /home/panchenko/.local/bin/mystem from http://download.cdn.yandex.net/mystem/mystem-3.1-linux-64bit.tar.gz

In [3]: m.lemmatize("Красивая мама красиво мыла раму")
Out[3]: ['красивый', ' ', 'мама', ' ', 'красиво', ' ', 'мыть', ' ', 'рама', '\n']
alexanderpanchenko commented

seems that colab uses some version of ubuntu 18.04:

image

on my ubuntu 18.04 box it works (anaconda 3):

image

alexanderpanchenko commented

i guess the problem is with some hardware/resource limits of the free colab accounts, not sure which ones however... (i managed to reproduce the error https://colab.research.google.com/drive/10LW6u0l_T63lUy2mV2mdApunIWb3GOYx)

ukoloff commented

Well, at my Ubuntu box pymystem3 works either.

Something happens inside Colab, but I have no idea. May be some network access is silently filtered?

In my example it hangs inside select.select([self._procout_no], [], [], 30) call. Not sure does this mean anything?

Kunya commented

Hi! Any luck so far? Run into the same issue)
My best guess it's related to how python wrapper exchange data with mystem binary. Colab seems to block this so output from binary never reach wrapper back. I might be totally wrong though

ukoloff commented

Hi! Any luck so far?

I'm afraid no progress yet.

alexanderpanchenko commented
nokados commented

It works with pymystem==0.1.10
(mystem from http://download.cdn.yandex.net/mystem/mystem-3.0-linux3.1-64bit.tar.gz)
If you have already installed pymystem==0.2.0, then it probably installed mystem 3.1. In this case, you need to delete mystem or reset your runtime.

  • 👍4
alexanderpanchenko commented

once the issue was with the versions i will close this issue

brs1977 commented

copy mystem3 to /bin

!wget http://download.cdn.yandex.net/mystem/mystem-3.0-linux3.1-64bit.tar.gz
!tar -xvf mystem-3.0-linux3.1-64bit.tar.gz
!cp mystem /bin

  • 👍3
gushart commented

14.08 edit copy:

!wget http://download.cdn.yandex.net/mystem/mystem-3.0-linux3.1-64bit.tar.gz
!tar -xvf mystem-3.0-linux3.1-64bit.tar.gz
!cp mystem /root/.local/bin/mystem

  • 👍10
vadimvik commented

14.08 edit copy:

!wget http://download.cdn.yandex.net/mystem/mystem-3.0-linux3.1-64bit.tar.gz
!tar -xvf mystem-3.0-linux3.1-64bit.tar.gz
!cp mystem /root/.local/bin/mystem

And thanks to "gushart", it works.

EvgenyNovashov commented

I tryed use advice gushart, but Colab gave error cp: failed to access '/root/.local/bin/mystem': Not a directory
I resolved problem: !pip install pymystem3==0.1.10

  • 👍3
NikitaTsekh commented

Same problem with Google Collab, and home version of Jupyter Notebook. While on the inner server of Yandex (I am a student and was given acess to inner Yandex Jupyter Notebook) lemmatization is done in seconds!

Raygeku commented

14.08 edit copy:

!wget http://download.cdn.yandex.net/mystem/mystem-3.0-linux3.1-64bit.tar.gz !tar -xvf mystem-3.0-linux3.1-64bit.tar.gz !cp mystem /root/.local/bin/mystem

I had same problem but with Py app in docker container and copying mystem unpacked file to /bin directory works for me.