Add python 3.X compatibility

Question

Add python 3.X compatibility

brenowca opened this issue 5 years ago · 9 comments

Code compatibility with python 3 would be a nice feature since python 2.7 is no longer continued.

Answer 1 · 2019-10-07T20:05:18.000Z

I made a pull request, #33, and I believe that it would close this issue and provide python 2.7 and 3.X compatibility to this repository.

Answer 2 · 2019-11-01T06:36:38.000Z

Reading model from logs/fn1.7-pretrained-targetid/best-targetid-1.7-model ...
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/wanicca/code/framebase/open-sesame/sesame/targetid.py", line 430, in <module>
    model.populate(model_file_name)
  File "_dynet.pyx", line 1461, in _dynet.ParameterCollection.populate
  File "_dynet.pyx", line 1516, in _dynet.ParameterCollection.populate_from_textfile
RuntimeError: Dimensions of lookup parameter /_0 lookup up from file ({100,400574}) do not match parameters to be populated ({100,400000})

I tried the pretrained model with your code, but there is something buggy...

Answer 3 · 2019-12-02T11:11:47.000Z

I have the same problem...

Answer 4 · 2019-12-09T13:17:34.000Z

Hi Wanicca, Sorry for the delay in the reply. I had a very busy November. Basically what I found is that there was some inconsistency in the original version of the code for dealing with 'utf-8' strings and binary strings. This is a known difficulty with Python 2.7, that Python 3.7 overcome. As far as I could understand, this situation leads to a shifty bug that made a sub-set of the strings in the vocabulary to be counted twice. That bug led to a vocabulary slightly bigger than the expected one (with two repeated words represented in binary and unicode). Anyways, I have trained the models again. I have made some more changes (on the encoding /decoding stuff) and will make a commit soon. After I make this new commit, I ask if you and *@swabhs *could kindly take some time to scrutinize the code and make sure that everything is making sense. After that, we can all validate if the vocabulary encoding issue is correct and the results are what you would expect. Thank you for your time and for spotting the problem. Breno

…

On Fri, Nov 1, 2019 at 3:36 AM wanicca ***@***.***> wrote: Reading model from logs/fn1.7-pretrained-targetid/best-targetid-1.7-model ... Traceback (most recent call last): File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/wanicca/code/framebase/open-sesame/sesame/targetid.py", line 430, in <module> model.populate(model_file_name) File "_dynet.pyx", line 1461, in _dynet.ParameterCollection.populate File "_dynet.pyx", line 1516, in _dynet.ParameterCollection.populate_from_textfile RuntimeError: Dimensions of lookup parameter /_0 lookup up from file ({100,400574}) do not match parameters to be populated ({100,400000}) I tried the pretrained model with your code, but there is something buggy... — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#32?email_source=notifications&email_token=AA3A5APEKS54CV5ETKARNY3QRPE7PA5CNFSM4I6HUBCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC2D5SA#issuecomment-548683464>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA3A5ANDOM7YAZ7IAI7CG4TQRPE7PANCNFSM4I6HUBCA> .

Answer 5 · 2019-12-19T20:51:45.000Z

This would seem like a very important thing to fix because, as far as I can tell, NLTK relies on Python 3 and Open-Sesame relies on NLTK. Until this is resolved, is there any way to get around this problem?

To be clear, I can easily install NLTK for python 3 but not python 2. When I try running the sesame.preprocessor script with python 3, however, there is a cascade of bugs. When I fix one, another bug pops up.

Is there an interim fix for this? Thank you!

Answer 6 · 2020-01-28T17:38:41.000Z

I have a branch running to resolve these issues. I have made it through training for the first 2 networks, and getting the 3rd now. On Python 3.6.X. Next, I will put it in a docker container.

dev branch with the latest: https://github.com/free-soellingeraj/open-sesame/tree/soellingeraj/python36-updates

I will push to master once I have gone through each model and start a new branch for the dockerfile.

Answer 7 · 2020-03-14T10:45:07.000Z

Here's a working version using Python3.6 and the option for GPU compute.
https://github.com/free-soellingeraj/open-sesame

Answer 8 · 2020-05-08T05:23:18.000Z

I struggled with the exact versions of libraries needed for running this 2.7 original version of open-sesame, so I'm sharing here what worked for me:
pip2 install numpy==1.15.0 dynet nltk==3.4
The remaining instructions, eg. downloading nltk modules, can run as usual.

Answer 9 · 2020-09-14T00:40:59.000Z

Thanks everyone for the discussion above. I've now switched to Python 3.7, and hope this resolves most of the issues above. Special thanks to @BrenoWSRCa and @free-soellingeraj for creating forks with this compatibility!