Unifying python bindings

Question

Unifying python bindings

ddemidov opened this issue 9 years ago · 46 comments

There is an ongoing discussion (see #3, ddemidov/ev3dev-lang-python#13, ddemidov/ev3dev-lang-python#17, ev3dev/ev3dev#404) about what to do with boost-based python bindings (developed by me) and pure python bindings (developed by @rhempel).

As I said earlier, using C++ and boost-python as a base for python bindings has a lot of drawbacks: it is cumbersome to setup the developement environment, and it requires from contributors knowledge of both C++ and python.

In my eyes performance of C++ based IO operations was the only reason to keep boost-based bindings, but @dlech says that drivers are not able to deliver unique measurements at that speed anyway. So at this point I vote for replacement of boost-python with pure-python bindings. The only minor drawback of this decision is that C++ bindings would be essentially orphaned by this: I only kept an eye on those to provide support for python bindings. But since C++ is now fully based on autogen scripts, keeping C++ uptodate should not be much of a problem.

If everyone agrees with replacement of boost-python with pure-python, I suggest we use this issue for discussion of the process. @rhempel did the most difficult part of implementing base Device class and file caching. Some motors and sensors are already implemented as well. I could help to introduce the rest of the features that were available in boost-python, setup doc generation, etc.

antonvh commented 9 years ago

Agree

jabrena commented 9 years ago

Oki

Answer 1 · 2015-10-01T13:04:16.000Z

Cool! This is what I mailed to @rhempel earlier.

Let's make a new repo ev3dev-lang-python-pure or something
Structure it like this:

ev3dev
    __intit__.py
    ev3dev.py <- @rhempel's file
README.md
setup.py
examples.py
etc...

__init__.py should of course contain from ev3dev import * - you know how it works.

To allow people to use it in virtualenv (very common with python
development) please subit it to Pypi like so:
http://peterdowns.com/posts/first-time-with-pypi.html (same as boost module)

At the ev3dev-lang repo in dev/master branch.

Rename python with current boostc to python-boostc or something similar
Add a new python folder
Add to the new repo ev3dev-lang-python-pure above as a submodule there.
https://git-scm.com/book/en/v2/Git-Tools-Submodules - you know it better than me, probably.
And finally we can get rid of the confusing pure-python branch.

I'm eager to fork and contribute to ev3dev-lang-python-pure with extra sensors and advanced motor classes.

Answer 2 · 2015-10-01T13:06:44.000Z

I started on a python tutorial too, but I'd like the above to be fixed first.

Answer 3 · 2015-10-02T01:15:50.000Z

What I plan to do is finish up the packaging at the end, for now, I'm going to continue pushing changes to the pure-python bindings here:

[https://github.com/ev3dev/ev3dev-lang/tree/pure-python]

in the pure-python branch. Please grab the latest code that is in there and let's work together to comment on the Class Names and other details - let's get what we have so far to everyone's liking, and then move on with the LCD stuff and Anton's structural suggestions.

Answer 4 · 2015-10-02T07:56:26.000Z

No prob, I'll just wait a while for the submodule then. Class names and namespace are important to get right! (Many open source libraries came into disuse because of weird naming or constant changes.)

Might be good to have a good return policy too. E.g. does motor.run_timed() return itself, or true if it succeeded, or nothing, or it's tacho, or it's speed, or it's duty cycle so you can elegantly measure if it's jammed.
Might be good to use true python getters and setters with decorators too
Also docstrings with triple quotes to make them more legible.

Answer 5 · 2015-10-02T12:12:21.000Z

I think the library already has true getters and setters for the class attributes, along with the doc portion.

What I'm most interested in first from the Python experts is the naming of the top level classes like Motor, Sensor, TouchSensor, etc...

Answer 6 · 2015-10-02T12:17:04.000Z

http://www.python-course.eu/python3_properties.php

It matters. You don't want other people's code to break when motor.set_position(0) becomes motor.position = 0

Answer 7 · 2015-10-02T12:18:34.000Z

The PEP 0008 says that class names should use CamelCase, so I'd say that is the most pythonic way. Same PEP says that it is allowed to keep 'wrong' naming schemes for backward compatibility:

do not break backwards compatibility just to comply with this PEP!

So we can stick to my initial naming scheme to avoid issues like ev3dev/ev3dev#396.

@antonvh, I think @rhempel means that motor.position = 0 works already with pure-python.

Answer 8 · 2015-10-02T12:24:15.000Z

Well, how much backwards compatibility will we actually break? There's the Rubik's solver, Anton's string drawer, the work @ensonic is doing with OpenRoberta - so if we're going to break it, let's break it now and get it right.
Believe me, I am reluctant to break old working projects again, but sometimes in the interest of the greater good, we may have to.

Answer 9 · 2015-10-02T12:28:59.000Z

There is also @cavenel's PrinterBot, but I don't have strong preferences here. Lets stick to PEP0008 then.

Answer 10 · 2015-10-02T12:35:04.000Z

OK, so, being the implementer of my classes, I am blind to their deficiencies :-) If you have time, I'd appreciate a review - note that almost all of the code is autogenerated. Actually, one thing I'd like to change in the autogen system is to be able to provide a stripped version of the file so we can more easily tell what's been hand coded and what has been autogene'd.

Answer 11 · 2015-10-02T12:41:38.000Z

That could probably be achieved by a set of empty templates.

pushd autogen/templates
mkdir bak
for f in cpp*; do
  mv $f bak
  touch $f
done
popd
node autogen/autogen.js cpp

works for me (with cpp bindings). This should be followed by

cd autogen/templates
mv bak/* .
rmdir bak

Answer 12 · 2015-10-02T12:57:28.000Z

I you need to break you api, please do it now, since my stuff has not been launched yet. Once thousand school kids have generated code on their ev3s does not work anymore it be bad.

Also if you change the API, please provide a simple list with the needed changes to make it painless. (e.g. wiki page).

Answer 13 · 2015-10-02T15:34:50.000Z

I do not own the Rubiks solver but have been working on it a lot lately. If you need to break it I'll volunteer to fix it.

Answer 14 · 2015-10-02T16:20:52.000Z

Thanks @dwalton76 ! @ensonic , please also put your opinions on the current API that we have in the pure-python branch - eventually this will be its own repo as a submodule of `ev3dev-lang'

Answer 15 · 2015-10-03T13:54:25.000Z

@ddemidov - I'm going to set up a repo in my part of GitHub also called ev3dev-lang-python. It will have the pure bindings and will be the python submodule in ev3dev-lang
If you like, you can create another submodule called python-boost-cpp (or similar) and use your ev3dev-lang-python repo for that.
One thing that would be handy though is that we make sure the demo code works with both bindings :-)
Does that work for everyone?

Answer 16 · 2015-10-03T14:06:38.000Z

@rhempel - Good for me!

@dwalton76 - if I'm correct, the rubiks cube solver doesn't use any python binding right for now (it uses ev3.py which I wrote that directly read and write in /sys/class/. Thus it will not be affected by this issue. Am I wrong?

For the print3rBot however, it may break the code. But if the changes are well documented I may try to rewrite it some time (in 16 year according to Ralph ;-) ).

Answer 17 · 2015-10-03T14:24:50.000Z

@rhempel, that sounds fine. I think there is no need in creating a submodule for boost based bindings, one version should be enough. I'll try to do some code review and general help when I have more time. Would you accept pull requests?

@cavenel, my elder daughter is just 6, and the mindstorms kit was intended for her. 16 years is a huge overestimate :). Anyway, I think the only changes would be in class names, and should be easy to fix.

Answer 18 · 2015-10-19T08:10:34.000Z

Being a newcomer to ev3dev, I've started with Denis's binding for a simple reason : the link attached to the list of supported languages on the ev3dev libraries page points to it :)
I confess that even if it works fine (for what I've tested until now), it is a bit dev environment unfriendly, since completion hints mechanisms have hard to do their work. So I'm glad to note that the community favours Ralph pure Python binding, since it will make beginners' life easier when working with tools such as the (excellent) PyCharm.
I would then suggest that the different links bringing to the official Python binding be harmonized, by making all of them point to Ralph's pure Python one. Currently, the one behind "language binding repository" on the libraries documentation page is correct, but the one behind "Python" in the list of supported languages points to Denis' work.
Having all links point to the same binding would avoid people ending in some dead end track by going with the boost based version. It will also solve concerns such as breaking compatibility as it has been mentioned earlier in this thread.
I could have done it myself (it the community agrees of course), but I'm way too fresh here to feel comfortable for the moment.

Answer 19 · 2015-10-19T08:45:10.000Z

Welcome Eric! I'm using pycharm too and the pure bindings have exactly the purpose of showing in-editor suggestions and documentations.

Our work is almost done, and we'll make the switch you are suggesting soon. In the meantime, please download ev3dev.py from the repo. (you don't need anything else) and add it to your project so you can import ev3dev.py I would love to know what you think.

On 19 Oct 2015, at 10:10, Eric PASCUAL notifications@github.com wrote:

Being a newcomer to ev3dev, I've started with Denis's binding for a simple reason : the link attached to the list of supported languages on the ev3dev libraries page points to it :)
I confess that even if it works fine (for what I've tested until now), it is a bit dev environment unfriendly, since completion hints mechanisms have hard to do their work. So I'm glad to note that the community favours Ralph pure Python binding, since it will make beginners' life easier when working with tools such as the (excellent) PyCharm.
I would then suggest that the different links bringing to the official Python binding be harmonized, by making all of them point to Ralph's pure Python one. Currently, the one behind "language binding repository" on the libraries documentation page is correct, but the one behind "Python" in the list of supported languages points to Denis' work.
I could have done it myself (it the community agrees of course), but I'm way too fresh here to feel comfortable for the moment.

—
Reply to this email directly or view it on GitHub #104 (comment).

Answer 20 · 2015-10-19T09:11:22.000Z

I'll test it ASAP (maybe tonight).
BTW, having worked with LeJOS in the past (and even contributed a bit to it), there are IMHO a couple of nice high level classes there that would be interesting to have ported to ev3dev (support for different robot mechanical architectures for instance). I'll start to work on this as soon as I'll be able to save some time for it and will contribute the results to the ev3dev community of course. However, I plan to work in pure Python only since my C++ skills have not been exercised since a while are thus far too rusted ;)

Answer 21 · 2015-10-19T10:47:02.000Z

Welcome Eric - glad to have another pair of hands and eyes on this project

Answer 22 · 2015-10-20T14:16:03.000Z

I took a look at pure-python. It is definitely not a drop-in replacement. What I did is:

cd /usr/bin
wget https://raw.githubusercontent.com/rhempel/ev3dev-lang-python/master/ev3dev.py
mv /usr/lib/python2.7/dist-packages/ev3dev{,.bak}

and then restarted my service. One thing that would really help is to have a 'sed' script that does the renaming. This would document the differences and given that people usually store their code in git, they can just run the 'porting' script and try it.

Some of the name are obvious, but e.g. I have not found the portnames (ev3dev.OUTPUT_A). And the LCD() seems to be not covered.

Answer 23 · 2015-10-20T14:24:38.000Z

There is a list of what needs to be done here: ev3dev/ev3dev-lang-python#15. Please add anything you think is missing.

Answer 24 · 2015-10-20T14:24:57.000Z

Thanks. Will fix OUTPUT_A right away. @ddemidov is still working on Lcd(). And class names are upercase now. You are welcome to create a porting script, but we estimated the number of projects that would break with the new names to be really low. Could you submit the issues where @ddemidov pointed you?

Answer 25 · 2015-10-20T15:20:59.000Z

I'm not very comfortable with the way ev3dev works WRT code generation. If I understood correctly, Python sources (and maybe other languages ones too) are not hand written but generated based on specs provided in JSON files. Is this right ?
How HW differences such as LCD, buttons, LEDs,... available or not, or available differently,are managed, knowing that it can differ a lot depending on the target ?
In addition, how higher level classes (such as the ones inspired from LeJOS I was referring too in a previous message) are supposed to be integrated in ev3dev ? As sibling packages ? By defining ev3dev as a namespace package and putting such additional material as sub-packages (which would imply migrating the current ev3dev modules in this namespace, perhaps renaming it to "core" or something similar to avoid name clash) ?
Apologizes if these questions sound stupid, I'm discovering a new universe ;)

Answer 26 · 2015-10-20T15:30:20.000Z

It's simple:

When doing a pull request don't touch anything that is inbetween
#~autogen [some statements here]
#~autogen

...or change the appropriate liquid files.

Basics like buttons and lcd go into ev3dev.py (our focus right now)

Higher level go in separate file, extras.py or advanced.py. Tbd. I have some ideas for them too, but haven't got around to build and test them.

On 20 Oct 2015, at 17:21, Eric PASCUAL notifications@github.com wrote:

I'm not very comfortable with the way ev3dev works WRT code generation. If I understood correctly, Python sources (and maybe other languages ones too) are not hand written but generated based on specs provided in JSON files. Is this right ?
How HW differences such as LCD, buttons, LEDs,... available or not, or available differently,are managed, knowing that it can differ a lot depending on the target ?
In addition, how higher level classes (such as the ones inspired from LeJOS I was referring too in a previous message) are supposed to be integrated in ev3dev ? As sibling packages ? By defining ev3dev as a namespace package and putting such additional material as sub-packages (which would imply migrating the current ev3dev modules in this namespace, perhaps renaming it to "core" or something similar to avoid name clash) ?
Apologizes if these questions sound stupid, I'm discovering a new universe ;)

—
Reply to this email directly or view it on GitHub #104 (comment).

Answer 27 · 2015-10-20T16:02:29.000Z

Thx for the autogen marker detail.

WRT higher level stuff, I do agree it must go into other modules. The point I raised was how all these modules (ev3dev core and additional ones) should be organised ? Under a single umbrella (aka namespace package) or as totally external stuff ? The first option is more user friendly, since you can them use them with imports such as "ev3dev.differential", "ev3dev.pathfind",...

Answer 28 · 2015-10-20T16:13:31.000Z

I agree with you. They should. But @rhempel and @ddemidov decided to flatten the library structure. The discussion is here: ev3dev/ev3dev-lang-python#10

Answer 29 · 2015-10-20T16:20:57.000Z

I think the purpose of the bindings is to provide basic device-interaction code. I believe @rhempel would agree with me (although his definition of 'basic' may be even more basic than mine :-) ). But things like differential or pathfind sound like they belong to a separate library, (based on the bindings). We could include those as demos/examples though.

Answer 30 · 2015-10-20T16:53:48.000Z

@ericpascual Why don't we start an separate library together with higher level functions? It think there should be MotorPair (with differential), an AdvancedPIDMotor. Or convenience classes around non-lego sensors like accelerometers and gyro's.

Answer 31 · 2015-10-20T16:57:24.000Z

Here's my take on this discussion starting with the code generation. It would be nice to have autogen be able to create a "blank" language binding so that it's easier to see what is autogen'ed. I can work around this by either:

Temporarily linking to blank template files
Linking the autogen script to another name and changing the way the code works
Adding a command line option to autogen to create a blank binding
The ev3dev.py binding should be to the lowest possible level for multiple platforms. I am still undecided whether the platform specific code should be the output of a run of autogen or if the Python binding shoud determine the platform it's running on.
Higher level functions and convenience classes belong (I think) in a separate library that can import ev3dev as a dependency. For example, not all Motor class implementations on platforms will support MotorPair or PID type functions.
Base level bindings support mechanisms, not policy. I am happy to discuss pros and cons, so feel free to comment.

Answer 32 · 2015-10-20T17:01:20.000Z

I am still undecided whether the platform specific code should be the output of a run of autogen or if the Python binding shoud determine the platform it's running on.

I think this should be done in python code at runtime, or we would have to support several versions of the distributed package. Platform discovery is only done once (during module import), and should not have any impact on program performance after that.

Answer 33 · 2015-10-20T17:03:34.000Z

I mostly agree with what all of you said.

But how should we consider what I would call "intermediate" convenient classes tied to a concrete product ? An example of such one would be a model of the EV3 brick, including a LCD, an a fixed set of LEDS and buttons already instantiated and ready to use ? Same applies of course to RasPi+BrickPi combo, or some day to RasPi+PiStorms (1) ? Shouldn't at least the EV3 model be part of the "standard" library ?

(1) which I found more attractive by the way than the BrickPi, which seemed a bit limited and unreliable as is when I tested it - see http://www.pobot.org/La-carte-d-extension-BrickPi.html (sorry, French version of the article only)

Answer 34 · 2015-10-20T17:05:07.000Z

And I'm OK with that. By keeping the base binding minimal, we won't need to get into too much code bloat as we add convenience classes.
The main goal right now should be to get the Python bindings stable enough for @ensonic to use in the OpenRoberta project.

Answer 35 · 2015-10-20T17:09:29.000Z

I think the ev3dev project would be stronger if the standard library would contain more convenience functions and classes. It would add to the magic. "Look how simple it is to this and that with ev3dev. Right out of the box! No fiddling with pip, easyinstall, virtualenv, yaddayadda. It's just like you know it from Lego's software except that you can type it.". Many people like to contribute and are confused by the alternative python libraries that exist.

When I do workshops and explain the basics of coding with python and mindstorms, for many creating their own MicroSD with ev3dev is usually enough Linux for them... Git cloning libraries is too much for them.

Answer 36 · 2015-10-20T17:19:29.000Z

Also regarding higher level classes (differential drive). I have so far written those in a python hal, which in turn calls low level code from ev3dev. The hal has a robot-configuration.
The lab.open-roberta.org code generator produces code that runs against the hal.

Answer 37 · 2015-10-20T17:23:22.000Z

By the way, this binding will ship with the next ev3dev image, I think - no git cloning needed!

Answer 38 · 2015-10-20T17:33:41.000Z

The git cloning is no big deal, no having a debian package to depend on is.

Answer 39 · 2015-10-30T08:42:01.000Z

Good morning,

I have created a repo to offer an API for Java developers using EV3Dev.
https://github.com/jabrena/ev3dev-lang-java

I think that Python development has good progress. For other language, what is the best strategy to integrate with the native API?
http://www.ev3dev.org/docs/sensors/
http://www.ev3dev.org/docs/motors/

I would like to create a easy example from Java to read a Sensor and use a motor for example.
Later, if the PoC goes well, I would like to generate the API for the rest of elements.
Is it possible to reuse low level development used from Python to other language?

Cheers

Juan Antonio

Answer 40 · 2015-10-30T09:08:08.000Z

Hi, you should open a separate issue for this.

Answer 41 · 2015-10-30T11:50:43.000Z

And we'll help you get set up so that you can retain the master copy of the binding and we'll pull from it. Please review the autogen folder in ev3dev-lang - that's how we create most of the Python binding.

Answer 42 · 2015-10-30T12:09:03.000Z

This weekend, I will study it.

I did an issue about this question here:
#114

Many thanks @ddemidov @rhempel

Answer 43 · 2015-11-29T22:58:01.000Z

What's the status on this one? Have we reached a state of convergence between the Python libs?

Answer 44 · 2015-11-30T00:29:12.000Z

I finally had the feeling that my way of thinking WRT autogen was not really shared by people involved in the project for longer than me, so I didn't insist any more. However, I've continued developing my alternative implementation (https://github.com/EricPobot/ev3dev-lang-python-alt) without autogen. It is there, in case it could be useful for anything. It has not been initialized as a fork since to much different from the original one.

The current version is aligned with the devices naming as defined in the latest distribution of the kernel (September), and not with the latest version of the sources. As soon as a new distribution will be released, I'll update my code. I initially planned to add some "try and guess" approach for making the code compatible with both versions, but I'm not sure this is really useful, since there will be no special need to maintain this kind of backward compatibility.

I have also started to "port" some concepts borrowed to LeJOS. This work is available here : https://github.com/EricPobot/ev3dev-lang-python-contrib

I am open to any suggestion WRT any of the above points.