Threading error when 2 Threads may control the same Motor. Errors encountered in EV3Dev2 with different error messages on MicroPython vs Python3. No such errors encountered in EV3Dev(1) and PyBricks.

Question

Threading error when 2 Threads may control the same Motor. Errors encountered in EV3Dev2 with different error messages on MicroPython vs Python3. No such errors encountered in EV3Dev(1) and PyBricks.

AntoniLuongPham opened this issue 4 years ago · 14 comments

ev3dev version: 4.14.117-ev3dev-2.3.5-ev3

ev3dev-lang-python version:
||/ Name Version Architecture Description
+++-==================================-======================-======================-=========================================================================
ii micropython-ev3dev2 2.1.0 all Python language bindings for ev3dev for MicroPython
ii python3-ev3dev 1.2.0 all Python language bindings for ev3dev
ii python3-ev3dev2 2.1.0 all Python language bindings for ev3dev

I have certain programs that crash on EV3Dev2 when 2 parallel Threads may control the same Motor. The crashes have different error messages on MicroPython vs. Python3. I have reproduced the errors in the enclosed test scripts "BUG.SameMotor.Threading.EV3Dev2.MicroPython.py" and "BUG.SameMotor.Threading.EV3Dev2.Python3.py".

For comparison, I have also tried and enclosed below other variations (e.g. using ev3dev(1.2.0) or pybricks instead of ev3dev2, or using multiprocessing or PyBricks's run_parallel instead of threading) that run successfully.

All these programs do the following: turn the Motor clockwise when the Touch Sensor is pressed, or turn the same Motor counter-clockwise when any IR Remote Control button is pressed. The observations from all the coding variations are as follows:

threading with ev3dev2 2.1.0 on micropython: CRASHED after pressing the IR Remote Control buttons a few times (making the Motor turn counter-clockwise), even if the Touch Sensor is NOT pressed at all (i.e. that Touch-Sensor-triggered Thread never commands the Motor at all).

File "ev3dev2/motor.py", line 1048, in on_for_seconds
File "ev3dev2/motor.py", line 928, in wait_until_not_moving
File "ev3dev2/motor.py", line 908, in wait
OSError: 4

threading with ev3dev2 2.1.0 on python3: CRASHED when the Touch Sensor and an IR Remote Control button are pressed at the same time, i.e. both Thread are trying to command the Motor at the same time. But the program runs ok when only 1 button is pressed.

File "/usr/lib/python3/dist-packages/ev3dev2/motor.py", line 1048, in on_for_seconds
self.wait_until_not_moving()
File "/usr/lib/python3/dist-packages/ev3dev2/motor.py", line 928, in wait_until_not_moving
return self.wait(lambda state: self.STATE_RUNNING not in state or self.STATE_STALLED in state, timeout)
File "/usr/lib/python3/dist-packages/ev3dev2/motor.py", line 908, in wait
self._poll.poll(poll_tm)
RuntimeError: concurrent poll() invocation

multiprocessing with ev3dev2 2.1.0 on micropython: both Processes run successfully WITH mutual blocking, i.e. 1 Process CANNOT interrupt/reverse the other Process's Motor movement mid-stream
multiprocessing with ev3dev2 2.1.0 on python3: both Processes run successfully WITH mutual blocking, i.e. 1 Process CANNOT interrupt/reverse the other Process's Motor movement mid-stream
threading with ev3dev (1.2.0) on python3: both Threads run successfully WITHOUT mutual blocking, i.e. 1 Thread CAN interrupt/reverse the other Thread's Motor movement mid-stream
multiprocessing with ev3dev (1.2.0) on python3: both Processes run successfully WITHOUT mutual blocking, i.e. 1 Process CAN interrupt/reverse the other Process's Motor movement mid-stream
threading with pybricks (2.0.0) on pybricks-micropython: both Threads run successfully WITH mutual blocking, i.e. 1 Thread CANNOT interrupt/reverse the other Thread's Motor movement mid-stream
run_parallel (experimental PyBricks feature) with pybricks (2.0.0) on pybricks-micropython: both Threads run successfully WITH mutual blocking, i.e. 1 Thread CANNOT interrupt/reverse the other Thread's Motor movement mid-stream

foocross commented 4 years ago

Thanks

Answer 1 · 2020-07-11T20:59:09.000Z

`BUG.SameMotor.Threading.EV3Dev2.MicroPython.py`

#!/usr/bin/env micropython


from ev3dev2.motor import MediumMotor, OUTPUT_A
from ev3dev2.sensor import INPUT_1, INPUT_4
from ev3dev2.sensor.lego import TouchSensor, InfraredSensor

from threading import Thread


TOUCH_SENSOR = TouchSensor(address=INPUT_1)
IR_SENSOR = InfraredSensor(address=INPUT_4)
MOTOR = MediumMotor(address=OUTPUT_A)


def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.is_pressed:
            MOTOR.on_for_seconds(
                speed=100,
                seconds=1,
                brake=True,
                block=True)


def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if IR_SENSOR.buttons_pressed(channel=1):
            MOTOR.on_for_seconds(
                speed=-100,
                seconds=1,
                brake=True,
                block=True)


Thread(target=touch_to_turn_motor_clockwise).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()
# *** BUG as of 2020 ***
# *** just the presence of the above parallel Thread makes this main thread crash ***
# Traceback (most recent call last):
#   File "/home/robot/test/BUG.SameMotor.Threading.EV3Dev2.MicroPython.py", line 38, in <module>
#   File "/home/robot/test/BUG.SameMotor.Threading.EV3Dev2.MicroPython.py", line 29, in press_any_ir_remote_button_to_turn_motor_counterclockwise
#   File "ev3dev2/motor.py", line 1048, in on_for_seconds
#   File "ev3dev2/motor.py", line 928, in wait_until_not_moving
#   File "ev3dev2/motor.py", line 908, in wait
# OSError: 4

Answer 2 · 2020-07-11T20:59:54.000Z

`BUG.SameMotor.Threading.EV3Dev2.Python3.py`

#!/usr/bin/env python3


from ev3dev2.motor import MediumMotor, OUTPUT_A
from ev3dev2.sensor import INPUT_1, INPUT_4
from ev3dev2.sensor.lego import TouchSensor, InfraredSensor

from threading import Thread


TOUCH_SENSOR = TouchSensor(address=INPUT_1)
IR_SENSOR = InfraredSensor(address=INPUT_4)
MOTOR = MediumMotor(address=OUTPUT_A)


def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.is_pressed:
            MOTOR.on_for_seconds(
                speed=100,
                seconds=1,
                brake=True,
                block=True)


def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if IR_SENSOR.buttons_pressed(channel=1):
            MOTOR.on_for_seconds(
                speed=-100,
                seconds=1,
                brake=True,
                block=True)


Thread(target=touch_to_turn_motor_clockwise,
       daemon=True).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()

# *** BUG as of 2020 ***
# *** as soon as both Touch Sensor and an IR button are pressed ***
# Traceback (most recent call last):
#   File "/home/robot/test/BUG.SameMotor.Threading.EV3Dev2.Python3.py", line 39, in <module>
#     press_any_ir_remote_button_to_turn_motor_counterclockwise()
#   File "/home/robot/test/BUG.SameMotor.Threading.EV3Dev2.Python3.py", line 33, in press_any_ir_remote_button_to_turn_motor_counterclockwise
#     block=True)
#   File "/usr/lib/python3/dist-packages/ev3dev2/motor.py", line 1048, in on_for_seconds
#     self.wait_until_not_moving()
#   File "/usr/lib/python3/dist-packages/ev3dev2/motor.py", line 928, in wait_until_not_moving
#     return self.wait(lambda state: self.STATE_RUNNING not in state or self.STATE_STALLED in state, timeout)
#   File "/usr/lib/python3/dist-packages/ev3dev2/motor.py", line 908, in wait
#     self._poll.poll(poll_tm)
# RuntimeError: concurrent poll() invocation

Answer 3 · 2020-07-11T21:00:45.000Z

`SameMotor.MultiProcessing.EV3Dev2.MicroPython.py`

#!/usr/bin/env micropython


from ev3dev2.motor import MediumMotor, OUTPUT_A
from ev3dev2.sensor import INPUT_1, INPUT_4
from ev3dev2.sensor.lego import TouchSensor, InfraredSensor

from multiprocessing import Process


TOUCH_SENSOR = TouchSensor(address=INPUT_1)
IR_SENSOR = InfraredSensor(address=INPUT_4)
MOTOR = MediumMotor(address=OUTPUT_A)


def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.is_pressed:
            MOTOR.on_for_seconds(
                speed=100,
                seconds=1,
                brake=True,
                block=True)


def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if IR_SENSOR.buttons_pressed(channel=1):
            MOTOR.on_for_seconds(
                speed=-100,
                seconds=1,
                brake=True,
                block=True)


Process(target=touch_to_turn_motor_clockwise).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()

# observation: both processes run successfully WITH mutual blocking
# i.e. 1 process CANNOT interrupt/reverse the other process's Motor movement mid-stream

Answer 4 · 2020-07-11T21:01:23.000Z

`SameMotor.MultiProcessing.EV3Dev2.Python3.py`

#!/usr/bin/env python3


from ev3dev2.motor import MediumMotor, OUTPUT_A
from ev3dev2.sensor import INPUT_1, INPUT_4
from ev3dev2.sensor.lego import TouchSensor, InfraredSensor

from multiprocessing import Process


TOUCH_SENSOR = TouchSensor(address=INPUT_1)
IR_SENSOR = InfraredSensor(address=INPUT_4)
MOTOR = MediumMotor(address=OUTPUT_A)


def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.is_pressed:
            MOTOR.on_for_seconds(
                speed=100,
                seconds=1,
                brake=True,
                block=True)


def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if IR_SENSOR.buttons_pressed(channel=1):
            MOTOR.on_for_seconds(
                speed=-100,
                seconds=1,
                brake=True,
                block=True)


Process(target=touch_to_turn_motor_clockwise,
        daemon=True).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()

# observation: both processes run successfully WITH mutual blocking
# i.e. 1 process CANNOT interrupt/reverse the other process's Motor movement mid-stream

Answer 5 · 2020-07-11T21:02:13.000Z

`SameMotor.Threading.EV3Dev1.py`

#!/usr/bin/env python3


from ev3dev.ev3 import TouchSensor, INPUT_1, InfraredSensor, INPUT_4, RemoteControl, MediumMotor, OUTPUT_A

from threading import Thread


TOUCH_SENSOR = TouchSensor(address=INPUT_1)
REMOTE_CONTROL = RemoteControl(sensor=InfraredSensor(address=INPUT_4),
                               channel=1)
MOTOR = MediumMotor(address=OUTPUT_A)


def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.is_pressed:
            MOTOR.run_timed(
                speed_sp=1000,   # deg/s
                time_sp=1000,   # ms
                stop_action=MediumMotor.STOP_ACTION_HOLD)


def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if REMOTE_CONTROL.buttons_pressed:
            MOTOR.run_timed(
                speed_sp=-1000,   # deg/s
                time_sp=1000,   # ms
                stop_action=MediumMotor.STOP_ACTION_HOLD)


Thread(target=touch_to_turn_motor_clockwise,
       daemon=True).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()

# observation: both threads run successfully WITHOUT mutual blocking
# i.e. 1 thread CAN interrupt/reverse the other thread's Motor movement mid-stream

Answer 6 · 2020-07-11T21:02:46.000Z

`SameMotor.MultiProcessing.EV3Dev1.py`

#!/usr/bin/env python3


from ev3dev.ev3 import TouchSensor, INPUT_1, InfraredSensor, INPUT_4, RemoteControl, MediumMotor, OUTPUT_A

from multiprocessing import Process


TOUCH_SENSOR = TouchSensor(address=INPUT_1)
REMOTE_CONTROL = RemoteControl(sensor=InfraredSensor(address=INPUT_4),
                               channel=1)
MOTOR = MediumMotor(address=OUTPUT_A)


def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.is_pressed:
            MOTOR.run_timed(
                speed_sp=1000,   # deg/s
                time_sp=1000,   # ms
                stop_action=MediumMotor.STOP_ACTION_HOLD)


def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if REMOTE_CONTROL.buttons_pressed:
            MOTOR.run_timed(
                speed_sp=-1000,   # deg/s
                time_sp=1000,   # ms
                stop_action=MediumMotor.STOP_ACTION_HOLD)


Process(target=touch_to_turn_motor_clockwise,
        daemon=True).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()

# observation: both processes run successfully WITHOUT any mutual blocking
# i.e. 1 process CAN interrupt/reverse the other process's Motor movement mid-stream

Answer 7 · 2020-07-11T21:03:51.000Z

`SameMotor.Threading.PyBricks.py`

#!/usr/bin/env pybricks-micropython


from pybricks.ev3devices import TouchSensor, InfraredSensor, Motor
from pybricks.parameters import Port, Stop

from threading import Thread


TOUCH_SENSOR = TouchSensor(port=Port.S1)
IR_SENSOR = InfraredSensor(port=Port.S4)
MOTOR = Motor(port=Port.A)


def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.pressed():
            MOTOR.run_time(
                speed=1000,   # deg/s
                time=1000,   # ms
                then=Stop.HOLD,
                wait=True)


def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if IR_SENSOR.buttons(channel=1):
            MOTOR.run_time(
                speed=-1000,   # deg/s
                time=1000,   # ms
                then=Stop.HOLD,
                wait=True)


Thread(target=touch_to_turn_motor_clockwise).start()

press_any_ir_remote_button_to_turn_motor_counterclockwise()

# observation: both threads run successfully WITH mutual blocking
# i.e. 1 thread CANNOT interrupt/reverse the other thread's Motor movement mid-stream

Answer 8 · 2020-07-11T21:04:21.000Z

`SameMotor.RunParallel.PyBricks.py`

#!/usr/bin/env pybricks-micropython


from pybricks.ev3devices import TouchSensor, InfraredSensor, Motor
from pybricks.parameters import Port, Stop

from pybricks.experimental import run_parallel


TOUCH_SENSOR = TouchSensor(port=Port.S1)
IR_SENSOR = InfraredSensor(port=Port.S4)
MOTOR = Motor(port=Port.A)


def touch_to_turn_motor_clockwise():
    while True:
        if TOUCH_SENSOR.pressed():
            MOTOR.run_time(
                speed=1000,   # deg/s
                time=1000,   # ms
                then=Stop.HOLD,
                wait=True)


def press_any_ir_remote_button_to_turn_motor_counterclockwise():
    while True:
        if IR_SENSOR.buttons(channel=1):
            MOTOR.run_time(
                speed=-1000,   # deg/s
                time=1000,   # ms
                then=Stop.HOLD,
                wait=True)


run_parallel(
    touch_to_turn_motor_clockwise,
    press_any_ir_remote_button_to_turn_motor_counterclockwise)

# observation: both threads run successfully WITH mutual blocking
# i.e. 1 thread CANNOT interrupt/reverse the other thread's Motor movement mid-stream

Answer 9 · 2020-07-11T22:13:02.000Z

Yeah, multi-threaded access to the same motor is not (officially) supported, unfortunately. It's curious that your results seem to indicate that stock motors work un-modified with the v1.0 series; I am not sure why there's a difference there, probably due to blocking logic changes. It might be worth diffing the motor code to compare, but then again, since we haven't intentionally designed it to work with concurrent access I don't expect that to be consistent.

My suggestion is to go with the mutex version to be safe.

Answer 10 · 2020-07-11T23:07:39.000Z

@WasabiFan the most curious result among the above is from Program no. 1 "BUG.SameMotor.Threading.EV3Dev2.MicroPython.py": the program crashes even when the Touch Sensor is never touched and hence its related Thread never actually attempts to access the Motor. What do you make of this?

Is there a MicroPython compiler that compiles this program and identifies that there is a chance that the Touch-Sensor-related Thread may access the Motor?

Answer 11 · 2020-07-12T07:15:36.000Z

That case sounds like this other one: #727

OSError 4 is EINTR, i.e. "system call interrupted". It means that a signal was raised while in a system call, and the system call was aborted to prevent deadlock. This happens because even just the presence of threads means the interpreter (Micropython specifically) will begin triggering signals to synchronize garbage collection.

Technical overview aside, it means that multithreading does not play nice with our MicroPython library, unfortunately. I had tried to work toward a true solution of this (starting with #732, to benchmark the speed of our library before adding error handling) but haven't circled back to finish it, implement the error handling, and re-benchmark. There are some notes at the bottom of the thread linked above which might help you work around it.

Answer 12 · 2020-08-26T23:50:30.000Z

We are also getting OSError: 4 errors in wait a significant number of times in micropython. If we shut off odometry_start() from MoveDifferential , would it remove the problem?

Answer 13 · 2020-08-27T04:40:13.000Z

Ah, yeah, odometry uses a thread in the background and may provoke EINTR.