ami-iit/yarp-openmct

Running the telemetry server from the folder `./iCubTelemVizServer` with the `node` command fails

Closed this issue · 9 comments

nunoguedelha commented

Refer to [Issue 1] in the issue recap below:

Test Report

Objectives

The tests focused on checking the data pipe and overall functionality related to the Yarp Text Logging of the yarprobotinterface process running on the robot head and the module WalkingModule running on some other machine connected to the robot network. The related telemetry domain entries in the Open MCT Visualiser, yarprobotinterface process logging through yarplogger and WalkingModule process logging through yarplogger under the telemetry folder Process Logging:
image

Test Sequence, Outcome and Issues

  • Installation went smoothly.

  • [Issue 1]: Running the telemetry server from the folder ./iCubTelemVizServer and node command fails:

    $> cd iCubTelemVizServer
$> node iCubTelemVizServer.js
...
/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:282
            this.state[id].ori.roll = sensorSample[3][0][0][0];
                                                        ^

TypeError: Cannot read property '0' of undefined
    at ICubTelemetry.parseFromId (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:282:57)
    at Object.<anonymous> (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:166:31)
    at /Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/iCubTelemVizServer.js:74:56
    at /Users/nunoguedelha/dev/yarp-openmct/node_modules/YarpJS/yarp.js:188:13
$>
    Full log 
    iCub Telemetry server launched!
[INFO] |yarp.os.Port|/yarpjs/inertial:i| Port /yarpjs/inertial:i active at tcp://192.168.1.70:10130/
[INFO] |yarp.os.Port|/yarpjs/head/inertials:i| Port /yarpjs/head/inertials:i active at tcp://192.168.1.70:10131/
[INFO] |yarp.os.Port|/yarpjs/leftArm/inertials:i| Port /yarpjs/leftArm/inertials:i active at tcp://192.168.1.70:10132/
[INFO] |yarp.os.Port|/yarpjs/rightArm/inertials:i| Port /yarpjs/rightArm/inertials:i active at tcp://192.168.1.70:10133/
[INFO] |yarp.os.Port|/yarpjs/leftLeg/inertials:i| Port /yarpjs/leftLeg/inertials:i active at tcp://192.168.1.70:10134/
[INFO] |yarp.os.Port|/yarpjs/rightLeg/inertials:i| Port /yarpjs/rightLeg/inertials:i active at tcp://192.168.1.70:10135/
[INFO] |yarp.os.Port|/yarpjs/leftFoot/inertials:i| Port /yarpjs/leftFoot/inertials:i active at tcp://192.168.1.70:10136/
[INFO] |yarp.os.Port|/yarpjs/rightFoot/inertials:i| Port /yarpjs/rightFoot/inertials:i active at tcp://192.168.1.70:10137/
[INFO] |yarp.os.Port|/yarpjs/left_leg/stateExt:o| Port /yarpjs/left_leg/stateExt:o active at tcp://192.168.1.70:10138/
[INFO] |yarp.os.Port|/yarpjs/right_leg/stateExt:o| Port /yarpjs/right_leg/stateExt:o active at tcp://192.168.1.70:10139/
[INFO] |yarp.os.Port|/yarpjs/left_arm/stateExt:o| Port /yarpjs/left_arm/stateExt:o active at tcp://192.168.1.70:10140/
[INFO] |yarp.os.Port|/yarpjs/right_arm/stateExt:o| Port /yarpjs/right_arm/stateExt:o active at tcp://192.168.1.70:10141/
[INFO] |yarp.os.Port|/yarpjs/torso/stateExt:o| Port /yarpjs/torso/stateExt:o active at tcp://192.168.1.70:10142/
[INFO] |yarp.os.Port|/yarpjs/head/stateExt:o| Port /yarpjs/head/stateExt:o active at tcp://192.168.1.70:10143/
[INFO] |yarp.os.Port|/yarpjs/camLeftEye:i| Port /yarpjs/camLeftEye:i active at tcp://192.168.1.70:10144/
[INFO] |yarp.os.Port|/yarpjs/camRightEye:i| Port /yarpjs/camRightEye:i active at tcp://192.168.1.70:10145/
[INFO] |yarp.os.Port|/yarpjs/left_arm/cartesianEndEffectorWrench:i| Port /yarpjs/left_arm/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10146/
[INFO] |yarp.os.Port|/yarpjs/right_arm/cartesianEndEffectorWrench:i| Port /yarpjs/right_arm/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10147/
[INFO] |yarp.os.Port|/yarpjs/left_upper_leg/cartesianEndEffectorWrench:i| Port /yarpjs/left_upper_leg/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10148/
[INFO] |yarp.os.Port|/yarpjs/left_lower_leg/cartesianEndEffectorWrench:i| Port /yarpjs/left_lower_leg/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10149/
[INFO] |yarp.os.Port|/yarpjs/right_upper_leg/cartesianEndEffectorWrench:i| Port /yarpjs/right_upper_leg/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10150/
[INFO] |yarp.os.Port|/yarpjs/right_lower_leg/cartesianEndEffectorWrench:i| Port /yarpjs/right_lower_leg/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10151/
[INFO] |yarp.os.Port|/yarpjs/left_foot_front/cartesianEndEffectorWrench:i| Port /yarpjs/left_foot_front/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10152/
[INFO] |yarp.os.Port|/yarpjs/left_foot_rear/cartesianEndEffectorWrench:i| Port /yarpjs/left_foot_rear/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10153/
[INFO] |yarp.os.Port|/yarpjs/right_foot_front/cartesianEndEffectorWrench:i| Port /yarpjs/right_foot_front/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10154/
[INFO] |yarp.os.Port|/yarpjs/right_foot_rear/cartesianEndEffectorWrench:i| Port /yarpjs/right_foot_rear/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10155/
[INFO] |yarp.os.Port|/yarpjs/left_arm/FT:i| Port /yarpjs/left_arm/FT:i active at tcp://192.168.1.70:10156/
[INFO] |yarp.os.Port|/yarpjs/right_arm/FT:i| Port /yarpjs/right_arm/FT:i active at tcp://192.168.1.70:10157/
[INFO] |yarp.os.Port|/yarpjs/left_leg_hip/FT:i| Port /yarpjs/left_leg_hip/FT:i active at tcp://192.168.1.70:10158/
[INFO] |yarp.os.Port|/yarpjs/right_leg_hip/FT:i| Port /yarpjs/right_leg_hip/FT:i active at tcp://192.168.1.70:10159/
[INFO] |yarp.os.Port|/yarpjs/left_foot_heel_tiptoe/FT:i| Port /yarpjs/left_foot_heel_tiptoe/FT:i active at tcp://192.168.1.70:10160/
[INFO] |yarp.os.Port|/yarpjs/right_foot_heel_tiptoe/FT:i| Port /yarpjs/right_foot_heel_tiptoe/FT:i active at tcp://192.168.1.70:10161/
[INFO] |yarp.os.Port|/yarpjs/battery/data:i| Port /yarpjs/battery/data:i active at tcp://192.168.1.70:10162/
[INFO] |yarp.os.Port|/yarpjs/walking-coordinator-logger/data:i| Port /yarpjs/walking-coordinator-logger/data:i active at tcp://192.168.1.70:10163/
[INFO] |yarp.os.Port|/yarpjs/proc-yarprobotinterface-yarptextlogging:i| Port /yarpjs/proc-yarprobotinterface-yarptextlogging:i active at tcp://192.168.1.70:10164/
[INFO] |yarp.os.Port|/yarpjs/proc-walkingmodule-yarptextlogging:i| Port /yarpjs/proc-walkingmodule-yarptextlogging:i active at tcp://192.168.1.70:10165/
[INFO] |yarp.os.Port|/yarpjs/sysCmdsGenerator/rpc| Port /yarpjs/sysCmdsGenerator/rpc active at tcp://192.168.1.70:10166/
{ status: 'OK', message: 'Opem-MCT static server process started.' }
info:Yarp command successfully started...
ICubTelemetry History Server listening on http://192.168.1.70:8081/history
ICubTelemetry Realtime Server listening on ws://192.168.1.70:8081/realtime
Control Console Server listening on http://192.168.1.70:3000
Yarp port names retrieval completed successfully (0)
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/inertial:i| Receiving input from /icubSim/inertial to /yarpjs/inertial:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/head/inertials:i| Receiving input from /icubSim/head/inertials/measures:o to /yarpjs/head/inertials:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_leg/stateExt:o| Receiving input from /icubSim/left_leg/stateExt:o to /yarpjs/left_leg/stateExt:o using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_leg/stateExt:o| Receiving input from /icubSim/right_leg/stateExt:o to /yarpjs/right_leg/stateExt:o using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_arm/stateExt:o| Receiving input from /icubSim/left_arm/stateExt:o to /yarpjs/left_arm/stateExt:o using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_arm/stateExt:o| Receiving input from /icubSim/right_arm/stateExt:o to /yarpjs/right_arm/stateExt:o using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/torso/stateExt:o| Receiving input from /icubSim/torso/stateExt:o to /yarpjs/torso/stateExt:o using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/head/stateExt:o| Receiving input from /icubSim/head/stateExt:o to /yarpjs/head/stateExt:o using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_arm/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_arm/cartesianEndEffectorWrench:o to    /yarpjs/left_arm/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_arm/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_arm/cartesianEndEffectorWrench:o to    /yarpjs/right_arm/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_upper_leg/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_upper_leg/cartesianEndEffectorWrench:o to    /yarpjs/left_upper_leg/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_lower_leg/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_lower_leg/cartesianEndEffectorWrench:o to    /yarpjs/left_lower_leg/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_upper_leg/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_upper_leg/cartesianEndEffectorWrench:o to    /yarpjs/right_upper_leg/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_lower_leg/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_lower_leg/cartesianEndEffectorWrench:o to    /yarpjs/right_lower_leg/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_foot_front/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_foot_front/cartesianEndEffectorWrench:o to    /yarpjs/left_foot_front/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_foot_rear/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_foot_rear/cartesianEndEffectorWrench:o to    /yarpjs/left_foot_rear/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_foot_front/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_foot_front/cartesianEndEffectorWrench:o to    /yarpjs/right_foot_front/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_foot_rear/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_foot_rear/cartesianEndEffectorWrench:o to    /yarpjs/right_foot_rear/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/proc-yarprobotinterface-yarptextlogging:i| Receiving input from /log/iiticublap199/yarprobotinterfa[yarprobotinterface]/85559 to    /yarpjs/proc-yarprobotinterface-yarptextlogging:i using tcp
[OPEN-MCT STATIC SERVER] ipc: {"pid":85881}
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/proc-walkingmodule-yarptextlogging:i| Receiving input from /log/iiticublap199/WalkingModule-0.[walkingModule]/85560 to /yarpjs/proc-   walkingmodule-yarptextlogging:i using tcp
/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:282
            this.state[id].ori.roll = sensorSample[3][0][0][0];
                                                        ^

TypeError: Cannot read property '0' of undefined
    at ICubTelemetry.parseFromId (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:282:57)
    at Object.<anonymous> (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:166:31)
    at /Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/iCubTelemVizServer.js:74:56
    at /Users/nunoguedelha/dev/yarp-openmct/node_modules/YarpJS/yarp.js:188:13

  • The app could be launched without issues from the repo root:

    npm start

  • [Issue 2]: As soon as the server starts, it crashes with the following issue:

    /Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:404
    this.history[id].push(telemetrySample); // update history
                     ^

TypeError: Cannot read property 'push' of undefined
    at ICubTelemetry.generateTelemetry (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:404:22)
    at ICubTelemetry.<anonymous> (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:183:18)
    at Array.forEach (<anonymous>)
    at ICubTelemetry.<anonymous> (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:182:33)
    at listOnTimeout (internal/timers.js:555:17)
    at processTimers (internal/timers.js:498:7)
    Extended log 
    NEW CONNECTION!!
CLOSE CONNECTION
[OPEN-MCT STATIC SERVER] stdout: CLOSE CONNECTION

NEW CONNECTION!!
CLOSE CONNECTION
[INFO] |yarp.os.impl.PortCoreInputUnit| Removing input from /log/iiticublap199/WalkingModule-0.[walkingModule]/85560 to /yarpjs/proc-walkingmodule-yarptextlogging:i
[INFO] |yarp.os.impl.PortCoreInputUnit| Receiving input from /icubSim/left_foot_heel_tiptoe/measures:o to /yarpjs/left_foot_heel_tiptoe/FT:i using tcp
/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:404
    this.history[id].push(telemetrySample); // update history
                     ^

TypeError: Cannot read property 'push' of undefined
    at ICubTelemetry.generateTelemetry (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:404:22)
    at ICubTelemetry.<anonymous> (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:183:18)
    at Array.forEach (<anonymous>)
    at ICubTelemetry.<anonymous> (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:182:33)
    at listOnTimeout (internal/timers.js:555:17)
    at processTimers (internal/timers.js:498:7)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! yarp-openmct@1.0.0 start: `cd iCubTelemVizServer && node ${NODE_DEBUG_OPTION} iCubTelemVizServer.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the yarp-openmct@1.0.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/nunoguedelha/.npm/_logs/2022-10-04T08_39_39_709Z-debug.log

  • The issue was fixed with a debug branch debug/test-with-stefano pushed to the remote repository. Meanwhile, the robot had to go to maintenance.

  • [Issue 3]: Later in the day, @S-Dafarra ran another trial with the debug branch. The issue observed earlier was fixed, no crash ever occurred. We could see the expected Yarplogger messages on the visualiser, but there was a glitch though:

    • As we ran some rpc command (e.g. calib all 300) triggering Yarplogger messages we could not see the messages right away.
    • Later on, we could only see the messages if we clicked on another entry in the left pane, then on the Yarp Text Logging entry back again.
    • There is no log for this issue.

Issues recap

  • [Issue 1]: Running the telemetry server from the folder ./iCubTelemVizServer with the node command fails
  • [Issue 2]: Telemetry server crashes at startup when parsing <robot>/left_foot_heel_tiptoe/measures:o port FT data.
  • [Issue 3]: We can't see the Yarplogger messages right away as they are received on the yarpjs input port, unless we click away from the Yarp Text Logging entry and back again.

Originally posted by @nunoguedelha in https://github.com/ami-iit/component_ANA-Avatar-XPRIZE/issues/760#issuecomment-1266616475

nunoguedelha commented

Issue still occurred in commit 2c3aa4b of branch main.
Not reproducible on latest commit of branch main: 390b246.

Checking commit 2c3aa4b, for making sure future code doesn't reproduce accidentally the code branching or processing causing this issue...

nunoguedelha commented

The command node iCubTelemVizServer.js starts the server but the server crashes right after with a segmentation fault Segmentation fault: 11.

This is not the same failure reported in this issue description. The log reported in the description was either wrong or obtained while some issue with the IMU port data occurred on the real robot (not reproduced).

We shall analyse here the only failure reproduced so far, i.e. the segmentation fault, which log is listed below.

Full log 
$ node iCubTelemVizServer.js
iCub Telemetry server launched!
[INFO] |yarp.os.Port|/yarpjs/inertial:i| Port /yarpjs/inertial:i active at tcp://192.168.1.70:10106/
[INFO] |yarp.os.Port|/yarpjs/head/inertials:i| Port /yarpjs/head/inertials:i active at tcp://192.168.1.70:10107/
[INFO] |yarp.os.Port|/yarpjs/leftArm/inertials:i| Port /yarpjs/leftArm/inertials:i active at tcp://192.168.1.70:10108/
[INFO] |yarp.os.Port|/yarpjs/rightArm/inertials:i| Port /yarpjs/rightArm/inertials:i active at tcp://192.168.1.70:10109/
[INFO] |yarp.os.Port|/yarpjs/leftLeg/inertials:i| Port /yarpjs/leftLeg/inertials:i active at tcp://192.168.1.70:10110/
[INFO] |yarp.os.Port|/yarpjs/rightLeg/inertials:i| Port /yarpjs/rightLeg/inertials:i active at tcp://192.168.1.70:10111/
[INFO] |yarp.os.Port|/yarpjs/leftFoot/inertials:i| Port /yarpjs/leftFoot/inertials:i active at tcp://192.168.1.70:10112/
[INFO] |yarp.os.Port|/yarpjs/rightFoot/inertials:i| Port /yarpjs/rightFoot/inertials:i active at tcp://192.168.1.70:10113/
[INFO] |yarp.os.Port|/yarpjs/left_leg/stateExt:o| Port /yarpjs/left_leg/stateExt:o active at tcp://192.168.1.70:10114/
[INFO] |yarp.os.Port|/yarpjs/right_leg/stateExt:o| Port /yarpjs/right_leg/stateExt:o active at tcp://192.168.1.70:10115/
[INFO] |yarp.os.Port|/yarpjs/left_arm/stateExt:o| Port /yarpjs/left_arm/stateExt:o active at tcp://192.168.1.70:10116/
[INFO] |yarp.os.Port|/yarpjs/right_arm/stateExt:o| Port /yarpjs/right_arm/stateExt:o active at tcp://192.168.1.70:10117/
[INFO] |yarp.os.Port|/yarpjs/torso/stateExt:o| Port /yarpjs/torso/stateExt:o active at tcp://192.168.1.70:10118/
[INFO] |yarp.os.Port|/yarpjs/head/stateExt:o| Port /yarpjs/head/stateExt:o active at tcp://192.168.1.70:10119/
[INFO] |yarp.os.Port|/yarpjs/camLeftEye:i| Port /yarpjs/camLeftEye:i active at tcp://192.168.1.70:10120/
[INFO] |yarp.os.Port|/yarpjs/camRightEye:i| Port /yarpjs/camRightEye:i active at tcp://192.168.1.70:10121/
[INFO] |yarp.os.Port|/yarpjs/left_arm/cartesianEndEffectorWrench:i| Port /yarpjs/left_arm/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10122/
[INFO] |yarp.os.Port|/yarpjs/right_arm/cartesianEndEffectorWrench:i| Port /yarpjs/right_arm/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10123/
[INFO] |yarp.os.Port|/yarpjs/left_upper_leg/cartesianEndEffectorWrench:i| Port /yarpjs/left_upper_leg/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10124/
[INFO] |yarp.os.Port|/yarpjs/left_lower_leg/cartesianEndEffectorWrench:i| Port /yarpjs/left_lower_leg/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10125/
[INFO] |yarp.os.Port|/yarpjs/right_upper_leg/cartesianEndEffectorWrench:i| Port /yarpjs/right_upper_leg/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10126/
[INFO] |yarp.os.Port|/yarpjs/right_lower_leg/cartesianEndEffectorWrench:i| Port /yarpjs/right_lower_leg/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10127/
[INFO] |yarp.os.Port|/yarpjs/left_foot_front/cartesianEndEffectorWrench:i| Port /yarpjs/left_foot_front/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10128/
[INFO] |yarp.os.Port|/yarpjs/left_foot_rear/cartesianEndEffectorWrench:i| Port /yarpjs/left_foot_rear/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10129/
[INFO] |yarp.os.Port|/yarpjs/right_foot_front/cartesianEndEffectorWrench:i| Port /yarpjs/right_foot_front/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10130/
[INFO] |yarp.os.Port|/yarpjs/right_foot_rear/cartesianEndEffectorWrench:i| Port /yarpjs/right_foot_rear/cartesianEndEffectorWrench:i active at tcp://192.168.1.70:10131/
[INFO] |yarp.os.Port|/yarpjs/left_arm/FT:i| Port /yarpjs/left_arm/FT:i active at tcp://192.168.1.70:10132/
[INFO] |yarp.os.Port|/yarpjs/right_arm/FT:i| Port /yarpjs/right_arm/FT:i active at tcp://192.168.1.70:10133/
[INFO] |yarp.os.Port|/yarpjs/left_leg_hip/FT:i| Port /yarpjs/left_leg_hip/FT:i active at tcp://192.168.1.70:10134/
[INFO] |yarp.os.Port|/yarpjs/right_leg_hip/FT:i| Port /yarpjs/right_leg_hip/FT:i active at tcp://192.168.1.70:10135/
[INFO] |yarp.os.Port|/yarpjs/left_foot_heel_tiptoe/FT:i| Port /yarpjs/left_foot_heel_tiptoe/FT:i active at tcp://192.168.1.70:10136/
[INFO] |yarp.os.Port|/yarpjs/right_foot_heel_tiptoe/FT:i| Port /yarpjs/right_foot_heel_tiptoe/FT:i active at tcp://192.168.1.70:10137/
[INFO] |yarp.os.Port|/yarpjs/battery/data:i| Port /yarpjs/battery/data:i active at tcp://192.168.1.70:10138/
[INFO] |yarp.os.Port|/yarpjs/walking-coordinator-logger/data:i| Port /yarpjs/walking-coordinator-logger/data:i active at tcp://192.168.1.70:10139/
[INFO] |yarp.os.Port|/yarpjs/proc-yarprobotinterface-yarptextlogging:i| Port /yarpjs/proc-yarprobotinterface-yarptextlogging:i active at tcp://192.168.1.70:10141/
[INFO] |yarp.os.Port|/yarpjs/proc-walkingmodule-yarptextlogging:i| Port /yarpjs/proc-walkingmodule-yarptextlogging:i active at tcp://192.168.1.70:10142/
[INFO] |yarp.os.Port|/yarpjs/sysCmdsGenerator/rpc| Port /yarpjs/sysCmdsGenerator/rpc active at tcp://192.168.1.70:10143/
{ status: 'OK', message: 'Opem-MCT static server process started.' }
info:Yarp command successfully started...
ICubTelemetry History Server listening on http://192.168.1.70:8081/history
ICubTelemetry Realtime Server listening on ws://192.168.1.70:8081/realtime
Control Console Server listening on http://192.168.1.70:3000
Yarp port names retrieval completed successfully (0)
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/inertial:i| Receiving input from /icubSim/inertial to /yarpjs/inertial:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/head/inertials:i| Receiving input from /icubSim/head/inertials/measures:o to /yarpjs/head/inertials:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_leg/stateExt:o| Receiving input from /icubSim/left_leg/stateExt:o to /yarpjs/left_leg/stateExt:o using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_leg/stateExt:o| Receiving input from /icubSim/right_leg/stateExt:o to /yarpjs/right_leg/stateExt:o using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_arm/stateExt:o| Receiving input from /icubSim/left_arm/stateExt:o to /yarpjs/left_arm/stateExt:o using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_arm/stateExt:o| Receiving input from /icubSim/right_arm/stateExt:o to /yarpjs/right_arm/stateExt:o using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/torso/stateExt:o| Receiving input from /icubSim/torso/stateExt:o to /yarpjs/torso/stateExt:o using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/head/stateExt:o| Receiving input from /icubSim/head/stateExt:o to /yarpjs/head/stateExt:o using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_arm/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_arm/cartesianEndEffectorWrench:o to /yarpjs/left_arm/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_arm/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_arm/cartesianEndEffectorWrench:o to /yarpjs/right_arm/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_upper_leg/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_upper_leg/cartesianEndEffectorWrench:o to /yarpjs/left_upper_leg/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_lower_leg/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_lower_leg/cartesianEndEffectorWrench:o to /yarpjs/left_lower_leg/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_upper_leg/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_upper_leg/cartesianEndEffectorWrench:o to /yarpjs/right_upper_leg/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_lower_leg/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_lower_leg/cartesianEndEffectorWrench:o to /yarpjs/right_lower_leg/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_foot_front/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_foot_front/cartesianEndEffectorWrench:o to /yarpjs/left_foot_front/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/left_foot_rear/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/left_foot_rear/cartesianEndEffectorWrench:o to /yarpjs/left_foot_rear/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_foot_front/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_foot_front/cartesianEndEffectorWrench:o to /yarpjs/right_foot_front/cartesianEndEffectorWrench:i using tcp
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/right_foot_rear/cartesianEndEffectorWrench:i| Receiving input from /wholeBodyDynamics/right_foot_rear/cartesianEndEffectorWrench:o to /yarpjs/right_foot_rear/cartesianEndEffectorWrench:i using tcp
[ERROR] |yarp.os.Network| Failure: no way to make connection, invalid source 'undefined'
[OPEN-MCT STATIC SERVER] ipc: {"pid":6977}
[INFO] |yarp.os.impl.PortCoreInputUnit|/yarpjs/proc-yarprobotinterface-yarptextlogging:i| Receiving input from /log/iiticublap199/yarprobotinterfa[yarprobotinterface]/3667 to /yarpjs/proc-yarprobotinterface-yarptextlogging:i using tcp
Segmentation fault: 11

If we list the running node processes (ps -ef | grep node), we can see that the openmctStaticServer server is still running while the telemetry server iCubTelemVizServer has stopped, so that was the process to crash ungracefully.

nunoguedelha commented

when adding a breakpoint to the line...

yarp-openmct/iCubTelemVizServer/openMctServerHandlerParentProc.js

Line 67 in 390b246

embeddedThis.outputCallback('[OPEN-MCT STATIC SERVER] ipc: ' + JSON.stringify(m));
...then continuing the execution, we get the same error reported in the issue description:

/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:282
            this.state[id].ori.roll = sensorSample[3][0][0][0];
                                                        ^

TypeError: Cannot read property '0' of undefined
    at ICubTelemetry.parseFromId (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:282:57)
    at Object.<anonymous> (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/icubtelemetry.js:166:31)
    at /Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/iCubTelemVizServer.js:74:56
    at /Users/nunoguedelha/dev/yarp-openmct/node_modules/YarpJS/yarp.js:188:13
    at callbackTrampoline (internal/async_hooks.js:134:14)
(robotologydeps3) nunoguedelha@iiticublap199 iCubTelemVizServer (main)
nunoguedelha commented

I've installed a stack tracer for Node.js, as explained here https://snyk.io/advisor/npm-package/node-segfault-handler#package-footer.

We get the following trace:

{ status: 'OK', message: 'Opem-MCT static server process started.' }
info:Yarp command successfully started...
ICubTelemetry History Server listening on http://192.168.1.70:8081/history
ICubTelemetry Realtime Server listening on ws://192.168.1.70:8081/realtime
Control Console Server listening on http://192.168.1.70:3000
=========== Caught a Segmentation Fault [pid=12043] ===========
-----[ Native Stacktraces ]-----
Cannot unwind stacktraces: Feature disabled / Missing libunwind on your system

---[ V8 JavaScript Stacktraces ]---
at (null) (/Users/nunoguedelha/dev/yarp-openmct/iCubTelemVizServer/iCubTelemVizServer.js:74:67)
at (null) (/Users/nunoguedelha/dev/yarp-openmct/node_modules/YarpJS/yarp.js:188:13)
============================================================
nunoguedelha commented

The trace shows that the segfault occurs always here...

yarp-openmct/iCubTelemVizServer/iCubTelemVizServer.js

Lines 73 to 74 in 2c3aa4b

portIn.onRead(function (bottle){
icubtelemetry.processOrDropYarpData[id](id,bottle.toArray());

...when calling processOrDropYarpData.

by adding a breaking to that line, we can see the systematic sequence leading to the segfault:
image

We break three times, each time we run bottle.toArray() in the console in order to check the bottle content. Last run crashes the debugger. This behaviour is systematic, which is good news.

nunoguedelha commented

The bottle with the left leg joint state data is causing the problem here (id = 'sens.leftLegJointState').

Analysing...

nunoguedelha commented

For the second call, with id = 'sens.headIMU' (M.A.S. sensor data),...

  • bottle.toString() returns (((-0.240112648394806865726 0.256409150712710476316 1.25491568355236471355) 4336.35999999999967258)) (((-0.152372687054658478845 -0.0560934592355309263878 -9.73042375287977989728) 4336.35999999999967258)) (((0.0 0.0 0.0) 4336.35999999999967258)) (((0.000133744890740640628643 -0.00548904149557477138294 2.25436842049552410572) 4336.35999999999967258)) () () () () () ()
  • bottle.toArray() returns Array(10) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0].

bottle.toArray() should return the same content instead of zeros. So, for nested bottles, either we get a wrong result or we get a segmentation fault. This is a YarpJS bug..

Moving the issue to YarpJS repo.

nunoguedelha commented

Side note: this problem occurred to @S-Dafarra a few times on his machine even when running npm start. If I remember correctly:

  • it was happening in this commit 2c3aa4b,
  • commit 47a887c was working initially without this issue, but then started failing also.

We had no clue on what was happening! We reproduced the issue here.

  • 👍1