Receiving and parsing NMEA2000 messages

Question

Receiving and parsing NMEA2000 messages

Closed this issue 6 months ago · 35 comments

Unlike NMEA0183, which uses a simple text format, NMEA2k messages are binary and unintelligible to the eye. Further, the binary forms is proprietary and only available to licence holders. However, the canboat project has reverse-engineered many of the PGNs and published PGN descriptors.

OCPN can receive NMEA2k messages and has built-in decoders for the common PGNs. These are used by the dashboard plugin.
Extending the JavaScript plugin to use all these built-in decoders would be significant work and a lot of code.

JavaScript is ideal for testing and experimentation, so rather than depending on the limited set of built-in decoders, I am aiming to decode any PGN for which there is a PGN descriptor within JavaScript.

There is a JavaScript version of canboat but it is large and complicated and relies heavily on the latest features of JavaScript not available in the Duktape JavaScript engine used in the plugin. It can be used for insights in how to do various bits but is not generally usable for our current purpose.

The task becomes:
Step 1: Extend the plugin to listen for NMEA2k messages and pass the raw payload to the script.
Step2: Write a function that will parse any payload for which there is a PGN descriptor and return the data as a JavaScript object.

Answer 1 · 2023-10-20T12:09:24.000Z

@duichan
Ready to start Step 1 testing - receiving NMEA2000 messages.
You will need build 9a03cee (20 Oct ca 09:030) or later.

Please run the following script

// capture samples of NMEA2000 data as returned to this plugin
// Alpha testing - details may change

pgns = [	   // pgns to sample
	129029, // GNSS Position Data
	129540, // GNSS Sats in View
	130306  // Wind Data
	];
timeLimit = 10;  // seconds

onSeconds(timeout, timeLimit);
count = pgns.length;
for (i = 0; i < pgns.length; i++){
	OCPNonNMEA2k(gotOne, pgns[i]);	// listen out
	}

function gotOne(input){
	print(count, "\t", input, "\n");
	if (count-- <= 1){	// all done
		onSeconds();	// cancel timer
		stopScript("All done");
		}
	}

function timeout(){
	scriptResult("Timed out with count=" + count);
	print(consoleDump());
	OCPNonNMEA2k();		// cancel outstanding listeners
	}

It sets up an array of pgns, listens out for them and displays the received data.
I need to see what this data looks like when received into the script.
The script will stay busy until all pgns have been received.
There is a timeout set of 10 seconds. If all messages have not been received in that time, it will dump internal stuff to help see what is going on. You can adjust the time as required.
You can amend or add to the pgn array if those I have chosen are not right for your set up.

I have run this script substituting all occurrences of OCPNonNMEA2k with OCPNonNMEA0183 and the pgns with NMEA0183 sentence types and it works as intended. I then cloned the OCPNonNMEA0183 code and adapted it for OCPNonNMEA2k. I have not been able to test this for lack of NMEA2k input.

The User Guide would help you understand the OCPN part of the plugin but does not include the new
OCPNonNMEA2k(functionToCall, pgn); // listen out for pgn and pass input to functionToCall
Calling it with no arguments will cancel outstanding listeners.

Answer 2 · 2023-10-22T17:42:01.000Z

Here is the result:

3 {"source":"nmea2000 socketCAN-can0","payload":[147,54,3,5,248,1,255,1,255,255,255,255,43,96,196,76,48,169,186,37,64,244,115,40,109,103,77,7,2,144,3,51,154,166,52,0,64,50,59,7,0,0,0,0,16,252,12,150,0,250,0,91,18,0,0,255,85]}
2 {"source":"nmea2000 socketCAN-can0","payload":[147,158,6,4,250,1,255,1,255,255,255,255,147,96,254,12,19,232,25,185,47,128,12,0,0,0,0,242,24,208,51,184,77,240,10,0,0,0,0,242,25,255,14,136,159,196,9,0,0,0,0,242,32,81,14,193,214,96,9,0,0,0,0,242,79,127,52,42,233,128,12,0,0,0,0,242,10,255,14,229,190,0,0,0,0,0,0,240,12,150,41,78,149,152,8,0,0,0,0,240,15,104,18,253,119,0,0,0,0,0,0,240,17,92,16,255,29,96,9,0,0,0,0,240,22,34,6,127,37,0,0,0,0,0,0,240,23,46,8,171,165,52,8,0,0,0,0,240,81,209,6,45,83,0,0,0,0,0,0,240,85]}
1 {"source":"nmea2000 socketCAN-can0","payload":[147,19,3,16,240,1,255,1,255,255,255,255,8,96,240,196,76,64,208,186,37,85]}
result: All done

Note that I have substituted PGN130306 in your script for PGN126992 (System Time) as I am testing this on the bench and don't have wind data available.

Answer 3 · 2023-10-22T18:45:23.000Z

Thanks @duichan for the test. That will keep me out of mischief for a good while while I work out how to decode that lot!

Answer 4 · 2023-10-24T14:04:36.000Z

I seem to be getting the N2k payload into the JavaScript but am having difficulty decoding it. Starting with the header section and abstracting from the third test above, I have

test3 = new Uint8Array([147,54, 3, 5,248,1, 255, 1, 255,255,255,255, 43,
		96,196,76,48,169,186,37,64,244,115,40,109,103,77,7,2,144,3,51,154,166,52,0,64,50,59,7,0,0,0,0,16,252,12,150,0,250,0,91,18,0,0,255,85]);

which in hex is

["93","36","03","05","F8","01","FF","01","FF","FF","FF","FF","2B","60","C4","4C","30","A9","BA","25","40","F4","73","28","6D","67","4D","07","02","90","03","33","9A","A6","34","00","40","32","3B","07","00","00","00","00","10","FC","0C","96","00","FA","00","5B","12","00","00","FF","55"]

Following what OCPN does in tN2kMsg MakeN2kMsg(std::vector<unsigned char> &v) I decode to get

Test 3	PGN should be 126992	System time
{
	"prio": 3,
	"PGN": 129025,
	"dest": 255,
	"src": 1,
	"MsgTime": -1,
	"dataLength": 43
}

The data length is correct and the first byte x93 is expected, so it is looking good.
But I find a PGN of 129025 when it should be 126992. I am masking it with 0x3FFFF because a pgn is only 11 bits.
I still cannot match the pgn being listened to. MakeN2kMsg seems to just take bytes 3-5 (counting from zero) but I have found no way of making that what I expect.
I read that pgns have an structure that can change what bytes are what but I do not see that being accounted for in OCPN.

What am I missing, anyone?

Answer 5 · 2023-10-24T18:01:47.000Z

You are decoding the wrong line.
PGN126992 is contained in this line:
1 {"source":"nmea2000 socketCAN-can0","payload":[147,19,3,16,240,1,255,1,255,255,255,255,8,96,240,196,76,64,208,186,37,85]}
since 16,240,1 -> 126992
so the date is 196,76 -> 19652 days since epoch -> 2023-10-22
and the time is 64,208,186,37 -> 63300.0000 secs -> 17:35:00

Answer 6 · 2023-10-24T18:09:49.000Z

Likewise PGN129029 is in the first displayed line (5,248,1) and PGN129540 in the second (4,250,1).

Answer 7 · 2023-10-25T08:57:39.000Z

Yes - was looking out for that but PGNs still wrong. Now discovered multibyte values are littleendeon - now working.
Can now move forward.

Answer 8 · 2023-11-02T14:45:56.000Z

@duichan I have a first version of a generalised parser although still "work in progress". Please
(1) download the N2K.zip attached, decompress and place the folder somewhere and name it, say, "N2k".
(2) In the plugin tools, set the current directory to be that folder.
(3) Run the first script below. It will listen out for the three pgns as before but should parse them and display as a JavaScript object and also display some of the info, such as position and time. Please let me have the output.
(4) Run the second script. This should listen out for 20 seconds for all the pgns known to the canboat project and list the ones received on your system.

N2K.zip

First script

// capture samples of NMEA2000 data as returned to this plugin
// Alpha testing - details may change

Nmea2kConstructor = require("NMEA2000.js");
Position = require("Position");

pgns = [	   // pgns to sample
	129029, // GNSS Position Data
	129540, // GNSS Sats in View
	126992  // System time
	];
timeLimit = 10;  // seconds

onSeconds(timeout, timeLimit);

OCPNonNMEA2000(gotTime, 126992);	// listen for system time
OCPNonNMEA2000(gotInView, 129540);	// listen for sats in view
OCPNonNMEA2000(gotPosition, 129029);	// listen for position


function gotTime(input){
	NMEA2k = new Nmea2kConstructor(input);
	print(JSON.stringify(NMEA2k, null, "\t"), "\n");
	print("System time: ",  new Date(NMEA2k.date * 24*60*60*1000 + NMEA2k.time*1000), "\n");
	}

function gotInView(input){
	NMEA2k = new Nmea2kConstructor(input);
	print(JSON.stringify(NMEA2k, null, "\t"), "\n");
	print("Satellites in view:", NMEA2k.satsInView, "\n");
	}

function gotPosition(input){
	NMEA2k = new Nmea2kConstructor(input);
	print(JSON.stringify(NMEA2k, null, "\t"), "\n");
	position = new Position(NMEA2k.latitude, NMEA2k.longitude);
	print("Position: ", position.formatted, "\n");
	print("Fix time: ", new Date(NMEA2k.date * 24*60*60*1000 + NMEA2k.time*1000), "\n");
	print("using ", NMEA2k.numberOfSvs, " satellites\n");
	}

function timeout(){
	scriptResult("Timed out");
	print(consoleDump());
	OCPNonNMEA2000();		// cancel outstanding listeners
	}

List all pgns

descriptors = require("pgnDescriptors.js")();
timeLimit = 20;  // seconds

pgns= [];

for (d = 0; d < descriptors.length; d++){
	pgn = descriptors[d].PGN;
	OCPNonNMEA2000(gotOne, pgn);
	}
onSeconds(timesUp, timeLimit);

function gotOne(result){
	payload = result.payload;
	pgn = getBytes(payload, 3, 3);	// extract the pgn
	pgns.push(pgn);
	}

function getBytes(v, start, bytes){	// little endean!
	offset = start+bytes-1;
	result = v[offset--];
	for ( ; offset >= start; offset--){
		// result = (result << 8) | v[offset];  shift uses 32 bits, so to be 64 bit safe we use...
		result = (result * 256) + v[offset];
		}
	return result;
	};

function timesUp(){
	OCPNonNMEA2000();	// cancel outstanding ones
	print(pgns, "\n");
	}

Answer 9 · 2023-11-02T16:42:19.000Z

Scripts as they stand both return: ReferenceError: identifier 'OCPNonNMEA2000' undefined
However changing the call to OCPNonNMEA2k the results are as follows:

SCRIPT1
Console name: JavaScript
this: 0x0003987300
mpNextConsole: nullptr
mStatus: (no status)
mRunningMain: false
m_time_to_allocate: 1000ms
m_timeout_check_counter: 1
mWaitingCached: true
mWaiting: true
mpMessageDialog: nullptr
isBusy(): true
isWaiting(): true
auto_run: false
position: screen x:0 y:64 DIP x:0 y:64
Size(): x:1920 y:1044 DIP x:1920 y:1044 MinSize() x:159 y:30
m_parked: false position x:0 y:0
isParked(): false
Messages callback table
OCPN_CORE_SIGNALK
OCPN_OPENGL_CONFIG
OpenCPN Config
m_timerActionBusy: true
Timers callback table
(empty)
Menus callback table
(empty)
m_dialog: None
m_alert: None
m_NMEAmessageFunction:
m_streamMessageCntlsVector
messageCntlId: 145527636
messageType: 2
id0183:
id2k: 126992
functionName: gotTime
--------------
messageType: 2
id0183:
id2k: 129540
functionName: gotInView
--------------
messageCntlId: 1686795594
messageType: 2
id0183:
id2k: 129029
functionName: gotPosition
m_activeLegFunction:
m_exitFunction:
m_explicitResult: true
m_result: Timed out
mscriptToBeRun: false
No brief
mConsoleRepliesAwaited 0
Duktape context dump:
ctx: top=0, stack=[]
result: Timed out messageCntlId: 2122763275

SCRIPT2
[]
result: undefined

Answer 10 · 2023-11-02T17:17:32.000Z

Whoops again.... yes I have changed API from OCPNonNMEA2k to OCPNonNMEA2000. Please update to latest alpha 9e256d0 from the repository. Then you can use as I had them.

Both scripts are timing out. Are you sure you have NMEA2000 data coming in?
If you go back to the original scripts I send, they should be work as before.

Answer 11 · 2023-11-02T17:34:40.000Z

Whoops me too. I unplugged the N2k network and forgot to plug it back again. So....

SCRIPT1
{
"PGN": 126992,
"prio": 3,
"dest": 255,
"src": 1,
"MsgTime": "undefined",
"description": "System Time",
"sid": 117,
"source": "invalid",
"date": 19663,
"time": 63115
}
System time: "2023-11-02T17:31:55.000Z"
{
"PGN": 129029,
"prio": 3,
"dest": 255,
"src": 1,
"MsgTime": "undefined",
"description": "GNSS Position Data",
"sid": 118,
"date": 19663,
"time": 63115,
"latitude": 52.6188915,
"longitude": 1.4820791666666666,
"altitude": 70.13,
"gnssType": "GLONASS",
"method": "no GNSS",
"integrity": "invalid",
"numberOfSvs": 12,
"hdop": 1.1,
"pdop": 1.7,
"geoidalSeparation": 46.99,
"referenceStations": "undefined",
"information": "Only first of 15 repeats of following decoded"
}
Position: 52° 37.133'N 001° 28.925'E
Fix time: "2023-11-02T17:31:55.000Z"
using 12 satellites
{
"PGN": 129540,
"prio": 6,
"dest": 255,
"src": 1,
"MsgTime": "undefined",
"description": "GNSS Sats in View",
"sid": 118,
"rangeResidualMode": "invalid",
"satsInView": 12,
"information": "Only first of 4 repeats of following decoded",
"prn": 10,
"elevation": 0.2094,
"azimuth": 4.607600000000001,
"snr": 21,
"rangeResiduals": 0,
"status": "invalid"
}
Satellites in view:12
Console name: JavaScript
this: 0x00072a0d18
mpNextConsole: nullptr
mStatus: (no status)
mRunningMain: false
m_time_to_allocate: 1000ms
m_timeout_check_counter: 1
mWaitingCached: true
mWaiting: true
mpMessageDialog: nullptr
isBusy(): true
isWaiting(): true
auto_run: false
position: screen x:-2 y:34 DIP x:-2 y:34
Size(): x:1920 y:1044 DIP x:1920 y:1044 MinSize() x:159 y:30
m_parked: false position x:0 y:0
isParked(): false
Messages callback table
OCPN_CORE_SIGNALK
OCPN_OPENGL_CONFIG
OpenCPN Config
m_timerActionBusy: true
Timers callback table
(empty)
Menus callback table
(empty)
m_dialog: None
m_alert: None
m_NMEAmessageFunction:
m_streamMessageCntlsVector
m_activeLegFunction:
m_exitFunction:
m_explicitResult: true
m_result: Timed out
mscriptToBeRun: false
No brief
mConsoleRepliesAwaited 0
Duktape context dump:
ctx: top=0, stack=[]
result: Timed out

SCRIPT2:
[129025,130824,130824,130310,130311,65280,129026,126992,127258,128267,129029,129283,129284,129539,129540,130312,129285,126993]
result: undefined

Answer 12 · 2023-11-02T18:45:45.000Z

Looking good so far, apart from getting the dump unnecessarily - my bad.
Issue for me is that I have not had opportunity to check decoding of negative numbers. Pity you are not west of Greenwich!

So I now have a list of the PGNs your system receives. I have hence discovered there are two different descriptions for PGN 130824. That really complicates things.

Here is a script that should display a dump for each PGN you receive. This will let me explore the decoding further.
Thanks for your help with this.

pgns = [129025,130824,130310,130311,65280,129026,126992,127258,128267,129029,129283,129284,129539,129540,130312,129285,126993];

descriptors = require("pgnDescriptors.js")();
pgns = pgns.sort();
for (p = 0; p < pgns.length; p++){
	pgn = pgns[p];
	OCPNonNMEA2000(gotOne, pgn);
	}

function gotOne(result){
	payload = result.payload;
	pgn = getBytes(payload, 3, 3);	// extract the pgn
	for (d = 0; d < descriptors.length; d++){
		descriptor = descriptors[d];
		if (descriptor.PGN == pgn){
			print(pgn, "\t", descriptor.Description, "\n");
			break;
			}
		}
	print("\t", payload, "\n\n");
	}

function getBytes(v, start, bytes){	// little endean!
	offset = start+bytes-1;
	result = v[offset--];
	for ( ; offset >= start; offset--){
		// result = (result << 8) | v[offset];  shift uses 32 bits, so to be 64 bit safe we use...
		result = (result * 256) + v[offset];
		}
	return result;
	};

Answer 13 · 2023-11-02T18:57:04.000Z

We've been trying to move west for some time, but the house market stagnating the way it is we haven't had much luck!
PGN130824 is a proprietary B&G(Navico) code so I'm not sure what it does.

Anyway here is the latest:
129025 Position, Rapid Update
[147,19,2,1,248,1,255,1,255,255,255,255,8,122,0,93,31,121,38,226,0,85]

65280 Furuno: Heave
[147,19,2,0,255,0,255,0,255,255,255,255,8,19,153,4,5,0,0,2,0,85]

130310 Environmental Parameters (obsolete)
[147,19,5,6,253,1,255,2,255,255,255,255,8,255,255,255,255,255,255,255,255,85]

129026 COG & SOG, Rapid Update
[147,19,2,2,248,1,255,1,255,255,255,255,8,13,252,196,73,20,0,255,255,85]

130311 Environmental Parameters
[147,19,5,7,253,1,255,2,255,255,255,255,8,255,255,255,255,255,127,255,255,85]

128267 Water Depth
[147,19,3,11,245,1,255,2,255,255,255,255,8,255,255,255,255,255,255,127,255,85]

129029 GNSS Position Data
[147,54,3,5,248,1,255,1,255,255,255,255,43,13,207,76,208,3,94,40,64,255,227,168,208,101,77,7,254,135,31,80,154,167,52,0,176,130,5,4,0,0,0,0,16,252,12,24,1,213,1,91,18,0,0,255,85]

129283 Cross Track Error
[147,19,3,3,249,1,255,3,255,255,255,255,8,13,48,255,255,255,127,255,255,85]

129284 Navigation Data
[147,45,3,4,249,1,255,3,255,255,255,255,34,13,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,127,255,255,255,127,255,127,85]

129540 GNSS Sats in View
[147,158,6,4,250,1,255,1,255,255,255,255,147,13,254,12,25,33,51,101,183,28,12,0,0,0,0,242,28,115,20,147,206,240,10,0,0,0,0,242,32,162,28,241,177,172,13,0,0,0,0,242,66,46,23,124,202,84,11,0,0,0,0,242,76,34,21,101,198,40,10,0,0,0,0,242,75,115,50,11,17,96,9,0,0,0,0,242,65,162,13,241,162,40,10,0,0,0,0,242,3,23,4,53,235,0,0,0,0,0,0,240,6,174,15,115,35,0,0,0,0,0,0,240,11,174,15,10,62,0,0,0,0,0,0,240,12,150,41,22,49,0,0,0,0,0,0,240,24,255,14,103,93,0,0,0,0,0,0,240,85]

130824 B&G: key-value data
[147,13,3,8,255,1,255,3,255,255,255,255,2,125,153,85]

126992 System Time
[147,19,3,16,240,1,255,1,255,255,255,255,8,13,240,207,76,208,3,94,40,85]

127258 Magnetic Variation
[147,19,7,26,241,1,255,1,255,255,255,255,8,14,255,255,255,151,0,255,255,85]

129285 Navigation - Route/WP Information
[147,50,7,5,249,1,255,3,255,255,255,255,39,255,255,2,0,255,255,255,255,231,3,1,0,0,0,0,3,1,0,255,255,255,127,255,255,255,127,1,0,3,1,0,255,255,255,127,255,255,255,127,85]

130312 Temperature
[147,19,5,8,253,1,255,2,255,255,255,255,8,4,0,0,255,255,255,255,255,85]

126993 Heartbeat
[147,19,6,17,240,1,255,4,255,255,255,255,8,96,234,1,255,255,255,255,255,85]

129539 GNSS DOPs
[147,19,6,3,250,1,255,1,255,255,255,255,8,13,208,24,1,124,1,255,127,85]

Answer 14 · 2023-11-02T20:19:11.000Z

To generate a negative number I have just set up a route from Poole Bar. The result with PGN129285 is as follows:
129285 Navigation - Route/WP Information
[147,65,7,5,249,1,255,3,255,255,255,255,54,255,255,2,0,255,255,255,255,224,9,1,83,69,78,48,48,52,0,91,0,0,3,1,0,255,255,255,127,255,255,255,127,1,0,12,1,80,111,111,108,101,32,66,97,114,0,88,80,49,30,185,47,219,254,85]

This should decode thus:
129285 Navigation - Route/WP Information: Start RPS# = Unknown; nItems = 2; Database ID = Unknown; Route ID = Unknown; Navigation direction in route = Forward; Supplementary Route/WP data available = Off; Route Name = SEN004; Reserved = 67; WP ID 1 = 0; WP Name 1 = Unknown; WP Latitude 1 = Unknown; WP Longitude 1 = Unknown; WP ID 2 = 1; WP Name 2 = Poole Bar; WP Latitude 2 = 50.6548312; WP Longitude 2 = -1.9189831

Answer 15 · 2023-11-02T21:04:19.000Z

Thanks hugely. This and the previous will take me time to digest and work on.

Answer 16 · 2023-11-02T21:06:08.000Z

Good luck!

Answer 17 · 2023-11-04T10:34:41.000Z

Your worked examples are helping me understand this stuff. But your understanding is clearly better than mine. For your last 129285 example we have:

[147,65,7,5,249,1,255,3,255,255,255,255,54,
	255,255,	// Start RPS#	unknown
	2,0,		// nItems 2
	255,255,	// databaseId	unknown
	255,255,	// route ID	unknown
	224,		// 0xE009 navigationDirectionInRoute (forward) + supplementaryRouteWpDataAvailable (off)
	9,1,83,69,78,48,48,52,0,	// "SEN004"
	91,		// reserved	0x5B  67 HOw does this make 67???
	0,0,		// WP ID 1		0
	3,1,0,	// WP Name1
	255,255,255,127,	// WP Latitude 1		unknown
	255,255,255,127,	// WP Longitude 1	unknown
	1,0,		// WP ID 2		0
	12,1,80,111,111,108,101,32,66,97,114,0,	// Poole Bar
	88,80,49,30,	// WP Latitude 2 = 50.6548312
	185,47,219,254,	// WP Longitude 2 = -1.9189831
	85]

Questions:

(1) Line 6: you have 224 giving Navigation direction in route = Forward; Supplementary Route/WP data available = Off;
The descriptor for this is

		{
			"Order": 5,
			"Id": "navigationDirectionInRoute",
			"Name": "Navigation direction in route",
			"BitLength": 3,
			"BitOffset": 64,
			"BitStart": 0,
			"Type": "Lookup table",
			"Resolution": 1,
			"Signed": false,
			"RangeMin": 0,
			"RangeMax": 6,
			"EnumValues": [
				{
					"name": "Forward",
					"value": 0
				},
				{
					"name": "Reverse",
					"value": 1
				}
			]
		},
		{
			"Order": 6,
			"Id": "supplementaryRouteWpDataAvailable",
			"Name": "Supplementary Route/WP data available",
			"BitLength": 2,
			"BitOffset": 67,
			"BitStart": 3,
			"Type": "Lookup table",
			"Resolution": 1,
			"Signed": false,
			"RangeMin": 0,
			"RangeMax": 2,
			"EnumValues": [
				{
					"name": "Off",
					"value": 0
				},
				{
					"name": "On",
					"value": 1
				}
			]
		},

224 is binary 1110 0000
The navigational direction is bits 1-3 which are all on, so I see it as 'unavailable' not Forward ???

(2) Reserved byte 91 you have interpreted as 67. Wondering how you got this? Reserved bits are supposed to be all on. and unused bits off.

(3) Variable length strings are of two possible types

STRING_LZ - A varying length string containing single byte codepoints encoded with a length byte and terminating zero.

It is unclear what character sets are allowed/supported. Possibly UTF-8 but it could also be that only ASCII values are supported.

Encoding: The length of the string is determined by a starting length byte. It also contains a terminating zero byte. The length byte includes the zero byte but not itself.

STRING_LAU - A varying length string containing double or single byte codepoints encoded with a length byte and terminating zero.

It is unclear what character sets are allowed/supported. For single byte, assume ASCII. For UNICODE, assume UTF-16, but this has not been seen in the wild yet.

Encoding: The length of the string is determined by a starting length byte. The 2nd byte contains 0 for UNICODE or 1 for ASCII.

It looks like these are STRING_LAU, as there is a two byte header. STRING_LZ explicitly states that the count includes the zero terminator but not itself. These counts seem to include the count itself.

(4) Negative numbers. I have

NUMBER - Number

Encoding: Binary numbers are little endian. Number fields that use two or three bits use one special encoding, for the maximum value. When present, this means that the field is not present. Number fields that use four bits or more use two special encodings. The maximum positive value means that the field is not present. The maximum positive value minus 1 means that the field has an error. For instance, a broken sensor. For signed numbers the maximum values are the maximum positive value and that minus 1, not the all-ones bit encoding which is the maximum negative value.

Can you show me how 185,47,219,254 gets to be -19189831 ?

Thanks

Answer 18 · 2023-11-04T12:18:22.000Z

Decoding NMEA2000 would tax the brains of GCHQ, and dealing with little endian doesn't help. Maybe that's what the NMEA committee intended?

FYI I did the decoding by running it through the Canboat analyzer:
candump can0 | candump2analyzer | analyzer

To answer your specific questions:
(1) reserved E0 | supp << 3 | dirn => E0
(2) Good question - I will investigate further
(3) The first byte is the count. The second byte is 0 for Unicode or 1 for ASCII. The last byte is the zero terminator. The count does indeed include everything from itself to the terminator. Maybe STRING_LAU differs from STRING_LZ in this respect?
(4) 185,47,219,254 reversed and as hex => FEDB2F89
converting from twos comp ~FEDB2F89 + 1 => -1918931

Answer 19 · 2023-11-04T12:33:11.000Z

I have just run a much longer candump analysis trying various different routes. It seems that reserved byte 91 can take all sorts of values. So I suspect that it has a function that Canboat has not discovered. That your javascript recorded a value of 91 whereas my decoding reported 67 is explained by the fact that these tests had to be done sequentially, not simultaneously.

Answer 20 · 2023-11-04T12:44:33.000Z

(1) reserved E0 | supp << 3 | dirn => E0
Ah! So not only are the whole bytes little endian but the bits count from right to left - bit 1 is the little-est. I was counting left to right within a byte.

Answer 21 · 2023-11-04T12:46:17.000Z

That's right!

Answer 22 · 2023-11-04T22:45:39.000Z

I am positing that within a given 'nibble' (a section of bits) the value is still left-most bit most significant.
So in the case above reserved E0 | supp << 3 | dirn, if we had 1100 1011, the first three bits are '011' which has a value of 3. We do not need to reverse the bits to 110 which would give a value of 6. The next 2 bits are `01' evaluating to 1 . We do not reverse the bits order to get 2.

I am assuming this because, with a multi-byte number such as 123, 456, we evaluate this as 456*256 + 123. We do not reverse the bit order within each byte.

Answer 23 · 2023-11-05T00:51:01.000Z

I've stumbled across this from your associated post on the OpenCPN repo.

A couple of observations:

Don't reinvent the wheel. OpenCPN makes use of a modified version of Timo's libraries. https://github.com/ttlappalainen/NMEA2000 You'll probably find O's version in O's repo under libraries. My TwoCan plugin has my own set of NMEA 2000 <--> NMEA 183 encoding/decoding routines that support a larger number of PGN's The Engine Dashboard also decodes engine and fluid level PGN's.
Values are of different types, bit values char, short, integer even a few long longs, in both signed and unsigned form and strings The byte ordering is LSB, but for bit masked values the bit ordering is MSB. (high order bit is leftmost).Most strings begin with a length byte, encoding byte (Unicode or ASCII) and then the unterminated string.
There are many proprietary PGN's, you will need to parse the initial contents in order to determine what to do with them.
Finally, on a personal note, I'm kind of interested in understanding the aim for NMEA 2000 support in your JavaScript plugin
Good luck.

Answer 24 · 2023-11-05T01:23:01.000Z

I should add, that if you just want to spit out json, then you could probably just shell out to canboat's analyzer. This is an example using I think a text entry from a canboat log. Not sure if you could just pass a text version of the Actisense format that OpenCPN spits out,

echo "2009-06-18Z09:46:01.129,2,129025,2,255,8,98,56,72,1f,65,a8,3b,03" | analyzer -json {"version":"4.12.0","units":"std","showLookupValues":false} INFO 2023-11-05T01:16:32.489Z [analyzer] Assuming normal format with one line per frame {"timestamp":"2009-06-18Z09:46:01.129","prio":2,"src":2,"dst":255,"pgn":129025,"description":"Position, Rapid Update","fields":{"Latitude":52.7586968,"Longitude": 5.4241381}}

As I said, no need to reinvent the wheel....

Answer 25 · 2023-11-05T16:49:57.000Z

I am positing that within a given 'nibble' (a section of bits) the value is still left-most bit most significant
As TwoCanPlugin has already confirmed, this is indeed the case.

Answer 26 · 2023-11-11T18:32:03.000Z

@duichan You may have seen the discussion about dropping the Actisense header.
In anticipation of this, I have reworked OCPNonNMEA2000 to do this and return the data , the pgn and source as separate parameters. This is easier for the user to understand.
Please update your JavaScript_pi to build 4706291 and test with the following script, which will run for 15 seconds. As well as testing, this will give me the new data format to adapt the parser to suite.
Thanks.

pgns = [129025,130310,130311,65280,129026,126992,127258,128267,129029,129283,129284,129539,129540,130312,129285,126993];

getDescriptor = require("pgnDescriptors.js");
pgns = pgns.sort();
for (p = 0; p < pgns.length; p++){
	pgn = pgns[p];
	OCPNonNMEA2000(gotone, pgn);
	}
onSeconds(timesup, 15);

function gotone(data, pgn, source){
	name = getDescriptor(pgn).Description;
	print(pgn, "\t", name, "\t", source, "\n", data, "\n\n");
	}

function timesup(){
	OCPNonNMEA2000();
	}

Answer 27 · 2023-11-11T19:18:36.000Z

Yes I have been following that discussion with interest.
Here is the latest output:
129025 Position, Rapid Update nmea2000 socketCAN-can0
[133,1,93,31,240,40,226,0]

130310 Environmental Parameters (obsolete) nmea2000 socketCAN-can0
[255,255,255,255,255,255,255,255]

130311 Environmental Parameters nmea2000 socketCAN-can0
[255,255,255,255,255,127,255,255]

129026 COG & SOG, Rapid Update nmea2000 socketCAN-can0
[22,252,0,0,5,0,255,255]

65280 Furuno: Heave nmea2000 socketCAN-can0
[19,153,4,5,0,0,2,0]

129283 Cross Track Error nmea2000 socketCAN-can0
[21,48,255,255,255,127,255,255]

129284 Navigation Data nmea2000 socketCAN-can0
[21,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,255,127,255,255,255,127,255,127]

129285 Navigation - Route/WP Information nmea2000 socketCAN-can0
[255,255,2,0,255,255,255,255,231,3,1,0,255,0,0,3,1,0,255,255,255,127,255,255,255,127,1,0,3,1,0,255,255,255,127,255,255,255,127]

129539 GNSS DOPs nmea2000 socketCAN-can0
[22,208,120,0,170,0,255,127]

129540 GNSS Sats in View nmea2000 socketCAN-can0
[22,254,12,6,220,8,22,19,184,11,0,0,0,0,242,25,115,50,150,56,116,14,0,0,0,0,242,28,45,38,101,198,72,13,0,0,0,0,242,31,209,21,240,207,184,11,0,0,0,0,242,32,174,15,218,158,96,9,0,0,0,0,242,68,92,16,112,215,252,8,0,0,0,0,242,86,138,39,254,59,140,10,0,0,0,0,242,85,185,17,115,20,196,9,0,0,0,0,242,11,197,19,150,41,108,7,0,0,0,0,240,12,46,23,161,58,52,8,0,0,0,0,240,29,231,40,67,132,140,10,0,0,0,0,240,26,116,5,217,188,0,0,0,0,0,0,240]

126992 System Time nmea2000 socketCAN-can0
[22,240,216,76,64,170,85,41]

127258 Magnetic Variation nmea2000 socketCAN-can0
[23,255,255,255,151,0,255,255]

128267 Water Depth nmea2000 socketCAN-can0
[255,255,255,255,255,255,127,255]

129029 GNSS Position Data nmea2000 socketCAN-can0
[23,216,76,80,209,85,41,0,92,124,233,15,102,77,7,0,206,250,128,41,168,52,0,32,238,247,3,0,0,0,0,16,252,12,120,0,209,0,91,18,0,0,255]

130312 Temperature nmea2000 socketCAN-can0
[133,0,0,255,255,255,255,255]

Answer 28 · 2023-11-15T08:16:28.000Z

@duichan I would like to check how the source relates to OCPN driver handles.
Please run the following script and let me have the output. Thanks.

handles = OCPNgetActiveDriverHandles();
for (h = 0; h < handles.length; h++){
	attributes = OCPNgetDriverAttributes(handles[h]);
	print(handles[h], "\n\t",attributes, "\n");
	}

Answer 29 · 2023-11-17T12:54:26.000Z

Hi Tony
Sorry for the delay - broadband from our local exchange has been out for two days!
Anyway here is what you wanted:

nmea2000!@!socketCAN-can0
{"canAddress":"72","canPort":"can0","protocol":"nmea2000"}

Regards
Andrew

Answer 30 · 2023-11-17T17:29:13.000Z

So std::string GetN2000Source(NMEA2000Id id, ObservedEvt ev) returns the source as nmea2000 socketCAN-can0
but std::vector<DriverHandle> GetActiveDrivers() returns it as nmea2000!@!socketCAN-can0.

We are told not to interpret what is in this string - just use it as a label. But this discrepancy complicates writing back to the source of a message. Any thoughts on how the !@! got there? Likely a non-Ascii character that gets mangled as it comes through? @leamas ?

One would have to identify the stream by protocol, which would work as long as there is only one nmea2000 stream.

Answer 31 · 2023-11-22T18:49:45.000Z

Latest alpha build 2d9a7af has:

The JavaScript 'require's are now built-in. You can dispense with the external folder of stuff.
I've gone back to returning the Actisense header before the N2K data as it may be useful when replying. Payload[7] is supposed to contain the source address - useful when replying.

I have attached the user guide covering the N2K work.
p15 covers the OCPNonNMEA2000 API
pp49-50 explains how to use the NMEA2000 object to decode messages using its built-in decoder.
p51 explains how to use the canboat analyzer instead of the built-in decoder.
Appendix A (pp63-64) covers the different results from the two methods.

@duichan If you can find the time, I would appreciate it if you could play around with this and let me have feedback, including if the documentation is not clear.
JavaScript_plugin_user_guide.pdf

Answer 32 · 2023-12-12T10:34:19.000Z

Running the first part of your example on p49 gives the result expected. The second part however returns an error:
[anon] line 65 TypeError: cannot read property 12 of undefined
called from handle129029 line 6
The code on p50 works fine. I haven't tested the canboatAnalyzer function yet but will do so shortly.

There are a few very minor typos in the documentation.
page 49: Sets up a function to process (delete: to handle) the next payload for the given PGN.
page 49: However, a few projects have worked out how the various PGNs are enclosed. (should be: encoded)
page 49: ... can be used to decode NMEA messages into JavaSCript (should be: JavaScript) objects
page 50: You could decide which on (should be: of) these is correct for your installation and construct using the relevant descriptor.

Answer 33 · 2023-12-19T17:36:31.000Z

@duichan An updated alpha build commit b8e07d6.
In addition to decoding NMEA2000 payloads, this version can also reverse that and encode a JavaSCript object into a payload.
Here is a test script that exercises the decoding and encoding - and checks we get back what we started with. I have only tested it on MacOS so please confirm it works on Linux. That would be reassuring.
Lines 2-4 let you also test using the canboat analyser as an extra goody.
NMEA2kTests.js.zip

Assuming all is OK, we now move on to see if we can push NMEA2000 out. Will pick that up in this issue.
Season's greetings.

Answer 34 · 2024-01-09T17:07:21.000Z

@duichan So we need to be able to listen persistently as well as one-time. I have now implemented two versions of listening

NMEAonNMEA2000()
NMEAonAllNMEA2000()
Also, I have adopted a change to allow for the dummy CRC to be dropped when that is implemented.
Alpha build 6f1d1ed has the changes and the script attached exercises both calls and returns a sample of data so I can check the fix for the CRC works. Please can you run it? I hope I have the script correct. I hope this is the last test!
Thank you.
Test_NMEA2000_one-off_and_All.js.zip

Answer 35 · 2024-01-09T19:54:02.000Z

This looks good to me, well done!

One-off response pgn:126992 source:nmea2000 socketCAN-can0 payload:[147,19,3,16,240,1,255,1,255,255,255,255,8,78,240,19,77,80,104,159,42]

Time is up. Responses 5

Source ProdCode ModelId softwareVersionCode modelSerialCode Certification loadEquivalency
0 25492 Vulcan 5 MFD 01000_E 18.3.61.1.155 001318# 2 1
1 25492 Vulcan 5 iGPS 01000_E 18.3.61.1.155 001318# 2 0
2 25492 Vulcan 5 Echo (This unit) 01000_E 18.3.61.1.155 001318# 2 0
4 1770 TwoCan Plugin 2.1 1219876 0 1
3 25492 Vulcan 5 Navigator 01000_E 18.3.61.1.155 001318# 2 0
result: undefined