Using Pm_Write to send System Exclusive inserts extraneous zeros

Question

Using Pm_Write to send System Exclusive inserts extraneous zeros

Closed this issue 3 years ago · 5 comments

Hi Roger,

I've been doing some further testing of my Pharo Smalltalk binding to PortMidi.

I'm able to send regular multiple regular messages fine. However when I send a System Exclusive message such as:

#[16rF0 16r01 16r02 16r03 16rF7]

PortMidi is inserting extraneous zeros. I have tested this by sending to the receiveMidi utility. You can see the extra zero in the output below:

receiveMidi output:

system-exclusive hex 01 02 00 03 dec

"Strange errors on shut down...not sure what is causing this:"
CoreMIDI error: 419 - ffffffce
CoreMIDI error: 425 - ffffffce
CoreMIDI error: 433 - ffffffce
MIDI input port "Virtual Output" got disconnected, waiting.
CoreMIDI error: 398 - ffffffce

There seems to be a pattern with progressively longer messages:

system-exclusive hex 01 dec
system-exclusive hex 01 02 00 dec
system-exclusive hex 01 02 00 03 dec
system-exclusive hex 01 02 00 03 04 dec
system-exclusive hex 01 02 00 03 04 05 00 dec
system-exclusive hex 01 02 00 03 04 05 00 06 dec
system-exclusive hex 01 02 00 03 04 05 00 06 07 dec

I've put a trace in the midi_write_byte routine. The output shows it is inserting an extra zero:

trace output:

driver com.apple.AppleMIDIIACDriver
driver
driver
driver com.apple.AppleMIDIIACDriver
driver
driver
midi_out_open isIACdevice 0
***** midi_write_byte => 240
***** midi_write_byte => 1
***** midi_write_byte => 2
***** midi_write_byte => 0
***** midi_write_byte => 3
***** midi_write_byte => 247
add 240 to packet 0x137887e20 len 6 timestamp 16472691953188 @ 686362164755541 ns (host 16472691954134)

If I conduct the same test, but sending to MidiMonitor the messages appear as expect.

I've also tried padding the messages to the boundary, but that has no effect either.

I'm not actually using SysEx, but I'm planning on publishing the binding for others to use, so it would be good to get fixed, assuming my analysis is correct.

Please let me know if you need me to conduct any testing. It's on a M1 Mac Mini running Monterey.

Answer 1 · 2022-06-22T03:40:52.000Z

I can't explain this behavior. Are you sending with Pm_WriteSysEx or Pm_Write?
Do the PortMidi tests work? Here are the tests (from pm_test/README.txt):
[I believe there's a mistake in the description in that the sysex program's l(oopback) test quits by itself after 100K bytes are sent.]
21. ./sysex [no latency]
[This requires either a hardware loopback (MIDI cable from OUT to IN
on interface) or a software loopback (macOS IAC bus or ALSA MIDI
Through Port)]

l [for loopback test]
Type output device number: >>0 [pick output device to loopback]
Latency in milliseconds: >>0
Type input device number: >>0 [pick input device for loopback]
[Program will run forever. After awhile, program will quit. You can read
the Cummulative bytes/sec value.]

./sysex [latency > 0]
[This requires either a hardware loopback (MIDI cable from OUT to IN
on interface) or a software loopback (macOS IAC bus or ALSA MIDI
Through Port)]

l [for loopback test]
Type output device number: >>0 [pick output device to loopback]
Latency in milliseconds: >>100
Type input device number: >>0 [pick input device for loopback]
[Program will run forever. After awhile, quit with ^C. You can read
the Cummulative bytes/sec value; it is affected by latency.]

Answer 2 · 2022-06-26T05:18:45.000Z

Hi Roger, Thanks for getting back to me and apologies for my delay in replying. I am sending using Pm_Write. According to the documentation: "Pm_Write() writes midi data from a buffer. This may contain: short messages or sysex messages that are converted into a sequence of PmEvent structures. Use Pm_WriteSysEx() to write a sysex message stored as a contiguous array of bytes." I've adapted the testio program to send such sysex messages and I'm getting the same result - extra zeros in the messages. I've created a little movie demonstrating this (as it is large, there's a Google Drive link). Please see attached. You can see there is a pattern where an extra zero is being inserted at the end of each three byte chunk. Hopefully this will help - please sing out with further questions. Cheers, Stewart Extraneous zeros in Pm_Write sysex.mov <https://drive.google.com/file/d/1aA4su6wjMLdkIwB1EViZhG7zWw3cdQYM/view?usp=drive_web>

…

On Wed, Jun 22, 2022 at 3:41 PM Roger B. Dannenberg < ***@***.***> wrote: I can't explain this behavior. Are you sending with Pm_WriteSysEx or Pm_Write? Do the PortMidi tests work? Here are the tests (from pm_test/README.txt): [I believe there's a mistake in the description in that the sysex program's l(oopback) test quits by itself after 100K bytes are sent.] 21. ./sysex [no latency] [This requires either a hardware loopback (MIDI cable from OUT to IN on interface) or a software loopback (macOS IAC bus or ALSA MIDI Through Port)] l [for loopback test] Type output device number: >>0 [pick output device to loopback] Latency in milliseconds: >>0 Type input device number: >>0 [pick input device for loopback] [Program will run forever. After awhile, program will quit. You can read the Cummulative bytes/sec value.] 1. ./sysex [latency > 0] [This requires either a hardware loopback (MIDI cable from OUT to IN on interface) or a software loopback (macOS IAC bus or ALSA MIDI Through Port)] l [for loopback test] Type output device number: >>0 [pick output device to loopback] Latency in milliseconds: >>100 Type input device number: >>0 [pick input device for loopback] [Program will run forever. After awhile, quit with ^C. You can read the Cummulative bytes/sec value; it is affected by latency.] — Reply to this email directly, view it on GitHub <#34 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXGQYQ46DKD67AYPNRSZTALVQKDM5ANCNFSM5ZFXGU5Q> . You are receiving this because you authored the thread.Message ID: ***@***.***>

Answer 3 · 2022-06-28T00:28:51.000Z

Hi Stewart, I just modified test/sysex.c to send with both Pm_Write() and Pm_WriteSysEx(). I think I might know what's going on. When you send with Pm_Write, are you sending 3 bytes or 4 bytes per message? PortMidi is expecting 4, so if you send 3 and leave the 4th at zero, that would explain seeing unexpected zeros. The documentation mentions 4-byte messages in the description of PmEvent, but it's not totally clear that Sysex messages should use all 4 bytes until you get to the end. If that's not the problem, maybe you can pull the latest from github with the modified sysex test and, assuming it works, compare how sysex.c works to what your code is doing. If that *is* the problem, let me know how to fix the documentation. I'm thinking the Pm_Write description should say: - sysex messages that are converted into a sequence of PmEvent structures, e.g. sending data from a file or forwarding them from midi input, with 4 SysEx bytes per PmEvent message, low-order byte first, until the last message, which may contain from 1 to 4 bytes ending in MIDI EOX (0xF7). - PortMidi allows 1-byte real-time messages to be embedded within SysEx messages, but only on 4-byte boundaries so that all but the last PmEvent representing a SysEx message contain a full 4 bytes. Each real-time message always occupies a full PmEvent (meaning that 3 of the 4 bytes in the PmEvent's message are ignored) even when embedded in a SysEx message. but your input is welcome.

…

-Roger

On 6/26/22 1:18 AM, Stewart MacLean wrote: Hi Roger, Thanks for getting back to me and apologies for my delay in replying. I am sending using Pm_Write. According to the documentation: "Pm_Write() writes midi data from a buffer. This may contain: short messages or sysex messages that are converted into a sequence of PmEvent structures. Use Pm_WriteSysEx() to write a sysex message stored as a contiguous array of bytes." I've adapted the testio program to send such sysex messages and I'm getting the same result - extra zeros in the messages. I've created a little movie demonstrating this (as it is large, there's a Google Drive link). Please see attached. You can see there is a pattern where an extra zero is being inserted at the end of each three byte chunk. Hopefully this will help - please sing out with further questions. Cheers, Stewart Extraneous zeros in Pm_Write sysex.mov <https://drive.google.com/file/d/1aA4su6wjMLdkIwB1EViZhG7zWw3cdQYM/view?usp=drive_web> On Wed, Jun 22, 2022 at 3:41 PM Roger B. Dannenberg < ***@***.***> wrote: > I can't explain this behavior. Are you sending with Pm_WriteSysEx or > Pm_Write? > Do the PortMidi tests work? Here are the tests (from pm_test/README.txt): > [I believe there's a mistake in the description in that the sysex > program's l(oopback) test quits by itself after 100K bytes are sent.] > 21. ./sysex [no latency] > [This requires either a hardware loopback (MIDI cable from OUT to IN > on interface) or a software loopback (macOS IAC bus or ALSA MIDI > Through Port)] > > l [for loopback test] > Type output device number: >>0 [pick output device to loopback] > Latency in milliseconds: >>0 > Type input device number: >>0 [pick input device for loopback] > [Program will run forever. After awhile, program will quit. You can read > the Cummulative bytes/sec value.] > > > 1. ./sysex [latency > 0] > [This requires either a hardware loopback (MIDI cable from OUT to IN > on interface) or a software loopback (macOS IAC bus or ALSA MIDI > Through Port)] > > l [for loopback test] > Type output device number: >>0 [pick output device to loopback] > Latency in milliseconds: >>100 > Type input device number: >>0 [pick input device for loopback] > [Program will run forever. After awhile, quit with ^C. You can read > the Cummulative bytes/sec value; it is affected by latency.] > > — > Reply to this email directly, view it on GitHub > <#34 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AXGQYQ46DKD67AYPNRSZTALVQKDM5ANCNFSM5ZFXGU5Q> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <#34 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALLUHETGSUUWZPKPD7EZU3VQ7R4BANCNFSM5ZFXGU5Q>. You are receiving this because you commented.Message ID: ***@***.***>

Answer 4 · 2022-06-28T07:20:22.000Z

Hi Roger, Yes, that is the problem! And the reason I was doing this is that I was encountering errors when I was setting the message value within the PmEvent structure. This was because I'd defined its type as int, as per the documentation. See https://portmidi.github.io/portmidi_docs/group__grp__events__filters.html#gad8455e4afe978a74c68e345bef99761d. I believe this should be unsigned as with SysEx messages you can exceed the max for signed integers. The timestamp should also be unsigned. I've changed the types, no errors and everything behaves as expected :) Yes, I think those documentation changes would be good, even though they are documented in detail elsewhere, ( https://portmidi.github.io/portmidi_docs/struct_pm_event.html) . I'd also include that the short messages are padded to a 4 byte boundary as well, as they are encoded as a 32 bit unsigned int. Thanks again for your help! Cheers, Stewart On Tue, Jun 28, 2022 at 12:29 PM Roger B. Dannenberg < ***@***.***> wrote:

…

Hi Stewart, I just modified test/sysex.c to send with both Pm_Write() and Pm_WriteSysEx(). I think I might know what's going on. When you send with Pm_Write, are you sending 3 bytes or 4 bytes per message? PortMidi is expecting 4, so if you send 3 and leave the 4th at zero, that would explain seeing unexpected zeros. The documentation mentions 4-byte messages in the description of PmEvent, but it's not totally clear that Sysex messages should use all 4 bytes until you get to the end. If that's not the problem, maybe you can pull the latest from github with the modified sysex test and, assuming it works, compare how sysex.c works to what your code is doing. If that *is* the problem, let me know how to fix the documentation. I'm thinking the Pm_Write description should say: - sysex messages that are converted into a sequence of PmEvent structures, e.g. sending data from a file or forwarding them from midi input, with 4 SysEx bytes per PmEvent message, low-order byte first, until the last message, which may contain from 1 to 4 bytes ending in MIDI EOX (0xF7). - PortMidi allows 1-byte real-time messages to be embedded within SysEx messages, but only on 4-byte boundaries so that all but the last PmEvent representing a SysEx message contain a full 4 bytes. Each real-time message always occupies a full PmEvent (meaning that 3 of the 4 bytes in the PmEvent's message are ignored) even when embedded in a SysEx message. but your input is welcome. -Roger On 6/26/22 1:18 AM, Stewart MacLean wrote: > Hi Roger, > > Thanks for getting back to me and apologies for my delay in replying. > > I am sending using Pm_Write. According to the documentation: > > "Pm_Write() writes midi data from a buffer. > > This may contain: > > short messages or > > sysex messages that are converted into a sequence of PmEvent structures. > > Use Pm_WriteSysEx() to write a sysex message stored as a contiguous array > of bytes." > > I've adapted the testio program to send such sysex messages and I'm > getting > the same result - extra zeros in the messages. > > I've created a little movie demonstrating this (as it is large, there's a > Google Drive link). Please see attached. You can see there is a pattern > where an extra zero is being inserted at the end of each three byte chunk. > > Hopefully this will help - please sing out with further questions. > > Cheers, > > Stewart > Extraneous zeros in Pm_Write sysex.mov > < https://drive.google.com/file/d/1aA4su6wjMLdkIwB1EViZhG7zWw3cdQYM/view?usp=drive_web > > > > On Wed, Jun 22, 2022 at 3:41 PM Roger B. Dannenberg < > ***@***.***> wrote: > > > I can't explain this behavior. Are you sending with Pm_WriteSysEx or > > Pm_Write? > > Do the PortMidi tests work? Here are the tests (from > pm_test/README.txt): > > [I believe there's a mistake in the description in that the sysex > > program's l(oopback) test quits by itself after 100K bytes are sent.] > > 21. ./sysex [no latency] > > [This requires either a hardware loopback (MIDI cable from OUT to IN > > on interface) or a software loopback (macOS IAC bus or ALSA MIDI > > Through Port)] > > > > l [for loopback test] > > Type output device number: >>0 [pick output device to loopback] > > Latency in milliseconds: >>0 > > Type input device number: >>0 [pick input device for loopback] > > [Program will run forever. After awhile, program will quit. You can read > > the Cummulative bytes/sec value.] > > > > > > 1. ./sysex [latency > 0] > > [This requires either a hardware loopback (MIDI cable from OUT to IN > > on interface) or a software loopback (macOS IAC bus or ALSA MIDI > > Through Port)] > > > > l [for loopback test] > > Type output device number: >>0 [pick output device to loopback] > > Latency in milliseconds: >>100 > > Type input device number: >>0 [pick input device for loopback] > > [Program will run forever. After awhile, quit with ^C. You can read > > the Cummulative bytes/sec value; it is affected by latency.] > > > > — > > Reply to this email directly, view it on GitHub > > > <#34 (comment) >, > > or unsubscribe > > > < https://github.com/notifications/unsubscribe-auth/AXGQYQ46DKD67AYPNRSZTALVQKDM5ANCNFSM5ZFXGU5Q > > > . > > You are receiving this because you authored the thread.Message ID: > > ***@***.***> > > > > — > Reply to this email directly, view it on GitHub > <#34 (comment)>, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AALLUHETGSUUWZPKPD7EZU3VQ7R4BANCNFSM5ZFXGU5Q >. > You are receiving this because you commented.Message ID: > ***@***.***> > — Reply to this email directly, view it on GitHub <#34 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXGQYQ4EMAPJYJJXKW642DTVRJBM5ANCNFSM5ZFXGU5Q> . You are receiving this because you authored the thread.Message ID: ***@***.***>

Answer 5 · 2022-08-08T00:21:49.000Z

Closing this a problem is resolved as above.

Thanks Roger