RT MIDI

Optimize your system for ultimate performance.

Moderators: MattKingUSA, khz

User avatar
khz
Established Member
Posts: 1648
Joined: Thu Apr 17, 2008 6:29 am
Location: German
Has thanked: 42 times
Been thanked: 92 times

RT MIDI

Post by khz »

Inspired by this thread viewtopic.php?f=27&t=19276 I'm interested in how to optimize MIDI for realtime.
Since audio runs in realtime it would be more than useful to run MIDI in realtime (reliably). If audio is processed in real time and MIDI is not processed in real time - i.e. MIDI is processed at some point (jitter) - this could be the problem. Two different priorities for a task (music).
Can MIDI (ALSA/jack) also be optimized for real-time?

What I have found so far on this topic: Source: http://jack-audio.10948.n7.nabble.com/C ... 19580.html

Are there any further optimizations or ideas?
. . . FZ - Does humor belongs in Music?
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
  • I don't care about the freedom of speech because I have nothing to say.
merlyn
Established Member
Posts: 1392
Joined: Thu Oct 11, 2018 4:13 pm
Has thanked: 168 times
Been thanked: 247 times

Re: RT MIDI

Post by merlyn »

What makes you think MIDI doesn't run in realtime? Compared to audio it's low bandwidth.
User avatar
khz
Established Member
Posts: 1648
Joined: Thu Apr 17, 2008 6:29 am
Location: German
Has thanked: 42 times
Been thanked: 92 times

Re: RT MIDI

Post by khz »

I've read little about MIDI RT so far, therefore.
A MIDI sequencer controls a software synthesizer, for example, which then calculates the sound in real time. This should have both (Audio/MIDI) the same priority, be processed in real time and reliably (MIDI: jitter-free).

It could be that MIDI is out of the Box RT, as I said, I have found little on the Internet on this topic.
I have similar problems as lilith in the above mentioned thread.
. . . FZ - Does humor belongs in Music?
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
  • I don't care about the freedom of speech because I have nothing to say.
merlyn
Established Member
Posts: 1392
Joined: Thu Oct 11, 2018 4:13 pm
Has thanked: 168 times
Been thanked: 247 times

Re: RT MIDI

Post by merlyn »

I see what you mean. If I find anything out I'll post it. :)

As an example if your tune is at 120bpm, then that is 2 beats per second. 16th notes at 120bpm is 8 notes per second, or 8Hz, which is orders of magnitude less than 44100Hz for audio. It seems it would be easy to process MIDI data. But I don't know for sure. :)
User avatar
khz
Established Member
Posts: 1648
Joined: Thu Apr 17, 2008 6:29 am
Location: German
Has thanked: 42 times
Been thanked: 92 times

Re: RT MIDI

Post by khz »

. . . FZ - Does humor belongs in Music?
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
  • I don't care about the freedom of speech because I have nothing to say.
User avatar
raboof
Established Member
Posts: 1855
Joined: Tue Apr 08, 2008 11:58 am
Location: Deventer, NL
Has thanked: 50 times
Been thanked: 74 times
Contact:

Re: RT MIDI

Post by raboof »

khz wrote:Inspired by this thread viewtopic.php?f=27&t=19276 I'm interested in how to optimize MIDI for realtime.
To go deeper into this topic, it might be good to be explicit about your use case. What exactly are you trying to do with MIDI? Live MIDI-to-MIDI processing, live MIDI-to-Audio, recording MIDI, or MIDI-to-Audio and then directly recording the result?
User avatar
khz
Established Member
Posts: 1648
Joined: Thu Apr 17, 2008 6:29 am
Location: German
Has thanked: 42 times
Been thanked: 92 times

Re: RT MIDI

Post by khz »

MIDI-to-Audio.
I sometimes play hardware and software synthesizers with a hardware keyboard or a hardware or software sequencer. I record all this sometimes.
In all cases where I play with MIDI I get more xruns than when I play my hardware e-bass through Linux effects and then possibly record.

Only audio I get as good as no xruns.
MIDI-to-Audio I get xruns. MIDI should run jitter-free (tight), I do the groove.


I suspect they are complex dependencies on the individual priorities assigned to the
A) RT kernel
B) RT-MIDI
C) RT-IRQ (If it is a USB MIDI keyboard (hardware))
D) foobar

If these are not optimally coordinated - MIDI does not run/work in real time but audio and irq run/work in real time - conflicts (xruns) occur.

The whole real-time priorities should always run below the operating system (RT less than 100%) otherwise the system will have no more processing time.
. . . FZ - Does humor belongs in Music?
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
  • I don't care about the freedom of speech because I have nothing to say.
j_e_f_f_g
Established Member
Posts: 2032
Joined: Fri Aug 10, 2012 10:48 pm
Been thanked: 357 times

Re: RT MIDI

Post by j_e_f_f_g »

khz wrote:MIDI-to-Audio I get xruns.
BackupBand is an example of MIDI-to-audio. Do you get xruns with that? If not, I suspect the problem is the way your other apps handle MIDI Input. Many devs treat midi as an afterthought, and sometimes do things that completely subvert its realtime properties (especially when they try to put a "timestamp" on each incoming message). Since MIDI is what internally drives BackupBand, its midi input pulls out every trick in the book to avoid needing to timestamp, or do any "waiting" when interacting with the audio output. It also uses a scheme that postpones all graphics processing until after midi input is done. (I don't just leave it up to the OS scheduler). A midi realtime arranger has to be as "tight" in its processing of midi as possible.

Author of BackupBand at https://sourceforge.net/projects/backupband/files/
My fans show their support by mentioning my name in their signature.

User avatar
khz
Established Member
Posts: 1648
Joined: Thu Apr 17, 2008 6:29 am
Location: German
Has thanked: 42 times
Been thanked: 92 times

Re: RT MIDI

Post by khz »

I'm doing a test with BackupBand. But I have just little time ... .
There's no hurry either: no panic, general meta question regarding MIDI.

The rtmidi also handles software sequencers/synthesizers in realtime? I suspect rather that it refers to the hardware MIDI inputs and outputs of the sound card. If that were so, what are there for jitter-free/tight software MIDI solutions?

A stable (jitter-free/tight) MIDI is just as important as a stable (jitter-free/tight) audio.


But even with hardware there are huge differences. https://www.innerclocksystems.com/litmus/ If now all hardware components have different processing times and the same (similar) also applies to the different software sequencers/instruments, electronic music is a miracle.
Somebody should be the master at MIDI (realtime) and ideally synchronized to the (realtime) audio.
I use a Cirklon as master clock/sequencer, but even there there are xruns.
EDIT:
A problem I haven't solved in years: If I use an external (hardware) clock there are very few programs that have an external clock input.
Ideally the external (hardware) clock should control qjackctl/Cadence and these should control all other programs.
. . . FZ - Does humor belongs in Music?
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
  • I don't care about the freedom of speech because I have nothing to say.
User avatar
raboof
Established Member
Posts: 1855
Joined: Tue Apr 08, 2008 11:58 am
Location: Deventer, NL
Has thanked: 50 times
Been thanked: 74 times
Contact:

Re: RT MIDI

Post by raboof »

j_e_f_f_g wrote:Many devs treat midi as an afterthought, and sometimes do things that completely subvert its realtime properties (especially when they try to put a "timestamp" on each incoming message). Since MIDI is what internally drives BackupBand, its midi input pulls out every trick in the book to avoid needing to timestamp, or do any "waiting" when interacting with the audio output.
Isn't taking into account timestamps needed to prevent jitter?

If you don't timestamp and just trigger the MIDI audio at the beginning of the next buffer, then MIDI messages that arrive close to the beginning of the previous buffer have a higher latency than MIDI messages that arrive close to the end of the previous buffer. By taking into account the timestamp inside the previous buffer, you can delay the audio to have a more constant latency.

Of course this trades jitter for latency: without taking into account the timestamp the average latency will be half a buffer size, taking into account the timestamp will increase it to average a whole buffer - but with much less jitter.

How do you deal with this trade-off?
merlyn
Established Member
Posts: 1392
Joined: Thu Oct 11, 2018 4:13 pm
Has thanked: 168 times
Been thanked: 247 times

Re: RT MIDI

Post by merlyn »

One possibility that came up on lilith's thread was that lilith had reached his hardware limit. Although Cadence reported a DSP load of 85%, that is an average, and the maximum could be 100%. No amount of tweaking or configuration can stop Xruns at 100% DSP load.

So in your case khz, do Xruns appear as soon as you add MIDI, or is it when you are working on a large project with a DSP load around 80%?

I've realised that this is based on nothing, but I thought that by using JACK MIDI I would have something like realtime MIDI, because JACK is working in realtime.
Last edited by merlyn on Tue Dec 04, 2018 5:27 pm, edited 1 time in total.
User avatar
raboof
Established Member
Posts: 1855
Joined: Tue Apr 08, 2008 11:58 am
Location: Deventer, NL
Has thanked: 50 times
Been thanked: 74 times
Contact:

Re: RT MIDI

Post by raboof »

khz wrote:In all cases where I play with MIDI I get more xruns than when I play my hardware e-bass through Linux effects and then possibly record.

Only audio I get as good as no xruns. MIDI-to-Audio I get xruns.

I suspect they are complex dependencies on the individual priorities assigned to the
A) RT kernel
B) RT-MIDI
C) RT-IRQ (If it is a USB MIDI keyboard (hardware))
D) foobar

If these are not optimally coordinated - MIDI does not run/work in real time but audio and irq run/work in real time - conflicts (xruns) occur.
This is interesting. xruns are typically basically an audio thing, independent of MIDI: a failure of the audio processing to consume or produce the audio fast enough to keep up with the clock. If the MIDI processing has too low a priority, you would expect no xruns, but jitter: the MIDI events would arrive too late, but not interfere with the audio processing.

I guess there are 3 ways MIDI could cause increased xruns:
* by consuming system resources that were needed for the audio processing to keep up with the clock
* by causing additional load on the audio processing
* if the audio processing threads would somehow 'wait' for MIDI data.

I think point '3' is pretty unlikely, so that leaves '1' and '2'.
khz wrote:I sometimes play hardware and software synthesizers with a hardware keyboard or a hardware or software sequencer
This would be interesting to test '2': could you program your hardware sequencer to produce some pattern, and also record the same pattern into a software sequencer? It would be interesting to see if the same pattern (and otherwise the same system) would produce xruns when driven by the hardware sequencer but not by the software sequencer.

Generally, how do you connect your MIDI devices? Via your sound card, or via a MIDI/USB bridge? How does your MIDI data flow into your audio applications? ALSA MIDI, Jack MIDI, ...?
User avatar
raboof
Established Member
Posts: 1855
Joined: Tue Apr 08, 2008 11:58 am
Location: Deventer, NL
Has thanked: 50 times
Been thanked: 74 times
Contact:

Re: RT MIDI

Post by raboof »

merlyn wrote:I've realised that this based on nothing, but I thought that by using JACK MIDI I would have something like realtime MIDI, because JACK is working in realtime.
Using JACK MIDI will make sure that, once observed, the MIDI events will flow through your system at predictable latency, just like for audio. Of course observing the MIDI events and having the applications using the JACK API to process the MIDI events still needs to be done properly. So it much depends on those parts of the system as well.
User avatar
lilith
Established Member
Posts: 1698
Joined: Fri May 27, 2016 11:41 pm
Location: bLACK fOREST
Has thanked: 117 times
Been thanked: 57 times
Contact:

Re: RT MIDI

Post by lilith »

merlyn wrote:One possibility that came up on lilith's thread was that lilith had reached his hardware limit. Although Cadence reported a DSP load of 85%, that is an average, and the maximum could be 100%. No amount of tweaking or configuration can stop Xruns at 100% DSP load.

So in your case khz, do Xruns appear as soon as you add MIDI, or is it when you are working on a large project with a DSP load around 80%?

I've realised that this based on nothing, but I thought that by using JACK MIDI I would have something like realtime MIDI, because JACK is working in realtime.
That seems to be exactly the case for me.
User avatar
khz
Established Member
Posts: 1648
Joined: Thu Apr 17, 2008 6:29 am
Location: German
Has thanked: 42 times
Been thanked: 92 times

Re: RT MIDI

Post by khz »

merlyn wrote: So in your case khz, do Xruns appear as soon as you add MIDI, or is it when you are working on a large project with a DSP load around 80%?
As soon as i add MIDI. There are also xruns with low CPU load (few software instruments controlled via MIDI).
merlyn wrote:I've realised that this based on nothing, but I thought that by using JACK MIDI I would have something like realtime MIDI, because JACK is working in realtime.
So I understood that ALSA/jack MIDI has the same priority as audio. That would be logical. The control signal is as important as the audio. Both depend on each other.
raboof wrote:If the MIDI processing has too low a priority, you would expect no xruns, but jitter:
Full approval. Strangely enough, I get a lot more xruns as soon as MIDI is in the game. It doesn't matter if I control a hardware synthesizer or if I control a software synthesizer from the hardware sequencer or if everything comes from software.
raboof wrote: This would be interesting to test '2'
Yes. But I have just little time ... at the weekend.
raboof wrote: Generally, how do you connect your MIDI devices? Via your sound card, or via a MIDI/USB bridge? How does your MIDI data flow into your audio applications? ALSA MIDI, Jack MIDI, ...?
To both points: both variants. Whereby I try to control as little as possible via USB (layer). I think "real" Midi is more reliable. I use a2j bridge, so I can't see if it's ALSA or jack MIDI.

# Thanks for the MIDI discussion.
. . . FZ - Does humor belongs in Music?
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
  • I don't care about the freedom of speech because I have nothing to say.
Post Reply