Standard test needed to benchmark XRUNs
Moderators: MattKingUSA, khz
-
- Established Member
- Posts: 64
- Joined: Mon Jul 30, 2018 11:04 pm
Standard test needed to benchmark XRUNs
Hello,
Still quite new to doing full-blown DAW via GNU/Linux and as we all know, there are plenty of knobs to tweak. I'd really like to come up with a standard test suite that attempts to generate XRUNs so I can track progress as I fine tune my system. I'm asking for opinions on what this test should be because I don't want to go through the tedious steps involved with launching a sequencer, opening a file, blah blah blah... I'm thinking along the lines of one of the phoronix test suites or something. Looking forward to hearing ideas.
Thanks in advance
Still quite new to doing full-blown DAW via GNU/Linux and as we all know, there are plenty of knobs to tweak. I'd really like to come up with a standard test suite that attempts to generate XRUNs so I can track progress as I fine tune my system. I'm asking for opinions on what this test should be because I don't want to go through the tedious steps involved with launching a sequencer, opening a file, blah blah blah... I'm thinking along the lines of one of the phoronix test suites or something. Looking forward to hearing ideas.
Thanks in advance
- lilith
- Established Member
- Posts: 1708
- Joined: Fri May 27, 2016 11:41 pm
- Location: bLACK fOREST
- Has thanked: 126 times
- Been thanked: 57 times
- Contact:
Re: Standard test needed to benchmark XRUNs
I had the same thought 5 seconds ago while discussing some xrun issues at IRC. So, if such a thing is technically possible it would be great.
Also, what does it mean when people mention latencies of 2ms. Maybe they can record one instrument at 2ms latency, but does that mean that a big session with lots of plugins also plays at 2ms without any xruns?
Also, what does it mean when people mention latencies of 2ms. Maybe they can record one instrument at 2ms latency, but does that mean that a big session with lots of plugins also plays at 2ms without any xruns?
-
- Established Member
- Posts: 64
- Joined: Mon Jul 30, 2018 11:04 pm
Re: Standard test needed to benchmark XRUNs
Right, system performance and xruns are relative to each user's workload. That's why I'm looking for a standard load to run (test, tweak, repeat) as it relates to A/V. I have a feeling it'd come down to one or more of the tests provided by http://phoronix-test-suite.com.
- khz
- Established Member
- Posts: 1648
- Joined: Thu Apr 17, 2008 6:29 am
- Location: German
- Has thanked: 42 times
- Been thanked: 92 times
Re: Standard test needed to benchmark XRUNs
All the software effects, instruments, channels (and mixing) must be calculated in real time. 10 - x Software synthesizers/effects require more computing power than an electric bass through hardware effects and software routing/recording.lilith wrote:but does that mean that a big session with lots of plugins also plays at 2ms without any xruns?
# rt-tests >> https://packages.debian.org/de/stretch/rt-tests;
# cyclictest - High resolution test program >> https://manpages.debian.org/testing/rt- ... .8.en.html
# Using and Understanding the Real-Time Cyclictest Benchmark >> https://events.static.linuxfound.org/si ... ictest.pdf
?
. . . FZ - Does humor belongs in Music?
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
- I don't care about the freedom of speech because I have nothing to say.
- lilith
- Established Member
- Posts: 1708
- Joined: Fri May 27, 2016 11:41 pm
- Location: bLACK fOREST
- Has thanked: 126 times
- Been thanked: 57 times
- Contact:
Re: Standard test needed to benchmark XRUNs
I have a test running with the RT kernel and discussed the results with JackWinter at IRC. There is obviously no problem. My maximum latency is 50 microseconds even with 100% CPU usage.
-
- Established Member
- Posts: 381
- Joined: Sun May 28, 2017 3:52 pm
Re: Standard test needed to benchmark XRUNs
Then the RT kernel is doing it's job well.lilith wrote:I have a test running with the RT kernel and discussed the results with JackWinter at IRC. There is obviously no problem. My maximum latency is 50 microseconds even with 100% CPU usage.
Reaper/KDE/Archlinux. i7-2600k/16GB + i7-4700HQ/16GB, RME Multiface/Babyface, Behringer X32, WA273-EQ, 2 x WA-412, ADL-600, Tegeler TRC, etc For REAPER on Linux information: https://wiki.cockos.com/wiki/index.php/REAPER_for_Linux
Re: Standard test needed to benchmark XRUNs
I started thinking about this too some time ago. We have utilities as those mentioned by khz, but results from those are hard to correlate to userland audio performance. My latest idea was to use FAUST and do something like this:windowsrefund wrote:Hello,
Still quite new to doing full-blown DAW via GNU/Linux and as we all know, there are plenty of knobs to tweak. I'd really like to come up with a standard test suite that attempts to generate XRUNs so I can track progress as I fine tune my system. I'm asking for opinions on what this test should be because I don't want to go through the tedious steps involved with launching a sequencer, opening a file, blah blah blah... I'm thinking along the lines of one of the phoronix test suites or something. Looking forward to hearing ideas.
Thanks in advance
- Have some FAUST based DSP worker: a DSP algorithm with a known computational complexity. The FAUST compiler can be made to optimize the build for the target platform, which perhaps is appropriate to make a fair test of a platform.
- Have a program that: sets up the audio stack (JACK, for example) witch certain buffering variables (Sample rate / buffer size, all that stuff).
- The program then spawn DSP workers while logging all XRUNS.
- DSP load.
- JACK (or possibly ALSA?) settings.
- khz
- Established Member
- Posts: 1648
- Joined: Thu Apr 17, 2008 6:29 am
- Location: German
- Has thanked: 42 times
- Been thanked: 92 times
Re: Standard test needed to benchmark XRUNs
In e.g. "qjackctl" settings > Advanced: "Server Prefix" select "Soft Mode".
I also have xruns depending on the audio/MIDI load, but I don't hear them. Does it also depend on the soundcard used?
Linux is talkative. No Panik!
I also have xruns depending on the audio/MIDI load, but I don't hear them. Does it also depend on the soundcard used?
Linux is talkative. No Panik!
. . . FZ - Does humor belongs in Music?
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
- I don't care about the freedom of speech because I have nothing to say.
-
- Established Member
- Posts: 1392
- Joined: Thu Oct 11, 2018 4:13 pm
- Has thanked: 168 times
- Been thanked: 247 times
Re: Standard test needed to benchmark XRUNs
I think a standard test is a good idea. We then need a standard unit of Audio Based Computational Demand. I drew a graph of a simplified model.
To measure performance across different systems the x-axis needs units. A standard unit of computational demand.
Ideally there would be no Xruns below 100% DSP load. In mathematical speak it would be good if the relationship between Xruns and 100% DSP Load was 'If and only if". The significance of that is that 'if and only if' is a two way relationship. Xruns means 100% DSP Load and 100% DSP Load means Xruns.
In practice I have found that not to be the case, but I suspect that Xruns at low DSP loads tells us that there is something wrong with the configuration.
Relevant points that come out the graph are : the slope of the graph is inversely proportional to the power of the system being tested. The lines don't go through the origin, so outputting a stream of zeros takes some work.To measure performance across different systems the x-axis needs units. A standard unit of computational demand.
Ideally there would be no Xruns below 100% DSP load. In mathematical speak it would be good if the relationship between Xruns and 100% DSP Load was 'If and only if". The significance of that is that 'if and only if' is a two way relationship. Xruns means 100% DSP Load and 100% DSP Load means Xruns.
In practice I have found that not to be the case, but I suspect that Xruns at low DSP loads tells us that there is something wrong with the configuration.
- khz
- Established Member
- Posts: 1648
- Joined: Thu Apr 17, 2008 6:29 am
- Location: German
- Has thanked: 42 times
- Been thanked: 92 times
Re: Standard test needed to benchmark XRUNs
Besides rt-tests https://git.kernel.org/pub/scm/utils/rt-tests/ for audio there is also test for MIDI: alsa-midi-latency-test https://github.com/koppi/alsa-midi-latency-test and jack_midi_latency https://github.com/x42/jack_midi_latency.
. . . FZ - Does humor belongs in Music?
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
- I don't care about the freedom of speech because I have nothing to say.
-
- Established Member
- Posts: 1392
- Joined: Thu Oct 11, 2018 4:13 pm
- Has thanked: 168 times
- Been thanked: 247 times
Re: Standard test needed to benchmark XRUNs
Thanks for the links. The proposed test has a different focus. The currently available tests make sure the foundations are solid to build an audio system on.khz wrote:Besides rt-tests https://git.kernel.org/pub/scm/utils/rt-tests/ for audio there is also test for MIDI: alsa-midi-latency-test https://github.com/koppi/alsa-midi-latency-test and jack_midi_latency https://github.com/x42/jack_midi_latency.
The proposed test would find out how much force is required to push the building over given solid foundations.
-
- Established Member
- Posts: 381
- Joined: Sun May 28, 2017 3:52 pm
Re: Standard test needed to benchmark XRUNs
IMO, the big question is what is an xrun, and what is dsp load..
The basic issue is that calculating the audio has a deadline, if it's not finished on time (minus overhead) there is a dropout/glitch/xrun. The deadline can be calculate by buffersize / samplerate.
The dsp load is an average given by the jack server indicating how much of that deadline actually was used.
Xruns can happen at even low dsp load, if there is either a hardware issue causing the hardware to respond too late, or a software issue causing the program to be late in delivering it's result.
IMO, the first step ought to be to make it obvious to the user if the xrun is caused by hardware or software...
It would also be very useful to have a max latency metric from the jack server, and not only an average..
JACK2 can be built with the --profile option which can give a lot of interesting metrics, you can even get information as audio signals to record into your DAW for later perusal, see: http://www.grame.fr/ressources/publications/Timing.pdf
The basic issue is that calculating the audio has a deadline, if it's not finished on time (minus overhead) there is a dropout/glitch/xrun. The deadline can be calculate by buffersize / samplerate.
The dsp load is an average given by the jack server indicating how much of that deadline actually was used.
Xruns can happen at even low dsp load, if there is either a hardware issue causing the hardware to respond too late, or a software issue causing the program to be late in delivering it's result.
IMO, the first step ought to be to make it obvious to the user if the xrun is caused by hardware or software...
It would also be very useful to have a max latency metric from the jack server, and not only an average..
JACK2 can be built with the --profile option which can give a lot of interesting metrics, you can even get information as audio signals to record into your DAW for later perusal, see: http://www.grame.fr/ressources/publications/Timing.pdf
Reaper/KDE/Archlinux. i7-2600k/16GB + i7-4700HQ/16GB, RME Multiface/Babyface, Behringer X32, WA273-EQ, 2 x WA-412, ADL-600, Tegeler TRC, etc For REAPER on Linux information: https://wiki.cockos.com/wiki/index.php/REAPER_for_Linux
-
- Established Member
- Posts: 64
- Joined: Mon Jul 30, 2018 11:04 pm
Re: Standard test needed to benchmark XRUNs
I found these a few days ago but failed to get any value from them as each requires a loop created with the MIDI cables. In my case, this is a problem since I'm using one of those USB to MIDI IN & MIDI Out cables. In other words, I don't have the ability to take the MIDI IN and connect it to a MIDI OUT. Years ago, this would have been possible when I used a dedicated MIDI card with individual IN and OUT ports.khz wrote:Besides rt-tests https://git.kernel.org/pub/scm/utils/rt-tests/ for audio there is also test for MIDI: alsa-midi-latency-test https://github.com/koppi/alsa-midi-latency-test and jack_midi_latency https://github.com/x42/jack_midi_latency.
- khz
- Established Member
- Posts: 1648
- Joined: Thu Apr 17, 2008 6:29 am
- Location: German
- Has thanked: 42 times
- Been thanked: 92 times
Re: Standard test needed to benchmark XRUNs
Yes these tests are for "old" real MIDI interfaces.
These "old" real MIDI interfaces, installed in a sound card / audio interface, send the data byte by byte.
USB MIDI sends the data block by block due to the USB technology. USB transmits the MIDI data (reliably?) over many layers.
This is a serious difference from a technical point of view.
####
Possibly helpful to the topic :
JACK Latency tests https://wiki.linuxaudio.org/wiki/jack_latency_tests
Latency: Myths and Facts. Part 3(2/1): A look at a quantitative study https://thecrocoduckspond.wordpress.com ... ive-study/
These "old" real MIDI interfaces, installed in a sound card / audio interface, send the data byte by byte.
USB MIDI sends the data block by block due to the USB technology. USB transmits the MIDI data (reliably?) over many layers.
This is a serious difference from a technical point of view.
####
Possibly helpful to the topic :
JACK Latency tests https://wiki.linuxaudio.org/wiki/jack_latency_tests
Latency: Myths and Facts. Part 3(2/1): A look at a quantitative study https://thecrocoduckspond.wordpress.com ... ive-study/
. . . FZ - Does humor belongs in Music?
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
. . GNU/LINUX@AUDIO ~ /Wiki $ Howto.Info && GNU/Linux Debian installing >> Linux Audio Workstation LAW
- I don't care about the freedom of speech because I have nothing to say.
-
- Established Member
- Posts: 1392
- Joined: Thu Oct 11, 2018 4:13 pm
- Has thanked: 168 times
- Been thanked: 247 times
Re: Standard test needed to benchmark XRUNs
I found an xrun counter that tramp wrote on this thread
xruncounter.c
So that's a start on one half of the test. The other half is to generate load.
If we compare the proposed 'xrun benchmark' test to using the rt-tests combination of hackbench and cyclictest, they are similiar, but not identical. In rt-tests hackbench (the load generator) is simple and cyclictest (the measurement part) is complex and flexible. In xrun benchmark it would be the load generating part that has more options -- buffer size, sample rate and other things yet to be specified. The load generator can be thought on as equivalent to 'number of soft synths and plugins'
xruncounter.c
Code: Select all
#include <stdio.h>
#include <errno.h>
#include <unistd.h>
#include <signal.h>
#include <stdlib.h>
#include <jack/jack.h>
/* gcc -Wall xruncounter.c -lm `pkg-config --cflags --libs jack` -o xruncounter */
jack_client_t *client;
void
jack_shutdown (void *arg)
{
exit (1);
}
int jack_xrun_callback(void *arg)
{
/* count xruns */
static int xruns = 0;
xruns += 1;
fprintf (stderr, "xrun %i \n", xruns);
return 0;
}
void
signal_handler (int sig)
{
jack_client_close (client);
fprintf (stderr, " signal received, exiting ...\n");
exit (0);
}
int
main (int argc, char *argv[])
{
if ((client = jack_client_open ("xruncounter", JackNullOption, NULL)) == 0) {
fprintf (stderr, "jack server not running?\n");
return 1;
}
signal (SIGQUIT, signal_handler);
signal (SIGTERM, signal_handler);
signal (SIGHUP, signal_handler);
signal (SIGINT, signal_handler);
jack_set_xrun_callback(client, jack_xrun_callback, 0);
jack_on_shutdown (client, jack_shutdown, 0);
if (jack_activate (client)) {
fprintf (stderr, "cannot activate client");
return 1;
}
while (1) {
usleep (100000);
}
jack_client_close (client);
exit (0);
}
If we compare the proposed 'xrun benchmark' test to using the rt-tests combination of hackbench and cyclictest, they are similiar, but not identical. In rt-tests hackbench (the load generator) is simple and cyclictest (the measurement part) is complex and flexible. In xrun benchmark it would be the load generating part that has more options -- buffer size, sample rate and other things yet to be specified. The load generator can be thought on as equivalent to 'number of soft synths and plugins'