Reviving Beatrix, JACK overhead

[j_e_f_f_g]

While recently working on the linuxaudio wiki, I kept running across software that looked extremely promising according to its description. But in practice, the software's development has been so long neglected that it's no longer functional. I call it abandonware. One example is Beatrix, a hammond drawbar organ, whose last development was 10 years ago. I couldn't get it to run. First, it requires a sound device that does 22 KHz. Most of today's sound chips no longer do less than 44K. Secondly, beatrix uses the pre-ALSA OSS. Even if there was some way to upsample its output to 44k, there would be horrible lag under alsa's oss emulation.

Unuseable.

But not anymore. I've done a major overhaul of the codebase and now we have Beatrix 2, the standalone version with GUI. There are still some features yet to be finished, such as patch saving/loading. And the LV2 plugin ain't done. But those interested get a sneak preview today. I've included a 64-bit executable. 32-bit folks will need to compile.

Consult readme.htm to get started.

http://wikisend.com/download/711880/beatrix.zip

[looplog]

Not sure if you are aware, but there is a history involved with Beatrix and the legality of using it's code (http://rg42.org/wiki/beatrix). The result is setBfree (http://setbfree.org/), which your efforts may be an unnecessary duplication of.

[j_e_f_f_g]

Wasn't aware of setBfree. This is why the linuxaudio app wiki needs even more updating. There's so much software out there that so few people know about.

I emailed the beatrix author a month ago about redistributing modified sources. Although the email didn't bounce, I never got a reply. Maybe something has happened to him. Anyway I looked over the setBfree sources, and I see permission is moot. setBfree are the Beatrix 0.4.1 sources verbatim (albeit a couple bug fixes), with the OSS stuff replaced with a JACK interface. (Then there is new GPL code added to implement plugins). If the orig author has given permission to license this source as gpl (and that's how it is licensed), then the Beatrix 0.4.1 sources, minus the OSS code, is gpl.

My version is the Beatrix 0.4.1 sources (albeit significantly reworked), with the OSS stuff replaced with a ALSA interface. The legality is settled. I'm putting gpl 2 on mine, as the gpl forbids even the original author from restricting the right to modify and redistribute gpl licensed code.

As far as duplication is concerned, that's not applicable here. setBfree is essentially a jack/lv2 interface to the orig code. Mine is a reworking to make the engine faster, and use less RAM. Moreover, now that there's both a jack, as well as alsa, versions, this gives people a splendid opportunity to experience the differences in round-trip latencies between alsa and jack. Run setBfree and see how low you can get the latency without underruns. Play it with a midi keyboard until you get a good sense of how responsive (ie tight) the note tracking is.

Now try my version. You should definitely be able to set my version to a much lower latency (the "Buffer" setting under the "Audio" menu) before you see underruns while playing. Beatrix defaults to a buffer size of 32, which gives me no underruns even on an ordinary non-RT kernel. You can try going all the way down to the ridiculous, dedicated-hardware-like setting of 1, but it probably will give underruns since beatrix is more computationally intensive than eDrummer. But on a well-tuned rt kernel, you should be able to go small. Now try playing beatrix. A musician with a decent sense of timing should sense a tighter note tracking than with setBfree.

This is an excellent opportunity to witness how jack's context switching, and many instances of file descriptor polling, force you to higher buffer (latency) settings, and by consequence how this impacts the realtime response of music software.

[autostatic]

I uncommented line 596 in AudioDevice.c, recompiled and the lowest setting I could achieve with your version of Beatrix without having too much underruns was this:
period_size=128, Buffer=256

I could go lower and the lowest possible setting without getting distortion was this:
period_size=32, Buffer=64

I can run JACK + setBfree with the same buffer sizes, assuming the buffer size is the value of -n (number of periods) * the value of -p (number of frames) as per the JACK man page hardware buffer reference. JACK + setBfree behave similarly to beatrix. There's some more crackling with the lowest possible setting but that could be the optimizations you've done to the beatrix code. This is with a generic kernel and using the onboard soundcard. But I could've misunderstood the code, I'm not a coder, so if I'm utterly wrong about all this and made the wrong assumptions then I stand corrected of course

[j_e_f_f_g]

period_size=32, Buffer=64

JACK + setBfree with the same buffer sizes, assuming the buffer size is the value of -n (number of periods) * the value of -p (number of frames)

Well no, those beatrix buffer and period sizes you see there are referenced in bytes -- not frames nor number of periods. To do a direct comparison, you also need the jack sizes in bytes. According to jackd docs...

The period size (-p) times --nperiods times four is the JACK buffer size in bytes.

Multiplying by that extra 4 gives you buffer size in bytes, rather than the number of 32-bit samples.

In other words, to get playback "without distortion", you had to run the jack version with 4 times as much buffering as the alsa version. This is the same behavior on my systems too.

But what about how responsive the note tracking is? As you were playing, didn't you feel that extra lag with setBfree? I definitely felt it when I was playing my keyboard with setBfree versus my beatrix port.

[x42]

Dear j_e_f_f_g,

Please let me address a few points in your recent post. I'll split them up to avoid confusion and separate issues that are not of general interst.

0) Wiki & PR

Well, setBfree is documented in the apps wiki. The Beatrix page even links to it: http://apps.linuxaudio.org/apps/all/beatrix
setBfree has also been announced on the Linux-Audio-Announce email list (which is picked up by planet.linuxaudio.org).
AVLinux mentions it in the 6.01 release notes and there are a couple of threads on the linux-audio-user's email list.
There'a g+ post by libregraphics world about it, it's mentioned in various "Friday Interviews" on zthmusic, linuxmao has an article about it..

Short of shoving it up your noise, I think it's doing rather fine :) Any help to spread the word further is certainly welcome, but calling it Abandonware is a bit of a stretch.


1) Licensing

You cannot simply slap on the GPL to your copy of beatrix. You can however fork the very first commit of setBfree, which is almost pristine Beatrix v41. Will does have a written permission from Fredrik to use that (and only that) in terms of the GPL. Doing this bit of extra work to re-base your changes is not strictly for *your* benefit but for the users of your code. You cannot legally grant them the freedom they deserve by simply putting the GPL on your copy of Beatrix. Doing so is in violation of beatrix' original license.


2) remarks on setBfree

"setBfree is essentially a jack/lv2 interface" is not wrong, but neglects a lot.

There have been countless bug fixes and improvements. While the general architecture has not been changed, the engine has been pretty much rewritten to support variable sample rates and is no longer limited to 16bits but rewritten to use floating point (allowing for FPU usage and SSE). Other known issues and shortcomings of Beatrix have been addressed, the complete changelog is available with the source (as is the git history). The general sound quality has been retained (A/B listening test) and in some cases improved, doing blind tests - even literally thanks to Julien :)

It's also a community effort, with contributions from over 15 people so far, in particular Dominique Michel, Fons Adriaensen, Jeremy Jongepier, Julien Claasen, Ken Restivo, Will Panther, myself and of course Fredrik Kilander. Murray Saul is teaching a workshop at his college and will do a presentation at FSOSS (Free software and Open Source Symposium) later this year. Will is playing it in his church. Glen aka AVlinux has not only been helpful with bug reports but also released music done with it. Julius Orion Smith (who wrote the paper that the Leslie emulation is modelled on) provided better coefficients for it and admitted that he got distracted for a whole afternoon just playing it, Ken plays it on stage... Community and Musician support is similarly important as CPU optimizing and worth much more than a few bytes of RAM these days.

There's also been major work to add a GUI, by popular request particularly one that allows mouse-click-binding of MIDI CCs, while still retaining .cfg and .pgm file formats for advanced and visually impaired users.


3) on JACK, ALSA, latency and Jitter

Your claim about jack latency and context-switches is similarly half true. I'm afraid that you knowledge of things is good enough to sound convincing but incomplete and incorrect. Which is a dangerous thing to new users when you explain things and make rather bold statements without citing any sources. Please don't take it personally. You're on a very good track in general and have lots of potential. But I ask you to refrain from making statements about things that you cannot explain in depth, or at least provide references. Elaborating on means to verify and measure the claims you make would be very helpful as well.


jack does indeed have a minimum latency (1 period) for audio+midi, while theoretically ALSA does not have this limitation.
In practice however the fact that ALSA MIDI (and audio) can provide a lower latency is mitigated by the fact that the jitter is an order or magnitude lager when processing ALSA midi+audio directly. If you synchronize the two, at best, you basically end up doing what jackd 0.124 is doing (using alsarawmidi). Short of pointing you to the jack codebase and elaborating on the complete underlying stack.. I recommend to simply measure it.

I did measurements (using setBfree which supports both ALSA midi and JACK midi). The author of alsa-midi-latency test suite kindly sent me one of his https://github.com/koppi/avr/tree/maste ... schematics boards to facilitate that using a scope.

external midi signal -> scope 1st chanel and soundcard midi in -> setBfree -> soundcard audio out -> scope 2nd channel.

The measurement was done using a USB UA-25 on a Thinkpad X60s with linux 3.2.35 SMP PREEMPT RT.

-> 5.3 ms +- 0.1 ms latency with setBfree here using jack-midi
-> 3.7 ms [-1 + 6ms] latency using the alsa-sequencer.


comparing https://github.com/koppi/alsa-midi-latency-test with https://github.com/x42/jack_midi_latency procudes similar results, getting alsa (even raw alsa) midi jitter down is not trivial. See also http://gareus.org/wiki/midilatency and http://lists.linuxaudio.org/pipermail/l ... 97296.html

Now these measurements are hardware (and to some extend also software/kernel) dependent. In my case they're all done on the same hardware using the same setup, to allow direct comparison.

Please try to reproduce the measurements on your machine. It may well be that it's different either because of your hardware or the configuration that you use. But if plain ALSA turns out to be more reliable than jack midi on your machine, I suspect some configuration mistake regarding jack on your box. Maybe try using the AVLinux or KXstudio live system and compare that against your setup.


4) JACK again.

Speaking of which.. Your claim about JACK, context switches and latency is plain wrong.

Context-switches could cause a higher CPU load but are entirely unrelated to latency.

The whole idea behind JACK is that it does not switch context at all. All process callbacks inside every jack client are run in the same context (jack's). You can read up on this jackd/engine.c of the jack1 source-code. JACK2's source is also fine but it's a bit harder to read. You can even 'witness' it using gdb.

jack does not poll anything, it uses a 'push' architecture on the client side, as for the backend side it subscribes to IRQs of the soudcard. Could you please explain what you mean by "many instances of file descriptor polling" and where you see that?

Your sloppy use of terminology e.g. 'realtime response' only supports the assumption that your basic understanding is lacking. 'realtime' usually refers to reliability to meet a given deadline, while in the context of your quote you meant to say 'total round trip latency'. There are strict definitions for 'response time', 'latency' and 'realtime system' and you can't just mix them. The last sentence in your statement may sound impressive but I can parse it only as nonsense jumble of buzzwords.


5) more on Beatrix, optimizations and misc rants

The original Beatrix was not sample accurate, she combines all midi events in a 128 sample cycle and can skip note on/off events that arrive in the same cycle.
Furthermore the code defines BUFFER_SIZE_SAMPLES to 128 in src/tonegen.c:231, and yes it's samples, not bytes.

In any case, organ players are usually vary good to compensate for latency (think church organs), 5ms is very acceptable (about 1.6 meters of sound in air, usual distance of a leslie to ear anyway), but a jitter of 6ms even drives most of those guys mad :)

As for CPU usage, I can run setBfree reliably on a 1GHz ARMv7 (armhf) with jackd at 64 frames with a DSP load of about 10% (without the additional new IR speaker emulation). The total RSS is about 80 MBytes. While the latter can certainly be optimized, I deem in not important at this point in time. The trade-offs during setBfree development were done keeping in mind that more RAM is easier to come by than faster CPUs.

Reliable low latency, robustness and reproducibility are much more important. The original beatrix failed on all these accounts (it can produce denormals, it's not sample-sync: feeding it the same input twice from a sequencer does not produce identical results,..)


6) last but not least.

I'll be happy to try your version, it currently does not even compile out of the box but that's easily remedied: missing -lm

There may be parts in your work that could be added to setBfree (which currently optionally supports ALSA MIDI, but no ALSA audio), heck it may even supersede (parts of) setBfree, however from skimming the code it looks like you just turn your back to users who have pre-existing beatrix .cfg and .pgm files (hardcoded the midi split and CCs), as for other parts of the source, I cannot yet comment on, but I'll check it out. Currently i'm a bit wary about the licensing situation of your endeavour and it'd be great if that could be remedied first.

I'm not interested in a pissing contest whose software is "better", my goal is to aim for best given the available resources and make that easily available to musicians. Fragmentation, while it may be necessary in the short term, is rarely beneficial in the long run. Let's hope we can consolidate things somewhere down the road.

yours truly,
robin

[j_e_f_f_g]

Hi Robin,

setBfree is documented in the apps wiki. calling it Abandonware is a bit of a stretch.

No, no, no, I was referring to beatrix as abandonware, not setBfree which I didn't even know about. The web page google found for me was http://people.dsv.su.se/~fk/beatrix_home.html and it seemed legit. I read the page, but didn't see any mention of setBfree at all, and no wiki. There appears to be just one outside link to a "Arnout Engelen's Beatrix page" which is 404. Right below this is an indication that the last released update is nearly a decade ago. I clicked on the download link and went to http://people.dsv.su.se/~fk/beatrix_download.html. Again, no mention of setBfree nor wiki, but there was the download. I don't recall seeing any mention of either in the materials I downloaded. On the otherhand, I did see a codebase that was clearly not updated in awhile. So that's why I assumed beatrix was abandonware.

I have no idea whether http://rg42.org/wiki/beatrix is the "real" beatrix page, and I got directed to something else. Confusing.

You cannot simply slap on the GPL to your copy of beatrix. You can however fork the very first commit of setBfree... re-base your changes

Yep, done. I've made a new version based on all the changes you've done with the following exceptions:

1) I already started my own GUI, so nothing GUI-related is included.
2) I don't see the point of making the osc waveform float instead of short. The latter uses less RAM, and is converted to float later at the mixing stage where the FPU, SSE, and added headroom matter.
3) I didn't include the programs nor config stuff because I was already planning to implement them in an entirely different way... until I discovered Calf Organ.

Your claim about jack latency and context-switches is similarly half true.

Everyone keeps claiming "JACK doesn't add latency" and saying I'm wrong to insist otherwise. But when the overhead of what JACK does forces you to set your alsa MMAP buffer size 4X as large on the jack version of your app as you can on the alsa version, to avoid underuns on either... then that is increased latency. Let me state this explicitly; that is genuine, bonafide, guaranteed, unavoidable, measurably 4 times as much latency. Jeremy's test results of setBfree versus my beatrix match my own finding almost exactly. The former requires several times greater an alsa dma period size (read:real-time latency) to achieve the same underrun performance. It's jack.

I have a friend who made his own loudspeakers. Trouble is, he didn't design the enclosure for his particular drivers, so the bass is really boomy and muddy. And he loves it. I could get a pink noise gen and test equipment, and show him why he "shouldn't" love it, but that doesn't matter. That boom and mud is more than fine for him. This is the same rationale behind the "JACK doesn't add latency" (when in actual practice it does) mantra. Yes, I understand that a difference between 4 ms and 40 ms may be meaningless to a given individual. But to someone else it may determine whether something is even playable.

I can feel the extra bit of lag with setBfree. And it's due to needing a bigger dma period to account for jack's overhead.

I did measurements (using jack-midi...<versus> the alsa-sequencer.

You lost me at "alsa-sequencer". Even though it still trounced jack-midi, alsa-sequencer is the abomination that started the whole misguided trend of attempting real-time interoperability between processes instead of coming up a well-developed plugin system. Jack took things further.

alsa-sequencer should be deprecated and buried, just like Microsoft had the sense to do with its similiar construct -- the stream api. The difference here -- every pro audio/midi Windows dev knew what a fundamentally flawed concept it was and instead went on to develop/adopt the now renowned and greatly sought-after Windows plugin ecosystem.

comparing https://github.com/koppi/alsa-midi-latency-test with https://github.com/x42/jack_midi_latency

line #263 of the aforementioned alsa-midi-latency-test.c:

Code: Select all

err = snd_seq_open(&seq, "default", SND_SEQ_OPEN_DUPLEX, 0);
check_snd("open sequencer", err);

Deprecate it, bury it, get the townsfolk to chase it with burning torches.

See also http://gareus.org/wiki/midilatency

Uses the above tool. Inexplicably does not chase it with a burning torch.

and http://lists.linuxaudio.org/pipermail/l ... 97296.html

Again.

Conclusion: Townsfolk have run out of matches.

All process callbacks inside every jack client are run in the same context (jack's).

No, you must be referring to what jack calls an "internal client". This is source code compiled into a "shared object" (i.e. .so file) that jack directly loads into its process space.

Every client that starts its own process, such as setBfree, ardour, carla, and every other jack app i've seen (never even seen an internal client in the wild) is instead an "external client". Your process calls the userspace jacklib function jack_activate() which makes your process create a pthread running in your address space. See line 2375 in libjack/client.c where your call to jack_activate calls jack_start_thread() which calls jack_client_create_thread (at line 2285), and then we're in libjack/thread.c where it looks like a call to jack_thread_creator() is made at line 156. jack_thread_creator is a function pointer pointing to pthread_create. In other words, this is a call to pthread_create(). (Oof! The abstraction!) That's your process creating a pthread running in your address space. And what's the code your process runs? It's right there in libjack.so loaded into your address space -- not anything in jackd's process which is where the data is eventually handed to alsa. It's jack_process_thread_work() at line 2087 in libjack/client.c.

Prior to calling jack_activate, note that you've called lots of other libjack functions such as jack_client_open and jack_port_register. I won't walk you through these, but I'll note that the whole purpose of these is to setup access to shared audio buffers that are read/written by 2 separate processes -- 1 thread running in jackd's address space (jack server()), and 1 thread running in your space (jack_process_thread_work()). This is what it means to be an external jack client. The purpose of your thread is call your own process callback. (yes your thread does that and i'm going to walk you through to show you where, when, and how). The purpose of jackd's thread is to tell each client's thread exactly when to call its own process() so that no 2 clients are writing to the shared buffers simultaneously. And of course to let alsa know when the last client thread is done and now alsa can have the data.

So your statement above is quite incorrect.

And with a shared buffer, you need to arbitrate access to it, and a way for jackd to tell you when to read/write, and for you to tell jack when you finished. Here's where the file descriptors and polling comes in.

Let me summarize the salient things done by jack_client_open et al (all the stuff prior to calling jack_activate). Jackd has given you several file descriptors it created for you. They are stored in the jack_client_t struct also allocated for you. The first couple (in the jack_client_t->pollfd[] array) are what your thread polls when you're waiting for jackd to tell you when to call your own process callback or when jackd needs to "alert" you about something. Another (jack_client_t->graph_next_fd) is what jackd's thread polls when it's waiting for you to report when your process callback is done.

The other thing the libjack (port register) functions do is make sure you obtain a pointer to a shared memory buffer where jackd wants your process callback to write your audio data, and/or a shared buffer where jackd will be giving you other app's data, plus the size of these buffers. (All are the same size). These pointers/size are also stored in the data struct allocated by and for you.

And do I need to mention that just coordinating this "client init/setup" with the jack daemon also involves use of several file descriptors, mutex locking, and context switches?

So the first thing your jack_process_thread_work() does (when it gets into its while loop, which loops continually until your main thread disconnects from jack) is call jack_cycle_wait at 2132 of libjack/client.c which calls jack_client_core_wait() at 2189. The first thing your thread does in jack_client_core_wait is poll() on the numerous file descriptors jack gave you (line 2007). Your thread's waiting for jackd to tell it to do something. What is jackd's thread doing? It's also waiting on a file descriptor (fd) it shares with alsa -- waiting for alsa to tell jackd to service the sound or midi hardware. Let's say alsa wants some audio data to play. Alsa signals the fd and jack wakes up. Where's the data? jackd doesn't have it. You do. So jackd writes some bytes ("jack instructions") to one of your fds which wakes your thread up. (Note: everytime jackd does this that means a process context switch). At line 2064 you call jack_client_process_events() which is where you read those jack bytes (line 1745) if jackd is notifying you of some special "event" such as an underrun. For such a special event, jackd will have written to your client->pollfd[EVENT_POLL_INDEX] fd. But that isn't the case if jackd wants you to call your process callback. In this case, jackd writes to your other client->pollfd[WAIT_POLL_INDEX] fd. jack_client_process_events() sees that it wasn't given a special event, so it does nothing, and just returns 0 (ie no error).

Now we're back in jack_client_core_wait line 2064. At line 2070 you detect that jackd has written to your client->pollfd[WAIT_POLL_INDEX] fd, so you say "Time for me to return from this wait function. jackd wants me to call my process callback." You break this while loop (2072) and return 0 (2080).

Now we're back in jack_cycle_wait (2189). You returned 0, so you drop down and call your own sync client callback (2205), if you previously registered one via jack_set_sync_callback (336 in libjack/transclient.c). Then you return client->engine->buffer_size. That's the size of the shared buffer your process callback is going to fill. libjack set this for you when you called jack_port_register.

Now we're back in jack_process_thread_work (2132). Here it comes. You did indeed return client->engine->buffer_size, so there's you calling your own process callback (2141), passing the buffersize and whatever void * you specified to jack_set_process_callback (2677). There it is.

Now your own thread is in your process() callback. You call jack_port_get_buffer which retrieves that shared buffer pointer libjack set for you when you called jack_port_register. You fill it in, and return.

Now we're back in jack_process_thread_work (2677). What have jackd/alsa been doing? Waiting for you to signal when you're done with the buffer. they can't know when because it's not them running your callback. It's your own process. Specifically, jackd's thread is polling your jack_client_t->graph_next_fd, waiting for you to write to it. You call jack_cycle_signal (2151) which does more things before it calls jack_wake_next_client (2234) which does that write (1891).

Now jackd's thread wakes up... after a process context switch of course. Jackd does more bookkeeping and then finally says to alsa "here's your next period of data". And this dance repeats for each and every cycle.

Now here's how my alsa apps do it. I wait for alsa to signal an fd that tells me it wants data. Alsa does, so I fill the buffer and say "here's your next period of data". And that's what accounts for the 4X difference in buffer size, and consequent latency.

You can read up on this jackd/engine.c of the jack1 source-code.

To paraphrase a movie line "I think you think I don't know as much about jack as I do".

Everything above I learned solely from my own code review of jack... especially engine.c

You can even 'witness' it using gdb.

How did you step through that code and not see the above, not to mention none of the fds/polls???

jack does not poll anything

jack_client_t->graph_next_fd and that's just one of many you'll find in jack.

Could you please explain what you mean by "many instances of file descriptor polling" and where you see that?

I pretty much stepped-debugged you through several of them just on an audio buffer fill. They're also used in midi handling, session handling, transport sync handling, client registering, all over. Paul loves fds.

skimming the code it looks like you just turn your back to users who have pre-existing beatrix .cfg and .pgm files (hardcoded the midi split and CCs)

That's the stuff I didn't finish, but had plans to. Not sure if I'll get to it. I just looked at Calf organ, and i think that codebase does what beatrix (and azr3) does and more, and is so far ahead in its development that it's where folks should consolidate the efforts. Have you evaluated it (and the calf leslie too!)?

[x42]

Hi Jeff,

No, no, no, I was referring to beatrix as abandonware, ..

The beatrix page on the wiki has a link to setBfree, as does the annoucement on LAU, LAA, etc, but that's water unter the bridge now.
Will asked Fred to add a link to his old beatrix page, but it seems he's AWOL or something

Everyone keeps claiming "JACK doesn't add latency"

I'm not sure who 'everyone' here is, but I'm pretty sure I never heard any of the jack-developers say it.

JACK does not add lateny as long as you do stay within JACK.

If you take the complete system (Soundcard, Bus, OS, ALSA, JACK) into account, things are a bit more complex but still JACK provides the same latency that you get with identical ALSA settings, nothing more, nothing less (unless you use jackd's -n option or jack2 in async mode but the former is pretty much an ALSA setting as well).

JACK does add some overhead (more CPU) which in turn may force you switch to larger periods, but that overhead is minimal. Please go ahead an measure or profile it. It's tiny and entirely neglectable on any well tuned system. Sure, one could construct cases where it matters, but they're usually of no real world concearn on any OS these days, and even less so on dedicated systems thanks to efforts of various GNU/Linux distributors who provide out-of-the-box well tuned systems.

I can feel the extra bit of lag...

I feel a lot of things as well. I cannot reproduce your feeling here.

Measurement here does not support your claim that it does add extra latency (see below).

Conclusion: Townsfolk have run out of matches.

Some of those tests do include alsa raw midi (not alsa sequencer). e.g. the ones at http://lists.linuxaudio.org/pipermail/l ... 97296.html

[long detailed jack codebase analysys

That's great! It looks like you did your homework properly. Kudos.

Can you also conclude that this overhead matters on an modern CPU? or any CPU built since the year 2000?

Now here's how my alsa apps do it. I wait for alsa to signal an fd that tells me it wants data. Alsa does, so I fill the buffer and say "here's your next period of data".

I fail to see what latency has to do with all this. CPU usage, yes, but latency, no.

And that's what accounts for the 4X difference in buffer size, and consequent latency.

Where does this 4x buffer suddenly come from? The number of samples processed and bufferd at any given time is identical in both approaches, is it not?

How did you step through that code and not see the above, not to mention none of the fds/polls???

using breakpoints and watches, I suppose.

I just looked at Calf organ, and i think that codebase does what beatrix (and azr3) does and more, and is so far ahead in its development that it's where folks should consolidate the efforts. Have you evaluated it (and the calf leslie too!)?

yes, also the Csound organ, bristol and various other projects, sadly they don't even come close.

best,
robin

[raboof]

Everyone keeps claiming "JACK doesn't add latency" and saying I'm wrong to insist otherwise. But when the overhead of what JACK does forces you to set your alsa MMAP buffer size 4X as large on the jack version of your app as you can on the alsa version, to avoid underuns on either... then that is increased latency.

Let me fix the emphasis: when the overhead of what JACK does forces you to set your alsa MMAP buffer size 4X as large on the jack version of your app as you can on the alsa version, to avoid underuns on either... then that is increased latency.

Jeremy's test results of setBfree versus my beatrix match my own finding almost exactly. The former requires several times greater an alsa dma period size (read:real-time latency) to achieve the same underrun performance.

That's cool, I've many times asked you for a realistic and reproducable test case to show JACK CPU overhead causing enough trouble to affect latency. Looks like finally we're getting close.

This is the same rationale behind the "JACK doesn't add latency" (when in actual practice it does) mantra. Yes, I understand that a difference between 4 ms and 40 ms may be meaningless to a given individual.

No. When I say JACK 'does not add latency', I mean the CPU overhead it adds is not sufficient to force me to increase my buffer size.

[x42]

Jeremy's test results of setBfree versus my beatrix match my own finding almost exactly. The former requires several times greater an alsa dma period size (read:real-time latency) to achieve the same underrun performance.

That's cool, I've many times asked you for a realistic and reproducable test case to show JACK CPU overhead causing enough trouble to affect latency. Looks like finally we're getting close.

Running jackd at -r 48000 -p64 -n2 setBfree adds about 1% DSP load according to jack_cpu_load on my 1.6GHz i386. and htop report both jackd as well as setBfree itself under 5% CPU. no crackles no x-runs.
The patched version of beatrix is the same, but i have no means to measure the DSP load or assess its I also did not set any RT priority to beatrix. The Total is about 5% CPU (same as jackd+setBfree).

There's no reason to increase any buffersize at all, nor any indication that beatrix performs better so far.

@Autostatic: Could you please clarify where you hear those crackles? I assume they're in jeff's version of beatrix otherwise you'd have called them x-runs, right? - If they're in setBfree, could you please provide more info about jack DSP load and jack settings that you use as well as the version and variant (LV2, standalone,...) of setBfree in question. Thanks.

[tramp]

In well-tuned systems I think is not worth the hassle (comparing a full featured JACK session vs one app only).

Exact!! A application which running as single client and block alsa is out of interest for me as well.

[autostatic]

@Autostatic: Could you please clarify where you hear those crackles? I assume they're in jeff's version of beatrix otherwise you'd have called them x-runs, right?

In Jeff's version of beatrix they coincide with 'underrun' being printed, so I assume those are x-runs. This happens as soon as I go lower than a buffer setting of 16. I'd like to know what this buffer setting refers to, how do I perceive this number? How does it relate to Jack's buffer settings?

- If they're in setBfree, could you please provide more info about jack DSP load and jack settings that you use as well as the version and variant (LV2, standalone,...) of setBfree in question. Thanks.

• setBfree standalone version 0.7.3 with the default config file patch
• jackd 0.124.1 with -Xalsa_midi
• I've tried with different JACK settings with my onboard card and the crackling starts with -p32 -n3 -r48000. If I increase the number of periods to 6 and decrease the number of frames/period to 16 the crackles disappear (this halves the JACK latency though).
• JACK DSP load with -p32 -n3: 18% | system load: JACK 13%, setBfree 6%
  with -p16 -n6: load varies from 18% to 50% | system load: JACK 10%, setBfree 5%
• Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz
• System not optimized for real-time, low-latency audio (generic 3.8 kernel)

With an USB2 audio interface crackling starts with -p8 -n3 -r48000, everything's ok at -p16 -n3. Jeff's beatrix starts crackling at a buffer setting of 1. Same set-up with some minor tweaks (disabling bluetooth and webcam). -p8 -n3 yields a JACK latency of 0.5 ms. Dividing that by four would yield a latency of 0.125ms, is that even possible?

[x42]

PS. To amend to my previous message

64 spp is the lowest reliable sampler/period that I can use my (USB1 and USB2) devices.
The UA-25 (USB1.1) can go down to 16 and the 1818VSL (USB2) to 32 but either way they do not run reliably with less than 64 spp.
This is true for plain ALSA as well as jack via ALSA likely due to USB limitations (either chipset and or the device itself).

In any case 64spp provides me with 6ms total round-trip latency and works reliably (no jack x-runs all for days under normal operation).
measured with jack_delay: http://robin.linuxaudio.org/tmp/vsl1818latency.png
The overhead is not due to JACK or ALSA but due to the stack underneath both.

Ideally one would measure the smallest period-size at which no underruns occur, but that is not possible with my hardware.
Jackd will certainly loose that against plain ALSA but probably only by a sample or two. The jack overhead is some microseconds per period.
But that also assumes that the ALSA I/O thread is using elevated scheduling privileges (jackd does that by default, many ALSA apps don't do that).

Another important factor is MIDI jitter. Getting a plain ALSA app to reliably synchronize audio + MIDI is far from trivial.

yes, you can do better than jack, but no, I have not yet seen a single app the actually does better.

[x42]

Thanks Jeremy,

That makes sense.

And frankly. for playing a live organ even slightly larger period settings are well acceptable sometimes even preferable.
Most of the real analog devices have similar response times due to mechanics involved.

-p8 -n3 yields a JACK latency of 0.5 ms. Dividing that by four would yield a latency of 0.125ms, is that even possible?

That's the rate jack runs at and yes that's possible. But it may not be the actual round-trip latency. Under the hood the kernel as well as bus do buffer and align.

The Application (jack or alsa) will receive the audio in chunks of 8 samples, while in fact the underlying system does exchange larger buffers. Unless you know the complete stack precisely, the only way to check this is to measure it.
jack_delay (or the fork of it that comes with jack: jack_iodelay) exists for this purpose. As a side note: this is how Apple does it, they do know exactly what is in their machines and they do not allow you to change it. For Windows, some ASIO drivers + professional soundcards do have a hidden built-in mode, equivalent to jack_delay, to measure the additional latency and then compensate for it.

[j_e_f_f_g]

Wait a minute! Call me slow, but I just looked at that fri interview with a Robin Gareus, and put 2+2 together. You're that robin, right?? You must be. I just looked at the AUTHORS file of setBfree. And in the interview, you say your linux involvement is "At the time of writing mostly Ardour3"?? You're working on the first, and still most jack-intensive, app of all -- the testbed for jack development and the baby of Paul Davis, who wrote jack? You're working with Paul? (He's still doing ardour, right?)

And you're the guy who replied to my jack statements with "All process callbacks inside every jack client are run in the same context (jack's)." and "jack does not poll anything"? And me, a Windows dev foremost, is the guy who did the above detailed line-by-line walkthrough of the jack sources explaining where, when, and how not only does jack do the exact opposite -- it does exactly what i initially said it does!

And look at this AUTHORS list! There's Takashi Iwai! You're working with the daddy of ALSA! How could you be working with these 2 and then have the above occur? I'm definitely not intending this to be a personal attack on anyone, and apologies if you see it otherwise, but i seriously must ask... Do you not find this situation as foreboding and unnerving as I do?? I, the Win dev who hasn't contributed a single line to jack or ardour, should have been the guy who wrote your post. And you should have been the guy who wrote mine! What exactly went wrong to careen us into this Bizarro world?

Is it even possible that, out of all the devs using jack, a guy who really dislikes it (and a Win dev too) may have become significantly more knowledgeable about what the jack source actually does, and how it works??? Because I've had these discussions with numerous jack-using devs, and I end up being the one who always talks about what the actual jack code does. And they always do that hand-waving about theoretical latency, and follow with mere truth-by-proclamation accusations "you don't understand, and don't know what you're talking about". I'm left wondering, among all these folks, how shocked I would be by the contrast between mine, and their answers to a question like "what does jack_open_client actually do?".

I find this alarming. It means the very foundations of linux audio aren't adequately understood/documented. So what's gonna happen when the old guard fades (and they will), and the new guard comes in, is the same vicious cycle that happens every so many years. The new guys are gonna look at the old guard work and say "this code is so involved, and so poorly documented and uncommented, and not even understood by anyone hacking at it today... Let's just throw the whole thing out, and start a new replacement from scratch. Yes, let's go back to square 1. It's easier.". If this behavior was happening at Microsoft as often as it does in linuxland, Ballmer and Gates would be throwing chairs and having every dev's head on a platter. Seriously, people would be getting fired right now.

This behavior does a few things:
1) It knocks linux support further back behind the competition, giving folks like MS and apple any breathing room needed to recover from any missteps/missed opportunities.
2) It typically results in the new guys making the same (or worse) mistakes as the old guys, because nobody even knows what his ancestors actually did, and generations aren't learning from each other, (ie, history repeats itself).

And so it goes. And here I am -- a guy who emphatically believes it's (what others dismiss as) "little things" most holding linux back. One example is the appalling lack of detailed, accurate linux documentation. I've seen too much fail from folks who think "If we just build it, they will come. We can get away with neglible attention to little details like documentation, marketing, support, etc". They built the Amiga. Nobody came. They built OS/2. Nobody came. They built NeXtstep. Nobody came. They built BeOS. Nobody came. It's not that they didn't build it, and often build something better than the market leaders. It's that they then let neglect of the little details kill them.

I was there for it all.

So I find this linux situation alarming, and I need to emphatically say to linux audio devs:

Stop! Put down your compilers right now! Stop, everyone! I don't care what you're writing! Stop now!

Don't write a single new line of code... nothing!... until AFTER you've gone back and documented everything you've already written and left out there!!! And I mean real docs like what we get on MSDN. Not grinding your code through doxygen. That ain't docs. That's barely above what MS puts in a .h header. I mean pick up a text editor. Write human words and arrange them into sentences where you at least try to make it understandable to mere laymen, but certainly an overwhelming number of devs. Human language written by the human authority on that code.

Go back and profusely comment your sources. You don't have to be as anal about that as me, where there's almost a 1x1 ratio of code lines to comments. But put comments all over. They should function like a flowchart inlined right in the source. If a dev spends a couple mins looking at one of your funcs, then I ask him how the func works and he can't tell me... then you haven't commented it enough. Go back and do it right.

Document, document, document!

For example, document jack so we don't end up up stuck in a confused parallel universe where the roles of Windows and ardour devs have been reversed.

Document ALSA so that the ratio between number of funcs versus amount/quality of docs is not bad enough to make it an unfathomable black box to so many devs.

And after you're done documenting/commenting, do not grab that compiler yet. First, code review others' (hopefully now well-documented) software you're relying upon, and learn what it really does and how it does this. Especially if you're potential new guard. So when it's your turn at the helm, you won't have to say "This is too much an fathomable mess. I have to start from square one".

Do it right now, before it gets worse.