Improve performance and reduce latency

[steinwayer]

Hello guys!!! Good to be talking with other linux musicians!!

I am writing this post to ask your advice in regards to the latency and performance issues i am experiencing on my machine.
But first of all, let's start with the specs!!

I am currently running Kubuntu 18.04 with low latency kernel, on a Laptop with I7-8750H 6 core 12 threads, 8GB RAM, GTX 1060 nvidia 6gb VRAM and 256GB SSD.

I always used Windows in the past for professional recording, but since Linux has been my daily driver for quite a lot now, i decided to give it a shot!

So after reading and tweaking around and a few broken systems (of course on test partitions eheh) i got my system at a decent stage.
I am running Jack through Cadence, as Daw I utilize BitWig Studio and run wine coupled with wineasio in order to run windows Vsts like Kontakt 5.
I mainly use this machine with my midi controller and tons of vst, for film scoring.

Now, everything works, don't get me wrong. But I am struggling with latency and xruns.

So, after doing several tests, my resultst are as follows:

I can play with a "decently" low latency with 256 samples at 2 periods/buffer, but i still get occasional xruns (especially when i play a good amount of notes together with my midi controller) . If i try setting it at 128 samples, xruns are going insane, even when i don't play anything and when i put 64, they just go by the hundreds in a matter of seconds , since the moment i start jack, without even opening the daw.

I tried a few tweaks and workarounds found online ( like editing the file /etc/security/limits.conf by adding the rtprio set to 99 etc) but none worked.

I'd like to point out that i am using the integrated audio interface, but WAIT. Before you start saying that i need to get an external soundcard etc, hear me out cause that was my first thoght too.

I started digging around and run a few tests. One of my favourite vsts is Pianoteq, which I use since forever. The funny thing is , if i run pianoteq standalone , not as a plugin, i have EXTREMELY DIFFERENT RESULTS, (and very weird as well). THe resulsts are:
If i run Pianoteq standalone with jack , i have pretty much the same issues i have in bitwig. If I run it with Alsa instead, i can play even setting it a 64 SAMPLES!! and NO CLICKS Whatsoever!!

So i started thinking, could jack be the issue? I tried running Bitwig Studio under Alsa instead of Jack, and while the resulsts are better (oddly enough) i still get clicks here and there (of course the clicks disappear once i increase the samples , after all all i am doing is recording midi instructions) but it gets very frustrating.

So i decided to see what was the situation under Windows. I install Pianoteq on Windows, and Bitwig Studio. I Install the Asio4all drivers and... i can play at 64 samples without any problems whatsoever , both in standalone and inside Bitwig.

I tried also changing DAW to see if Bitwig was the issue, but i had the same exact results.


At this point, i really don't know what could be causing theese issues...

Do you guys have any suggestions? thougts?


Thanks so much in advance!

[CrocoDuck]

Hi!

I think the first port of call is realTimeConfigQuickScan: https://github.com/raboof/realtimeconfigquickscan

Run it and post the output back here.

[steinwayer]

Oh!!!Thanks for the reply!!
here is the output:

== GUI-enabled checks ==
Checking if you are root... no - good
Checking filesystem 'noatime' parameter... 4.15.0 kernel - good
(relatime is default since 2.6.30)
Checking CPU Governors... CPU 0: 'powersave' CPU 1: 'powersave' CPU 10: 'powersave' CPU 11: 'powersave' CPU 2: 'powersave' CPU 3: 'powersave' CPU 4: 'powersave' CPU 5: 'powersave' CPU 6: 'powersave' CPU 7: 'powersave' CPU 8: 'powersave' CPU 9: 'powersave' - not good
Set CPU Governors to 'performance' with 'cpufreq-set -c <cpunr> -g performance'
See also: http://linuxmusicians.com/viewtopic.php?f=27&t=844
Checking swappiness... 60 - not good
** vm.swappiness is larger than 10
set it with '/sbin/sysctl -w vm.swappiness=10'
See also: http://linuxmusicians.com/viewtopic.php ... t=30#p8916
Checking for resource-intensive background processes... none found - good
Checking checking sysctl inotify max_user_watches... < 524288 - not good
increase max_user_watches by adding 'fs.inotify.max_user_watches = 524288' to /etc/sysctl.conf and rebooting
For more information, see http://wiki.linuxaudio.org/wiki/system_ ... sysctlconf
Checking access to the high precision event timer... not readable - not good
/dev/hpet found, but not readable.
make /dev/hpet readable by the 'audio' group
For more information, see http://wiki.linuxaudio.org/wiki/system_ ... are_timers
Checking access to the real-time clock... not readable - not good
/dev/rtc found, but not readable.
make /dev/rtc readable by the 'audio' group
For more information, see http://wiki.linuxaudio.org/wiki/system_ ... are_timers
Checking whether you're in the 'audio' group... yes - good
Checking for multiple 'audio' groups... no - good
Checking the ability to prioritize processes with chrt... yes - good
Checking kernel support for high resolution timers... found - good
Kernel with Real-Time Preemption... not found - not good
Kernel without real-time capabilities found
For more information, see http://wiki.linuxaudio.org/wiki/system_ ... ime_kernel
Checking if kernel system timer is high-resolution... found - good
Checking kernel support for tickless timer... found - good
== Other checks ==
Checking filesystem types... ok.
** Set $SOUND_CARD_IRQ to the IRQ of your soundcard to enable more checks.
Find your sound card's IRQ by looking at '/proc/interrupts' and lspci.

[khz]

Welcome \o/

Except "max_user_watches" you can make the optimizations described in the respective link. There you can also see what they do and how they are implemented.
Use among other things the [ code ] text [ /code ] function.

Have a look at the links in my signature for meta information.

[CrocoDuck]

I see that the most important settings for low latency audio are all properly tuned. You might want to follow the links at the not-goot outputs and configure the system according to the instructions. Let's leave the realtime kernel out for now, and maybe tune the CPU to performance only while during music.

Can you share the jack settings you use? If you use qjackctl the output of this should suffice:

Code: Select all

cat ~/.jackdrc


[steinwayer]

I see that the most important settings for low latency audio are all properly tuned. You might want to follow the links at the not-goot outputs and configure the system according to the instructions. Let's leave the realtime kernel out for now, and maybe tune the CPU to performance only while during music.

Can you share the jack settings you use? If you use qjackctl the output of this should suffice:

Code: Select all

cat ~/.jackdrc


I am using Cadence, any idea on how to get the output with it?

P.S. i tried setting the cpu governor to performance, it does improve performance a bit but nowhere near the windows ones.
Just to give some context, on Windows, without the laptop being even connected to the power, on battery, i can still play at 64 samples ( the cpu of this laptop really consumes a lot, 45W and when on battery it throttles down itself, it's not capable of reachig its max potential on battery because of the consumption, i ve tested this extensively with benchmarks) and even then, it still manages to make me play at 64 samples (under windows)

Which surprised me a lot, considering it's an internal audio card!

Bit at least shows that the hardware is well capable of low latency without any hickups. So that i know that's not what we need to blame

[CrocoDuck]

I am using Cadence, any idea on how to get the output with it?

Not sure, maybe just make a screenshot of the window.

Which surprised me a lot, considering it's an internal audio card!

Not entirely surprising actually.

Let's have a look at iRQs, maybe we can do something with prioritization. Do you have rtirq installed?

it still manages to make me play at 64 samples (under windows)

Beware that you are maybe still experiencing overruns on windows too, but the system does not report them and maybe they cannot quite be heard. For example, I often have xruns that I cannot hear on Linux, while I only discovered I have buffer over/underruns in Mac Os while using a spectrum analyser.

Bet let's try to get rid of them.

[steinwayer]

I am using Cadence, any idea on how to get the output with it?

Not sure, maybe just make a screenshot of the window.

Which surprised me a lot, considering it's an internal audio card!

Not entirely surprising actually.

Let's have a look at iRQs, maybe we can do something with prioritization. Do you have rtirq installed?

it still manages to make me play at 64 samples (under windows)

Beware that you are maybe still experiencing overruns on windows too, but the system does not report them and maybe they cannot quite be heard. For example, I often have xruns that I cannot hear on Linux, while I only discovered I have buffer over/underruns in Mac Os while using a spectrum analyser.

Bet let's try to get rid of them.

Here are the screenshots





Uhm, i do not have rtirq, also i would like to point out that i didn't see any differences so far from low latency kernel and generic kernel (at least in my system)

[CrocoDuck]

Settings look good. Maybe try to uncheck "Server Synchronous Mode".

Repost the outputs of

Code: Select all

aplay -l


and

Code: Select all

cat /proc/interrupts


[steinwayer]

Settings look good. Maybe try to uncheck "Server Synchronous Mode".

Repost the outputs of

Code: Select all

aplay -l


and

Code: Select all

cat /proc/interrupts


Synchronous Mode actually reduces the latency issue. If i disable it, it gets worse.

Output from aplay -l :
**** List of PLAYBACK Hardware Devices ****
card 0: PCH [HDA Intel PCH], device 0: ALC3246 Analog [ALC3246 Analog]
Subdevices: 0/1
Subdevice #0: subdevice #0
card 0: PCH [HDA Intel PCH], device 3: HDMI 0 [HDMI 0]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 0: PCH [HDA Intel PCH], device 7: HDMI 1 [HDMI 1]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 0: PCH [HDA Intel PCH], device 8: HDMI 2 [HDMI 2]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 0: PCH [HDA Intel PCH], device 9: HDMI 3 [HDMI 3]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 0: PCH [HDA Intel PCH], device 10: HDMI 4 [HDMI 4]
Subdevices: 1/1
Subdevice #0: subdevice #0


output from cat /proc/interrupts:
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11
0: 12 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC 2-edge timer
1: 0 0 0 0 0 0 0 628 0 0 0 0 IR-IO-APIC 1-edge i8042
8: 0 0 0 0 0 0 0 0 1 0 0 0 IR-IO-APIC 8-edge rtc0
9: 0 3542 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC 9-fasteoi acpi
12: 0 0 0 0 0 0 211 0 0 0 0 0 IR-IO-APIC 12-edge i8042
14: 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC 14-fasteoi INT3450:00
16: 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC 16-fasteoi idma64.0, i2c_designware.0
17: 0 0 21323 0 188637 0 0 0 0 0 0 0 IR-IO-APIC 17-fasteoi idma64.1, i2c_designware.1
51: 0 0 0 5304 0 0 0 0 0 0 0 0 IR-IO-APIC 51-fasteoi DELL0825:00
120: 0 0 0 0 0 0 0 0 0 0 0 0 DMAR-MSI 0-edge dmar0
121: 0 0 0 0 0 0 0 0 0 0 0 0 DMAR-MSI 1-edge dmar1
122: 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 442368-edge pcie-dpc
123: 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 475136-edge pcie-dpc
124: 0 0 0 0 0 4865 0 0 0 0 0 0 IR-PCI-MSI 327680-edge xhci_hcd
125: 0 15799 0 0 0 0 0 0 0 37881 0 0 IR-PCI-MSI 376832-edge ahci[0000:00:17.0]
126: 0 0 0 0 0 54457 0 0 0 0 7422 0 IR-PCI-MSI 32768-edge i915
127: 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 30932992-edge enp59s0
128: 0 0 0 0 25 0 0 0 0 0 0 0 IR-PCI-MSI 360448-edge mei_me
129: 0 0 0 347 0 782 0 0 0 0 317 0 IR-PCI-MSI 333824-edge iwlwifi: default queue
130: 1 3 13 0 0 0 0 0 0 0 24 0 IR-PCI-MSI 333825-edge iwlwifi: queue 1
131: 0 9 0 0 0 0 0 0 0 0 0 8 IR-PCI-MSI 333826-edge iwlwifi: queue 2
132: 0 0 3 0 13 0 0 0 0 0 0 0 IR-PCI-MSI 333827-edge iwlwifi: queue 3
133: 0 11 0 132 60 0 0 0 65 32 0 0 IR-PCI-MSI 333828-edge iwlwifi: queue 4
134: 0 0 0 0 1 0 0 0 0 12 0 0 IR-PCI-MSI 333829-edge iwlwifi: queue 5
135: 0 0 0 0 0 1 0 0 0 0 13 0 IR-PCI-MSI 333830-edge iwlwifi: queue 6
136: 0 0 0 0 0 0 0 0 0 0 0 150 IR-PCI-MSI 333831-edge iwlwifi: queue 7
137: 0 0 0 0 0 0 15 10 0 0 0 0 IR-PCI-MSI 333832-edge iwlwifi: queue 8
138: 0 0 0 0 0 0 0 0 24 26 0 0 IR-PCI-MSI 333833-edge iwlwifi: queue 9
139: 0 0 0 0 0 50 0 0 18 24 0 0 IR-PCI-MSI 333834-edge iwlwifi: queue 10
140: 0 0 8 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 333835-edge iwlwifi: queue 11
141: 0 0 0 0 0 0 0 160 0 0 0 242 IR-PCI-MSI 333836-edge iwlwifi: queue 12
142: 0 0 0 0 0 0 0 33 0 0 0 0 IR-PCI-MSI 333837-edge iwlwifi: exception
143: 0 0 0 0 0 0 40865 0 0 0 0 0 IR-PCI-MSI 514048-edge snd_hda_intel:card0
144: 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 30932993-edge enp59s0-TxRx-0
145: 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 30932994-edge enp59s0-tx-1
146: 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 30932995-edge enp59s0-tx-2
147: 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 30932996-edge enp59s0-tx-3
NMI: 1 1 0 0 1 1 1 0 0 1 1 0 Non-maskable interrupts
LOC: 9287 11629 10938 9698 8763 30498 9961 112776 9096 9060 12353 9222 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 1 1 0 0 1 1 1 0 0 1 1 0 Performance monitoring interrupts
IWI: 1 1 0 1 9 4116 1 1 5 0 580 3 IRQ work interrupts
RTR: 0 0 0 0 0 0 0 0 0 0 0 0 APIC ICR read retries
RES: 3783 1104 949 373 1111 1678 1188 2200 280 324 735 267 Rescheduling interrupts
CAL: 3447 3805 3229 3166 3351 3384 3232 3286 3317 3097 3133 3363 Function call interrupts
TLB: 505 867 549 424 609 368 337 361 402 413 623 405 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 0 0 0 0 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 1 2 2 2 2 2 2 2 2 2 2 2 Machine check polls
HYP: 0 0 0 0 0 0 0 0 0 0 0 0 Hypervisor callback interrupts
ERR: 0
MIS: 0
PIN: 0 0 0 0 0 0 0 0 0 0 0 0 Posted-interrupt notification event
NPI: 0 0 0 0 0 0 0 0 0 0 0 0 Nested posted-interrupt event
PIW: 0 0 0 0 0 0 0 0 0 0 0 0 Posted-interrupt wakeup event



What is very odd again, is that if i run Pianoteq and select Alsa instead of Jack, i get very good performance and i can play at 64 samples (only in standalone tho, in bitwig or any other daw for that matter, if i select alsa, i can't play at 64 samples at all)

[CrocoDuck]

Synchronous Mode actually reduces the latency issue. If i disable it, it gets worse.

Yes, you will have longer latency. How big of a latency can you have? We can try various configurations of sample rate. buffer size and periods to get the latency we want.

By the way, I would recommend not to obsess too much over latency. According to experimental evidence that I reviewed here, latency between 5 ms and 10 ms will be good in most situations.

By the way, you can measure latency with jack_iodelay under Linux, see here.

I think there are similar tools for windows too, I think it could be interesting to measure win latency and see how it compares to the Linux latency. Maybe win is running asynchronous, with higher latency, and we don't know it.

143: 0 0 0 0 0 0 40865 0 0 0 0 0 IR-PCI-MSI 514048-edge snd_hda_intel:card0

Looks like your soundcard got a low priority, and maybe things will be at odd with jack's high priority. I would suggest to install rtirq. After you do it, reboot and repost the output of this:

Code: Select all

/usr/bin/rtirq status


It should tell us if rtirq is working, and what configuration we should try.

What is very odd again, is that if i run Pianoteq and select Alsa instead of Jack, i get very good performance and i can play at 64 samples (only in standalone tho, in bitwig or any other daw for that matter, if i select alsa, i can't play at 64 samples at all)

I think it is because BitWig is really heavy.

[steinwayer]

Synchronous Mode actually reduces the latency issue. If i disable it, it gets worse.

Yes, you will have longer latency. How big of a latency can you have? We can try various configurations of sample rate. buffer size and periods to get the latency we want.

By the way, I would recommend not to obsess too much over latency. According to experimental evidence that I reviewed here, latency between 5 ms and 10 ms will be good in most situations.

By the way, you can measure latency with jack_iodelay under Linux, see here.

I think there are similar tools for windows too, I think it could be interesting to measure win latency and see how it compares to the Linux latency. Maybe win is running asynchronous, with higher latency, and we don't know it.

143: 0 0 0 0 0 0 40865 0 0 0 0 0 IR-PCI-MSI 514048-edge snd_hda_intel:card0

Looks like your soundcard got a low priority, and maybe things will be at odd with jack's high priority. I would suggest to install rtirq. After you do it, reboot and repost the output of this:

Code: Select all

/usr/bin/rtirq status


It should tell us if rtirq is working, and what configuration we should try.

What is very odd again, is that if i run Pianoteq and select Alsa instead of Jack, i get very good performance and i can play at 64 samples (only in standalone tho, in bitwig or any other daw for that matter, if i select alsa, i can't play at 64 samples at all)

I think it is because BitWig is really heavy.

Oh i am not obsessed with latency at all, i know 5ms is more than enough, and I agree with you, Bitwig is heavy, but so it is under Windows, but in Windows i can set it at 64 samples with asio4all and have no clicks. I'd be happy to keep it at 256 (which is the highest setting at which the latency is acceptable for real time playing, over that ,i can feel it when i play)

I tried installing rtirq, but the command you gave me does not work. I installed form the official repos, it's called rtirq-init, after i install it and reboot, i dont' seem to find a command to launch it.

Any suggestions?

[khz]

rtirq-init is automatically started as a service at system startup.
The command should rather be

Code: Select all

/etc/init.d/rtirq status

(i use Init, for Systemd possibly different?), this will display an overview.
For example:

Code: Select all

GNU /home/Linux # /etc/init.d/rtirq status

PID CLS RTPRIO  NI PRI %CPU STAT COMMAND   
   81 FF      50   -  90  0.0 S    irq/9-acpi   
  105 FF      50   -  90  0.0 S    irq/24-PCIe PME   
  ...

JACK’s performance on Linux is much improved if it can use /dev/shm. See links in my signature.

[CrocoDuck]

I tried installing rtirq, but the command you gave me does not work. I installed form the official repos, it's called rtirq-init, after i install it and reboot, i dont' seem to find a command to launch it.

Any suggestions?

Right. I use rtirq on Arch Linux with systemd, so things are a bit different maybe. I will have a look on a Ubuntu virtual machine maybe. In the meanwhile other Ubuntu users will maybe fill this gap.

[CrocoDuck]

rtirq-init is automatically started as a service at system startup.
The command should rather be

Code: Select all

/etc/init.d/rtirq status

Yes, that's how it works on Ubuntu.

I tried installing rtirq, but the command you gave me does not work. I installed form the official repos, it's called rtirq-init, after i install it and reboot, i dont' seem to find a command to launch it.

Any suggestions?

Try the command above.