How to install low-latency kernel

Optimize your system for ultimate performance.

Moderators: MattKingUSA, khz

finotti
Established Member
Posts: 528
Joined: Thu Sep 01, 2011 9:07 pm
Has thanked: 86 times
Been thanked: 23 times

Re: How to install low-latency kernel

Post by finotti »

sunrat wrote: Mon Mar 13, 2023 9:40 pm

I get "Sanity Check Failed!" with the backports 6.0 kernel on Debian. Will reboot to Liquorix and check that.

Back in the latest Liquorix 6.1.18-1-liquorix-amd64 now.
Sanity check passes "Sanity Check OK!" when using performance governor, but fails with ondemand governor. CPU governor seems to be the critical factor.

Interesting... My system fails Ardour's test even with performance governor:

Code: Select all

$ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
performance
performance
performance
performance
performance
performance
performance
performance

$ /opt/Ardour-7.3.0/bin/sanityCheck -a
Sanity Check Failed!
sunrat wrote: Mon Mar 13, 2023 9:40 pm

Rebooted to the Debian kernel 6.0.0-0.deb11.6-amd64 and found the same result. Sanity check OK with performance governor and fails with powersave (this kernel doesn't have ondemand governor).

We must remember the sanity check is just a simple parameter check afaik; it doesn't check actual performance.

I was curious about that. If that is really the case, I need to do some more tests to see if there is any loss in performance with my new kernels.

sunrat wrote: Mon Mar 13, 2023 9:40 pm

There's a simple GUI utility available in Debian to switch governors - cpupower-gui. :wink:
And it is well worth checking setup with rtcqs - https://codeberg.org/rtcqs/rtcqs . It would tell you to set performance governor.

I've just ran it:

Code: Select all

$ rtcqs
rtcqs - version 0.5.3

Root User
=========
[ OK ] Not running as root.

Audio Group
===========
[ OK ] User finotti is in the audio group.

CPU Frequency Scaling
=====================
[ OK ] The scaling governor of all CPU's is set at performance.

Kernel Configuration
====================
[ OK ] Valid kernel configuration found.

High Resolution Timers
======================
[ OK ] High resolution timers are enabled.

Tickless Kernel
===============
[ OK ] System is using a tickless kernel.

Preempt RT
==========
[ WARNING ] Kernel 6.2.0-5.slh.1-aptosid-amd64 without 'threadirqs' parameter or real-time capabilities found. See also https://wiki.linuxaudio.org/wiki/system_configuration#do_i_really_need_a_real-time_kernel

Spectre/Meltdown Mitigations
============================
[ WARNING ] Kernel with Spectre/Meltdown mitigations found. This could have a negative impact on the performance of your system. See also https://wiki.linuxaudio.org/wiki/system_configuration#disabling_spectre_and_meltdown_mitigations

RT Priorities
=============
[ WARNING ] Could not assign a 80 rtprio SCHED_FIFO value due to the following error: [Errno 1] Operation not permitted. Set up limits.conf. See also https://wiki.linuxaudio.org/wiki/system_configuration#limitsconfaudioconf

Swappiness
==========
[ OK ] Swappiness is set at 10.

Filesystems
===========
[ OK ] The following mounts can be used for audio purposes: /, /gnu/store, /media/music, /media/sd_data, /media/data3, /media/media, /media/data2, /media/sg4tb
[ WARNING ] The following mounts should be avoided for audio purposes: /run/user/1000/doc. See also https://wiki.linuxaudio.org/wiki/system_configuration#filesystems

IRQs
=====
[ OK ] USB port ehci_hcd:usb2 with IRQ 16 does not share its IRQ.
USB port ehci_hcd:usb1 with IRQ 23 does not share its IRQ.
USB port xhci_hcd with IRQ 29 does not share its IRQ.

Power Management
================
[ WARNING ] Power management can't be controlled from user space, the device node /dev/cpu_dma_latency can't be accessed by your user. This prohibits DAW's like Ardour and Reaper to set CPU DMA latency which could help prevent xruns. For enabling access see https://wiki.linuxaudio.org/wiki/system_configuration#quality_of_service_interface

I will investigate the results later. Thanks for pointing it out. (I had seen it before, but forgotten...)

sunrat wrote: Mon Mar 13, 2023 9:40 pm

I see @finotti also has an interesting thread going at https://discourse.ardour.org/t/system-d ... /108331/13

Yes, I meant to link it above, but it seems that I forgot.

merlyn
Established Member
Posts: 1392
Joined: Thu Oct 11, 2018 4:13 pm
Has thanked: 168 times
Been thanked: 247 times

Re: How to install low-latency kernel

Post by merlyn »

The 'dynamic' part of PREEMPT_DYNAMIC happens at boot time. It's kind of One kernel to rule them all. After the kernel has booted it's the same pre-emption that's in a low-latency kernel. A dynamic kernel has boot options preempt=none/voluntary/full. The same kernel can then be used for a server or a desktop with the appropriate boot option. The default is full.

It seems the 6.0 kernel you tried may have had the default changed for preempt=.

User avatar
sunrat
Established Member
Posts: 915
Joined: Wed Jul 22, 2020 2:08 pm
Has thanked: 151 times
Been thanked: 242 times

Re: How to install low-latency kernel

Post by sunrat »

finotti wrote: Wed Mar 15, 2023 12:57 pm

Code: Select all

$ rtcqs
rtcqs - version 0.5.3
...
Preempt RT
==========
[ WARNING ] Kernel 6.2.0-5.slh.1-aptosid-amd64 without 'threadirqs' parameter or real-time capabilities found. See also https://wiki.linuxaudio.org/wiki/system_configuration#do_i_really_need_a_real-time_kernel
...
RT Priorities
=============
[ WARNING ] Could not assign a 80 rtprio SCHED_FIFO value due to the following error: [Errno 1] Operation not permitted. Set up limits.conf. See also https://wiki.linuxaudio.org/wiki/system_configuration#limitsconfaudioconf
...
Power Management
================
[ WARNING ] Power management can't be controlled from user space, the device node /dev/cpu_dma_latency can't be accessed by your user. This prohibits DAW's like Ardour and Reaper to set CPU DMA latency which could help prevent xruns. For enabling access see https://wiki.linuxaudio.org/wiki/system_configuration#quality_of_service_interface

All of these are OK on my system with Liquorix kernel. Do you have threadirqs set as a kernel boot option? Do you have limits.conf?
I also have the udev rule for cpu_dma_latency.

What proportion of gain is attainable from each of these settings I don't know, but it doesn't hurt to set them. If you don't get Preempt RT showing OK after setting threadirqs, it's the kernel as @merlyn said.
I would have thought the slh kernel is fine for low latency; the siduction kernel is IIRC.

I just had a quick look at the SanityCheck utility in Okteta. It basically just checks some of the same parameters as rtcqs.

User avatar
miuzik
Established Member
Posts: 76
Joined: Sun Aug 25, 2013 1:42 pm
Has thanked: 4 times
Been thanked: 17 times

Re: How to install low-latency kernel

Post by miuzik »

Well, now I have the kernel 6.0, but What is the modifications that I have do?

In the file /etc/default/grub Should I add or modify the next lines?:

GRUB_CMDLINE_LINUX_DEFAULT="quiet threadirqs"
CONFIG_PREEMPT_DYNAMIC=full

And the grub file is an only one file or exist a file for each kernel that I have installed (?).

User avatar
sunrat
Established Member
Posts: 915
Joined: Wed Jul 22, 2020 2:08 pm
Has thanked: 151 times
Been thanked: 242 times

Re: How to install low-latency kernel

Post by sunrat »

miuzik wrote: Fri Mar 17, 2023 12:46 am

Well, now I have the kernel 6.0, but What is the modifications that I have do?

In the file /etc/default/grub Should I add or modify the next lines?:

GRUB_CMDLINE_LINUX_DEFAULT="quiet threadirqs"

Yes

CONFIG_PREEMPT_DYNAMIC=full

No. As previously mentioned, it's a compile time option. You can check if your kernel has preempt options with:

Code: Select all

grep -i preempt /boot/config-$(uname -r)

If it doesn't have the options you want, you need a different kernel.

And the grub file is an only one file or exist a file for each kernel that I have installed (?).

Only one.

User avatar
miuzik
Established Member
Posts: 76
Joined: Sun Aug 25, 2013 1:42 pm
Has thanked: 4 times
Been thanked: 17 times

Re: How to install low-latency kernel

Post by miuzik »

sunrat wrote: Fri Mar 17, 2023 3:15 am

CONFIG_PREEMPT_DYNAMIC=full

No. As previously mentioned, it's a compile time option. You can check if your kernel has preempt options with:

But Where I write the "preempt=full" option? I suppose that kernel v. 6 has the dynamic option active

Here my output for grep -i preempt /boot/config-$(uname -r):

Code: Select all

CONFIG_PREEMPT_BUILD=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_COUNT=y
CONFIG_PREEMPTION=y
CONFIG_PREEMPT_DYNAMIC=y
CONFIG_PREEMPT_RCU=y
CONFIG_HAVE_PREEMPT_DYNAMIC=y
CONFIG_HAVE_PREEMPT_DYNAMIC_CALL=y
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_DRM_I915_PREEMPT_TIMEOUT=640
CONFIG_DEBUG_PREEMPT=y
# CONFIG_PREEMPT_TRACER is not set
# CONFIG_PREEMPTIRQ_DELAY_TEST is not set
User avatar
miuzik
Established Member
Posts: 76
Joined: Sun Aug 25, 2013 1:42 pm
Has thanked: 4 times
Been thanked: 17 times

Re: How to install low-latency kernel

Post by miuzik »

I think the Dynamic Preemption Support allows me to setting the "preempt" option at boot time without rebuilding the kernel or install a new kernel. Am I wrong?

EDIT 1:

In kernel v.6 I search in "dmesg" the "Dynamic" line:

sudo dmesg | grep Dynamic

Give me the next output:

Code: Select all

[    0.075938] Dynamic Preempt: voluntary
[    0.168080] ACPI: Dynamic OEM Table Load:

The options for the Dynamic Preemption are none, voluntary or full. You can see that I have active voluntary and I need full

EDIT 2:

Edit /etc/default/grub:

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="quiet threadirqs preempt=full"

sudo update-grub and restart

Now sudo dmesg | grep Dynamic and the output:

Code: Select all

[    0.038792] Dynamic Preempt: full
[    0.171827] ACPI: Dynamic OEM Table Load:
User avatar
Audiojunkie
Established Member
Posts: 392
Joined: Thu Feb 21, 2019 4:27 pm
Has thanked: 382 times
Been thanked: 153 times

Re: How to install low-latency kernel

Post by Audiojunkie »

This is correct. 🙂 You should be ready to test things now. 🙂

User avatar
sunrat
Established Member
Posts: 915
Joined: Wed Jul 22, 2020 2:08 pm
Has thanked: 151 times
Been thanked: 242 times

Re: How to install low-latency kernel

Post by sunrat »

miuzik wrote: Fri Mar 17, 2023 3:47 am

EDIT 2:

Edit /etc/default/grub:

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="quiet threadirqs preempt=full"

sudo update-grub and restart

Now sudo dmesg | grep Dynamic and the output:

Code: Select all

[    0.038792] Dynamic Preempt: full
[    0.171827] ACPI: Dynamic OEM Table Load:

Excellent! I learned something new today too. :wink:

merlyn
Established Member
Posts: 1392
Joined: Thu Oct 11, 2018 4:13 pm
Has thanked: 168 times
Been thanked: 247 times

Re: How to install low-latency kernel

Post by merlyn »

To be clear there is no problem with a preempt_dynamic kernel if it has been booted with full preemption. The boot option preempt=full may solve @finotti's problem. The default is supposed to be full, so the kernels that are booting with voluntary -- I would say they're a bit messed up.

It's not that there's anything wrong with preempt_dynamic, it's these particular kernels. To test this I ran a script that draws a plot of kernel scheduling latency for a preempt_dynamic kernel. It uses cyclictest, part of the rt-tests package.

Image

What does that mean? Well, it ran a hundred million realtime threads, (it takes about five hours) and most of them started running in less than 50us. The maximum kernel scheduling latency is 196us, and that only happened once out of a hundred million. 196us is 0.196 ms, and that's not really enough to cause an xrun. At a 64 sample buffer and 44.1kHz sample rate, there is 64/44100 = 1.45ms available to calculate a buffer, so, unless the DSP load is over 90%, 0.196ms won't be an issue. And that's the worst case, so a preempt_dynamic kernel is good for realtime audio, if booted with preempt=full.

User avatar
Audiojunkie
Established Member
Posts: 392
Joined: Thu Feb 21, 2019 4:27 pm
Has thanked: 382 times
Been thanked: 153 times

Re: How to install low-latency kernel

Post by Audiojunkie »

The nice thing too is that there is more improvement coming, once “all” of the realtime patches are mainlined. (There’s still a few more that aren’t in there yet). Last I heard, there were about 50 patches left to go.

finotti
Established Member
Posts: 528
Joined: Thu Sep 01, 2011 9:07 pm
Has thanked: 86 times
Been thanked: 23 times

Re: How to install low-latency kernel

Post by finotti »

First, thanks to all for the comments! I was out of town these past few days and could not test it.

But, I have it all almost figured out:

Code: Select all

$ rtcqs
rtcqs - version 0.5.3

Root User
=========
[ OK ] Not running as root.

Audio Group
===========
[ OK ] User finotti is in the audio group.

CPU Frequency Scaling
=====================
[ OK ] The scaling governor of all CPU's is set at performance.

Kernel Configuration
====================
[ OK ] Valid kernel configuration found.

High Resolution Timers
======================
[ OK ] High resolution timers are enabled.

Tickless Kernel
===============
[ OK ] System is using a tickless kernel.

Preempt RT
==========
[ OK ] Kernel 6.2.0-6.slh.1-aptosid-amd64 is using threaded IRQ's.

Spectre/Meltdown Mitigations
============================
[ WARNING ] Kernel with Spectre/Meltdown mitigations found. This could have a negative impact on the performance of your system. See also https://wiki.linuxaudio.org/wiki/system_configuration#disabling_spectre_and_meltdown_mitigations

RT Priorities
=============
[ WARNING ] Could not assign a 80 rtprio SCHED_FIFO value due to the following error: [Errno 1] Operation not permitted. Set up limits.conf. See also https://wiki.linuxaudio.org/wiki/system_configuration#limitsconfaudioconf

Swappiness
==========
[ OK ] Swappiness is set at 10.

Filesystems
===========
[ OK ] The following mounts can be used for audio purposes: /, /gnu/store, /media/music, /media/sd_data, /media/data2, /media/data3, /media/media, /media/sg4tb
[ WARNING ] The following mounts should be avoided for audio purposes: /run/user/1000/doc. See also https://wiki.linuxaudio.org/wiki/system_configuration#filesystems

IRQs
=====
[ OK ] USB port ehci_hcd:usb4 with IRQ 16 does not share its IRQ.
USB port ehci_hcd:usb2 with IRQ 23 does not share its IRQ.
USB port xhci_hcd with IRQ 28 does not share its IRQ.

Power Management
================
[ OK ] Power management can be controlled from user space. This enables DAW's like Ardour and Reaper to set CPU DMA latency which could help prevent xruns.

The only real problem is :

Code: Select all

RT Priorities
=============
[ WARNING ] Could not assign a 80 rtprio SCHED_FIFO value due to the following error: [Errno 1] Operation not permitted. Set up limits.conf. See also https://wiki.linuxaudio.org/wiki/system_configuration#limitsconfaudioconf

But I do have/etc/security/limits.d/audio.conf set :

Code: Select all

$ cat /etc/security/limits.d/audio.conf
# Provided by the jackd package.
#
# Changes to this file will be preserved.
#
# If you want to enable/disable realtime permissions, run
#
#    dpkg-reconfigure -p high jackd2

@audio   -  rtprio     95
@audio   -  memlock    unlimited
#@audio   -  nice      -19

Now, here is something is strange: yesterday, after coming back to town, and starting to reply to this message, my computer passed the rtcqs's RT Priorities test (and Ardour's). I really don't remember having done anything that could affect it, since I did not even know what could be the problem. The kernel was 6.2.0-5.slh.1-aptosid-amd64, the same from my failed test in the previous message.

But, a new kernel came in, so I installed and tried it, and it failed. So I rebooted with the old kernel (6.2.0-5.slh.1-aptosid-amd64 again), and it also failed!

Does anyone have any ideas on what could cause this problem?

merlyn
Established Member
Posts: 1392
Joined: Thu Oct 11, 2018 4:13 pm
Has thanked: 168 times
Been thanked: 247 times

Re: How to install low-latency kernel

Post by merlyn »

It's worth trying what miuzik did :

miuzik wrote: Fri Mar 17, 2023 3:47 am

Edit /etc/default/grub:

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="quiet threadirqs preempt=full"

sudo update-grub and restart

If the kernel you're using is booting with preempt=voluntary, changing the option to full should fix it.

finotti
Established Member
Posts: 528
Joined: Thu Sep 01, 2011 9:07 pm
Has thanked: 86 times
Been thanked: 23 times

Re: How to install low-latency kernel

Post by finotti »

merlyn wrote: Wed Mar 22, 2023 1:09 pm

It's worth trying what miuzik did :

miuzik wrote: Fri Mar 17, 2023 3:47 am

Edit /etc/default/grub:

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="quiet threadirqs preempt=full"

sudo update-grub and restart

If the kernel you're using is booting with preempt=voluntary, changing the option to full should fix it.

Thanks, but I already had that. I appreciate your help, though!

merlyn
Established Member
Posts: 1392
Joined: Thu Oct 11, 2018 4:13 pm
Has thanked: 168 times
Been thanked: 247 times

Re: How to install low-latency kernel

Post by merlyn »

Does update-grub also rebuild initcpio? Because I'm on Arch I have to do that separately. New kernels messing things up -- what does a new kernel change? grub and initcpio, the initial RAM disk the kernel boots into.

Post Reply