How to install low-latency kernel

Optimize your system for ultimate performance.

Moderators: MattKingUSA, khz

merlyn
Established Member
Posts: 1392
Joined: Thu Oct 11, 2018 4:13 pm
Has thanked: 168 times
Been thanked: 247 times

Re: How to install low-latency kernel

Post by merlyn »

@finotti

You can check the boot option worked with

Code: Select all

$ sudo dmesg | grep Dynamic

Then you can check limits.conf worked with

Code: Select all

$ ulimit -a
finotti
Established Member
Posts: 528
Joined: Thu Sep 01, 2011 9:07 pm
Has thanked: 86 times
Been thanked: 23 times

Re: How to install low-latency kernel

Post by finotti »

Thanks again @merlyn for the reply!

merlyn wrote: Fri Mar 24, 2023 1:40 pm

@finotti

You can check the boot option worked with

Code: Select all

$ sudo dmesg | grep Dynamic

That worked:

Code: Select all

$ sudo dmesg | grep Dynamic
[    0.067983] Dynamic Preempt: full
[    0.201552] ACPI: Dynamic OEM Table Load:
[    0.201990] ACPI: Dynamic OEM Table Load:
[    0.202441] ACPI: Dynamic OEM Table Load:
merlyn wrote: Fri Mar 24, 2023 1:40 pm

Then you can check limits.conf worked with

Code: Select all

$ ulimit -a

I am not sure about this one... I think it worked.

Code: Select all

 $ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 128021
max locked memory           (kbytes, -l) unlimited
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 95
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 128021
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

Again, I have

Code: Select all

$ cat /etc/security/limits.d/audio.conf
# Provided by the jackd package.
#
# Changes to this file will be preserved.
#
# If you want to enable/disable realtime permissions, run
#
#    dpkg-reconfigure -p high jackd2

@audio   -  rtprio     95
@audio   -  memlock    unlimited
#@audio   -  nice      -19

And I still fail RT Priorities in rtcqs and with Ardour's sanityCheck.

merlyn
Established Member
Posts: 1392
Joined: Thu Oct 11, 2018 4:13 pm
Has thanked: 168 times
Been thanked: 247 times

Re: How to install low-latency kernel

Post by merlyn »

Yes, it worked :

Code: Select all

real-time priority                  (-r) 95

You should be able to run realtime threads. At this point you could dig in deeper, but it would be quicker to use a kernel that does allow you to run realtime threads, if that's possible. I can't see why it isn't working.

User avatar
Audiojunkie
Established Member
Posts: 392
Joined: Thu Feb 21, 2019 4:27 pm
Has thanked: 382 times
Been thanked: 153 times

Re: How to install low-latency kernel

Post by Audiojunkie »

I think it is time to shed some light on some of the limitations of RTCQS. It truly is not the be all, end all of analysis scripts, although it is indeed very useful. It's the best we currently have, but people need to be aware of the shortcomings. RTCQS checks for the essential items, but doesn't take into account the differences in distros and settings. The last time I checked, which admittedly was a couple of years ago, the rtprio checks (in RTCQS) only looked for "Audio" as the group that a person sets up to assign themselves realtime priorities. To my knowledge, it does not consider that the Pipewire team's setup recommendations suggest to use a group named "pipewire" instead of "audio". In truth, you could name the group "pineapple" and still get a working result as long as you assigned the correct realtime priorities and memlock and added yourself to the "pineapple" group. The point I'm making, is that the script doesn't (and honestly can't) take all of these things into account. As long as you are using the group names and other associations that the script is expecting, it will be able to find and report that it has found (or not found) what it is looking for. One needs to be aware of these things when using this script. I've seen people add two identical groups for realtime priorities, to make sure at least one is the "audio" group, just to be safe. As far as I understand it, it is unnecessary, but it makes the script report correctly. I personally don't see a need for that at all.

The point I'm making, is that the script serves as a quick useful tool to check your settings, as long as you are aware of the limitations of its design. My current suspicion is that using a generic kernel with the realtime boot paramaters may not be something the script looks for--it is a relatively new development in the Linux world. I can't state this as a fact, but it is my suspicion. I would check to see that the system is utilizing the /preempt=full, and assume that you are running realtime, even if the RTCQS reports differently. There are other things you can do to check and compare to see if you are getting benefit from the kernel boot parameter, other than relying only on RTCQS--for example, before and after latency checks.

That said, I like and use RTCQS myself, and I'm grateful to have it! It is a useful tool. One just needs to be aware of its limitations. :)

merlyn
Established Member
Posts: 1392
Joined: Thu Oct 11, 2018 4:13 pm
Has thanked: 168 times
Been thanked: 247 times

Re: How to install low-latency kernel

Post by merlyn »

It's a good point -- it could be a false negative. I don't think it is though, as I get 'OK' for realtime priorities with rtcqs. Another way to test if you have realtime permissions is to run cyclictest. Install rt-tests and run cyclictest. It needs realtime permissions to run.

finotti
Established Member
Posts: 528
Joined: Thu Sep 01, 2011 9:07 pm
Has thanked: 86 times
Been thanked: 23 times

Re: How to install low-latency kernel

Post by finotti »

I get:

Code: Select all

$ cyclictest 
Unable to change scheduling policy!
Probably missing capabilities, either run as root or increase RLIMIT_RTPRIO limits.

Probably not a good sign...

merlyn
Established Member
Posts: 1392
Joined: Thu Oct 11, 2018 4:13 pm
Has thanked: 168 times
Been thanked: 247 times

Re: How to install low-latency kernel

Post by merlyn »

Totally bizarre. As you saw, you do have the realtime priority set correctly. Is there a kernel that works on your system?

finotti
Established Member
Posts: 528
Joined: Thu Sep 01, 2011 9:07 pm
Has thanked: 86 times
Been thanked: 23 times

Re: How to install low-latency kernel

Post by finotti »

merlyn wrote: Fri Mar 24, 2023 4:41 pm

Totally bizarre. As you saw, you do have the realtime priority set correctly. Is there a kernel that works on your system?

Yes, debian's own rt-kernel works. It just really has me stumped, so I have a hard time letting it go... But I do have an alternative.

User avatar
Audiojunkie
Established Member
Posts: 392
Joined: Thu Feb 21, 2019 4:27 pm
Has thanked: 382 times
Been thanked: 153 times

Re: How to install low-latency kernel

Post by Audiojunkie »

finotti wrote: Fri Mar 24, 2023 4:47 pm
merlyn wrote: Fri Mar 24, 2023 4:41 pm

Totally bizarre. As you saw, you do have the realtime priority set correctly. Is there a kernel that works on your system?

Yes, debian's own rt-kernel works. It just really has me stumped, so I have a hard time letting it go... But I do have an alternative.

It may be something not compile for the preempt_dynamic in the Debian kernel. It’s working fine (as far as I can tell) in Fedora, which I use.

finotti
Established Member
Posts: 528
Joined: Thu Sep 01, 2011 9:07 pm
Has thanked: 86 times
Been thanked: 23 times

Re: How to install low-latency kernel

Post by finotti »

finotti wrote: Mon Mar 20, 2023 8:16 am

First, thanks to all for the comments! I was out of town these past few days and could not test it.

But, I have it all almost figured out:

Code: Select all

$ rtcqs
rtcqs - version 0.5.3

Root User
=========
[ OK ] Not running as root.

Audio Group
===========
[ OK ] User finotti is in the audio group.

CPU Frequency Scaling
=====================
[ OK ] The scaling governor of all CPU's is set at performance.

Kernel Configuration
====================
[ OK ] Valid kernel configuration found.

High Resolution Timers
======================
[ OK ] High resolution timers are enabled.

Tickless Kernel
===============
[ OK ] System is using a tickless kernel.

Preempt RT
==========
[ OK ] Kernel 6.2.0-6.slh.1-aptosid-amd64 is using threaded IRQ's.

Spectre/Meltdown Mitigations
============================
[ WARNING ] Kernel with Spectre/Meltdown mitigations found. This could have a negative impact on the performance of your system. See also https://wiki.linuxaudio.org/wiki/system_configuration#disabling_spectre_and_meltdown_mitigations

RT Priorities
=============
[ WARNING ] Could not assign a 80 rtprio SCHED_FIFO value due to the following error: [Errno 1] Operation not permitted. Set up limits.conf. See also https://wiki.linuxaudio.org/wiki/system_configuration#limitsconfaudioconf

Swappiness
==========
[ OK ] Swappiness is set at 10.

Filesystems
===========
[ OK ] The following mounts can be used for audio purposes: /, /gnu/store, /media/music, /media/sd_data, /media/data2, /media/data3, /media/media, /media/sg4tb
[ WARNING ] The following mounts should be avoided for audio purposes: /run/user/1000/doc. See also https://wiki.linuxaudio.org/wiki/system_configuration#filesystems

IRQs
=====
[ OK ] USB port ehci_hcd:usb4 with IRQ 16 does not share its IRQ.
USB port ehci_hcd:usb2 with IRQ 23 does not share its IRQ.
USB port xhci_hcd with IRQ 28 does not share its IRQ.

Power Management
================
[ OK ] Power management can be controlled from user space. This enables DAW's like Ardour and Reaper to set CPU DMA latency which could help prevent xruns.

The only real problem is :

Code: Select all

RT Priorities
=============
[ WARNING ] Could not assign a 80 rtprio SCHED_FIFO value due to the following error: [Errno 1] Operation not permitted. Set up limits.conf. See also https://wiki.linuxaudio.org/wiki/system_configuration#limitsconfaudioconf

But I do have/etc/security/limits.d/audio.conf set :

Code: Select all

$ cat /etc/security/limits.d/audio.conf
# Provided by the jackd package.
#
# Changes to this file will be preserved.
#
# If you want to enable/disable realtime permissions, run
#
#    dpkg-reconfigure -p high jackd2

@audio   -  rtprio     95
@audio   -  memlock    unlimited
#@audio   -  nice      -19

Now, here is something is strange: yesterday, after coming back to town, and starting to reply to this message, my computer passed the rtcqs's RT Priorities test (and Ardour's). I really don't remember having done anything that could affect it, since I did not even know what could be the problem. The kernel was 6.2.0-5.slh.1-aptosid-amd64, the same from my failed test in the previous message.

But, a new kernel came in, so I installed and tried it, and it failed. So I rebooted with the old kernel (6.2.0-5.slh.1-aptosid-amd64 again), and it also failed!

Does anyone have any ideas on what could cause this problem?

Someone at the Ardour forum found the solution for not being able to set RT priorities.

I could manually set the content of /proc/sys/kernel/sched_rt_runtime_us from 950000 to -1 and the tests passed.

To make it permanent, I created a file 60-sched_rt_runtime.conf with kernel.sched.rt_runtime_us = -1.

Further testing might be needed, but it seems to be working now.

oddy.o.lynx
Established Member
Posts: 115
Joined: Sat Jun 30, 2018 2:54 am
Has thanked: 22 times
Been thanked: 22 times

Re: How to install low-latency kernel

Post by oddy.o.lynx »

For Power Management I fixed it this way:

Code: Select all

sudo chown root:audio /dev/cpu_dma_latency
sudo chmod 664 /dev/cpu_dma_latency

Power Management
================
[ OK ] Power management can be controlled from user space. This enables DAW's like Ardour and Reaper to set CPU DMA latency which could help prevent xruns.

User avatar
autostatic
Established Member
Posts: 1994
Joined: Wed Dec 09, 2009 5:26 pm
Location: Beverwijk, The Netherlands
Has thanked: 32 times
Been thanked: 104 times
Contact:

Re: How to install low-latency kernel

Post by autostatic »

After a reboot those permissions will be reset. To set it automatically on boot you could use an udev rule as described here: https://wiki.linuxaudio.org/wiki/system ... _interface

User avatar
autostatic
Established Member
Posts: 1994
Joined: Wed Dec 09, 2009 5:26 pm
Location: Beverwijk, The Netherlands
Has thanked: 32 times
Been thanked: 104 times
Contact:

Re: How to install low-latency kernel

Post by autostatic »

Audiojunkie wrote: Fri Mar 24, 2023 3:24 pm

The last time I checked, which admittedly was a couple of years ago, the rtprio checks (in RTCQS) only looked for "Audio" as the group that a person sets up to assign themselves realtime priorities. To my knowledge, it does not consider that the Pipewire team's setup recommendations suggest to use a group named "pipewire" instead of "audio". In truth, you could name the group "pineapple" and still get a working result as long as you assigned the correct realtime priorities and memlock and added yourself to the "pineapple" group. The point I'm making, is that the script doesn't (and honestly can't) take all of these things into account.

It can, rtcqs should just check if the user can set real-time prio's and if memory lock is set to unlimited. It shouldn't check for any group membership. It's on the TODO list: https://codeberg.org/rtcqs/rtcqs/issues/4

Audiojunkie wrote: Fri Mar 24, 2023 3:24 pm

The point I'm making, is that the script serves as a quick useful tool to check your settings, as long as you are aware of the limitations of its design. My current suspicion is that using a generic kernel with the realtime boot paramaters may not be something the script looks for--it is a relatively new development in the Linux world. I can't state this as a fact, but it is my suspicion.

Your suspicion is correct, rtcqs does not check for this yet. I've added checking for preempt=full to the TODO list also: https://codeberg.org/rtcqs/rtcqs/issues/7

Neil W
Posts: 1
Joined: Thu Feb 22, 2024 10:46 pm

Re: How to install low-latency kernel

Post by Neil W »

Ubuntu servers usually keep the kernel version to a stable, LTS kernel, so perhaps Pop OS or you installed 6.2 manually. The version lag shouldn't be an issue - in fact, it may prevent some issues associated with newer kernel versions.

You can install a mainline kernel, but you'd probably have to find it online or build it from source, as the Mainline ppa doesn't support it anymore

As for how to install it, after you run sudo apt install linux-lowlatency, you will need to update GRUB: sudo update-grub, and maybe change the boot order with grub-customizer if necessary.

The Ubuntu low latency kernel is usually not intended for desktop use, and is for embedded applications with high latency requirements though. You shouldn't expect big gains from performance or the like.

tseaver
Established Member
Posts: 398
Joined: Mon Mar 13, 2017 6:07 am
Has thanked: 11 times
Been thanked: 98 times

Re: How to install low-latency kernel

Post by tseaver »

@Neil W

The Ubuntu low latency kernel is usually not intended for desktop use, and is for embedded applications with high latency requirements though. You shouldn't expect big gains from performance or the like.

Low-latency / realtime use cases relevant to this forum:

  • Linux musicians who record audio into a DAW, and want to monitor the recorded performance in software. This can be especially important for recodring vocalists, who need to hear themselves clearly to sing well.

  • Linux musicians who play an instrument into another piece of software (e.g., an amp / effects simulator like Guitarix): being able to hear the sound from the "amp" with as little latency as possible makes for a better playing performance. Keeping the latency induced by the interface, the OS, and the DAW down to the same region as a physical amp at normal distances (10 ms or so) makes playing through the simulator feel like playing through an amp.

  • Linux musicians who record MIDI instrument performances, and need predictable, low latency to ensure the best human playing.

Interestingly, the "performance" that matters is that of the human, not so much the computer: performers need to hear themselves to play well.

Ubuntu, Mixbus32C; acoustic blues / country / jazz
Post Reply