I just had an interesting conversation on Matrix with Wim Taymans and some others on the pipewire forum to get some clarifications and further info. Things didn't go quite the way I expected, but I thought it would be of interest to the whole pipewire debate, here on the LinuxMusicians forum. After this conversation, I'm looking at pipewire a bit differently than before. Some may also feel the same. I am copy/pasting the conversation here in its entirety to continue with the conversation (including my mistakes and all). Note: Some pieces of the conversation didn't copy over correctly from the copy of the browser.
sercanbrack
Disclaimer: I am a long-time linux user, but I am by no means an expert. In many ways, I feel like I have Imposter syndrome when it comes to linux--everyone knows I love linux and use linux, and everyone thinks I know everything, but I don't. Also, if there is a better (more preferable) place to post long, detailed questions, please direct me where I need to go.
That said, I have some questions that I am hoping that people who know might be able to answer and help me with.
I am trying to understand and follow the pipewire performance tuning reference page listed here:
https://gitlab.freedesktop.org/pipewire ... nce-tuning
Inside this document, the following is stated:
Realtime priorities are given to the data processing threads in both the clients and the server.
Since 0.3.44 there is a single module-rt that can use Portal/RTKit and fall back to native thread implementation automatically.
module-rt first tries to use the org.freedesktop.portal.Realtime DBus service to request realtime privileges to the processing thread. This Portal service knows about flatpak namespaces and can map flatpak pids to pids used in org.freedesktop.RealtimeKit1.
org.freedesktop.RealtimeKit1 is used as a fallback when the Realtime portal doesn't exist.
The Portal/RTKit implementation is potentially better because it can implement a global policy and does not require extra permissions from the client. It however needs DBus and is currently not configured optimally in many distributions.
So, as I understand this, module-rt is the pipewire module that manages the real time management of the processing threads. Does this mean that those of us using generic kernels no longer need the /threadirqs kernel boot parameter to separate the kernel threads for priority processing? Or is the threadirqs boot parameter still useful?
Also, the document states that Portal/RTKit implementation is "potentially better", but doesn't state what it is potentially better than. I'm not a developer, but I'm trying to understand this reference info and use it. What is Portal/RTKit better than?
In the next section, "Configuration" shows the various paths to pipewire configuration files, where one can configure realtime priorities. I used to believe that this was where we needed to configure the sample rate and buffer size. However, I have since learned that we can access and change these things with the pw-metadata command. So then, is the Configuration section of the document pointing out that these paths are where we need to look to enable the module-rt in the context.modules section? Am I understanding this correctly?
Performance tuning · Wiki · PipeWire / pipewire · GitLab - GitLab
Multimedia processing graphs
wtay
sercanbrack
, threadirq is still useful
sercanbrack
, portal/rtkit is better because you don't need to add permissions to users to boost priorities themselves, which can reduce potential denial of service risks
flok left the room: Part: Leaving
wtay
sercanbrack
, I don't understand the question about context.modules
sercanbrack
The document states:
Configuration
The configuration of the realtime priorities and the implementation can be done in the different server and client config files:
/usr/share/pipewire/pipewire.conf for the PipeWire daemon settings
/usr/share/pipewire/pipewire-pulse.conf for the PipeWire pulseaudio server
/usr/share/pipewire/client-rt.conf for the PipeWire native client settings
/usr/share/pipewire/jack.conf for the PipeWire JACK clients
The relevant module is enabled in the context.modules section, see
module-rt documentation.
I'm trying to understand the line: The relevant module is enabled in the context.modules section, see
module-rt documentation.
wtay
sercanbrack
, https://gitlab.freedesktop.org/pipewire ... onf.in#L92
src/daemon/pipewire.conf.in · master · PipeWire / pipewire · GitLab - GitLab
Multimedia processing graphs
sercanbrack
Does this mean that we have to go into one (or more) of the above configuration files to "enable" the module-rt?
wtay
no, it's enabled by default and the default settings should be good
sercanbrack
OK. So we don't need to change anything there, and that part is just information so that we can understand what is going on and where the following configuration info is located in the config files:
Uses realtime scheduling to boost the audio thread priorities. This uses
1
2
3
4
5
6
7
8
RTKit if the user doesn't have permission to use regular realtime
scheduling.
{ name = libpipewire-module-rt
args = {
nice.level = -11
#rt.prio = 88
#rt.time.soft = -1
#rt.time.hard = -1
So, then, that brings me to the RKIT part of the documentation...
wtay
yes, lots of information
sercanbrack
For Red Hat/Arch Based Distributions
RTKit can be configured by passing extra arguments to the /usr/libexec/rtkit-daemon process. You can try /usr/libexec/rtkit-daemon --help to see the options.
Depending on the distribution, you would do something like this:
Make /usr/lib/systemd/system/rtkit-daemon.service.d/limits.conf with:
[Service]
EnvironmentFile=/etc/sysconfig/rtkit
ExecStart=
ExecStart=/usr/libexec/rtkit-daemon $RTKIT_ARGS
Does this mean that I have to create a rtkit-daemon.service.d/limits.conf file and put the listed service info into the new limits.conf file?
wtay
sercanbrack
, only if you are unhappy with the currently configured values for your rtkit setup
sercanbrack
OK, so this should be configured by default, but it depends on how the distribution is prepared. Correct?
wtay
yes
sercanbrack
And the same with:
Make /etc/sysconfig/rtkit with:
RTKIT_ARGS="--scheduling-policy=FIFO
--our-realtime-priority=89
--max-realtime-priority=88
--min-nice-level=-19
--rttime-usec-max=2000000
--users-max=100
--processes-per-user-max=1000
--threads-per-user-max=10000
--actions-burst-sec=10
--actions-per-burst-max=1000
--canary-cheep-msec=30000
--canary-watchdog-msec=60000
"
It should already be there if the distro put it in correctly?
pinkflames
Edu4rdSHL
: I wonder if it's some format conversion issue... Maybe file an issue at the FDO Gitlab, be sure to use the template and append that screenshot you posted here
wtay
sercanbrack
, yes
pinkflames
Edu4rdSHL
: I guess one way to find out for sure if it's a device or PipeWire bug would be to record the raw PCM being sent to ALSA but I'll provide instructions for that at a later point (probably via the Wiki)
wtay
: sorry for not remembering the details but isn't the reason threadirq is not the default, because it's kinda hacky port from kernel-rt and was never meant to be used on a general purpose kernel (as usual, I could be very wrong)?
sercanbrack
Great! This is really helping me understand things! So then, if I wanted to use a generic kernel and tune my linux from scratch, I would need to do the following:
"On recent kernels configured with PREEMPT_DYNAMIC, you can boot the kernel with and extra
preempt=full option. " .....and also set my /threadirqs kernel parameter
RLIMITs
Real-time priority limits are usually stored in /etc/security/limits.conf and /etc/security/limits.d/. The best option is to add a new file 95-pipewire.conf in /etc/security/limits.d/ with this content:
Default limits for users of pipewire
@pipewire - rtprio 95
@pipewire - nice -19
@pipewire - memlock 4194304
Then add your user to the PipeWire group so that you can use these priorities.
Confirm that the RTKit configuration files are like listed below:
RTkit
For Red Hat/Arch Based Distributions
RTKit can be configured by passing extra arguments to the /usr/libexec/rtkit-daemon process. You can try /usr/libexec/rtkit-daemon --help to see the options.
Depending on the distribution, you would do something like this:
Make /usr/lib/systemd/system/rtkit-daemon.service.d/limits.conf with:
[Service]
EnvironmentFile=/etc/sysconfig/rtkit
ExecStart=
ExecStart=/usr/libexec/rtkit-daemon $RTKIT_ARGS
Make /etc/sysconfig/rtkit with:
RTKIT_ARGS="--scheduling-policy=FIFO
--our-realtime-priority=89
--max-realtime-priority=88
--min-nice-level=-19
--rttime-usec-max=2000000
--users-max=100
--processes-per-user-max=1000
--threads-per-user-max=10000
--actions-burst-sec=10
--actions-per-burst-max=1000
--canary-cheep-msec=30000
--canary-watchdog-msec=60000
"
wtay
I don't know, really. I don't do tuning, it should work without
sercanbrack
Hmmm.... I am a member of the LinuxMusicians group. I've seen you post there, so I'm pretty sure you know what we do and why. Low latency without xruns are very essential to making pipewire useful for us. I'm trying to find the best (and easiest) way to make use of Pipewire for our purposes. This is why the tuning is so essential for us.
There isn't much documentation out there about this stuff. I've been using the Performance Tuning page as the only pipewire based tuning info I can find:
https://gitlab.freedesktop.org/pipewire ... nce-tuning
Performance tuning · Wiki · PipeWire / pipewire · GitLab - GitLab
Multimedia processing graphs
I've been trying to decipher all this so that I can teach it to the others.
pinkflames
Edu4rdSHL
: btw, have you perhaps modified PipeWire or WIrePlumber settings? Such as changing the ["dither.noise"] property?
oh, and if you do file an issue report, it might be very helpful if you also attach a recording of that noise, since it seems like it's more like a background hiss?
sercanbrack
index.php
LinuxMusicians - Index page
LinuxMusicians creating music freely Skip to content Search Advanced search It is currently Fri Jun 02, 2023 3:46 pm
pinkflames
sercanbrack
: the main issue is that there's not much to tune, it either works well, or something is going wrong and it's not easy to debug
Unless, of course you think it's a good idea to run at quantum 32 May CPU speed be with you on that one (and bugfree drivers/devices)
wtay
sercanbrack
, as long as you can give FIFO scheduling with some high priority >50 to the realtime thread you're good to go
sercanbrack
, either by setting limits.conf or using rtkit, pipewire will pick it up automatically
pinkflames
sercanbrack
: and to expand upon what Wim said, all you need for that is to add your user to the pipewire group and reboot
sercanbrack
Do we still need to run the rtirq startup script for for raising the thread priorities, or is that handled with the module-rt and RTKit settings in pipewire?
https://github.com/rncbc/rtirq
GitHub - rncbc/rtirq: rtirq - Startup script for realtime-preempt enabled kernels - GitHub
rtirq - Startup script for realtime-preempt enabled kernels - GitHub - rncbc/rtirq: rtirq - Startup script for realtime-preempt enabled kernels
wtay
sercanbrack
, no, that's not handled
pinkflames
sercanbrack
: no, rtirq is dubious for jackd and probably a bad idea for anything else
sercanbrack
So that part is still useful to prioritize threads?
pinkflames
I was not aware it was also doing that...
sercanbrack
My understanding is that this is the whole point of the rtirq script. You need to be running a realtime kernel or using the /threadirqs option to make use of the script, and then the script prioritizes the threads to give higher priority to the audio threads to increase performance
That's how I understand it, anyway.
pinkflames
I sure hope people who came up with that knew what they were doing
wtay
I have no idea what that would do or if it would help..
pinkflames
wtay
: I think it's trying to identify kernel threads involved in audio and boost their priority. Yeah, I'd be scared to do that, myself
sercanbrack
rncbc, the devloper of qpwgraph set it up
...wrote it
wtay
I guess you can still do that..
sercanbrack
xrun free low latency is an essential part of music production, and without it, people in this field don't want to move away from JACKD to pipewire
pinkflames
Yes, I know who he is. I still wonder if it's a good idea
Yes, but at what latency or JACK buffer size?
sercanbrack
...and yes, a quantum of 32 would be an ideal. Most people want to run with a quantum of 64.
pinkflames
There's a big difference between 0 xruns at 42 ms vs. 0 runs at 0.6 ms...
sercanbrack
Agreed. But that's the whole point of music production
pinkflames
sercanbrack
: by most people you mean most musicians or normal people? :/
sercanbrack
: and just out of curiosity, could you provide that value in terms of how jackd would be set up?
sercanbrack
I mean most musicians I don't think normal people would care at all. But then again, Pipewire is being built as a replacement for both jack and pulseaudio. If it can't perform the functions that Jack and do, it isn't really serving its purpose of replacing jack
pinkflames
Okay, that is quite nice to know, if you're right about that.
But what is the jackd config you'd use for the equivalant?
sercanbrack
pinkflames
sercanbrack
: and just out of curiosity, could you provide that value in terms of how jackd would be set up?
I'm not sure I understand this question
Edu4rdSHL
pinkflames
, the only settings I have changed is loading the switch-on-connect module for pipewire-pulse, which I doubt will be the cause of the problem
pinkflames
Basically, what is the jackd or jackdbus configuration that people use?
because the quantum and jack period are not the same
Edu4rdSHL
: right, I think you should just file an issue report and attach both that screenshot and a recording for now
It would be nice to also do a PCM dump but that will take a while, unless you happen to know how to do that yourself
Edu4rdSHL
pinkflames
, a record as per with pw-record, right?
sercanbrack
OK, I think I understand what you mean. You are referring to the sample rate and buffer sizes. As always, lower is better. 48khz with a buffer of 64 is usually good for most people a round trip latency time of under 5ms is ideal
pinkflames
Edu4rdSHL
: as long as it's attached to a microphone. But I guess pw-record might be interesting, too
So maybe both a pw-record or similar of the relevant monitor port and, if that's clean, then a microphone recording of the noise as heard
sercanbrack
: so when you said 64 quantum, you actually meant jackd set to period size 64? That is actually quantum 128 for PipeWire
And trying to use quantum 64 is like trying to run jackd with period size 32
wtay
it doesn't work like that...
pipewire doesn't use period size for scheduling
it uses timers
sercanbrack
I still get the quantum and buffer stuff confused. I'm used to saying latency. The point is having an xrun-free audio with lower than 5ms latency "round trip"
pinkflames
wtay
: well, it's a simplification but that's the usually similar added latency, since jackd latency is the buffer size, which is period size multiplied by the period count
wtay
for batch devices we play with the period-size to get more accurate readings but the quantum is still supposed to be the same
pinkflames
and jackd period count is at least 2, so it will be at least twice that in PW quantum
wtay
sercanbrack
, it depends on the hardware, I can do this here with what I have but many people can't get this with pipewire (but they can with jack)
pinkflames
wtay
: yeah but batch devices with jackd are supposed to have a period size 3, so it should still be the same as the 2*period in quantum terms
wtay
somewhat
pinkflames
sercanbrack
: in theory 5.0 ms round trip requires quantum 64, if I got my math right but, if you were okay with a bit more (5.4 ms, IIRC), then that can be done with quantum 128. In all cases the ALSA will still add some overhead which is hopefully small but can be big, depending on ALSA
sercanbrack
wtay
sercanbrack
, it depends on the hardware, I can do this here with what I have but many people can't get this with pipewire (but they can with jack)
That's the argument that is taking place on the LinuxMusicians forum. People are saying that pipewire isn't a real jack replacement and is more of a PulseAudio 2. My understanding was that pipewire is supposed to be a replacement for jack, so I've been arguing that pipewire will serve as a true replacmet that can provide xrun free low latency audio. NOTE: I'm "Audiojunkie" in the LinuxMusican's forum. There is a lot of skepticism about pipewire being able to do the task. I've been trying to find the best way to set pipewire up to run as a replacement for jack, but I've found very little documentation. That's my whole point of all of the questions.
sam_
have the skeptics tried using pipewire?
pinkflames
sercanbrack
: I recommend not caring about other people and being happy with what you have
If they want to use jackd, let them use it
sercanbrack
Yes. and that's the problem. It performs worse than jack and with more xruns. We have no high level detailed documentation for the hundreds of linux musicians to know how to configure pipewire to work properly
pinkflames
sercanbrack
: I recommend not caring about other people and being happy with what you have
That's kind of missing the point if jack is being depreciated, which it is.
pinkflames
sercanbrack
: in short, you set the quantum and it hopefully works. If not, something is wrong
sercanbrack
: yes but still let people use deprecated software, it's not like you're paid for every PW converrt
sercanbrack
That doesn't sound confident enough for those wanting to move away from Windows and use linux for audio in the professional realm
sercanbrack
Hehehe.... that's not the point. Pipewire has stated very clearly that it aims to replace jack and pulseaudio and provide the same functionality. Suggesting that the pro audio industry just rely on the depreciated jack server that even falkTX has agreed is depreciated and not going to get patches is not going to help the Linux cause with the pro audio industry. "Hoping" things will work without having any laid out tuning strategies to get xrun free sub 5ms audio (round-trip) is not going to cut it for the pro audio industry. If you spent some time reading the issues on the LinuxMusicians forums (where many of the pro audio professionals communicate), you'll see that xrun free sub 5ms audio (round-trip) is pretty essential. I'm a non professional, but I care enough about pro audio on linux to be here trying to discuss with the developers the best and easiest strategies to accomplish this, so that I can teach it to the others.
I mean no disrespect, by the way. I'm just passionate about this.
pinkflames
It's not like PipeWire xruns at that config, because we like it that way... It's due to various bugs or deployment deficiencies that need to be identified and fixed.
In other words, there's no tuning
sercanbrack
OK. Then how can we help with troubleshooting and debugging pipewire to get jack level results on a reliable basis?
What information would be needed and where do we report it?
https://gitlab.freedesktop.org/pipewire ... nce-tuning
Performance tuning · Wiki · PipeWire / pipewire · GitLab - GitLab
Multimedia processing graphs
pinkflames
sercanbrack
: where appropriate? Just to prove my point I just launched a YouTube Music song at quantum 64, that is ALSA period 32..
It did a few xruns when initializing but 0 reported now and also perfectly clean
sercanbrack
This page seems to indicate that there is at least some tuning involved
pinkflames
And that's Firefox running using pipewire-pulse
and being filtered via EasyEffects for good measure, too
sercanbrack
: if you read it, it's basically a list of things to get right.. like using the right preemption mode, IIRC
sercanbrack
pinkflames
It did a few xruns when initializing but 0 reported now and also perfectly clean
Yes, but that load doesn't represent the music industry at all. Try doing the same with 24 recorded audio tracks and a dozen plugins. Linux should be as equally performant with the same hardware as on Windows or Macs
pinkflames
But that's for distros and not users
sercanbrack
: is Windows and macOS really doing the equivalent work at sub 2.5 ms latency one way?
It's easy to claim the same when it's recording a single track at a time...
nedko
wtay: https://gitea.ladish.org/LADI/jack-head ... 6beb8f1fdd
jack-headers - Gitea: LADI
JACK API headers
pinkflames
Or with way higher latency
nedko
wtay: i sent email to sletz about this
sercanbrack
Yes. This is throughout the entire pro audio and media industry, including video editing
pinkflames
Cool. Do file a proper issue report about any regressions you encounter
sercanbrack
So, as I understand it, as long as the distro has the performance tuning page instructions set up properly, everything should work equivalent to the jack server with the equivalent latency and xruns. The problem is, how has this been tested? By your own admission, you tested pipewire and stated your results without testing under a proper load. What kind of load was all of this tested under:
https://gitlab.freedesktop.org/pipewire ... erformance
Performance · Wiki · PipeWire / pipewire · GitLab - GitLab
Multimedia processing graphs
?
I get the feeling that pro audio requirements are still, after all of this time, not fully understood by those who don't work with it in the industry.
I know that Wim didn't know about how jack session management works until WAY, Way later in the process. I'm refering to the "Non" protocol and session management.
nedko
sercanbrack
: nonsm provides no notion of client subgrouping at all. ladish for pipewire will use the pipewire powers for this.
sercanbrack
If there is no tuning involved with pipewire, which everyone in the industry that uses JACK does have to do, and if there is no proper testing of low latencies and 0 xruns, then the numbers listed don't really apply and pipewire can't really do what it promises to do.
- nedko aims at implementing the jack2 settings control on top of pipewire
sercanbrack
nedko
sercanbrack
: nonsm provides no notion of client subgrouping at all. ladish for pipewire will use the pipewire powers for this.
ladish is dead as a standard. Non is the replacement. It does everything ladish can do and more. FalkTX has even admitted it.
pinkflames
sercanbrack
: try not to imagine things or that we're all incompetent, please. PipeWire is very well designed, so it's highly insensitive to system load, as long as everything is working correctly. It might be possible to make a client hang but that's due to the client being written in a way that resource contention will make it hang...
sercanbrack
At any rate, if was not my intention to start an argument with all of the developers. I'm just disappointed with the things that I am hearing and the lack of concern. If pipewire was supposed to be the future and solve the problems of the separate pulseaudio and jack servers, I expected more.
nedko
sercanbrack
: falktx formally left (it was pseudo-participation anyway) the LADI project a year ago or so. de factio i see people on the street locally (yeah) who have bad experience with ladish. even carla has/had its "ladish support" broken. actually it is currently a code to restrict Carla to NSM-like behaviour if it detects LADISH, even if no save/restore is available.
pinkflames
sercanbrack
: but do you yourself have an issue? No one can fix bugs that are not reported...
nedko
sercanbrack
: combine this with 15 years of debian and other distro mispachaging of jackdbus
sercanbrack
pinkflames
sercanbrack
: try not to imagine things or that we're all incompetent, please. PipeWire is very well designed, so it's highly insensitive to system load, as long as everything is working correctly. It might be possible to make a client hang but that's due to the client being written in a way that resource contention will make it hang...
I am truly sorry that my words were understood that I felt you are incompetent. It was not my intent or desire. I just went off of your example you gave about your low latency with youtube.
pinkflames
I'd say that Firefox and PulseAudio are about as far away from low latency as possible but they still work
nedko
sercanbrack
: calling other projects "dead" is not really human. i've signed the ubuntu code of conduct in years ago
pinkflames
Anyway, gotta go now. Not sure when I'll be back but I'll probably try to work on the Wiki, when I get the time
sercanbrack
I appreciate that. Thank you for your time and sorry to be offensive
pinkflames
No problem, I'm not a developer (only a contributor) and I'm not that easily offended, unless you happened to find the right angle