Ultralite AVB

Talk about your MIDI interfaces, microphones, keyboards...

Moderators: MattKingUSA, khz

buonamorte.records
Established Member
Posts: 98
Joined: Tue Apr 07, 2020 4:31 am
Has thanked: 11 times
Been thanked: 11 times

Re: Ultralite AVB

Post by buonamorte.records »

I've sent it via pvt message

buonamorte.records
Established Member
Posts: 98
Joined: Tue Apr 07, 2020 4:31 am
Has thanked: 11 times
Been thanked: 11 times

Re: Ultralite AVB

Post by buonamorte.records »

About the new patch: in my system, it works perfectly in the playback side, now i can't test the recordings, because i'm at home with the covid and i don't have an xlr cable...have someone tested the rec side?

buonamorte.records
Established Member
Posts: 98
Joined: Tue Apr 07, 2020 4:31 am
Has thanked: 11 times
Been thanked: 11 times

Re: Ultralite AVB

Post by buonamorte.records »

Dylan:

I spent a little time reading through Drumfix's driver. It has the right idea doing implicit feedback and batch submitting some transfers at startup. However, and I say this without having tested it myself so I might be barking up the wrong tree, but I suspect the issue is similar to the one my patch attempts to address (xHCI controller underruns). A few factors could be contributing to this. The first factor is that each URB in Drumfix's driver seems to only contain one microframe's worth of data. This likely means that both the capture and playback completion handlers are getting called 8000 times per second. This high callback frequency might be taxing to the system and making it more difficult for the kernel to maintain real-time performance. The second factor could be that while some transfers are batched up to be submitted on startup, only about 2ms of transfers are by default. This means that if anything causes a system hang-up for 2ms or more, or if there is a a long list of URB completion handlers to call that take too much time, the xHCI controller might get all the way to the end of its transfer list. This would cause a gap in the isoc stream when the audio driver eventually gets around to submitting the next set of URBs and that gap would ultimately cause a hardware buffer underrun in the audio interface. It would be possible to verify this hypothesis by looking at the dynamic_debug logs of the xHCI driver when the channel walking or distortion start occurring. (Look around line 2666 in drivers/usb/host/xhci-ring.c). Given that it is basically impossible to guarantee real-time performance on modern systems, this underrun will always be a possibility no matter how good the drivers are, the best we can do is try to minimize the likelihood by scheduling things as far in advance as possible. In practice, if we are careful should be able to make it so the likelihood of an underrun is practically 0 (it will just never be 100% guaranteed). For this reason, it would be useful if the xHCI driver were able to propagate this underrun event back to USB drivers, so that they could handle the event by restarting the isochronous audio stream. Unfortunately, I don't think the current implementation supports this.

For those reasons, with my patch, I think the class audio driver stands a better chance of working reliably given its current design of submitting URBs consisting of 8 microframes at a time and the patch change to batch submit 12ms of audio on startup. I just wanted to share these thoughts to see if they jibe with your past experiences or not.

buonamorte.records
Established Member
Posts: 98
Joined: Tue Apr 07, 2020 4:31 am
Has thanked: 11 times
Been thanked: 11 times

Re: Ultralite AVB

Post by buonamorte.records »

Dylan:
The latency is something I can also look at here, I just haven't gotten around to it yet. However, at least one version of this test should be fairly straightforward even without an analog input source just by using the router in the UltraLiteAVB to loop the USB Output channels back to the USB Input channels. Then, if you play a track from a DAW while recording the looped back audio into another track you can examine the offset between the channels. This could then be compared to the unpatched behavior to see if the offset grows. (If there is any change I would expect it to be around 10-12ms). I need to study more about how ALSA manages client sync and latency in order to have a better intuition about how this patch could impact it. I raise the possibility of an issue here because when I shared this idea with Takashi he had reservations regarding latency. I'm just not completely sure what kind of latency and where to look for it yet.

Drumfix
Established Member
Posts: 299
Joined: Mon Jan 26, 2009 5:15 pm
Been thanked: 11 times

Re: Ultralite AVB

Post by Drumfix »

The real fix would be to fix the firmware. As Dylan pointed out, Motus firmware does not do the necessary recovery actions in case a packet gets lost.
Having an ring underrun is just one of the cases in which the host controller can detect a data loss. However there are other cases (link a crc error due to a bad connection) where the host controller has no knowledge about whether the package has been successfully received by the device.
You'd better ask Dylan, why Notu refuses to fix their firmware. Everybody else in the audio industry is doing this right.

buonamorte.records
Established Member
Posts: 98
Joined: Tue Apr 07, 2020 4:31 am
Has thanked: 11 times
Been thanked: 11 times

Re: Ultralite AVB

Post by buonamorte.records »

@Drumfix I think you're right but they said they've changed the way to use the usb because of an Apple's change, so probably we have to ask directly to Apple...
I also think it will be a good idea if you and @shellwalker can talk directly with Dylan. I'm not an expert in programming stuff, I already feel :D when i learn something like how to compile and patch a kernel...If you want, i can ask to Dylan if I can give you his motu-mail contact.

User avatar
AudioNarwhal
Established Member
Posts: 39
Joined: Thu Apr 29, 2021 12:43 pm
Has thanked: 2 times
Been thanked: 35 times

Re: Ultralite AVB

Post by AudioNarwhal »

Maybe I can join in here directly. I just skimmed through this thread and I am thoroughly impressed with how much @Drumfix and collaborators have been able to reverse engineer. I understand a lot of the frustrations I'm reading about in here—Hopefully I can help out.

As Dylan pointed out, Motus firmware does not do the necessary recovery actions in case a packet gets lost.

This is a fair point. It is something that has been addressed in more recent hardware such as the UltraLite-mk5, and will be addressed in all devices moving forward.

The real fix would be to fix the firmware.

Making the firmware recover after a missed packet would definitely improve the situation. That said, implicit feedback implementations on linux seem to be generating many more missed isoc intervals than I have seen on Windows and Mac platforms. Ideally the audio interface should be able to recover on its own—and—these events should be happening much less frequently.

Early experiments seem to show that pre-queueing more out transfers on startup either eliminates or greatly reduces the likelihood these underrun on the computers I have tested.

You'd better ask Dylan, why Notu refuses to fix their firmware.

Some products involve 3rd party contributions that we don't always have full control over. Starting with the UltraLite-mk5 and moving forward significant steps have been taken to avoid such situations in the future.

@Drumfix I think you're right but they said they've changed the way to use the usb because of an Apple's change, so probably we have to ask directly to Apple...

The comment regarding Apple was more about the decision to use implicit feedback vs explicit feedback for the USB audio class implementation. As far as I know neither the implicit (newer) nor explicit (older) firmwares support automatically recovering after a missed isoc interval. The reported higher reliability of the explicit feedback firmwares on linux tells me that we should be able to see similar performance with an implicit implementation. The implicit implementations being used on linux so far have many fewer isoc packets queued ahead of the ongoing re-submissions than the class explicit implementation does. On Windows and Mac platforms we typically schedule isoc much further in advance than any of these linux implementations do and as a result host controller underruns aren't really an issue we have to contend with. We have seen missed intervals, but usually for other reasons, and it is generally a rare occurrence. Even with our recovery system in the new hardware, we still spend a lot of timing making sure it is never exercised. Generally the expectation is for completely clean audio even under heavy CPU loads, with the goal of providing completely uninterrupted audio.

Last edited by AudioNarwhal on Sat Oct 29, 2022 4:33 am, edited 1 time in total.
shellwalker
Established Member
Posts: 68
Joined: Sun Jan 03, 2021 1:54 pm
Has thanked: 21 times
Been thanked: 11 times

Re: Ultralite AVB

Post by shellwalker »

AudioNarwhal wrote: Sat Oct 29, 2022 4:01 am

Maybe I can join in here directly. I just skimmed through this thread and I am thoroughly impressed with how much @Drumfix and collaborators have been able to reverse engineer. I understand a lot of the frustrations I'm reading about in here—Hopefully I can help out.
...

Hi AudioNarwhal and welcome to the thread!
Is my assumption correct that you are Dylan from MOTU that buonamorte.records has mentioned?
If yes, then thanks very much for joining the discussion. I really appreciate that!! We have been waiting for this kind of move from MOTU for a loooong while.
I think the most of us here appreciate the hardware and robust build of the MOTU interfaces and really want to get creative with them under Linux. We have paid some considerable amount of money to own such units (I own 2!!) and now it appears to be just a small step away to be able to unleash the full potential of MOTU units on our favorite OS!!

Thank you also for providing such detailed and very much needed insight into why things have been going wrong.

BTW: all debug-engineering kudos go to Drumfix!! He is our Thread-hero!! :D

My own involvement is just a meek attempt to weed out some confusion, distillate the significant information and bring in some structured approaches...
In this spirit, let me try to paraphrase some key take-aways from what you've written (please correct my errors):

  1. The MOTU firmware currently cannot be fixed, due to 3rd party dependencies that is beyond MOTUs control. Newer hardware platforms (e.g. UltraLite-mk5) no longer have this issue.

  2. The other OS platforms (Mac and Windows) would potentially face the same issue if their drivers did not schedule many isocs well in advance (no idea what this really is, but I'm trying to condense the core points here).

  3. Under Linux, we could potentially achieve the same performance as Windows & Mac with implicit feedback, if Linux would also schedule enough isocs in advance. You are currently working on achieving this in Linux.

  4. Since the explicit mode worked reliably under Linux, that is a good sign that the implicit mode can be made to work as reliably as the explicit mode.

Some questions from my side:

  • Regarding the 3rd party contributions beyond MOTUs control, how high are the chances that MOTU can find a true Firmware fix after all?
  • Do you have any test-automation we could use so that we stress-test your patches on our systems and provide feedback easily?
Last edited by shellwalker on Tue Nov 01, 2022 5:27 am, edited 1 time in total.

MOTU 828mk3, MOTU 828ES, Cakewalk by Bandlab, Kubuntu 22.04.1 LTS, https://github.com/shellwalker-coder/motu_patch_testing

User avatar
AudioNarwhal
Established Member
Posts: 39
Joined: Thu Apr 29, 2021 12:43 pm
Has thanked: 2 times
Been thanked: 35 times

Re: Ultralite AVB

Post by AudioNarwhal »

Is my assumption correct that you are Dylan from MOTU that buonamorte.records has mentioned?

Yes, it's me!

  1. The MOTU firmware currently cannot be fixed, due to 3rd party dependencies that is beyond MOTUs control. Newer hardware platforms (e.g. UltraLite-mk5) no longer have this issue.

  2. The other OS platforms (Mac and Windows) would potentially face the same issue if their drivers did not schedule many isocs well in advance (no idea what this really is, but I'm trying to condense the core points here).

  3. Under Linux, we could potentially achieve the same performance as Windows & Mac with implicit feedback, if Linux would also schedule enough isocs in advance. You are currently working on achieving this in Linux.

  4. Since the explicit mode worked reliably under Linux, that is a good sign that the implicit mode can be made to work as reliably as the explicit mode.

This is a good summary. The first statement is only true for the parts of the firmware that involve these dependencies, and yes the newer platforms no longer have those dependencies.

  • Regarding the 3rd party contributions beyond MOTUs control, how high are the chances that MOTU can find a true Firmware fix after all?

Well, I think we are stuck with the implicit feedback mode on this generation of devices. As for the missed interval recovery, that one is also tough, but you all are giving me the motivation to look and see if I can find a creative solution. I have an idea for a kind of workaround, but I need to do some experiments to see how viable it is. I am looking into this now.

  • Do you have any test-automation we could use so that we stress-test your patches on our systems and provide feedback easily?

We do not have any Linux based test-automation at the moment. One thing I have been thinking about is a way to stress test the system while audio is running. In my experiments using benchmark tools, stress, and other utilities designed to make heavy use of the IO, RAM and CPU don't always result in the kinds of audio dropouts that typical computer use results in. For example, prior to my patch I found that the playback packet stream would reliably miss a service interval whenever I opened Firefox on Ubuntu. However, none of the stress tools, or other GUI applications would create the same problem. I would be interested to hear about any ideas for automating this type of computer use. If we had something like that could reliably induce the failures we are seeing. We could write a script for Linux that monitors audio looped through the device looking for issues.

User avatar
bluebell
Established Member
Posts: 1909
Joined: Sat Sep 15, 2012 11:44 am
Location: Saarland, Germany
Has thanked: 111 times
Been thanked: 116 times

Re: Ultralite AVB

Post by bluebell »

AudioNarwhal wrote: Sun Oct 30, 2022 2:35 pm

One thing I have been thinking about is a way to stress test the system while audio is running. In my experiments using benchmark tools, stress, and other utilities designed to make heavy use of the IO, RAM and CPU don't always result in the kinds of audio dropouts that typical computer use results in.

Motu Ultralite AVB non-ESS:
I think the latest firmware with explicit feedback was 1.3.4+558 and ran with a Linux standard kernel.
I could easily force a modulo-8 channel hop by simply changing jackd's buffer size on the fly with the jack_bufsize command.

So I recommend adding this test to the list of test procedures.

Linux – MOTU UltraLite AVB – Qtractor – http://suedwestlicht.saar.de/

User avatar
AudioNarwhal
Established Member
Posts: 39
Joined: Thu Apr 29, 2021 12:43 pm
Has thanked: 2 times
Been thanked: 35 times

Re: Ultralite AVB

Post by AudioNarwhal »

Motu Ultralite AVB non-ESS:
I think the latest firmware with explicit feedback was 1.3.4+558 and ran with a Linux standard kernel.
I could easily force a modulo-8 channel hop by simply changing jackd's buffer size on the fly with the jack_bufsize command.

Thank you for this information. I would like to reproduce this setup to see if it is the same root cause. However, it looks like our publicly posted firmware versions jump from version "1.2.9+479" to "1.3.4+603". I can search internally for a "1.3.4+558" version, as it may have been released via another channel. Can you please confirm the firmware version and kernel version under which you were producing this issue? Thanks again.

User avatar
bluebell
Established Member
Posts: 1909
Joined: Sat Sep 15, 2012 11:44 am
Location: Saarland, Germany
Has thanked: 111 times
Been thanked: 116 times

Re: Ultralite AVB

Post by bluebell »

AudioNarwhal wrote: Mon Oct 31, 2022 12:32 am

Motu Ultralite AVB non-ESS:
I think the latest firmware with explicit feedback was 1.3.4+558 and ran with a Linux standard kernel.
I could easily force a modulo-8 channel hop by simply changing jackd's buffer size on the fly with the jack_bufsize command.

Thank you for this information. I would like to reproduce this setup to see if it is the same root cause. However, it looks like our publicly posted firmware versions jump from version "1.2.9+479" to "1.3.4+603". I can search internally for a "1.3.4+558" version, as it may have been released via another channel. Can you please confirm the firmware version and kernel version under which you were producing this issue? Thanks again.

I have 1.3.2+520, 1.3.4+558, 1.3.5+626 and 1.4.0+90954. IIRC only one of them was not downloaded from motu.com.
I used a Ubuntu stock lowlatency kernel 4.13 with MOTU firmware up to 1.3.4+558 IIRC – all with the channel hopping issue.

I think with 1.3.5+626 I had to use a self compiled kernel and Drumfix' ua101 driver because that was the version that switched to implicit feedback. With Drumfix' driver the channel hopping vanished. I have a notebook computer with Core i7
viewtopic.php?f=6&t=18046&p=122814#p122814

The Ultralite AVB's interrupts are managed by an onboard xhci controller. I count 16.000 interrupts per second regardless of jackd's bufsize.

Linux – MOTU UltraLite AVB – Qtractor – http://suedwestlicht.saar.de/

User avatar
AudioNarwhal
Established Member
Posts: 39
Joined: Thu Apr 29, 2021 12:43 pm
Has thanked: 2 times
Been thanked: 35 times

Re: Ultralite AVB

Post by AudioNarwhal »

Thank you @bluebell! I have located the 1.3.4+558 release. It looks like we released that as a beta firmware which is why it didn't come up in my initial search.

User avatar
AudioNarwhal
Established Member
Posts: 39
Joined: Thu Apr 29, 2021 12:43 pm
Has thanked: 2 times
Been thanked: 35 times

Re: Ultralite AVB

Post by AudioNarwhal »

I am still looking into all of this as I have time, but I wanted to give an update.

It looks like there is still hope of making a firmware that can monitor the USB buffer health and force a resynchronization if things get bad. I have done some experiments that seem promising. If this pans out, I'd like to share a test firmware sometime soon-ish to see if it improves the situation. With this, glitches would still happen (pops and ticks when a service interval is missed), but the device shouldn't get stuck in the bad state.

Digging a little deeper still, there may actually be a hope of supporting explicit feedback again (at least to some extent), there is more hope here than I suspected, however I need more time to investigate and test.

Lastly, I am still hoping to improve the implicit feedback support in the Linux kernel, as I think this will result in the best long term solution for Linux users being able to use class compliant devices that rely on this synchronization method. Takashi sent me an idea that may be even better than what I shared already, however its just proof of concept code for now.

supereb
Established Member
Posts: 93
Joined: Fri Aug 17, 2018 10:55 pm
Has thanked: 36 times
Been thanked: 5 times

Re: Ultralite AVB

Post by supereb »

@AudioNarwhal Thanks for getting involved and for the update. I have an 8pre-ES. Worked beautifully and never had a dropout using the 4.19 kernel complied with @Drumfix 's kernel quirks. I'm on 5.15 now using @Drumfix 's linux driver. That works well most of the time but I still get intermittent channel hopping on the playback side. It might go a 4 hour session without a problem, or drop out repeatedly in the span of a half hour. Restarting jack restores playback.

Looking forward to seeing what you come up with for a fix. Thanks again!

Last edited by supereb on Thu Nov 10, 2022 8:23 pm, edited 1 time in total.
Post Reply