Atomic Operations in RT

[ssj71]

Hi all:

I've been thinking about porting the rbdrum2midi program to Lv2. This would make the plugin be a midi generator tied to hardware. I'm not aware of any plugin that does this, but it should be feasible.

Thats not my main question and is nearly irrelevant (other than context). There will be 2ish threads the plugin is concerned with. The main plugin callback and the asynchronous interrupt callback thread(s?) from libusb (when the drum is hit). A struct containing all the parameters (controls) which is mostly arrays of unsigned chars is shared between the threads, the plugin callback gets the control port values from the host, and then writes those values to the host. When a USB event occurs the interrupt callback is run which acts on these values and writes midi data to a circular buffer. That buffer is read next time the plugin callback runs and midi events are sent through the atom event port.

So the question is this: as I understand the writes and reads to the struct values will be atomic (meaning the action is completed in a single cpu instruction) because they are each a single byte, therefore this will be RT thread safe. Is this correct?

Thanks!

[ssj71]

I'll have to look at your try-locks. I'm only worried about the interrupt being called in the middle of a write and then acting on corrupt data. But I think since they are single bytes I can do it lockless. Something like:

Code: Select all

//in plugin callback
unsigned char tmp = (unsigned char)*plug->port;
plug->mystruct->myval = tmp; // this I *think* is atomic so it can't be interrupted
...

//in interrupt
if(mystruct->myval)//this may not be atomic, but the interrupt won't be interrupted (pre-empted?) so I'm not so worried
{
   //more stuff
}

I'd love for an expert help us understand for sure though.

[ssj71]

rt safe or not depends if the read/write uses locks or not.
afaik, atomic operations means either using something like std::atomic<int> or memory barriers.
(I think it's a very advanced topic).

So I think its more a question of if its thread safe. Lockless is of course RT safe.
From what I've read atomic means small and can't be broken up, so atomic operations compile to a single instruction. If the cpu begins the operation nothing can happen before the operation is complete (because its a single instruction any interrupt or anything has to come as the next instruction).

And yes most of the Stack exchange threads regarding it mention its fairly advanced. But I want to LEARN!

Perhaps the best way to answer my question is to look at the disassembly test and look if its a single instruction.

[raboof]

btw, one can fake atomics by using locks, see:
https://github.com/muse-sequencer/muse/ ... node.h#L29

since the lock only happens when modifying or reading a single value, it might considered rt-safe. not 100% sure though.

There's the problem of priority inversion lurking there though: say you have 3 threads with rtprio 10 (gui thread), 20 (event processing thread) and 30 (jack process thread).

In such a scenario you could have the following sequence of events:

• The user turns a knob in the GUI.
• The gui thread takes the lock to atomically write the new value
• A burst of events arrives.
• As the event processing thread has a higher priority than the gui thread, this thread is scheduled
• The JACK process callback is triggered. It tries to take the lock, but is blocked because the gui thread hasn't yet released the lock
• As the event processing thread has a higher priority than the gui thread, this thread is scheduled again
• The process is booted from JACK because it missed the deadline

Luckily there is a solution: priority inheritance. Quite frankly, unfortunately, I'm not sure whether Linux supports that at all... anyone?

[diizy]

Luckily there is a solution: priority inheritance.

Sounds interesting. How exactly does it work? If you don't mind explaining...

[Drumfix]

Priority inheritance is a kernel thing. The linux kernel with rt-patch provides priority inheritance, the stanard linux kernels do not.

Priority inheritance works like this: If a lower priority thread has ownership of a lock and a higher priority thread wants to take that lock,
then the priority of the lower priority thread is temporarily raised to the priority of the higher priority thread until it realeases the lock.

So in the 3 threads example with priority inheritance, thread A prio 10, thread B prio 20, thread C prio 30:

A aquires lock, B preempts A, C wants to aquire lock, Prio of A is raised to prio of C, thus A preempts B, A releases lock, C aquires lock, prio of A is set
back to its own prio.

In the 3 threads example without priority inheritance:

A aquires lock, B preempts A, C wants to aquire lock, A has to wait until B is finished (which could be a very long time), A release the lock, C aquires lock.

[ssj71]

btw, one can fake atomics by using locks, see:
https://github.com/muse-sequencer/muse/ ... node.h#L29

since the lock only happens when modifying or reading a single value, it might considered rt-safe.
not 100% sure though.


not exactly about atomics, but here's some good info about memory barriers:
http://en.wikipedia.org/wiki/Memory_barrier
https://www.kernel.org/doc/Documentatio ... rriers.txt

I'd never heard about memory barriers. It makes sense theoretically. I don't really know how to apply it really.

raboof is right though, that code is definitely not RT safe precisely because it is not atomic at all. And I assume drumfix to be correct, then it is safe if you have an rt patched kernel. Though I'm reading about Pi-futex which is a user-space priority inheritance fast mutex implementation at kernel.org: https://www.kernel.org/doc/Documentation/pi-futex.txt
are patches hosted at kernel.org too or just mainline vanilla kernel stuff?

[Drumfix]

Oh, maybe PI-futexes have made it in into the mainline kernel already. Would be great.

For parallel programming (including memory barriers), i recommend reading this (esp. chapter 14):

http://kernel.org/pub/linux/kernel/peop ... ok-e1p.pdf

[diizy]

Priority inheritance is a kernel thing. The linux kernel with rt-patch provides priority inheritance, the stanard linux kernels do not.

Priority inheritance works like this: If a lower priority thread has ownership of a lock and a higher priority thread wants to take that lock,
then the priority of the lower priority thread is temporarily raised to the priority of the higher priority thread until it realeases the lock.

Oh, that makes sense. Thanks for the explanation.

[tramp]

I prefer very much "signaling" over all. sigc could be used not only in threads, it suits as well processes.

[kmatheussen]

Priority inheritance is a kernel thing. The linux kernel with rt-patch provides priority inheritance, the stanard linux kernels do not.

Priority inheritance works like this: If a lower priority thread has ownership of a lock and a higher priority thread wants to take that lock,
then the priority of the lower priority thread is temporarily raised to the priority of the higher priority thread until it realeases the lock.

So in the 3 threads example with priority inheritance, thread A prio 10, thread B prio 20, thread C prio 30:

A aquires lock, B preempts A, C wants to aquire lock, Prio of A is raised to prio of C, thus A preempts B, A releases lock, C aquires lock, prio of A is set
back to its own prio.

In the 3 threads example without priority inheritance:

A aquires lock, B preempts A, C wants to aquire lock, A has to wait until B is finished (which could be a very long time), A release the lock, C aquires lock.

You can do this manually too.

In a jack program, it should be enough to call "jack_acquire_real_time_scheduling(pthread_self(), jack_client_real_time_priority(client))" before obtaining the lock.