How to detect which notes are played? (like in Rocksmith/Yousician)

[linuxo]

Hello!
I regularly play the guitar with Rocksmith and Yousician but I've found a lot of problems with these softwares which I'd like to fix by creating my own (simpler) version of those.

Since I'm completely new to music with computer (but not new to programming since I'm a Python developer myself) I'd like to have some directions about how to do it:
The main issue is how to detect what is played. If I've understood how music is working with linux, we can use JACK, which looks like a audio driver specifically for jack connection. The signal received from the guitar to the computer is a (digital) audio stream apparently.
So how can I convert it to the notes which are actually played?

Thanks a lot in advance

[noedig]

Incidentally I just saw this this morning: https://basicpitch.spotify.com/
Spotify released an open source tool to convert music (audio) to MIDI. I think you can install it with PIP to use in Python.

MOD devices also brought out a new plugin to detect which notes are played on a guitar, so you can play synths with your guitar. I *think* they use Aubio, but you can probably find their source on GitHub (also check falkTX's repos).

[tavasti]

Fishman TriplePlay midi pickup would be pretty 'in a box' solution for that.

[scott.thomason]

Here are some resources to get you started...pay particular attention to Faust and JUCE as you go. I found these highly interesting and valuable.

https://cs.gmu.edu/~sean/book/synthesis/Synthesis.pdf
https://github.com/olilarkin/awesome-musicdsp
https://timiskhakov.github.io/posts/pro ... itar-music
https://greenteapress.com/thinkdsp/html/index.html (this one covers Python specifically)

[scott.thomason]

Also, I just want to mention that the exact thing that you want to do, extract accurate and usable note information from an audio stream, happens to be one of the larger challenges in the audio/dsp world. There are lots of people trying hard to make it work well.

[khz]

@ModDevices
GuitarMidi
https://forum.moddevices.com/t/new-plug ... i-lv2/7589
https://github.com/geraldmwangi/GuitarMidi-LV2

Audio2CV
https://pedalboards.moddevices.com/plug ... 10by1jdg==
Audio2CV Pitch
https://pedalboards.moddevices.com/plug ... tpbmcjY3Y=

[scott.thomason]

@ModDevices
GuitarMidi
https://forum.moddevices.com/t/new-plug ... i-lv2/7589
https://github.com/geraldmwangi/GuitarMidi-LV2

Audio2CV
https://pedalboards.moddevices.com/plug ... 10by1jdg==
Audio2CV Pitch
https://pedalboards.moddevices.com/plug ... tpbmcjY3Y=

I feel it is worth pointing out that if you read the reviews for various guitar/audio/midi devices, pretty much everyone concludes that it's just not quite right yet.

[khz]

I feel it is worth pointing out that if you read the reviews for various guitar/audio/midi devices, pretty much everyone concludes that it's just not quite right yet.

Yes, MIDI is problematic because of the rounding up/down (no matter if with 7 bit = 127 steps or with 14 bit = 16384 steps (14 bit should become standard!)). This can cause unwanted errors and also increases computing power. Whereas the GuitarMidi-LV2 tool may be able to do it very well, who want test it and send a bug report to the developer.
That's why I mentioned the CV tools that reproduce a sound exactly and in RT. If you then build a synth with e.g. Cardinal/AMS/... you can play synthesizer $guitar very well. See/hear https://moddevices.com/guitar-synth/

[tavasti]

Yes, MIDI is problematic because of the rounding up/down (no matter if with 7 bit = 127 steps or with 14 bit = 16384 steps (14 bit should become standard!)). This can cause unwanted errors and also increases computing power.

I don't think that resolution is problem, problem is recognition of fundamental frequency of sound in real-time.

[wjl]

I sometimes use this if I can't figure out the chords of a song:

https://www.youtube.com/watch?v=bQrleXXQmWY

Lots of possibilities with these vamp plugins, and it all seems to be open source (which I haven't checked tho)

[merlyn]

I don't think that resolution is problem, problem is recognition of fundamental frequency of sound in real-time.

Yes, the problem is to detect a pitch with minimal latency. Bottom E on a guitar is ~82.4Hz. One cycle takes 1/82.4 = ~12.1ms which is far too long. So algorithms must have some way of working out the frequency without needing a full cycle. Tricky I imagine.

[Basslint]

I don't think that resolution is problem, problem is recognition of fundamental frequency of sound in real-time.

Yes, the problem is to detect a pitch with minimal latency. Bottom E on a guitar is ~82.4Hz. One cycle takes 1/82.4 = ~12.1ms which is far too long. So algorithms must have some way of working out the frequency without needing a full cycle. Tricky I imagine.

Some thoughts:
1) Would not focusing on guitar timbres make the algorithm faster? There are some frequencies which are highly unusual on guitars. My guess is that even taking into account bass guitars, this could save some processing time.
2) A game like Rocksmith expects a given note/chord in a given moment. Isn't comparison between two sounds, one right and one possibly wrong, always faster than having to identify label the note/chord in real time?
3) Is ML necessarily faster than physics algorithms? I would think so, ML nowadays is used for real-time feature detection in many fields where time is critical. I think that a ML algorithm optimized for guitars (maybe a neural network? Not an expert) could take an unusual route and be faster. After all, very few people have absolute pitch so I don't understand why we expect a machine to do it faster than us!

In the past I was interested in polyphonic pitch tracking but lately, I've come to my own conclusion that it distracts from ear training, solfege and good musicianship.

[Kott]

https://github.com/luciamarock/Polyphon ... or-guitars

(I haven't test it much yet)

[merlyn]

Some thoughts:
1) Would not focusing on guitar timbres make the algorithm faster? There are some frequencies which are highly unusual on guitars. My guess is that even taking into account bass guitars, this could save some processing time.

I have a Roland GR-55 guitar synth and it tracks pretty well. I use it with a Godin nylon string guitar that has a GK output (Roland's 13 pin guitar synth connector.) The pickups are made by RMC, and there is one per string, so the output is split into strings.

There's also the GK3A pickup which can be fitted to most electric guitars. When fitting this it is recommended to get the pickup as close to the bridge as possible. There is a also a setting on the synth for how many millimetres from the bridge the pickup is.

So being close to the bridge suggests to me the pickup is going for the harmonics, which are higher frequency, rather than the fundamental to cut down on the detection time.

A guitar synth is, I think, a different instrument. After trying a guitar synth Frank Zappa is quoted as saying "This is a different instrument. I don't have time for this." There's a loose, kind of sloppy style that suits an electric guitar. Think Nile Rogers -- lots of muffled percussive hits. Well, that doesn't work on a guitar synth, not with a piano sound anyway. Also for a sound like a piano it makes sense to turn off any pitch bend. You can't bend notes on a piano, so a guitar synth kind of does become a different instrument.