Can WebRTC help musicians?

[Rainmak3r]

Hi all,

first of all, a disclaimer. I'm a WebRTC developer myself: specifically, I'm the main author of Janus, an open source WebRTC server that has gained some popularity these past few years. That said, I love making music and I love open source, which is the main reason why I've been hanging out around here for more than a year now (mostly sharing my "creations" with folks). As such, I wanted to figure out whether or not you think there's anything WebRTC could do to help musicians in the FOSS world.

If you're not familiar with WebRTC, in a nutshell it's a technology that allows you to create peer-to-peer real time media channels in a browser, with no need for custom plugins like Flash or Java applets. It was originally born to allow you to make video calls from a browser in JavaScript, but it soon ended up becoming so much more, involving many more different devices as well. Thanks to the help of dedicated applications and servers (that do break the peer-to-peer paradigm, but mostly with good reason), it's currently being used in a whole variety of vastly heterogeneous use cases. If you'd like to know more about it in general, please don't hesitate to ask: I have plenty of material to point you to.

That said, I've often wondered what WebRTC can do for arts, and music in particular. While my WebRTC project is labelled as "general purpose", since it tries to be as generic as possible and allow for different things, quite obviously most people use it for traditional and, if you will, "boring" applications like conferencing, e-learning, webinars, and stuff like that. I'm always looking forward to more "artistic" usages of the project, being a hobbyist musician myself, but that has unfortunately rarely happened. I do know a few have used it to stream their concerts live: WebRTC is real-time, and there's no requirement for sessions to be bidirectional, meaning it's a quite effective tool to stream live content with very low latency (especially compared, e.g., to HLS). Others have used it more simply as a tool for music lessons, e.g., to learn an intrument: while a more interesting use case than the boring business Skype call, it's still quite traditional though. An interesting exception has been a theater company, that recently wrote me to tell me they used the software to implement an online interactive play. But apart from that, there hasn't been much music-related.

I did try to experiment a bit in the past, just for fun. For instance, if you watch this video, you'll see a very short demo (~3 minutes) I made at a conference a few months ago, using open source tools. This demo had a piano keyboard in a web application; the web application would create a WebRTC datachannel to send notes being hit by the player to a Lua script in Janus; the Lua script would then use ALSA to translate those notes to MIDI signals and pass them to Jack; I'd then hook that up to something that could play the notes (e.g., QSynth or Yoshimi), and via Jack I would pass the output of that application to a gstreamer pipeline; finally, the gstreamer pipeline would be configured to encode the received audio in real-time to Opus (the default audio codec supported in WebRTC) and send it via RTP back to Janus; Janus would finally broadcast the encoded stream in real-time back to the user via WebRTC. The "fun" part in the demo was that I allowed multiple people to open the same web application at the same time: if I played something, and you had the page open too, you could see the notes being played on your keyboard (different people had different colors) and everyone could listen to what everyone was playing. Of course, it soon turned into a mess (10 people trying to play the same keyboard would be a mess in real-life too ), but it was a fun experiment nevertheless.

Anyway, I'm honestly interested to understand if you think there's actually anything that such a technology could do to help with music. Jam sessions come to mind, of course, even though I fear the latency involved (although very low most of the times) may be too high. Possibly integrations in DAWs for some use case?

Looking forward to your thoughts!

[Basslint]

Yes, of course I think WebRTC can help musicians! A great thing would be to implement an alterative to programs like Jamulus (or the NINJAM clients) so that people can just plug in their instruments and play without any technical knowledge.

A bit OT but since you are Italian like myself, what do you think about the fact that while standardized, low-cost, FLOSS, WebRTC, browser-based solutions exist many public schools forced students to use proprietary solutions like Cisco WebEX, while others used Jitsi Meet without any problems?

[Rainmak3r]

Yes, of course I think WebRTC can help musicians! A great thing would be to implement an alterative to programs like Jamulus (or the NINJAM clients) so that people can just plug in their instruments and play without any technical knowledge.

I'm not familiar with those, but yeah, jam sessions came to mind as one of the ideal use cases. The main issue I see is that one of the greatest advantages of WebRTC is that it "just works" in a browser, which makes it tricky in a music environment though. None of the browsers I know supports Jack, for instance (apart from an obscure Firefox fork somewhere, I read), which means you can't really integrate them with your typical music production flow. This means the only alternative is some more native approach, which works but clearly requires more effort and is harder to make really platform independent: that's why most WebRTC desktop clients out there are just simply Electron apps, which basically just wrap a browser in a thin native layer, and so wouldn't help here.

A relatively easy solution I guess might be leveraging Gstreamer for the purpose, which does support Jack in both directions, and supports WebRTC as well via its webrtcbin modules; I used it successfully in a couple of client-side projects, and in the simple demo I introduced in my post above. It would make capturing video along the real-time audio quite easy too. WebRTC frameworks like aiortc (python) and Pion (go) would make it even easier to write applications that wrap that functionality on the client side. Once you have a way to capture and render WebRTC streams on the client side, you can hook those up to pretty much any WebRTC server out there to distribute them, or keep them peer-to-peer (which can be problematic in jam sessions where you have more people in, though). It would definitely be interesting to experiment with all this: that said, while the lower level stuff is clear, I never cared much about learning more on client side frameworks And that would indeed be key to your "just plug in their instruments and play without any technical knowledge" part.

A bit OT but since you are Italian like myself, what do you think about the fact that while standardized, low-cost, FLOSS, WebRTC, browser-based solutions exist many public schools forced students to use proprietary solutions like Cisco WebEX, while others used Jitsi Meet without any problems?

I think there are different answers to that question.

One one side, there definitely is a cultural aspect: while things have started to change in Italy, most still see open source as either just a toy that can't really compare to or compete with commercial software, or "free" stuff that they can exploit. To give you an idea, while Janus is quite widespread and used all over the world, you'd expect a lot of Italian companies to actually use it, especially considering the company backing it is from Italy as well: that doesn't happen, instead, and I can tell you that the Italian companies we've done business with so far you can count on your hands. There may be more that just use it and don't tell it, but I don't think they would increase that number much. Most just go with well known commercial solutions instead.

Which brings us to schools and universities in this problematic times. My background is in academia, and I know that so many of them are actually using proprietary tools (mainly Teams, but WebEx and Zoom too) simply because they've received it for free on a temporary basis. While they can be lauded for that in this time and age, it's also clear that companies like Microsoft are giving their tools away for free to get organizations hooked and used to it, so that they can start charging for them once things settle down. Some may argue that they have the advantage of a more integrated experience: Teams will obviously be easier to integrate with Outlook if you're using it already, to make a dumb example. I'm not sure about that, since open source gives you much more freedom, and so with a bit of work you could actually have something that is not only better integrated, but actually perfectly fits your requirements and architecture like a glove. But that's the main issue: that bit of work, while low cost, is still a cost in terms of money and possibly time, which is why when you're given something for free, that you can use RIGHT NOW, you may just stick to that.

But I agree that for the sake of the individual sessions (let's say, a room to host lessons every day), there's so much in open source that could be used out of the box as a perfect replacement. We never invested in a sleek UI (we're more on the "engine" side of things, since we cover more scenarios), but the guys from Jitsi did and so it would be quite easy to use even for people that are not familiar with conferencing. The problem is that you only start using things when you know about them, and that's where the cultural aspect comes back.

I do know of efforts taking place to work on more tailored solutions, though. One teaching area people are looking at is proctoring, for instance, that is performing online exams that may involve tons of students at the same time: it's clear that with so many remote students, tracking them all and ensuring they don't "cheat" is quite problematic, and that's something that off-the-shelf commercial products don't really help with. Hopefully that will show, even here, how open source software can actually help you craft a much better solution to the problem.

Apologies, I just realized I went a bit too far on the OT topic, rather than the IT one

[d.healey]

I'm building an electron app to allow users to demo synths/sample libraries over the internet. I'm using node MIDI for the MIDI routing and was planning to use something like VNC for the GUI and Audio side of things, but I'm wondering if WebRTC might be able to handle that aspect. What do you think?

[Rainmak3r]

I'm building an electron app to allow users to demo synths/sample libraries over the internet. I'm using node MIDI for the MIDI routing and was planning to use something like VNC for the GUI and Audio side of things, but I'm wondering if WebRTC might be able to handle that aspect. What do you think?

For audio I'd say most definitely, as it's very close to the demo I mentioned in my original post. In that case I was using datachannels to relay the MIDI events (that a Lua script would transllate to actual MIDI via alsa on the server side), but I guess node MIDI would work too. I should probably put the code for that demo on github, so that you can play with it and see if there's anything you can do similarly: I'll post the link here when I've done that.

For the GUI, it's a bit more complex. WebRTC does support screen and application sharing, both of which are sent as plain video. This means it should be easy to capture the synth/sample you want to demo via ffmpeg or gstreamer, and have it encoded/streamed via RTP to a server that that turn it to WebRTC for you (like Janus or others). Controlling it would be a different matter, though, as WebRTC for security reasons doesn't provide ways to remotely control a screen or application itself: this is something you'd have to enforce out of band: a potential approach might be using a <canvas> element to draw the remote video, and intercept mouse movements and clicks on that; then you can send these events via data channels, and have them enforced on the remote GUI somehow via your server (xdotool?). Of course, just using VNC for the screensharing and just WebRTC for audio works too, even though I suspect there might be synchronization issues, and I'm not sure how much more/less VNC would use for bandwidth. A completely different alternative might be not exposing the actual GUI at all, but instead provide a controllable (REST/WebSocket/datachannel?) API with the synth/sample properties that the user can tweak directly in the browser, and that gets enforced on the server side: of course, this only works if the synth/sample is indeed controllable via command line somehow, and means you need to do some work for each of the ones you want to support, but would probably be more scalable (if you screenshare, you probably won't be able to do more than one session per remote server, since the actual desktop space would be "busy").

I hope this helps and is not even more confusing

[Rainmak3r]

FYI, I created a new repo and put the code for that old demo here: https://github.com/lminiero/webrtc-piano
Please let me know if you have trouble getting it to run.

[d.healey]

FYI, I created a new repo and put the code for that old demo here: https://github.com/lminiero/webrtc-piano
Please let me know if you have trouble getting it to run.

Thanks for the info. I'll take a look at your repo.

I'm exploring using Xpra for the GUI/remote control, it also handles audio too although I don't know if there will be a latency issue.

[Rainmak3r]

Hi all,

FYI, I submitted (and already recorded) a talk on this for the upcoming virtual FOSDEM meeting. If you're interested in checking this out, it will be streamed on the 6th of February in the RTC devroom: https://fosdem.org/2021/schedule/event/ ... musicians/

It's mostly meant as a high level and general overview on use cases, WebRTC-related challenges and opportunities, etc. (I couldn't go in too much detail of anything in 20 minutes), but hopefully you'll find something of interest there. While the presentation is pre-recorded, there will be a 5 minutes Q&A right after it ends (I think using Jitsi), so in case you feel like discussing this in "person" (or even just say hello ), please do pass by! I know I'd definitely like to finally "meet" @Basslint for instance I think a Matrix chat will also be available, but still not sure about that (I'm in one that's limited to speakers).

Looking forward to your feedback!

[Basslint]

Hi all,

FYI, I submitted (and already recorded) a talk on this for the upcoming virtual FOSDEM meeting. If you're interested in checking this out, it will be streamed on the 6th of February in the RTC devroom: https://fosdem.org/2021/schedule/event/ ... musicians/

It's mostly meant as a high level and general overview on use cases, WebRTC-related challenges and opportunities, etc. (I couldn't go in too much detail of anything in 20 minutes), but hopefully you'll find something of interest there. While the presentation is pre-recorded, there will be a 5 minutes Q&A right after it ends (I think using Jitsi), so in case you feel like discussing this in "person" (or even just say hello ), please do pass by! I know I'd definitely like to finally "meet" @Basslint for instance I think a Matrix chat will also be available, but still not sure about that (I'm in one that's limited to speakers).

Looking forward to your feedback!

I would more than gladly drop by and say hi! Jitsi Meet is more than fine

[Basslint]

Just watched the talk Lorenzo, it was great! I did not realize how big the impact of your work could be for educational purposes. I really wish your efforts will be widely adopted at the very least in European public administrations, where I definitely see a place for it. Also liked your web-based examples.

You cited NINJAM at the end. Since it has already wide adoption and it's FLOSS, why not implement a WebRTC-based version of it?

Also, thanks for the LinuxMusicians plug

PS: Element (and Quaternion) sadly did not play well with my Matrix server, so I couldn't interact with you during the talk...The platform was great overall but I seriously hope Matrix improves if it wants to become the standard messaging protocol...

[Rainmak3r]

Just watched the talk Lorenzo, it was great! I did not realize how big the impact of your work could be for educational purposes. I really wish your efforts will be widely adopted at the very least in European public administrations, where I definitely see a place for it. Also liked your web-based examples.

Thanks Fabio, this is very appreciated! I wasn't sure I had enough "meat" for a presentation, feeling it might be considered a bit too vague, so I'm glad to hear it came out interesting enough anyway.

You cited NINJAM at the end. Since it has already wide adoption and it's FLOSS, why not implement a WebRTC-based version of it?

Jamulus and NINJAM actually ended up in the slides just thanks to YOU!
You mentioned them in response to this post, a few months ago, and so I definitely wanted to cite them as existing solutions.
I'm afraid I'm not familiar with either, though, as I haven't checked how they work yet. I fear that a WebRTC endpoint for NINJAM would require a bit of a translation process somewhere (e.g., if they don't use Opus, audio would need to be transcoded), and that might add a bit of latency. But that's just thinking out loud: it might indeed make sense to at least check what can be done there.

Also, thanks for the LinuxMusicians plug

That was the very least I could do, I love this place! It's a pity there's no "music software" devroom at FOSDEM: considering how much rich the ecosystem is in terms of applications and solutions, I feel it would definitely have its place there.

PS: Element (and Quaternion) sadly did not play well with my Matrix server, so I couldn't interact with you during the talk...The platform was great overall but I seriously hope Matrix improves if it wants to become the standard messaging protocol...

There wasn't any live Q&A for my talk at all, unfortunately... I think it was a capacity issue of some sorts, and what was not really working actually was not Matrix, but the ad-hoc widgets they created for the purpose. The live stream worked when opened from the FOSDEM website, for instance, and the Jitsi room they used for Q&A worked on their own too, but were not properly integrated. I managed to get in the room with Saul (the devroom chair) eventually, but we were not being broadcast back to the stream. But that said, considering the humongous amounts of sessions and users they're serving at the same time, I still consider the setup impressive, and even just answering questions via chat (and interacting with the audience in real-time) was great!

[SpotlightKid]

Unfortunately I missed the stream The slides had some very interesting links in them, though. Is the video of the talk or the stream available for download or viewing again (or will it be)?

[Rainmak3r]

Unfortunately I missed the stream The slides had some very interesting links in them, though. Is the video of the talk or the stream available for download or viewing again (or will it be)?

It will become available soon, on the talk page: https://fosdem.org/2021/schedule/event/ ... musicians/
Usually they're quite fast, so I suspect either later today or tomorrow it should be there!

[Basslint]

Jamulus and NINJAM actually ended up in the slides just thanks to YOU!
You mentioned them in response to this post, a few months ago, and so I definitely wanted to cite them as existing solutions.
I'm afraid I'm not familiar with either, though, as I haven't checked how they work yet. I fear that a WebRTC endpoint for NINJAM would require a bit of a translation process somewhere (e.g., if they don't use Opus, audio would need to be transcoded), and that might add a bit of latency. But that's just thinking out loud: it might indeed make sense to at least check what can be done there.

Glad to have been of any use! The way NINJAM works is that it delays all jammers by the same interval, so that additional latency would not be a problem. NINJAM uses Ogg Vorbis for compression right now. Anyway I think that it would be very cool to have something which can get those people who aren't very computer literate jamming from their browsers with FLOSS. NINJAM has quite a follow already (http://autosong.ninjam.com/server-list.php) and is an open protocol with a FLOSS official implementation, that's why I push for it despite there being alternatives.

That was the very least I could do, I love this place! It's a pity there's no "music software" devroom at FOSDEM: considering how much rich the ecosystem is in terms of applications and solutions, I feel it would definitely have its place there.

I hope next year virtual talks will be allowed. First because they are friendlier to the environment, second because I think many of us here should participate!

There wasn't any live Q&A for my talk at all, unfortunately... I think it was a capacity issue of some sorts, and what was not really working actually was not Matrix, but the ad-hoc widgets they created for the purpose. The live stream worked when opened from the FOSDEM website, for instance, and the Jitsi room they used for Q&A worked on their own too, but were not properly integrated. I managed to get in the room with Saul (the devroom chair) eventually, but we were not being broadcast back to the stream. But that said, considering the humongous amounts of sessions and users they're serving at the same time, I still consider the setup impressive, and even just answering questions via chat (and interacting with the audience in real-time) was great!

All in all, it went decently for something so massive. I hope next year they will learn from these issues. I agree with what you said about Skype and sadly there are people getting conservatory lessons from there...

[Rainmak3r]

For those who missed it, the video of the presentation is now available, feedback welcome!

http://bofh.nikhef.nl/events/FOSDEM/202 ... cians.webm