[SOLVED] Live streaming and recording video

[nils]

Your script gives me:

Code: Select all

Setting pipeline to PAUSED ...
libv4l2: error getting pixformat: Invalid argument
ERROR: Pipeline doesn't want to pause.
ERROR: from element /GstPipeline:pipeline0/GstJackAudioSrc:jackaudiosrc0: Could not negotiate format
Additional debug info:
gstbasesrc.c(2830): gst_base_src_start (): /GstPipeline:pipeline0/GstJackAudioSrc:jackaudiosrc0:
Check your filtered caps, if any
Setting pipeline to NULL ...
Freeing pipeline ...

Can you derive any information from this?

[ToddMWorth]

Hi nils,

Sorry about that!

That command has a lot of stuff going on in it... You might want to break it down a bit, to ensure all the separate parts work. First, here are some comments that explain it a bit. Please let me know if you want more detail. (PS, don't try to run this. It's the same as the one I posted last, but with comments)

Code: Select all

# Start gstreamer. It makes sense to read from the bottom to the top from here on... 
    gst-launch -e \
# this mixer, mixes the desktop image on the bottom, and puts the webcam over the top. The parts of the webcam image that match your RGB values, will be transparent by now, so the desktop can be seen behind them. We now have all the video mixed together.
       videomixer name=chromamix sink_0::zorder=0 sink_1::zorder=1 ! \
  # here, we direct all the mixed video to three places:
         ffmpegcolorspace ! videoscale ! videorate ! video/x-raw-yuv, framerate=10/1, width=1280, height=720 ! tee name="vtee" ! \
  # to be saved to local disk as uncompressed AVI, with the audio.
         queue ! jpegenc ! videorate ! queue max-size-bytes=100000000 max-size-time=0 ! avimux name=avimux ! filesink location=video.avi sync=false vtee. ! \
  # to be shown on-screen, so you can locally monitor your stream
         queue leaky=1 ! ffmpegcolorspace ! xvimagesink sync=false vtee. ! \
  # and to be muxed to a flash video, and streamed to your ustream channel. The end. :)
         queue leaky=1 ! ffenc_flv ! flvmux name="flashmux" ! rtmpsink location="rtmp://1.12942718.fme.ustream.tv/ustreamVideo/channelnumber/streamkey flashver=FME/2.5\20(compatible;\20FMSc\201.0)" sync=false \
# grab audio by creating a stereo jack client. You'll see it in Claudia, and you can patch whatever you like into it. Send the raw audio to two places - the avimux muxer, and the flashmux muxer.
       jackaudiosrc connect=none ! \
         audioconvert ! audiorate ! 'audio/x-raw-int,rate=44100,channels=2' ! tee name="atee" ! \
           queue ! avimux. sync=true atee. ! \
           queue leaky=1 ! ffenc_aac ! flashmux. \
# create a video mixer called vmix, and apply chromakey transparency - in other words, let the desktop image in the background, be visible through your specified colour. Change that colour with RGB values in target-r, target-g, target-b. Send it to the chromamix second sink.
       videomixer name=vmix sink_1::zorder=0 sink_0::xpos=20 sink_0::ypos=20 sink_0::zorder=1 ! alpha method=custom target-r=123 target-g=21 target-b=17 angle=20 ! ffmpegcolorspace ! videoscale ! videorate ! video/x-raw-yuv, framerate=10/1, width=1280, height=720 ! chromamix.sink_1 \
# grab the first cam at low resolution to be used as the overhead 'deck-cam'. Send it to the mixer named vmix, on it's first sink.
       v4l2src device=/dev/video0 ! image/jpeg, width=320, height=240, framerate=10/1 ! jpegdec ! ffmpegcolorspace ! videorate ! \
         vmix.sink_0 sync=true \
# grab the second cam at high resolution (width and height parameters) to show the whole room. Send it to the mixer named vmix, on it's second sink.
       v4l2src device=/dev/video1 ! image/jpeg, width=1280, height=720, framerate=10/1 ! jpegdec ! ffmpegcolorspace ! videorate ! \
         vmix.sink_1 sync=true \
# grab the top part of the desktop, to be used as a background. You can change startx,starty, endx and endy, to capture a different area of your desktop. Send it to the mixer called chromamix, as it will be green-screened in.
       ximagesrc use-damage=false show-pointer=true startx=0 starty=40 endx=1680 endy=400 ! ffmpegcolorspace ! videoscale method=0 ! videorate ! video/x-raw-yuv, framerate=10/1, width=1280, height=720 ! chromamix.

So, now we can get the code from above into a text editor (eg Kate) and just get the stuff we want for testing. Let's test out the cameras, by using the v4l2src (Video4Linux 2 Source) module as above, and we will send it to the xvimagesink, to show it on screen.

Code: Select all

    gst-launch -e \
       v4l2src device=/dev/video0 ! image/jpeg, width=320, height=240, framerate=10/1 ! jpegdec ! ffmpegcolorspace ! videorate ! \
       ffmpegcolorspace ! xvimagesink sync=false

Code: Select all

    gst-launch -e \
       v4l2src device=/dev/video1 ! image/jpeg, width=1280, height=720, framerate=10/1 ! jpegdec ! ffmpegcolorspace ! videorate ! \
       ffmpegcolorspace ! xvimagesink sync=false

If those fail, maybe try different resolutions? Let me know how you go

[nils]

(PS, don't try to run this. It's the same as the one I posted last, but with comments)

I can't run it with the comments anyway since they disturb the new-lines "\"

Code: Select all

gst-launch -e \
       v4l2src device=/dev/video1 ! image/jpeg, width=320, height=240, framerate=10/1 ! jpegdec ! ffmpegcolorspace ! videorate ! \
       ffmpegcolorspace ! xvimagesink sync=false

/dev/video0 is my real camera. /dev/video1 is the loopback device.

At first I was confused because nothing worked at all. The forum uses '\302\240' characters as indentation spaces and I copied them. But after clean formatting, using the real cam (/dev/vide0) the command above works at least!
The commented version is nice to see. I'll crawl my way through once again.

My goal is actually much simpler: I just want to capture X-server 1:0 (second X server, first screen) and stream the video to a virtual webcam. Sound comes after that, but at least I am much more experienced with sound routing, commands and syntax. A real webcam picture-in-picture overlay would be nice to have, but is only my second step.

Nils

[nils]

Code: Select all

gst-launch -e \
    jackaudiosrc connect=none ! audioconvert ! audiorate ! 'audio/x-raw-int,rate=48000,channels=2' ! ffenc_aac ! flashmux. \
    ximagesrc use-damage=false show-pointer=true display-name=:1.0 screen-num=0 ! ffmpegcolorspace ! videoscale method=0 ! videorate ! video/x-raw-yuv, framerate=25/1 ! ffenc_flv ! flvmux name="flashmux" ! \
    rtmpsink location="rtmp://live.twitch.tv/app/*******************" sync=false

This works well to stream video witha audio to my rtmp account.
I do not do any conversion except the framerate here because I am recording my second X server which is already set to a low resolution.

With twitch.tv I get around 8 seconds delay which is still ok for a live stream. The important thing is: Audio and video are in sync and I can use JACK, that was not possible with ffmpeg at all. Even using alsa with ffmpeg generated audio that was
4 seconds behind the video!

I get a few x-run, or the gstreamer equivalent since I don't see any in qjackctl:

Code: Select all

gstbaseaudiosrc.c(840): gst_base_audio_src_create (): /GstPipeline:pipeline0/GstJackAudioSrc:jackaudiosrc0:
Dropped 114176 samples. This is most likely because downstream can't keep up and is consuming samples too slowly.

but that is no comparison to ffmpeg where I got plenty of those per second, no matter what audio driver or setting I used.
Maybe If reduce the audio-quality here I can still improve things.

Now all I need is the webcam overlay

Any comments or advices?

[ToddMWorth]

This works well to stream video witha audio to my rtmp account.

Sweet!

I do not do any conversion except the framerate here because I am recording my second X server which is already set to a low resolution.

Good call. I have lots of conversions in there, because I am mixing a 16:9 and 4:3 ratio... but the less conversion, the better quality, and the less CPU.

With twitch.tv I get around 8 seconds delay which is still ok for a live stream. The important thing is: Audio and video are in sync and I can use JACK, that was not possible with ffmpeg at all. Even using alsa with ffmpeg generated audio that was
4 seconds behind the video!

Glad the sync is working well for you, too. I had the same trouble. The 8 second delay is the lag of network traffic to the server, but it is good in one way because it does allow you a killswitch - if you stop gstreamer, the video stops almost instantly, meaning that the viewers never see or hear the last 8 seconds of your broadcast.

I get a few x-run, or the gstreamer equivalent since I don't see any in qjackctl:

Code: Select all

gstbaseaudiosrc.c(840): gst_base_audio_src_create (): /GstPipeline:pipeline0/GstJackAudioSrc:jackaudiosrc0:
Dropped 114176 samples. This is most likely because downstream can't keep up and is consuming samples too slowly.

This is where the 'queue' commands for gstreamer come in. Each of those, causes the prior input to be handled in it's own thread. The leaky=1, allows the buffer to become empty, and it will just fill the space with zeros. By using that before the network-facing sinks, the viewer would still get a glitch, but, the sync is still good. It's not a 'must', but it's recommended anywhere you have a mixer, muxer, tee, etc, where things could get messed up.

Maybe If reduce the audio-quality here I can still improve things.

I would also try a lower framerate. It really effects things (even when the high framerate does not overload the CPU). Especially if you are only making flash video, and not capturing raw data locally, the higher framerate won't count for much.

Now all I need is the webcam overlay

Any comments or advices?

I would do this first by adding the video mixer. Note the way I specify the inputs to the mixer, by 'sink_0' and 'sink_1'. This is the trick to making it work. By doing this, you can use '::zorder=' to force the Z-order of the images - in other words, you can tell it to put your webcam on top, and the background on the bottom; and you can use '::xpos=' and '::ypos=' to position the image. This is the hardest part, after this you'll cruise through it To make it easier, don't try to add both video sources. The mixer will happily run with just one, so you can get each one working separately and then add them when it is convenient.

HTH!

[nils]

I was now able to combine jack-audio, my webcam as overlay and one desktop on the second X server (dual screen) into an rtmp stream. Since twitch.tv really degrades the quality I have to export at 1280x720 for a 320x240 video game stream, only so you can see the actual webcam picture instead of a blurry blob.

One task remains undone: using the v4l2loopback driver to stream the thing into a virtual webcam. Either with sound (for full compatibility with Google Hangout and other services that only support a webcam) or without sound for One to One video conferences in skype etc. to teach people stuff.

Well I guess that last case works better with a remote desktop tool, but people have to install that first and route it through their firewall/router, so I think for non-interactive tasks Skype+Virtual Webcam might be the better choice.

[ToddMWorth]

v4l2sink is what you need. I tested this out successfully with skype, google and something else I forget.... Just as simple as this:

Code: Select all

! v4l2sink device=/dev/video1

[nils]

Then this is another indictator that something on my system is wrong. I can't get v4l2loopback working (v4lsink), even the simplest test fails.
Yesterday I filed a bugreports on their github tracker, so I'll have to wait for that.

https://github.com/umlaeute/v4l2loopback/issues/36

[ToddMWorth]

Bummer, works here, using either the ubuntu v4l2loopback-dkms package, or with the loopback included with webcamstudio.

Looks like your last issue now, is just that gstreamer/v4l2loopback/arch bug.... Good luck man!

[nils]

hey Todd,

I compiled and installed the experimental branch of v4l2loopback where a fix available. Now I can get the testvideo of gestreamer to the virtual webcam.

It still says "Could not negotiate format" but this time I believe it to be a real error.
Here is my small script, without sound to stream to a webcam. I tried to remove the ffmpegcolorspace before v4l2sink but that changes nothing.

Any comment or help for this?
I want the /video1 stream in the relatively high resolution of 1280x720 in the end.

Code: Select all

gst-launch -e \
    videomixer name=mixi sink_0::zorder=0 sink_1::zorder=1 sink_1::xpos=1000 sink_1::ypos=20 ! \
        ffmpegcolorspace ! videoscale ! videorate ! video/x-raw-yuv, framerate=30/1, width=1280, height=720 ! tee name="vtee" ! \
        queue leaky=1 ! ffmpegcolorspace ! v4l2sink device=/dev/video1 sync=false vtee. ! \
        queue leaky=1 ! ffmpegcolorspace ! xvimagesink sync=false \
    ximagesrc use-damage=false show-pointer=true  startx=0 starty=0 endx=1280 endy=720  display-name=:1.0 screen-num=0 ! ffmpegcolorspace ! videoscale method=0 ! videorate ! video/x-raw-yuv, framerate=30/1 ! mixi.sink_0 \
    v4l2src device=/dev/video1 ! image/jpeg, width=320, height=240, framerate=30/1  ! jpegdec ! ffmpegcolorspace ! videorate ! ffmpegcolorspace ! mixi.sink_1 sync=false

[ToddMWorth]

Have you tried it at a framerate of 10? I know 30 should work, but it's fairly high...

If you find that it works with the test pattern but not your stream, I recommend re-building your command using that as a starting point... As in, swap the test pattern for your webcam, then add the mixer, then the tee, etc etc... Might help to isolate the cause of the fault...?

[nils]

after querying and "verbosing" gstreamer a bit more I finally figured out the right format.
The v4l2loopback is very picky.

If find the framerate=30000/1001 a bit strange. But who am I to judge a kernel module

If anyone has suggestions how to improve things, maybe remove performance holes (except the framerate and resolution), please tell me!

Thank you ToddMWorth for you help!

Code: Select all

gst-launch-0.10 -e \
    videomixer name=mixi sink_0::zorder=0 sink_1::zorder=1 sink_1::xpos=10 sink_1::ypos=0 ! queue leaky=1 ! v4l2sink device=/dev/video1 \
    ximagesrc use-damage=false show-pointer=true display-name=:1.0 screen-num=0 startx=0 starty=0 endx=1280 endy=720 ! videoflip method=horizontal-flip ! ffmpegcolorspace ! videoscale method=0 ! videorate ! video/x-raw-yuv, format=\(fourcc\)YUY2, framerate=30000/1001, width=1280, height=720, interlaced=false, pixel-aspect-ratio=1/1, color-matrix=sdtv, chroma-site=mpeg2 ! mixi.sink_0 \
    v4l2src device=/dev/video0 ! video/x-raw-yuv, format=\(fourcc\)YUY2, width=160, height=120,  framerate=30/1 ! videorate !  mixi.sink_1 sync=false

[ToddMWorth]

Nice work dude!! LOL 30000/1001... okayyyyyy Yeh now that you mention it mine has shown me some strange rates like 300/11 and 299/10 and the like. No clue what is up with that, I just remember when I saw it, thinking, "that's uhm... weird. Oh well, it works..."

Have you tried just using "normal" framerates elsewhere, and then put videorate right before your v42lsink? In theory, it would automatically rate adapt according to the caps of the sink?

[nils]

I tried various thinks, I didn't log my attempts, I only kept the working ones. I am sure there are more ways to streamline that.

In the mean I discovered that my stream is good, but the point-to-point clients are useless. Skype adds so much compression and quality degeneration into the signal that you can't do screencasts/live-tutorials. it also cropped my image to 4:3. Not streched, it actually cuts! So I can't stream my whole desktop.

I guess for live lessons over the internet or help desks, paid support contracts etc. that is not the right approach. Showing a little webcam overlay on both ends and the rests is remote desktop control over X or whatever there is.

Nils