Neither one nor Many

 
September 1 2013

Live video streaming

I was interested in specifically the generation of H264 videos directly from code (without audio). This was something I used to do by generating separate .BMP files that I then convert with mencoder. This costs quite a lot of hdd space and was performance-wise not optimal. Of course I already figured that out, but figuring out how to encode directly to video was something on my To Do list.

Believe it or not, I actually had to search a while for libraries and SDK's, before concluding that ffmpeg was the best option. I've had a bad experience with ffmpeg 5 years ago, so I was probably ignoring ffmpeg in the google search results. You can link to ffmpeg's libraries and use them in your own program, they also provide code samples. This made it all pretty easy to setup. So when I completed generating H264 videos from code I wondered how difficult it would be to directly stream video with ffmpeg.

One cool application would be to use it in games, f.i. make a small livestream for actual games being played on the server. img1

Ffmpeg supports a few protocols, I chose an RTMP server for my setup, which is Flash only. If that works it'll probably work for other protocols as well. The best alternative for HTML5 is probably Apple's HLS Streaming, which actually is pretty awesome, as a side-note from source:

If multiple qualities of a stream are available, the player will continuously monitor the current bandwidth and pick the next fragment from the highest quality it can load. This is called Adaptive Streaming:

For the videoplayer I chose JWplayer version 6.

Setting up an rtmp server with Jwplayer

The servers I tested: rtmplite on linux, crtmpserver on linux and Red5 Media Server on windows. The linux server I used was a Suse (SLED11) micro EC2 instance on Amazon AWS. But these should all work on linux and windows.

RTMP is flash-only, traffics over port 1935.

Some notes on using rtmplite with JWplayer

Eventually I chose not to use rtmplite, but crtmpserver instead, as you have more output there as to what's going on in the server. rtmplite output is less useful to me, by default it outputs nothing, with verbose you get messages at packet level. img1 rtmplite used quite a bit more of my CPU compared to crtmpserver (not very accurate due to EC2, but crtmpserver at 0.3% CPU versus > 10% CPU for rtmplite) and it does not understand aspect ratio other than 4:3, so 16:9 will be "squished" into 4:3. It's not my intent to be negative about rtmplite, if you're a python guy It's probably very easy to tweak and I must say it works out-of-the-box with a simple python rtmp.py, ready for video at rtmp://servername/app/video.

With JWplayer you can view the stream with:

jwplayer("myElement").setup({
    file:       "rtmp://localhost/appflv:video",
    autostart:  true,
    width:      480,
    height:     320
});

If you attempt to use that config, please note that file URL! It took me an hour to get that correct, in all JWPlayer samples you see rtmp://server/app/flv:video.flv or rtmp://server/app/mp4:video.mp4. The prefix flv: or mp4: being just for JWplayer so that it knows what encoding to expect. As the .mp4 or .flv extension in URL's is not mandatory. Eventually JWplayer will strip away flv:. An rtmp url like rtmp://server/app/video consists of an application called app and a video called video. Stripping away the prefix from rtmp://cppse.nl/app/flv:video will provide app/ for the application which looks like app but isn't quite the same. img1

So... crtmpserver then!

After compiling you fire it up with default settings with: ./crtmpserver/crtmpserver crtmpserver/crtmpserver.lua. You should see something similar to the following output.

ip-10-224-83-43:/usr/local/src/crtmpserver/builders/cmake # ./crtmpserver/crtmpserver ./crtmpserver/crtmpserver.lua
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:203 Initialize I/O handlers manager: epoll
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:206 Configure modules
/usr/local/src/crtmpserver/sources/thelib/src/configuration/module.cpp:84 Module /usr/local/src/crtmpserver/builders/cmake/applications/appselector/libappselector.so loaded
/usr/local/src/crtmpserver/sources/thelib/src/configuration/module.cpp:84 Module /usr/local/src/crtmpserver/builders/cmake/applications/flvplayback/libflvplayback.so loaded
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:212 Plug in the default protocol factory
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:219 Configure factories
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:225 Configure acceptors
/usr/local/src/crtmpserver/sources/thelib/src/netio/epoll/iohandlermanager.cpp:100 Handlers count changed: 0->1 IOHT_ACCEPTOR
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:231 Configure instances
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:237 Start I/O handlers manager: epoll
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:240 Configure applications
/usr/local/src/crtmpserver/sources/thelib/src/configuration/module.cpp:177 Application appselector instantiated
/usr/local/src/crtmpserver/sources/thelib/src/configuration/module.cpp:177 Application flvplayback instantiated
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:246 Install the quit signal
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:257
+-----------------------------------------------------------------------------+
|                                                                     Services|
+---+---------------+-----+-------------------------+-------------------------+
| c |      ip       | port|   protocol stack name   |     application name    |
+---+---------------+-----+-------------------------+-------------------------+
|tcp|        0.0.0.0| 1935|              inboundRtmp|              appselector|
+---+---------------+-----+-------------------------+-------------------------+
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:258 GO! GO! GO! (19664)

With the default configuration anyone can stream a video to the RTMP server, as inbound and outbound are over port 1935. I tried separating that with the configs, but I couldn't find any configuration setting to do this, so I hacked something in the source to only allow inbound RTMP for 127.0.0.1 (see patch.txt). With crtmpserver you don't have the URL problem in JWPlayer.

jwplayer("myElement").setup({
    file:       "rtmp://server/flvplayback/flv:video",
    autostart:  true,
    width:      480,
    height:     320
});

Sending video to the server with "sendstream"

The proof-of-concept tool I set out to make is now a useful test tool you can use to stream a test video with optional audio at a given resolution to a given streaming server. I have compiled it for windows and linux 32 bit (opensuse 12) and also have that in a working chroot for download (~28 mb zipped).

usage: sendstream.exe < RTMP URL > [ < width > < height > < sound > < max_fps >]
 params: width default = 320, height default = 240 pixels, sound default = 1, max_fps default = 25.
 i.e: sendstream.exe "rtmp://localhost/app/video" 800 600

Example output for sendstream.exe rtmp://cppse.nl/flvplayback/video 1920 1080 1:

[libmp3lame @ 058a6320] Channel layout not specified
Output #0, flv, to 'rtmp://cppse.nl/flvplayback/video':
    Stream #0:0: Video: flv1, yuv420p, 1920x1080, q=2-31, 100 kb/s, 90k tbn, 20
tbc
    Stream #0:1: Audio: mp3, 44100 Hz, 2 channels, s16p, 24 kb/s
HandShake: client signature does not match!
writing frame 0 at 0.044793... +V
writing frame 1 at 0.112187... +A+A+A[flv @ 058a2b60] Encoder did not produce pr
oper pts, making some up.
+A+V
writing frame 2 at 0.324518... +A+A+A+V
writing frame 3 at 0.813168... +A+A+A+A+A+A+A+A+A+V
writing frame 4 at 1.03129... +A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+V
writing frame 5 at 1.16484... +A+A+A+A+A+A+A+A+A+V
writing frame 6 at 1.23609... +A+A+A+A+A+V
writing frame 7 at 1.28777... +A+A+A+V
writing frame 8 at 1.33955... +A+A+V
writing frame 9 at 1.38883... +A+A+V
writing frame 10 at 1.43901... +A+V
writing frame 11 at 1.48954... +A+A+V
writing frame 12 at 1.53965... +A+A+V
writing frame 13 at 1.58858... +A+A+V
writing frame 14 at 1.6456... +A+A+V
writing frame 15 at 1.69497... +A+A+V
writing frame 16 at 1.74456... +A+A+V

Audio and video are combined with interweaving and if rendering of video frames is relatively slow compared to audio it may require more audio frames in between and vica versa. In the output +A is a written audio and +V a written video frame.

I use SFML for generating the graphics and for audio something similar to the example code by ffmpeg. The interweaving part was the most difficult but important thing to get right. This is the easiest version of the code I could come up with that works properly. img1 Something I didn't know in the beginning was that you don't need to keep your framerate of sending frames to the server consistent at a given frames per second-rate.

sfml_init(&pixels, &pixelSize);

auto timer = TimerFactory::factory(TimerFactory::Type::WindowsHRTimerImpl);
timer->start();

frame->pts = timer->end();

int64_t currentframe = 0;
while (true) {

    sfml_generate_frame();

    std::cout << "writing frame " << currentframe++ << " at " << (timer->end() / 1000.0) << "... ";

    // Make sure to write enough audio frames until the audio timer lines up
    while (true) {
        double audio_pts = (audio_st) ? (double)audio_st->pts.val * audio_st->time_base.num / audio_st->time_base.den : 0.0;
        double video_pts = (video_st) ? (double)video_st->pts.val * video_st->time_base.num / video_st->time_base.den : 0.0;

        if (audio_pts >= video_pts)
            break;

        std::cout << "+A";
        write_audio_frame(oc, audio_st);
    }

    // Write video frame
    write_video_frame(oc, video_st);
    std::cout << "+V" << std::endl;

    // Set elapsed time as frame time
    frame->pts = timer->end();
}

One final side-note if you are going to use sendstream on linux with Xvfb (X virtual framebuffer). On SUSE it was a pain to get it working, I ended up installing it on an openSuse VMWare VM where it did work and then made a minimal chroot for it, one that includes Xvfb and sendstream and a run.sh script (you might want to edit that). Not all screen resolutions you can think off that work fine with RTMP server will work, i.e.: I wanted a 980 x 100 resolution video, that didn't work for Xvfb, it does work if you create a larger screen with a more common resolution like 1280 x 768. Also 980 x 100 didn't work for SFML, 1000 x 100 did, probably the OpenGL underneath complaining or something. AFAIK these problems do not exist without Xvfb.

The result

Sendstream runs on my webserver capped at 2 FPS and without audio, so that it doesn't hurt the CPU too much.

Download: sendstream-1.0.zip for windows. 11.2MB

Download: sendstream-1.0-chroot.zip for linux (chroot, use: "chroot /path/to/sendstream /run.sh"). 27.2MB

a930805b8867232755da195d34587476 *sendstream-1.0-chroot.zip
6ed8cf5102dbe10c5f4722c80d38cbbd *sendstream-1.0.zip

 

C++ Comments (4)

pardeep

website: www.softsolutiontechnologies.com @

2014-04-22 13:02:05

dear sir

my query is that how can i use above code on my web form with jw player , when i use this code" file: "rtmp://server/app/flv:video.flv"" then it show 'error loading stream could not connect to server'
i have already download all dll and one exe file which link you have on your site in last of paragraph
and i have put them in bin folder of website so that exe can execute itself

so pls help me how can implement it

Mark Lee

2014-09-04 11:38:01

Dear Ray Burgemeestre,

Your blog's contents help me and thank you.
Can I get the source code of sendstream.exe or any related document?
I tried many time for understanding timestamp in H.264 but I could't yet solve the problem.
Thank you in advance.

rayburgemeestre

2014-09-09 00:37:20

Hi Mark,

Sorry for my delayed reply, I already replied to you by e-mail, but didn't get the chance to dig into it until now.

As you also e-mailed me when my blog wasn't updating properly, I'll e-mail you this as well...

I had to search a bit for the sendstream code actually.. Unfortunately the code I have is not a simple file I can send you, as it's entangled with a lot of closed source stuff as well. But after a little searching I found the ffmpeg example I studied at the time called "muxing.c", you can find it here:

http://git.videolan.org/?p=ffmpeg.git;a=tree;f=doc/examples;h=c7b9e34f78cffcb72a490d1947f2abdc3dd1b171;hb=HEAD
(you can download the raw file there as well)

However, that example code has changed a little since I studied it =). Since this commit by Stefano Sabatini on 12th of January 2014 (a few months after I wrote this blog post):

http://git.videolan.org/?p=ffmpeg.git;a=blobdiff;f=doc/examples/muxing.c;h=3ec2eb66d0b105f6375ced9ced391eb6db9fe21d;hp=f771c2f9df48c2ca6845d8e039641c26b9d26fad;hb=HEAD;hpb=d497141b859488225a5869cee66ca91675cd07d9

The code on my blog was my own attempt at simplifying it for myself. And I can give you a diff on that example's main() and my own code: http://cdn.cppse.nl/68-Report.html I cherry picked it a bit

In my implementation I simply have an endless loop where I write video every iteration, and simply--before writing that frame--see how many audio frames I need to write according to the pts value(s). I can imagine the audio writing more often then video in case it uses more bitrate than the video stream uses, or vice versa if the audio bitrate is lower.

You can get source properly with git here and compile it etc: https://www.ffmpeg.org/download.html

Benchmarklib I use in my code is my own, http://blog.cppse.nl/benchmarklib

If you need more details please let me know..

Also another tip, if you're familiar with IRC, there is an #ffmpeg channel on freenode (http://webchat.freenode.net/) if you have more questions. There are ~270 people there online, so you probably get faster feedback as well :D

Cheers,
Ray

Mark Lee

2014-09-11 02:02:29

Thank you for your kind reply.
I was on a vacation trip and checked on today :)
Your comment will help my problem solving.
Thank you again.

Sincerely,
Mark


Leave a Reply


Author:
Ray Burgemeestre
february 23th, 1984

Topics:
C++, Lisp, Music, Philosophy

Other interests:
Music, Art, Zen, Webdevelopment