September 1 2013

Video streaming from code with SFML and ffmpeg

Live video streaming

I was interested in specifically the generation of H264 videos directly from code (without audio). This was something I used to do by generating separate .BMP files that I then convert with mencoder. This costs quite a lot of hdd space and was performance-wise not optimal. Of course I already figured that out, but figuring out how to encode directly to video was something on my To Do list.

Believe it or not, I actually had to search a while for libraries and SDK's, before concluding that ffmpeg was the best option. I've had a bad experience with ffmpeg 5 years ago, so I was probably ignoring ffmpeg in the google search results. You can link to ffmpeg's libraries and use them in your own program, they also provide code samples. This made it all pretty easy to setup. So when I completed generating H264 videos from code I wondered how difficult it would be to directly stream video with ffmpeg.

One cool application would be to use it in games, f.i. make a small livestream for actual games being played on the server.

Ffmpeg supports a few protocols, I chose an RTMP server for my setup, which is Flash only. If that works it'll probably work for other protocols as well. The best alternative for HTML5 is probably Apple's HLS Streaming, which actually is pretty awesome, as a side-note from source:

If multiple qualities of a stream are available, the player will continuously monitor the current bandwidth and pick the next fragment from the highest quality it can load. This is called Adaptive Streaming:

For the videoplayer I chose JWplayer version 6.

Setting up an rtmp server with Jwplayer

The servers I tested: rtmplite on linux, crtmpserver on linux and Red5 Media Server on windows. The linux server I used was a Suse (SLED11) micro EC2 instance on Amazon AWS. But these should all work on linux and windows.

RTMP is flash-only, traffics over port 1935.

Some notes on using rtmplite with JWplayer

Eventually I chose not to use rtmplite, but crtmpserver instead, as you have more output there as to what's going on in the server. rtmplite output is less useful to me, by default it outputs nothing, with verbose you get messages at packet level. rtmplite used quite a bit more of my CPU compared to crtmpserver (not very accurate due to EC2, but crtmpserver at 0.3% CPU versus > 10% CPU for rtmplite) and it does not understand aspect ratio other than 4:3, so 16:9 will be "squished" into 4:3. It's not my intent to be negative about rtmplite, if you're a python guy It's probably very easy to tweak and I must say it works out-of-the-box with a simple python rtmp.py, ready for video at rtmp://servername/app/video.

With JWplayer you can view the stream with:

jwplayer("myElement").setup({
    file:       "rtmp://localhost/appflv:video",
    autostart:  true,
    width:      480,
    height:     320
});

If you attempt to use that config, please note that file URL! It took me an hour to get that correct, in all JWPlayer samples you see rtmp://server/app/flv:video.flv or rtmp://server/app/mp4:video.mp4. The prefix flv: or mp4: being just for JWplayer so that it knows what encoding to expect. As the .mp4 or .flv extension in URL's is not mandatory. Eventually JWplayer will strip away flv:. An rtmp url like rtmp://server/app/video consists of an application called app and a video called video. Stripping away the prefix from rtmp://cppse.nl/app/flv:video will provide app/ for the application which looks like app but isn't quite the same.

So... crtmpserver then!

After compiling you fire it up with default settings with: ./crtmpserver/crtmpserver crtmpserver/crtmpserver.lua. You should see something similar to the following output.

ip-10-224-83-43:/usr/local/src/crtmpserver/builders/cmake # ./crtmpserver/crtmpserver ./crtmpserver/crtmpserver.lua
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:203 Initialize I/O handlers manager: epoll
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:206 Configure modules
/usr/local/src/crtmpserver/sources/thelib/src/configuration/module.cpp:84 Module /usr/local/src/crtmpserver/builders/cmake/applications/appselector/libappselector.so loaded
/usr/local/src/crtmpserver/sources/thelib/src/configuration/module.cpp:84 Module /usr/local/src/crtmpserver/builders/cmake/applications/flvplayback/libflvplayback.so loaded
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:212 Plug in the default protocol factory
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:219 Configure factories
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:225 Configure acceptors
/usr/local/src/crtmpserver/sources/thelib/src/netio/epoll/iohandlermanager.cpp:100 Handlers count changed: 0->1 IOHT_ACCEPTOR
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:231 Configure instances
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:237 Start I/O handlers manager: epoll
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:240 Configure applications
/usr/local/src/crtmpserver/sources/thelib/src/configuration/module.cpp:177 Application appselector instantiated
/usr/local/src/crtmpserver/sources/thelib/src/configuration/module.cpp:177 Application flvplayback instantiated
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:246 Install the quit signal
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:257
+-----------------------------------------------------------------------------+
|                                                                     Services|
+---+---------------+-----+-------------------------+-------------------------+
| c |      ip       | port|   protocol stack name   |     application name    |
+---+---------------+-----+-------------------------+-------------------------+
|tcp|        0.0.0.0| 1935|              inboundRtmp|              appselector|
+---+---------------+-----+-------------------------+-------------------------+
/usr/local/src/crtmpserver/sources/crtmpserver/src/crtmpserver.cpp:258 GO! GO! GO! (19664)

With the default configuration anyone can stream a video to the RTMP server, as inbound and outbound are over port 1935. I tried separating that with the configs, but I couldn't find any configuration setting to do this, so I hacked something in the source to only allow inbound RTMP for 127.0.0.1 (see patch.txt). With crtmpserver you don't have the URL problem in JWPlayer.

jwplayer("myElement").setup({
    file:       "rtmp://server/flvplayback/flv:video",
    autostart:  true,
    width:      480,
    height:     320
});

Sending video to the server with "sendstream"

The proof-of-concept tool I set out to make is now a useful test tool you can use to stream a test video with optional audio at a given resolution to a given streaming server. I have compiled it for windows and linux 32 bit (opensuse 12) and also have that in a working chroot for download (~28 mb zipped).

usage: sendstream.exe < RTMP URL > [ < width > < height > < sound > < max_fps >]
 params: width default = 320, height default = 240 pixels, sound default = 1, max_fps default = 25.
 i.e: sendstream.exe "rtmp://localhost/app/video" 800 600

Example output for sendstream.exe rtmp://cppse.nl/flvplayback/video 1920 1080 1:

[libmp3lame @ 058a6320] Channel layout not specified
Output #0, flv, to 'rtmp://cppse.nl/flvplayback/video':
    Stream #0:0: Video: flv1, yuv420p, 1920x1080, q=2-31, 100 kb/s, 90k tbn, 20
tbc
    Stream #0:1: Audio: mp3, 44100 Hz, 2 channels, s16p, 24 kb/s
HandShake: client signature does not match!
writing frame 0 at 0.044793... +V
writing frame 1 at 0.112187... +A+A+A[flv @ 058a2b60] Encoder did not produce pr
oper pts, making some up.
+A+V
writing frame 2 at 0.324518... +A+A+A+V
writing frame 3 at 0.813168... +A+A+A+A+A+A+A+A+A+V
writing frame 4 at 1.03129... +A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+A+V
writing frame 5 at 1.16484... +A+A+A+A+A+A+A+A+A+V
writing frame 6 at 1.23609... +A+A+A+A+A+V
writing frame 7 at 1.28777... +A+A+A+V
writing frame 8 at 1.33955... +A+A+V
writing frame 9 at 1.38883... +A+A+V
writing frame 10 at 1.43901... +A+V
writing frame 11 at 1.48954... +A+A+V
writing frame 12 at 1.53965... +A+A+V
writing frame 13 at 1.58858... +A+A+V
writing frame 14 at 1.6456... +A+A+V
writing frame 15 at 1.69497... +A+A+V
writing frame 16 at 1.74456... +A+A+V

Audio and video are combined with interweaving and if rendering of video frames is relatively slow compared to audio it may require more audio frames in between and vica versa. In the output +A is a written audio and +V a written video frame.

I use SFML for generating the graphics and for audio something similar to the example code by ffmpeg. The interweaving part was the most difficult but important thing to get right. This is the easiest version of the code I could come up with that works properly. Something I didn't know in the beginning was that you don't need to keep your framerate of sending frames to the server consistent at a given frames per second-rate.

sfml_init(&pixels, &pixelSize);

auto timer = TimerFactory::factory(TimerFactory::Type::WindowsHRTimerImpl);
timer->start();

frame->pts = timer->end();

int64_t currentframe = 0;
while (true) {

    sfml_generate_frame();

    std::cout << "writing frame " << currentframe++ << " at " << (timer->end() / 1000.0) << "... ";

    // Make sure to write enough audio frames until the audio timer lines up
    while (true) {
        double audio_pts = (audio_st) ? (double)audio_st->pts.val * audio_st->time_base.num / audio_st->time_base.den : 0.0;
        double video_pts = (video_st) ? (double)video_st->pts.val * video_st->time_base.num / video_st->time_base.den : 0.0;

        if (audio_pts >= video_pts)
            break;

        std::cout << "+A";
        write_audio_frame(oc, audio_st);
    }

    // Write video frame
    write_video_frame(oc, video_st);
    std::cout << "+V" << std::endl;

    // Set elapsed time as frame time
    frame->pts = timer->end();
}

One final side-note if you are going to use sendstream on linux with Xvfb (X virtual framebuffer). On SUSE it was a pain to get it working, I ended up installing it on an openSuse VMWare VM where it did work and then made a minimal chroot for it, one that includes Xvfb and sendstream and a run.sh script (you might want to edit that). Not all screen resolutions you can think off that work fine with RTMP server will work, i.e.: I wanted a 980 x 100 resolution video, that didn't work for Xvfb, it does work if you create a larger screen with a more common resolution like 1280 x 768. Also 980 x 100 didn't work for SFML, 1000 x 100 did, probably the OpenGL underneath complaining or something. AFAIK these problems do not exist without Xvfb.