2012 video

Video at FOSDEM 2012

A year ago, during FOSDEM 2011, I walked up to the NamurLUG folks who'd been doing video coverage of FOSDEM for years, and suggested that this year, maybe they should consider using dvswitch for video coverage. While they seemed agreeable to this idea, they simply had not had the time to look at it in detail, and were therefore not using it. Since I've used dvswitch in the past, both as part of the DebConf video team and for my own concert recordings, I reasoned this a problem that could be solved by joining the video team.

However, when it came to be time to start preparing for FOSDEM 2012, we found that many of the NamurLUG people were not going to have as much time to prepare for video coverage as they had during past years, and therefore responsibility of FOSDEM video coverage fell entirely to me. This was fairly unexpected, though not too daunting.

For those of you who don't know dvswitch very well: a typical dvswitch installation for coverage of a talk requires:

Two cameras; one to capture the speaker, one to capture audience members asking questions.
A twinpact 100, used to convert a VGA output into a DV stream that we can load into dvswitch
Three laptops: one for the audience camera, one for the twinpact, and one for the speaker camera and the dvswitch installation
A GigE network between the three laptops, with an uplink to do streaming
A large amount of diskspace to store the recordings on. Since dvswitch records in raw DV, you require approximately 13GB per hour of recorded video.
Some powerful systems to do transcoding from DV to a streaming format such as Ogg Theora or WebM
A fairly extensive audio setup: one wireless headset microphone for the speaker, one or two wireless handheld microphones for people in the audience, and a number of microphones at select positions in the room to pick up ambient noise (which for obvious reasons you don't want to overdo on, but which if done well makes the audio on your recording sound much more natural). This also requires a mixing console.
And last but not least a (network of) streaming server(s) to do the streaming to people watching from home.

This is a rather large amount of work, and therefore it is not too unexpected that for DebConf, the DebConf video team takes a week to properly set up two rooms so that they can be used for recording. However, for FOSDEM we only have less than a day on-site to set up things, and we were going to cover five rooms. So, I tried to cut corners as much as possible:

We were going to work with one camera per room, rather than two. Not only would this reduce the number of required volunteers (of which I wasn't sure yet at that point how I'd find them), it would also reduce the amount of preparation needing to be done, and it would reduce the number of things that could go wrong.
We were going to have the network team do our network, rather than try to do it ourselves. The result was that we were not going to have to lay out cables, cut them, crimp connectors on them, test them, etc, and would only need to worry about using the network, instead of building it.
Since I'd read Thomas' announcement about an ''offer I cannot refuse'', I contacted him to see about that offer and get it all set up for FOSDEM. This way, I wouldn't have to worry about streaming servers etc, instead just about getting a stream on the network, into a flumotion server running somewhere; and after that, it would no longer be my problem.
I decided to simplify the audio setup somewhat, and not use ambient mics. These are nice, but not essential, so we could do without them, and that would be (at least) two microphones per room that the team wouldn't have to rent, place, wire up, and tune.

So in the end, I would need to take care of ten firewire-capable laptops, five cameras, five mixing consoles, several microphones, a number of transcoding machines, and shitloads of cabling (power, firewire, and audio). And about a week or two before the conference, as the massiveness of what we were about to do started to sink in, I was starting to have bad dreams of what would happen, wondering what I'd gotten myself into this time.

Now, one day after FOSDEM, I have to say it all turned out okay, but there are some things where there's still room for improvement. So that I don't forget, I thought I'd make a list of things that went well, and one of things that didn't go well. And since that might be interesting for other people, I thought I'd do it here, rather than in a private file.

First, things that did not go so well:

Set-up for some rooms took more time than we had, and as such some rooms did not get properly streamed or recorded for their first few talks. This was due to the fact that there was some confused communication between some members of the team, which meant that we lost a day that we'd planned for preparation, and as such we couldn't test as much as we'd hoped. We need to improve on that next year. Ideally, we should run the full set-up somewhere, with all cameras and all laptops running, and making sure that everything is ready to be set up and that we have every cable we need, before dividing everything over a number of boxes (one per room) and bringing it to the conference.
While some of our cameras were semi-professional Panasonic cameras that had a balanced XLR audio input, others were much lower-level camcorder-style cameras that did not have such a thing. When a camera has a good audio input, it's fairly easy to set up a link from the mixing console to the camera and get the audio into the stream that way. When a camera does not have such a thing, the set-up gets much more complex. Since we had so much to worry about, we did not notice that in one room, on saturday, something went wrong with audio. To combat this, we should improve on our audio set-up for next year, and also verify the streams every once in a while (I did the latter on sunday, so everything should be fine there).
Inherent to an ad-hoc network, there were some network-related issues. For instance, on sunday evening, during the last session in that room, Ben informed me that the link to room H.1301 had gone down. As we were investigating, we easily found the source of the problem:

The cable had been put under a door, and had been 'protected' with loads and loads of duct tape, but apparently that wasn't enough. This wasn't something we wanted to fix anymore, at that point; instead, we just let the cable be and focused on other things.
Storage. None of the systems used for the recordings were my own; instead, some were owned by IRILL and some were rented. If I want to do anything with my recordings, that means I have to copy the data off. The system I chose was to bring a USB3 hard disk and copy everything over; but for saturday, this took an hour and thirty minutes. On sunday, my USB3 disk was fairly full, so I had to revert to a USB2 one, which increased the time by much. In the end, we had to abort copying files, and we'll now have to be a bit creative to get them.
Volunteer management. I originally set up a wiki page to allow people to sign up for talks that they wanted to help out with. Unfortunately, that didn't work so well as it did for DebConf; there were several talks that did not have enough people, resulting in the volunteers from the previous talk remaining on post and finishing their talk. Also, there were some people that signed up without contacting me, or without telling me what name they used on the wiki page, which made it somewhat harder to know who to talk to. I lost count of who had signed up to help out. On Sunday, I decided to set up an IRC channel instead, through which volunteers could communicate more directly and just ask for people to take over, instead of registering for a talk first and then hoping nobody would forget. Also, since I'm terribly bad with faces and video volunteers did not have anything that made them stand out from the rest, I couldn't just walk around until I would see someone involved with video work and ask them to take over in a room. Having a separate 'video team' t-shirt could help there.

After the bad news, here's the list of things that did go well:

Streaming. Even though choosing flumotion rather than icecast (as we'd done for streaming in the past) involved extra work for me before the conference (I had to write some code so that I could stream from dvswitch into flumotion, which wasn't possible out of the box), the guys at Flumotion decided to send over one of their support engineers. As such, I simply did not have to worry about streaming servers crashing or failing to do what they were supposed to do. While I have no reason to think any of the Flumotion servers were having issues, even if they were I wouldn't have known about it, since Francesc took care of it all and never ever required my involvement about something on the flumotion side. This was extremely valuable.
Training right before the welcome talk. Explaining to people in a few words how dvswitch works, and then immediately following that up with a live demonstration, isn't a bad way for them to understand.
Despite all the problems we did have, once everything had been set up and all the gear was more or less manned, actually recording and streaming did go fairly well. The feedback I received so far was mostly positive, which makes me very happy.

In the end, while there certainly is still room for improvement and things did not go as smoothly as I'd hoped, they have gone much better than my worst fears. There's still some work to do, which I'll be doing with the NamurLUG people this week, but all in all, it's gone pretty good.

As to the usage of the streams, I asked Flumotion for some statistics. Since I was fairly late in asking, they could only provide me with statistics about sunday:

Number of clients connecting:
Traffic: