Tue, 07 Feb 2012
A year ago, during FOSDEM 2011, I walked up to the NamurLUG folks who'd been doing video
coverage of FOSDEM for years, and suggested that this year, maybe they
should consider using dvswitch for video
coverage. While they seemed agreeable to this idea, they simply had not
had the time to look at it in detail, and were therefore not using it.
Since I've used dvswitch in the past, both as part of the DebConf video team and for my own
concert recordings, I reasoned this a problem that could be solved by
joining the video team.
However, when it came to be time to start preparing for FOSDEM 2012,
we found that many of the NamurLUG people were not going to have as much
time to prepare for video coverage as they had during past years, and
therefore responsibility of FOSDEM video coverage fell entirely to
me. This was fairly unexpected, though not too daunting.
For those of you who don't know dvswitch very well: a typical
dvswitch installation for coverage of a talk requires:
- Two cameras; one to capture the speaker, one to capture audience
members asking questions.
- A twinpact
100, used to convert a VGA output into a DV stream that we can load
- Three laptops: one for the audience camera, one for the twinpact,
and one for the speaker camera and the dvswitch installation
- A GigE network between the three laptops, with an uplink to do
- A large amount of diskspace to store the recordings on. Since
dvswitch records in raw DV, you require approximately 13GB per hour of
- Some powerful systems to do transcoding from DV to a streaming
format such as Ogg Theora or WebM
- A fairly extensive audio setup: one wireless headset microphone for
the speaker, one or two wireless handheld microphones for people in the
audience, and a number of microphones at select positions in the room to
pick up ambient noise (which for obvious reasons you don't want to
overdo on, but which if done well makes the audio on your recording
sound much more natural). This also requires a mixing console.
- And last but not least a (network of) streaming server(s) to do the
streaming to people watching from home.
This is a rather large amount of work, and therefore it is not too
unexpected that for DebConf, the DebConf video team takes a week to
properly set up two rooms so that they can be used for recording.
However, for FOSDEM we only have less than a day on-site to set
up things, and we were going to cover five rooms. So, I tried
to cut corners as much as possible:
- We were going to work with one camera per room, rather than two. Not
only would this reduce the number of required volunteers (of which I
wasn't sure yet at that point how I'd find them), it would also reduce
the amount of preparation needing to be done, and it would reduce the
number of things that could go wrong.
- We were going to have the network team do our network, rather than
try to do it ourselves. The result was that we were not going to have to
lay out cables, cut them, crimp connectors on them, test them, etc, and
would only need to worry about using the network, instead of building
- Since I'd read Thomas' announcement
about an ''offer I cannot refuse'', I contacted him to see
about that offer and get it all set up for FOSDEM. This way, I wouldn't
have to worry about streaming servers etc, instead just about getting a
stream on the network, into a flumotion server running somewhere; and
after that, it would no longer be my problem.
- I decided to simplify the audio setup somewhat, and not use ambient
mics. These are nice, but not essential, so we could do without
them, and that would be (at least) two microphones per room that the
team wouldn't have to rent, place, wire up, and tune.
So in the end, I would need to take care of ten firewire-capable
laptops, five cameras, five mixing consoles, several microphones, a
number of transcoding machines, and shitloads of cabling (power,
firewire, and audio). And about a week or two before the conference, as
the massiveness of what we were about to do started to sink in, I was
starting to have bad dreams of what would happen, wondering what I'd
gotten myself into this time.
Now, one day after FOSDEM, I have to say it all turned out okay, but
there are some things where there's still room for improvement. So that
I don't forget, I thought I'd make a list of things that went well, and
one of things that didn't go well. And since that might be interesting
for other people, I thought I'd do it here, rather than in a private
First, things that did not go so well:
- Set-up for some rooms took more time than we had, and as such
some rooms did not get properly streamed or recorded for their first few
talks. This was due to the fact that there was some confused
communication between some members of the team, which meant that we lost
a day that we'd planned for preparation, and as such we couldn't test
as much as we'd hoped. We need to improve on that next year. Ideally, we
should run the full set-up somewhere, with all cameras and all laptops
running, and making sure that everything is ready to be set up and that
we have every cable we need, before dividing everything over a number of
boxes (one per room) and bringing it to the conference.
- While some of our cameras were semi-professional Panasonic cameras
that had a balanced XLR audio input, others were much lower-level
camcorder-style cameras that did not have such a thing. When a camera
has a good audio input, it's fairly easy to set up a link from the
mixing console to the camera and get the audio into the stream that way.
When a camera does not have such a thing, the set-up gets much more
complex. Since we had so much to worry about, we did not notice that in
one room, on saturday, something went wrong with audio. To combat this,
we should improve on our audio set-up for next year, and also verify the
streams every once in a while (I did the latter on sunday, so everything
should be fine there).
- Inherent to an ad-hoc network, there were some network-related
issues. For instance, on sunday evening, during the last session in that
room, Ben informed me
that the link to room H.1301 had gone down. As we were investigating, we
easily found the source of the problem:
The cable had been put under a door, and had been 'protected' with loads
and loads of duct tape, but apparently that wasn't enough. This wasn't
something we wanted to fix anymore, at that point; instead, we just let
the cable be and focused on other things.
- Storage. None of the systems used for the recordings were my own;
instead, some were owned by IRILL and some were rented. If I want to
do anything with my recordings, that means I have to copy the data off.
The system I chose was to bring a USB3 hard disk and copy everything
over; but for saturday, this took an hour and thirty minutes. On sunday,
my USB3 disk was fairly full, so I had to revert to a USB2 one, which
increased the time by much. In the end, we had to abort copying
files, and we'll now have to be a bit creative to get them.
- Volunteer management. I originally set up a wiki page to allow
people to sign up for talks that they wanted to help out with.
Unfortunately, that didn't work so well as it did for DebConf; there
were several talks that did not have enough people, resulting in the
volunteers from the previous talk remaining on post and finishing their
talk. Also, there were some people that signed up without contacting me,
or without telling me what name they used on the wiki page, which made
it somewhat harder to know who to talk to. I lost count of who had
signed up to help out. On Sunday, I decided to set up an IRC channel
instead, through which volunteers could communicate more directly and
just ask for people to take over, instead of registering for a
talk first and then hoping nobody would forget. Also, since I'm terribly
bad with faces and video volunteers did not have anything that made them
stand out from the rest, I couldn't just walk around until I would see
someone involved with video work and ask them to take over in a room.
Having a separate 'video team' t-shirt could help there.
After the bad news, here's the list of things that did go well:
- Streaming. Even though choosing flumotion rather than icecast
(as we'd done for streaming in the past) involved extra work for me
before the conference (I had to write some code so that I could stream
from dvswitch into flumotion, which wasn't possible out of the box), the
guys at Flumotion decided to send over one of their support engineers.
As such, I simply did not have to worry about streaming servers crashing
or failing to do what they were supposed to do. While I have no reason
to think any of the Flumotion servers were having issues, even if they
were I wouldn't have known about it, since Francesc took care of it all
and never ever required my involvement about something on the flumotion
side. This was extremely valuable.
- Training right before the welcome talk. Explaining to people in a
few words how dvswitch works, and then immediately following that up
with a live demonstration, isn't a bad way for them to understand.
- Despite all the problems we did have, once everything had been set
up and all the gear was more or less manned, actually recording and
streaming did go fairly well. The feedback I received so far was mostly
positive, which makes me very happy.
In the end, while there certainly is still room for improvement and
things did not go as smoothly as I'd hoped, they have gone much better
than my worst fears. There's still some work to do, which I'll be doing
with the NamurLUG people this week, but all in all, it's gone pretty
As to the usage of the streams, I asked Flumotion for some
statistics. Since I was fairly late in asking, they could only provide
me with statistics about sunday:
- Number of clients connecting: