[Music]
i'm doing this not because i'm
professionally engaged with 360 video
just just an amateur i like taking 360
photos on vacation
and during covet i was thinking about
how to how the experience of
remote video
conferencing could be improved
and that's how i got involved
so um
stream is running right now so if you
want to make sure you're not on camera
consider sitting on that side
yeah so um
live streaming 360 video
is normally a pretty much a no-brainer
you just
you know
if your camera supports it you just
click on it and say i want to stream to
youtube i want to stream to facebook
and
most of the time it works
but i think it's important to do it with
your own infrastructure for the for the
usual reasons
no one is tracking who's watching what
and
also
by using free software you're making
sure that there are no backdoors no
surveillance
and
[Music]
by using
if everyone hosts his own stream it's
decentralized and uh if if one stream
fails then it's
not not everything is down as we've seen
with the large outages of aws and so on
well
um
using free software is unfortunately not
possible end-to-end because the cameras
themselves of course are running some
proprietary firmware at least all those
that i'm aware of
and then some of them can stream
directly and some of them rely on on an
app on your phone or
on
some software running on a pc which is
often also proprietary
but
from that step on in the chain we can
we can put our own stuff in so um i will
cover them during this talk
and the end device is often
also proprietary so if you have like a
vr headset or a phone
not everything running on your phone may
be open source
maybe maybe this maybe it isn't
but um we see how much we can
how much we can do
okay for those of you who are not
familiar with 360 cameras
basically the idea is that you have a
camera with
multiple lenses
so it can see everything around around
it at once
so
on the left you have uh
a diy early camera basically consists of
six gopro cameras
formed shaped in a cube so every every
lens is looking a different direction
and on the right we have
also
somewhat dated samsung cameras with two
fisheye lenses
and um
the one lens on the front sees the one
half of the image and the other one sees
the other half
and actually they're a bit more than 180
degrees so it's
you may think that's not even possible
but there are lenses that can
i've seen up to 220 degrees so basically
the camera is kind of looking behind
itself a bit
and this is important because if you
only have two lenses you need some
overlap between the two lenses
there will be a stitching
involved in this overlap area and it
ideally it's completely invisible
but
because if you look at the um
at the samsung camera on the right
um obviously there are two sensors one
looking in this way and one looking the
other way and there's some space in
between yeah so this is uh
this causes a parallax error
and this has to be hidden
by
clever stitching
and also it means the larger the
parallax
the further away you have to be from the
camera in the void in order to avoid
stitching errors
so
for this type of camera the stitching
arrow is pretty small
and
here for example we have some
on the left there's a camera google jump
with 16 gopros
and
it has a pretty large
parallax error but it does has a
different approach so it uses
some
advanced
computer vision to create a 3d
video out of it so um
may not be such a huge problem with that
type of camera
and on the right you have a another pro
camera with eight lenses
and
it can also generate a 3d video out of
out of the individual lenses
um
today i'm using for the for the live
stream i'm using the camera on the left
and yes you can see it has eight lenses
uh
four times two looking in all four
directions and they are wide angle
lenses so they can look up and down also
obviously the camera
cannot see what's directly above or
below it just a bit further away
that's a
common feature
of these cameras for example if you
if you have the camera on the left you
can put it on the monopod and then if
you're wondering how can i illuminate my
scene uh
if you don't want the the camera light
to be in the in the picture everything
is in the picture right so
but by putting the um maybe an led strip
on your on your monopod you can
illuminate the scene and it will be
invisible
yeah
okay so
um these cameras deliver
um the lenses are usually fisheye lenses
and um we have to turn it into one
coherent picture
and you're probably familiar with this
view of the globe that's called an
equirectangular projection
and
basically it projects a sphere onto a
plane
and
some people believe the earth is flat
but i can assure you
it's not
and
there this projection has a an error
um
oh yeah first something else
so you can also do a 360 camera
broadcast and just transmit a normal
video by
choosing a small part of the image and
re it's called reframing
for example you can have a camera on
your table and you can
some cameras offer
tracking features so if there's a person
in the image you can select it in your
app and then it will track the person
across a room
and it's like a virtual camera movement
but it's all done digitally and the
result is just a normal image so this is
not what this talk is about
um
yeah we want the viewer to be able to do
the reframing
himself or herself
so um coming back to this projection
here you can see uh the distortion
the closer you get to the
zenith and to the nettier
um the larger the distortion is and it's
basically infinite at this point because
the poles would be like only a point
and in this projection they go across
the whole image
and then it's the task of the the video
viewer
you need a special video viewer for 360
video to
um to undo this distortion yeah
i will show you later um
a little live demo if you haven't seen
there was also the url on the first
slide if you want to check it on your
phone maybe i can go back to it real
quick
for what
yeah you need some certain amount of
bandwidth so if you all check it it will
remember it might break actually so it's
t one p dot d e
m c h 20 20.
okay
okay so um
let's talk about cameras some more about
uh cameras that you can actually use for
live streaming
i mean there are
there are quite a few cameras on the
market these days
but
many of them have some severe
restrictions
so
for example all insta 360 cameras
that are non-pro cameras
they
cannot really stream in 4k and 4k is
considered the minimum i would say
i will talk a bit later about why y4k is
the minimum requirement
also the gopro max which is quite
popular can only do full hd
and
[Music]
the samsung gear 360 that i showed you
which is quite a few years old it can do
2k basically
yeah so
there are like
three that i'm aware of of four if you
count the voods and the vuvuz plus that
can that cost less than one thousand
euros and can do it
uh the voos can also do 3d
and then if you go to 2 000 euros you
can
those two cameras the lap panel and the
q cam they can also do 8k streaming
which obviously requires a lot of
bandwidth
um
yeah so
some some practical advice if you have a
360 camera don't don't hold it directly
in your hand because your hand will be
so close to the camera it will look
really creepy oh something i have to
check i think something went wrong with
the stream
it's okay because it was beeping
okay
so don't hold it directly in your hand
ideally
you put it on a tripod because if people
watch your
video on a
vr headset
they might get sick if if it's shaking
too much if you want to hold it in your
hand use a gimbal
and uh
make sure you observe the minimum
distance
if your
this camera also has a quite large
minimum distance
but if you make sure that
that you are
not on top of one of the stitching lines
between the lenses
then you can get away with being closer
than the minimum distance without being
cut off
yeah so um
we want a resolution of 4k
why 4k
because
um
the 4000 horizontal pixels are 3840 or
whatever they are spread out across
360 degrees
and normally we are used to maybe 90
degrees
or even less
depending on the kind of lens you have
so if you if you compare a 360 camera
with 4k to a regular camera
then you quickly see that it's um
after reframing it will be much much
lower resolution maybe it will be look
like 720p
which is
which is not so great but you know it's
acceptable
um
yeah so 4k is quite important
we use rtmp
because that's what most cameras
and
the bit rate
yeah mostly depends on what's what
bandwidth you have available if you are
somewhere on the go
here i managed to get an ethernet cable
so
with wi-fi it's it's quite quite quite
bad
so here's a screenshot of the uh
of the wu software running on a pc
it's quite nice it has a lot of features
but basically what you always have is
you can
enter your rtmp server address
sometimes you yeah most of the time you
can
choose your resolution
sometimes the bit rate
and
fps if your camera allows that and in
this case we can also toggle between 2d
and 3d
and for those of you
who are doing the token challenge
there's also a token on this slide
um okay so let's talk about rtmp
rtmp is a real-time messaging protocol
it's a proprietary protocol developed by
macromedia back in the days of flash
to communicate between the streaming
server and the flash running on the pc
and so it's proprietary so why is
everyone using it well after
adobe bought macromedia they published
an incomplete spec for public use
it uses tcp port
1935
and
which is sometimes blocked in firewalls
these days but
for the first mile delivery which is the
the from the camera to the streaming
server it's it's a protocol of choice
for sure
the last mile delivery which used to be
your browser running flash
browsers can no longer play rtmp
i mean you could use something like vlc
to view an rtmp stream
but vlc has some particular issues that
it often doesn't recognize the video as
being 360 and it
offers no manual override so you cannot
say this is a 360 video please distort
it i mean undistorted
so
that's uh really a bummer
and yeah there are also some proprietary
apps on different platforms that can do
rtmp but today i want to
talk about how to do the streaming
without proprietary plugins so
we're using hls which is http live
streaming
which was pretty much invented to fill
the gap that uh
that
the disappearance of flash left behind
and
which it's funny because uh hls was
actually invented by apple so
they killed flash so they i guess they
had to uh come up with something else
so hls
has
chops up the video in small
small smaller files and they are just
downloaded normally using http or https
so there is a problem with latency so it
can can be
between 10 20 30 seconds sometimes as
much as one minute
there's also a newer variant called low
latency hls
and
some browsers have some
restrictions
with
with hls
in terms of uh requiring https
so
to avoid it all just
get a certificate right away and don't
bother
setting it up without https
yeah
okay
so
what we want to do is we want to stream
the camera image via rtmp to an rtmp
server and the rtmp server will provide
hls
to which can then be streamed to the
browser
and
in terms of free software
there are four that i'm aware of
there's nginx
with the rtmp module
which is by far the most popular option
and it's also easy to set up
uh it's what i'm using today
it has some some drawbacks um
there hasn't been any development really
since since 2017
and so that means there's no low latency
hls support
and media server they they have a dual
license server so they have a lot of
enterprise features which is which are
only available in the in the
expensive version
something i don't really like that much
because usually
sooner or later the free version ends up
being useless with this kind of setup
yeah oven media looks looks really
interesting but i kind of ran out of
time otherwise i would have tried
setting up that one because it also
supports low latency hls
okay so
nginx super easy to set up
basically install nginx then install the
mod on with you have like a debian based
linux
that's the command to install it
then make sure you have ssl running
with certbot for example
and then
this is basically
um
your minimum
nginx configuration file yeah so you
have a block inside the file
without for the rtmp module
and then you can define an application
and the name of the application is
actually the the url
so
it will be slash app something
then you turn on live
you can also record
using nginx if you want to have
a recording of your stream which is
quite nice i think
um
yeah and then
using the hls option
you define it to
i mean you configure it to provide an
hls stream
at this point you can also
restrict the rtmp stream if you if you
want to force your users to to use the
hls stream for some reason
and you can try fiddling with the
fragment size
to reduce latency but usually
if you
get it too small then
you get stuttering or
quality suffers
okay
now we have
the rtmp stream going into the server
and we have hls from the server
available so how can we get it to play
in the browser
as i said hls is
in theory it's supported
by browsers more recent browsers
but in practice only safari supports it
directly in the video tag
but there are some
various javascript libraries available
that will
make sure it works
by
using all kinds of different
technologies under the hood
there's some multimedia extension
which is available on most browsers
and hls uses
it but for example
on ios that's
missing again so
if you just use hls js by itself it will
not work on ios
yeah and then you have a video playing
in your browser but remember we have an
acrylic rectangular image so it will
look like crap
so you also need um
to to reframe the video to the 360 video
and let the users
pick what they're looking at
usually this is done with if you're
looking on a desktop you can click on
the video and
drag with your mouse to look around in
in the sphere basically or if you have a
touch screen device you can just
drag with your finger
and if you have a device that has an imu
like a phone a positional sensor you can
also hold it in front of you and look
around in in your image
which is kind of nice and the vr goggles
also use imu so it's like standing
inside
standing where the camera is standing
and you're looking around in the image
so
kaleidoscope is one of those 360 video
viewers
it's it's quite simple and it lacking a
few features in particular um
yeah it doesn't support
vr goggles and so on
so another one i found was eg js
and that's what i ended up using
egs is a
java component group
and
one of the components is called vue 360
and it has
a sub component called
panoviewer which is a panoramic media
viewer
and it supports
tons of projections aquary rectangular
cube maps which is if you what you have
if you have like
remember the the gopro camera i showed
you in the beginning with six cameras
shaped like a cube
and
so basically you have six videos
and uh this viewer can
arrange it so
it works seamlessly
and it also supports stereoscopy
rectangular equirectangular
images which is just saying
so you have an acrylic rectangular image
and in the case of this camera you have
actually two of them
and uh
one is always the left eye the left
camera of the two and the other one is
over the right
and then this viewer supports it so if
if you watch it
it will actually be stereo
it runs on browsers with stable webgl
which means it runs on chrome firefox
safari edge even internet explorer l11
and on mobile it works on mobile safari
not just on ipad os but also on ios it
works with chrome on android
and it works with a samsung internet
browser which is what's running on
oculus go for example
yeah and it uses under the hood it uses
hls.js to play the to play the stream
um
yeah so some more
features that can run full screen you
can change orientation with the mouse
um
and one particular feature which i think
is really cool um if you
if you have a phone
maybe you remember the old google
cardboard
so basically um
cardboard is
having two lenses in front of your phone
display
and then
having the phone display
divided into two halves
and you have two lenses and um
every eye sees something else
and um so which means it's it's stereo
and uh the panel viewer supports this so
yeah that's that's quite nice so let's
let's check it out
i'm switching hopefully
to the
browser here
yeah let's see if the stream is still
running looks good
so now
i can look around here
maybe i can show you some things
you will see
for example here this this line this is
the stitching line
which is very obvious because the table
is too close to the camera here ideally
we would have a higher tripod
and the stitching line is also apparent
here somewhat in the
in the mch logo
yeah and you can also
zoom out a bit and zoom in
and it supports full screen
so
this is a really nice viewer
and if you're running it on a phone you
will get an extra icon in the top right
corner for the
cardboard mode
and if you're running it on a virtual
reality device such a oculus go it will
also work
so
pretty pretty neat i'm totally
oh
something is wrong with the stream
obviously
because it's not this hasn't happened in
a while i mean
this person is not on the on the stage
i have to check anyway
what
yeah i will restart it later
anyway um
let's go back to the presentation
so some possible possible improvements
besides the fact that the stream stopped
is to switch the rtmp server to get low
latency hls
and there's also depending on your
audience
if you have a small
audience
you can also use a webrtc instead of hls
which means a lot lower latency
less than one second mostly
but it doesn't really scale but if you
have like a small number of
clients maybe like
tens of clients
then webrtc is definitely a very
interesting option
and i saw that the oven media server
also
supports webrtc
yeah and you can also do some
interesting stuff on your rtmp server
switching between multiple cameras you
can
add an overlay using ffmpeg into the 360
image
and
yeah those are the two main improvements
i would say
okay
that's it
for the talk um
do you have any questions
okay if not we i will be around here so
you can just get in touch after
afterwards if you have any questions you
can line up in the microphones in the
middle
and i'm not sure yeah i'm and do we have
any questions on the internet
also not
okay
so if no one wants to ask a question
he will be around yes
please give him a round of applause oh
no there is a question maybe you have
one
then just come to the microphone please
close close close hello can you hear me
a while yes awesome um i was very
curious for this presentation thank you
so much for preparing this material and
presenting i have kind of like a hobby
project
um which is just like it's about like
showing outdoor adventures and so i'm
very curious to use
this tech egs and so on not just to
stream like a 360 video but to stream
multiple
multiple video streams into one and like
enabling people that are remote to to
have uh to have a multiple views into
what is happening uh so like uh i don't
know maybe an analogy is thinking think
of like
those multiple screens on a nasa launch
of the rocket like yeah yeah what is the
best way to like bring multiple streams
of video 360 or 360
and give
people the the opportunity to
participate as they want
yeah so the eg js is basically also an
api so
you would have to write a little bit of
javascript to add maybe a button or
something to switch between different
sources
but it will also be not live right so um
uh so
can't it be live or could you no i mean
it could be live but i'm saying uh
you you cannot so you will have maybe
you will have
either you will have multiple live
cameras or you will have
pre-recorded videos and you can allow
your viewers to switch between them
yeah but you would have to do a little
bit of programming for sure yes
out of the box all right no but it's not
i mean
i think it would be pretty easy actually
awesome yeah all right thank you very
welcome
next question please yes another
question um
moreover remark the engine xrtmp
module was kind of discontinued and
another group forked it and i think they
renamed it to engine xts and it has a
few more features
and also the current replacement for
rtmp is srt
are there 360 cameras supporting srt yet
or ndi for local streaming
well um ndi for sure but those are
usually pro cameras uh in the 5000 plus
euro range
srt i don't know i'm not familiar with
well
but of course you can usually also use
rtmps if you want to encrypt your stream
to the rtmp server that should be
supported by most cameras already
firefox is required
facebook is requiring it by now yeah
no
okay
so
you have one more question perfect then
come to the microphone please
i'm always a bit queer tinker
have you you have a device with multiple
cameras on it for
have you ever
done or built someone let's say a room
with the cameras around and then put the
360 software on
yeah i know what you mean yeah actually
egs supports that also
basically it's
reversing the direction you're looking
no longer looking outside from from a
sphere but you're looking onto something
yes you can do that but i haven't played
around with it but i saw that it's
supported yes
i now know let's support it so yeah
maybe so if you want to play around with
it yeah yeah
and one more question
have you seen um
front ends for the end user to do the
reframing thing so
actually allowing the viewer to select a
part of the the image or a moving target
and keep following that
um
no automatic tracking in the front end i
haven't seen that you know i mean i i've
i've seen it in the insta360 cameras
basically it's built into the into the
app
and it works pretty well
but no that's actually
would be a nice feature
but also i think some some cameras
support it in camera so
if you
if you want to do
automatic re re-tracking anyway why
don't you do it on the camera side and
then you you're no longer dealing with a
360 video which needs more bandwidth you
know yeah the use case would be for
instance for sports defense if you have
your favorite player okay say i want to
see this basketball player all game long
yeah and not the rest but i can provide
one
360 video to everyone and they just pick
whatever person they want to follow for
them yeah that would be a nice
i don't know
premium feature or something
okay and i see we have one question from
the internet microphone yes exactly um
the question was if you could
go a little bit more into detail on
bandwidth requirements
yeah so
basically um
the resolution and the the frame rate
are the most important factors
and also uh you have on the one hand
h.264 as a codec and or h.265 which is
almost twice as efficient
and
the
i would say the normal bandwidth
estimates apply so if you have a 4k
video stream
at 30fps maybe
you probably want ideally i mean ideally
you want to have as much bandwidth as
you can so
this camera supports 40 megabits for
stitched or 120 megabits for unstitched
images
so if you want to get the best quality
you would probably stitch it
on the server um
but
you can also get away with
20 megabits i think that's what i'm
using right now and it's for 3d i mean
it doesn't look really great
i was testing it during the camp the
whole time with wi-fi and it dropped out
a lot of the time
and then now i have ethernet here so i
can crank it up a bit
thanks a lot
okay and if there are no more questions
thank you very much for your talk thanks
you