hi I'm sarjit Singh and I'll be talking
about our work Hina Hina is a work
towards practical privacy preserving
inference Hina is based on homo
encryption which is a cryptographic tool
that allows compute over encrypted data
hence homic encryption is a vital tool
in realizing sensitive applications like
Medical Imaging genomics compute
Outsourcing
ET however practical adoption of homic
encryption has been limited by its
performance and
complexity homic encryption works with
large degree polinomial which are in
megabytes of size leading to significant
memory accesses and hence poor
performance additionally modular
arithmetic over large finite fields and
complex operations like domain
Transformations and morphism make this
tool extremely difficult to
implement as a result researchers have
seen many orders of slowdown in private
inference as compared to an unencrypted
one packing is a widely adopted
mechanism to elevate the cost of large
pols instead of encrypting a single
value a vctor of values are packed into
the slots of polom hence elevating the
cost cost of single Cipher
operation however in real applications
these slots might need to be rotated in
order to fully exploit them as you can
see from uh this picture representation
the colors or the slots have
moved rotations are expensive since they
require additional metadata called keys
to perform key switching and these keys
are in megabytes of
size moreover
a non-cyclic permutation of slots is
requires a series of many such cyclic
rotations and hence is even more
expensive in this work we target homic
encryption based convolution near
networks we identify these rotations as
the major performance botl lck in
encrypted
inference a typical convolution looks
something like this each output pixel is
being generated by a convolution of
input wids and inputs along the input
channels similar colors are multiplied
and the result across colors are added
to generate one single complete output
feature map which is represented by the
red output pixel
here a few things change when performed
with uh with encrypted
ciphers first inputs can be bashed
together into polom slots using packing
as we saw that from a previous slide
second the multiply accumulate step now
has an additional step called rotation
to fully able to accumulate
things on the left I'm presenting one
way to pack multiple inputs together
this example strategy backs all inputs
that are required to complete one output
pixel this this packing is very
intuitive however it requires many
rotations per multiplication which is
bad in fact after each multiplication it
requires n rotation or n minus one
rotations which is a huge
cost one other strategy which I present
on the right attempts to achieve zero
almost zero
rotations however such packing is also
inefficient as it
utilizes the polom slots leading to
ineffectual computations which can be
seen from these white boxes over
here both these strategies have been
explored in Prior work in fact the left
uh left packing is called Channel
packing and the right packing is called
Lola packing both these work try to come
up with a uh with OP with optimizing one
certain thing either slot utilization or
rotation managing this trade-off between
slot utilization and rotation is the
target of our
work wees Hina a packing and data flow
for private CNN inference the key
objectives while designing Hina are left
listed on the left these are to First
maximize slot
utilization if uh by maximiz slot
utilization we make sure that the total
number of ciphers that we're working
with is lower hence the working set and
memory activity is
smaller second to minimize rotation as
we learned rotation are expensive
because of expensive key switchings
operations hence we would like to
minimize them more importantly we would
like to minimize the permute type of
rotations uh as we remember permute type
rotations require many cyclic rotations
and hence many key switching operations
lastly to maximize on chip
reuse we start by packing a single phas
of weight and its corresponding phase in
it input feature map the output of this
multiplication contributes to the first
output pixels partial sub which is
highlighted by the dark red over
here next we pack the neighboring out uh
neighboring non-overlapping phas and
input uh feature my poly which is
highlighted by the blue color
here instead of instead of packing the
same weight we pack a weight a different
weight phas from a different output
Channel which is the brown weight
here by doing this we generate output
pixel across different output channels
which is out uh represented by the brown
output feature map here this avoids
redundant packing a weight packing
similar to uh similar to Lola packing
hence improving slot
utilization once we exhaust input faces
or output channels we go across input
channels of the same weight values which
is represented by the lighter colors
here these products accumulate to the
same output feature map since input
channels are are added together
so this highlights hina's packing
strategy It Is complemented by a data
flow that fully exploits the slots with
minimum rotations so far we have only
talked about a single multiplication and
let's talk about what happens next first
we reorder these slots so that the
rotations required to accumulate partial
sums are only
cyclic
next using the same input feature
features and weight polinomial we can
generate different output pixels by
rotating our weight polinomial across
different output feature Maps notice the
color change in the weight polinomial
here this generating uh this has
generated new colors uh in output
feature Maps as
well this allows us to reuse the same
operant to generate results across many
output channels hence minimizing the off
chest movement
when we go
across all our input channels all their
results uh accumulate to complete the
same output pixels notice how after step
B we don't iMed immediately rotated all
our output partial sums and added the
similar colors
instead we perform this rotation
assisted partial some accumulation only
after we have added across all
separately packed input channels by
delaying this rotation we have reduced
rotation calls by a factor of almost the
input Channel
count next we complete the output pixels
by going over all output channels we
reuse the input features to generate new
com new combinations using permute type
rotations and finally we perform these
operations s for all output
pixels overall the key strategies that
have gone into this packing and data
flow are first carefully tailored slot
packing that helps us convert by mutes
to shift rotates and hence maximizing
slot
utilization second a data flow that
delays partial sum accumulation until
all input channs have been added this
allows the St reduced rotation
counts lastly generating new inputs
using the already on chip cached inputs
this Maxim this minimizes memory
activity so we evaluate Hina on a Asic
design to run reset 50 mobile net and
gnmt models we employ a hybrid strategy
where MPC is used to preserve model
privacy while the client performs the
nonlinear layers
more importantly we have released a tool
for Community to explore novel packing
data flow strategies and Hardware
designs for private influence this tool
is currently uh public on GitHub and is
termed as H
packim more details about how we modeled
prior Baseline packings and
Architectural implementations is
presenting in our full
paper so moving on to the evaluation
results on the top top we present
rotation call counts and on below we
present data footprint in
gigabytes the rotation count is
distributed to permutes and cyclic
shifts we have compared with baselines
Lola pack Channel pack and c2pc pack we
note that compared to the baselines we
had a very small count of rotations and
data footprint in fact Hina has almost
zero permute type rotations and at least
10 times lower memory activity than the
other packings this is mainly due to our
efficiently packed and efficiently
managed slots using data
flow as a result our encrypted latencies
are in tens to hundreds of milliseconds
only which is at least 2x faster than
the best Baseline and way more energy
efficient this is represented by the two
graphs here the left hand side
represents the inference latency across
different B
benchmarks the right hand side graph is
a stacked graph that demonstrates the
distribution of energy consumption by
each Hardware
component as noticed from the previous
Slide the high data footprint of Prior
packing have led them to high memory
accesses which is also represented by
the Blue by the uh blue bar in the right
figure overall care carefully tailored
slots and well-managed data flow has
allowed Hina to reduce both compute and
data movement
cost with that I would like to conclude
my presentation in conclusion we
demonstrate a practical encrypted
inference solution we identify rotation
as the key bottom lck we explore various
backing and data flow strategies we
finally publish an open source to called
H Pac SIM for more explorat studies for
more information about this work please
refer to our full paper and IA
Symposium thank you