hi I'm VES Dudu I'm a PhD student in
Secure systems group at the University
of watero uh today I'll be talking about
our systematization on unintended
interactions among machine learning
defenses and risks this is a joint work
with Sebastian Schiller and our adviser
Professor Ana shokan machine learning
models are susceptible to a wide range
of risk to security privacy and fairness
uh prior Works have uh proposed defenses
to mitigate against each of these risks
uh however the effectiveness of these
defenses is only evaluated with respect
to the risks that they protect against
um but in practice uh machine learning
models have to be deployed while uh
incorporating multiple defenses
simultaneously this raises two questions
one uh can two defenses uh interact
negatively with each other and the
second is uh when a defense uh against a
specific risk is effective uh can it
increase or decrease the susceptibility
to some other unrelated risk uh these
are what we refer to as unintended
interactions so to answer the first
question uh we have unintended
interactions among defenses uh where um
prior work including ours have shown
that combining multiple defenses
together may result in
conflicts um the second uh the second is
um unint intended interactions between a
defense and other unrelated risk and
here we find that there is limited
evaluation because prior work consider
only specific risks uh defenses
interactions or do not consider the
underlying
causes uh so there is no systematic
framework to uh explore unintended
interactions and this is the focus of uh
our paper um so we present a systematic
framework to understand un unintended
interactions and we conjecture that
overfitting and memorization are the
underlying causes and their uh and
factors which influence them are likely
to influence these interactions as
well um then we uh survey existing
literature on unintended interactions
and situate them uh within our framework
uh and finally we present a guideline to
conjecture about previously unexploded
interactions uh and we consider uh
and empirically validate the conjecture
from the guideline for two previously
unexplored
interactions so I would I want to give a
background of different machine learning
risks and differences considered in our
work so we considered the risks to
security uh which includes uh evasion
poisoning and unauthorized model
ownership uh several risk to privacy and
discriminatory Behavior as a risk to
fairness and on the left left we have uh
corresponding defenses which uh protect
against each of the risks to
systematically explore unintended
interactions we consider all pairwise
combination between a defense and
unrelated risk um so for instance
considering adversarial training uh
which protects against the risk of
evasion we want to see what the
interaction with all the remaining risks
are other than evasion so we previously
mentioned that overfitting and
memorization are the underlying causes
um and there are two reasons for this
one um effective defenses May induce
reduce or rely on overfitting and
memorization and uh all the risks tend
to exploit overfitting and memorization
overfitting and memorization are
distinct and can occur simultaneously so
overfitting is measured as the
difference between the accuracy on train
and test data sets and this is an
aggregate metric as it is as accounts
for all data records in both the data
sets uh memorization on the other hand
is a score that is assigned to
individual data records in the training
data set it is measured as the
difference in the model's predictions on
a data record with and without that data
record in the training data set so to
illustrate the relationship between
overfitting and memorization we consider
a simple experiment uh we take a
synthetic data set with two claes uh
orange CLA and blue CLA and uh here the
training data records are indicated in
circle while the crosses indicate test
data records uh the Bas case is uh where
there is no overfitting and memorization
because the training data records are
linearly separable um and uh and the
test data distribution is similar to the
training data distribution so when we
train a multi-layer perceptron it learns
a linear uh uh decision boundary to
distinguish between both these claes we
take this base example and add noise to
the testing data records of both the
claes such that they fall on the wrong
side of the decision boundary as seen
here and uh this results in a decrease
in test accuracy and we have a case
where there is overfitting but no
memorization
now instead of adding noise to the test
data records we take the base example
and add noise to the train data records
now this results in the training data
records being closer uh together and now
they're no longer linearly separable and
hence the uh the the neural network
cannot learn a simple um linear
classifier but it learns this complex
decision boundary to fit um uh to fit
each of the train in data records for
both the classes perfectly here um
memorization is non zero uh but there is
no overfitting finally we have a case
where we add noise to uh both the
training and testing data records for
both classes and this is where we
observe overfitting and memorization to
occur simultaneously now given the
complexity of the data sets and the
capacity of the model in in current uh
machine learning Paradigm uh this is the
setting that is most likely to occur in
practice and this is what we assume for
the rest of the paper that is
overfitting and memorization uh occur
simultaneously now I want to describe a
framework uh which is used to evaluate
uh unintended interactions now a Fame
framework consists of different factors
which influence uh overfitting and
memorization um and this allows us to
have a fine grained uh understanding of
what influences these unintended
interactions so we start by factors
which influence overfitting uh bias and
variance are two uh two underlying
reasons why overfitting occur where bias
is the error from poor hyperparameter
choices for the machine learning model
um so as an example if uh you consider a
very small model uh which indicates a
high bias it prevents learning relations
between attributes and labels well uh
variance on the other hand is the error
from sensitivity to changes in the
training data set and high variance is
when the model fits the noise in the
training data set now there's a tradeoff
between bias and variance uh which uh
which will result in overfitting uh and
this trade-off can be balanced using two
factors one is the size of the training
data and and the model capacity we
categorize the factors influencing
memorization depending on whether
they're related to the data set
objective function or the model um so
for data set related factors we have day
length of the distribution where data
records constituting the tail of the
distribution are likely to be more
memorized we also note that number of
attributes um and the and whether the
model focuses on learning stable
attributes which do not change with
change in distribution of the data also
correlates with uh
memorization um we also have objective
function related factors which includes
curvature smoothness and
distinguishability in model observables
across data set subgroups and and models
themselves where these model observables
are basically uh predictions or
intermediate activations and finally the
distance of training data records to to
the decision boundary also uh uh
influences memorization finally uh we
have model capacity which is a factor
that influences both overfitting and
memorization given this framework with
all these different factors we now
situate prior work um and we indicate
whether the prior work shows uh an
increase in the risk for a specific
combination of defense and risk or
whether it's a decrease in the RIS risk
or a particular combination has not been
explored we also indicate whether prior
evaluates the influence of factors
empirically theoretically or simply
conjectured uh about a
factor now we revisit different risks
and defenses that we uh showed in the
background for each defense D we
indicate whether uh the effectiveness of
the defense correlates with the change
in a particular Factor
and we also U show whether a change in a
factor correlates with the change in
susceptibility to a risk R we use the
upward arrow for positive correlation
and a downward arrow for negative
correlation and our table basically U
enumerates all the defenses and risks
and the factors and how they correlate
with those
factors Now using this we want to
present a guideline by which uh
researchers and practition can
conjecture about unintended interactions
so for a defense d a risk R and a common
factor F uh we use a pair of arrows that
describe how D and R correspond to F for
a given common factor if both the arrows
align then it indicates that the risk
increases when a defense is effective uh
and that's depicted by a red circle and
otherwise um if the arrows are not
aligned then it's a green circle now
there could be multiple factors which
are common for a combination of uh
defense and a risk and uh if all the
factors suggest the same thing uh then
the conjectured overall interaction is
what those factors indicate otherwise we
have to prioritize the conjecture from
dominant factor and this notion of
dominance of factors uh depends on the
attack and we'll we'll come to this in
the next slide finally there could be uh
non-common factors which may affect the
overall interaction as
well so for dominant factors um we note
that there are we we can categorize the
factors as active or passive depending
on whether the factors are directly
exploited by the attacks which is the
case for uh o1 O2 and O3 and passive
factors like data or model configuration
are the rest so at attacks generally
exploit active factors and we deem them
as dominant because any change in those
factors will result in a significant
change in the susceptibility to a
particular uh uh uh
risk so uh there were two cases in our
framework where we had to use uh this
this notion of dominant factors so
differential privacy increases the
susceptibility to evasion as shown in uh
prior work um and our conjecture uh and
using our guideline we found three
factors in common and the uh and by
identifying the dominant factors which
was o1 and O3 uh we could decide that um
the interaction is given by a red which
also matches uh the empirical results um
second we had group fairness which uh
increases the susceptibility to
membership inference and um the work
that empirically showed this also points
out that um the distance to the decision
boundary plays a role in the
susceptibility to the risk and uh using
a guideline and identifying the dominant
Factor we can um indicate that um the
interaction is is red which matches with
the empirical results now using this
guideline we conjecture uh about two
unexplored interactions and empirically
validate them uh the first is uh group
aess and data reconstruction and uh the
common factor between both of them is uh
is distinguishability across subgroups
which suggests green um and there is one
non-common factor which is the number of
attributes that may uh affect the
susceptibility uh of of the risk so what
we do is we uh perform the attack when
the model is strained without fairness
and with fa fairness and we find that um
the using a fair model the attack sucess
is lower um which confirms our
conjecture um on evaluating the
non-common factor we find that for less
number of attributes um the conjecture
holds uh but as we increase the number
of attributes uh the memorization of uh
individual attributes decreases and
hence um this uh success of the attack
um is is not the same anymore and there
is no change in attack success on using
Fair
model now the second uh interaction is
explanations and distribution inference
and here the common factor it suggests
that there is a uh there's a negative
interaction between them um and there
are two non-common factors which is the
number of attributes and model capacity
which may affect the susceptibility to
different uh distribution inference now
uh for validating this empirically uh we
train a bunch of models on a training
data set with a
distribution um uh alpha 1 which could
be which is the ratio of males to
females and uh another set of models are
trained on a data set with distribution
Alpha 2 um and then the goal of the
attack is to use the model explanations
from both set of models to predict
whether the uh whether the model was
strained on a data set with alpha 1 or
Alpha 2 um and here we find that across
different explanation Al gorithms uh
there is an increased susceptibility to
inference as indicated by an attack
accuracy of greater than 15% 50% for
most ratios now we evaluate to see how
the non-common factors uh influence the
interaction uh starting with the number
of attributes uh on increasing the
number of attributes we find um that the
susceptibility to distribution inference
decreases and we attribute this to uh
lower memorization of the relevant
attributes which are responsible for uh
uh which are required for distribution
inference uh the second is model
capacity and we find that increasing the
model capacity uh results in a higher
attack success due to increased
memorization these examples show that
our guideline can be used to conjecture
about unex explor interactions uh and
also we validate our uh guidelines
predictions with uh uh interactions that
have already been considered in Prior
work uh however there are some
exceptions to our guideline uh first uh
the differences in adversary model can
change the interaction type and we find
uh three such cases in our survey where
uh difference in adversary assumptions
can uh result in different interaction
type but our guideline continues can
only predict one of them because uh uh
because it does not account for the
adversary model uh the second is uh that
there are some risks and defenses which
have too few factors uh evaluated in
Prior work and as a result all their on
all the uh interactions corresponding to
these defenses and risk uh are hard to
predict using our
guideline so as part of our current work
we are developing a software framework
for systematic empirical evaluation of
unexplored interactions this acts as a
tool for practitioners and researchers
um to evaluate the models uh and assess
the risk of uh using a defense on a
particular model there's also a need to
understand uh how different variants of
defenses and risk impact the
interactions so to summarize unintended
interactions are an important concern in
practice and uh by looking at the common
factors between a defense and a risk uh
we can conjecture about the nature of
such interactions and as part of future
work uh we want to look at how to design
defenses which can minimize such uh
unintended increases in other risks
thank you