all right thanks for joining us today i
want to welcome nick selby a good friend
who
has spoken at corncon before he's the
chief security officer
at paxos trust company and he's going to
speak today on tech debt
burn down kill it with fire so
i'll stop sharing my screen and you can
share yours if you want nick thanks for
coming
outstanding thanks a lot can you hear me
okay everything everything is good
yup i can hear you excellent i'm going
to see if i can
actually do this since i've never
actually believe it or not done
a um a zoom before
so am i you guys see my screen and
everything's yep your screen looks great
okay hey thanks a lot everybody um so
yeah burning technical debt it's
something that we all have
and um i get asked about it quite a bit
because it's it's a
it's a big focus of mine i will
jump right in and hopefully we'll get
some questions at the end
um i am as john said cso of paxos trust
where
we have something to do with blockchain
um actually we
empower large financial organizations by
providing blockchain infrastructure to
do
really cool things and we also have a um
a couple of the biggest stable coins out
there tied to gold
and uh us dollars right before
paxos i was at the nypd where i was the
director of cyber intelligence
and uh investigations and i also was a
detective in
the middle of the police department in
texas i was also a um
i had a startup in texas um for
for about five years and uh that that
fed some of my knowledge about tech debt
um
from you know the hard way uh but i also
spent about ten years doing
uh incident response so you know if you
saw me something terrible had just
happened
uh and prior to that in 2005 i
established the security practice at 451
research or
451 group which is now 451 research um
my son and i were talking last night
about
tech debt and he was we were sort of
going back and forth about whether this
is a
digital phenomenon whether this is a
phenomenon of the the new
you know cloud world and no it's it's
actually a human phenomenon
and uh we were talking and he just
immediately said oh yeah the great
escape
is is really a good example of that and
we both read the book and seen the movie
and
um while they're different there's
there's a few things that actually are
are the same and he's absolutely right
it's a perfect example of technical debt
and the reason i say that
um you know you probably know that the
basic story
which is a an officer a prisoner camp in
world war ii and there were current
canadians americans
brits and australians uh they were all
officers
mostly aviators they really wanted to
innovate in
in how they did escapes and they wanted
to come up with a new product
to get more people out than ever the
most important thing that they wanted to
do was not actually get the people out
but they wanted to
make the germans use too much of their
resources um burn up all their resources
at home so that
uh their comrades and arms could could
launch uh successful attacks
and the way they thought about this was
they said let's let's build a lot of
tunnels
in a place that the germans put us with
sandy soil so that we couldn't tunnel at
all and boy will that be innovative and
surprising
and so they they began work on three
simultaneous tunnels they called tom
dick and harry there was actually a
fourth called george that they didn't
they didn't end up doing but um
they pushed really hard to get all the
features and the the features were that
you know you had to be able to fit a man
but they had to make some
they had to make a bunch of compromises
as they went they had to make them
uh less deep than they than they wanted
they had to show them up less
less than they wanted and they had to
make them thinner and um
there were more choke points than they
wanted and so they they suffered from
collapse as they suffered from them
getting stuck in
the narrow tunnel and and a bunch of
other challenges but in the end
77 of the expected 250 people managed to
get through
76 actually escaped the 77th actually
surrendered
um there were 600 people who worked on
the thing
so what i'm saying is is that by shoring
up the infrastructure
even when you have not great
infrastructure you can make things work
if there is technical debt
it's just that they don't work as well
and that they don't work as safely
um if you take a look at some of some of
the stuff that they did to
shore up their infrastructure i mean
they stole everything and that was
actually part of the downfall part of
the ways that they got caught a couple
of times was they were just taking
everything to try to get the
infrastructure to work starting off as
small as they could but building up as
they needed to
so to define tech deck i'm just going to
say that it
is and we're all guilty of this i was
guilty of this as a ceo i
you know i want to make a product and
the first thing i want to do is get it
in customer hands so i need something
that's as close to mvp as possible and
so i just want to get there as fast as i
can and damn the torpedoes full speed
ahead let's just go
and get those features done so we're
going to accept things for now
we're going to accept things or ignore
things and
that is the infrastructural and
configuration or or information security
work that we delay
to let us focus on what we think is
important right to
to get the thing that we're building in
the hands of the people
we want to use it so that we can bring
in money so
our revenue teams are our business
people they're telling us what the
customers want
um and we're just moving ahead as fast
as we can
if you've spent any time in information
security or in
in information technology you know that
nothing is more permanent than a
temporary fix
um and so a lot of these fixes that
a lot of these things that we accept
accumulate and you end up with this huge
backlog
of accumulation of technical debt and
this is aggravated by
the fact and it's just a fact that no
matter how much you love our employees
no matter how much we love our
co-workers people leave people move on
to different things people get fired
uh people get sick and we try to do
knowledge transfer we try to share the
information
that we that we have and no but we're
when i'm doing that
if i'm a developer or if i'm an
application guy or i'm a business guy
i'm talking about the stuff that works
i'm talking about the stuff
that i think is important and a lot of
the
compromises that i might have made along
the way don't get knowledge transferred
so it's very
very common and i would say this all the
time in incident response
that you know i'm dealing with a team
who inherited
uh a group of a bunch of technology from
a team
who's no longer there and they inherited
it from a team was no longer there and
they inherited from teams so you've
actually got
no built-in cultural institutional
knowledge of how your technology got
there
and so people start to just assume
things about the bedrock on which
they're working
and if you don't stir it up and if you
don't look
to see whether those those assumptions
are true
you can get into some real trouble
developers
too and you'd think that developers are
in the best position to
uh to see these things and you know in
some ways they are but the problem is
developers sort of they like to see the
the things that are working and make
them better
um and they're not really interested in
the things that they haven't that they
don't need to worry about because
developers are constantly trying to push
forward they're very creative people
and they want to move forward and it's
it's completely understandable that
that's how they go so like if i think
about
you know and i've said this before in
different places but
you know developers business they're a
lot like
the the traffic cops they just want to
keep things going let's just get
everything going move as fast as you
possibly can let's just get where we're
going
and everything will be okay and security
people are much more like homicide
investigators
whoa let's just slow down and let me
just ask
a few questions now there's a natural
tension that gets built between these
two groups and that's great
that that should be there the the
problem is when one group is too
powerful
uh and and usually one group is too
powerful because senior executives are
not recognizing
that you need that balance you need to
be able to have
really good and cordial and
collaborative relationships across both
groups so you don't want the security
guy having too much
input to the extent that they are dr
know
that you know security is where
everything goes to die and they never
want to do anything and they're just no
fun at all
because as soon as you get into that
what will end up happening is your
product
um they'll just start to go around you
and they'll just you'll end up with not
not just technical debt but you'll also
end up with shadow i.t you'll end up
with a whole bunch of other things
that are part and parcel of security
having too much influence or security
uh workers who don't understand what is
trying to be created they don't
understand the business and i know we
always say you know hey we're security
we should we should understand the
business but a lot of us don't
we're just focused on what you can't do
um if you go the other way then the
security guy gets kind of left out
the developers are running the show and
the security guy's just yelling until
he's blue in the face
and if you've got a security person with
integrity they're going to leave and
and you're going to keep going through
people uh until you find somebody who
isn't going to leave and then you're
just going to have somebody who's
basically powerless
and that's not good either so finding
that good balance between
developers business and security is
really important that's got to come from
the top down
it can't be um it can't be something
that is imposed by just the security guy
or mid-level management
um excuse me bottom-up and top down or
it fails ceo has to put his arm around
the security guy and say yes this is
this is what we're doing
i heard this this morning we were
watching a
video interview of this uh he's a guy
who makes tools by hand
and everything and he had on carolyn
baker who writes a bunch of books and
and this
quote about the death of uncertainty and
the trauma in a torrent of trauma
uh something that she was talking about
uh
the pandemic we all felt this when this
happened and and
i know that with technical debt
that can happen too so when you've
accumulated technical debt and you might
not even notice it and everything seems
like it's just humming along and you're
all
you know everything's right down the
middle and all of a sudden you crash
into a tree
that phrase is perfect for the dynamic
that happens in companies because people
just get they really get stunned and
they don't quite know where to go
and they and they have to go back and
start questioning things they've never
actually questioned
everything that they've counted on you
know it's kind of like if your water
gets turned off in your house it's
really a surprise and i don't know if
it's just happened to you but when it
does it's really a shock or the internet
goes out
this stuff has all shifted that is the
moment
where it's really a you know come to
your
your day to hear moment where you
suddenly are
really rethinking everything about the
decisions that you've made and i think
it's pretty clear that you know we're on
zoom right now
we don't um we've seen that zoom
like many many companies um
had a product that they wanted to get
out and what they wanted to do was end
the horror and pain of webex they wanted
to end the horror and pain of other
video
video stuff is terrible i love zoom i
was just talking to my friend ian who i
mentioned later
and he was like i look better on zoom i
sound better on zoom right zoom is
awesome
but they were so interested in getting
all their stuff
out there that they forgot about this
basic
this bedrock this this foundational
stuff and
it cost them dearly as soon as it became
it went from
something that was nice to have to an
absolutely essential resource
and that's when all the security people
started playing with it that's when all
the criminals started playing with it
and suddenly it comes out that there's a
lot of stuff shifting the ground has
shifted underneath zoom's feet
and you know i'll talk about a little
bit about that later
so if i think about a technical debt
life cycle um you have a company that's
got a great idea this is going to be
awesome we're going to get into the
hands of the customers and so you get
something out there that's basically you
know
mvp or something and then there's a
reaction and the customers always say
the same thing
well you know this is pretty good you
know i like it um i get to do a few more
things and when they say that
of course the business is like oh of
course we're going to do a few more
things you know that's absolutely what
we're going to do and so
we start to make more and more of those
compromises because they by the way they
still haven't paid us right they're just
we're just trying to get this in the
hands so that they'll say it's great so
that we can go to our other people
and so like our business people like
this is awesome keep going keep pushing
keep pushing
and then you eventually and absolutely
without fail you'd get to a crash
and wow we we just never thought that
so you know that's that's the definition
of technical debt i hope that uh
everybody is is on the same page
and you probably recognize a lot of the
things about this
um because as i said it's human nature
to to accumulate technical debt
uh it's it's actually not human nature
to not do it it's
actually really difficult to to to
tune our minds to not build technical
debt
um and it it's not just in tech it's in
everything you do so plowing ahead so
what the hell do you do about it that's
that's actually what i read
um and i'm sorry it's boring
this is actually the very boring bit
about
what you have to do because the only way
to deal with tech debt is to look for
tech debt
and if you're not looking for it you're
just not going to find it you're not
going to see it until it
until it finds you so um you need the
smartest people in your organization
the smartest people on your teams you
need to get together and
you have to gain the executive authority
to actually do this and take time out of
all the things that are on your sprint
list all the things that you're supposed
to be doing
and say no we're going to actually spend
some time and we're going to be looking
at this because we need to understand
where we are and just just get a level
set
of what is the reality in our
organization how
good are we how bad are we what are the
deltas between how good we think we are
and how bad we actually are
or how good we think we are and how
great we actually are whatever it is you
need to know what that is
um i like to go across the different
um teams and find maybe some you know
there's some obvious people like
engineering managers security managers
people like that but there's also some
non-obvious people in in almost every
company in almost every group
there's a guy who everybody goes to
because he knows where their bodies are
very there's a guy
or or a woman who knows where all the
decisions that have been made over time
they're the person people seek out when
they need to fix something
personal like they just bought a new
watch that is really complicated and
they'll go to this person right that's
the person you want helping you find
tech debt
um you want the person who actually
thinks that this stuff is cool not
because it's security or not because
it's development but because it's
interesting
get down that's really important in the
beginning
you're looking for low hanging fruit
you've got to start somewhere um
you know when you're eating an elephant
you should just
start with the first bite um start with
the low-hanging fruit
look at the scans that you have of
things that are public get scans of
things that are public this should not
cost you any money you should be able to
find this
actually at almost no cost right but
take a look at
what's showing up like are you actually
allowing stuff to go out uh using ssl
version three i mean i hope it's not
this any older than that
um are you still a tls one like just
find out make a catalog
show what it is that's facing forward
and this is how they talk and this is
what they do
do this absolutely everywhere um what do
all your domains look like
how many domains do you have who holds
your domains where do you
uh you know where do you register them
and who has the registry of domains and
who's making sure that that those things
are all done
uh correctly are you spending too much
money there is that is that tech debt uh
just just the simple way right some of
them at godaddy and some of them are at
google
or do you actually have a plan so figure
that out and write that down
what do i p intelligence firms think
about you how do you look to the
security community do you show up
in uh in threat feeds do you show up in
different places right this is
really really low hanging fruit and it's
just the first way to go this will take
you some time
to to get to now
the best thing to do is take a look at
the work that you've already done
um strictly in security right so you've
probably done external pinterest you've
probably done
maybe even some internal pen tests go
back and take a look at all the pen
tests you can find over the last couple
of years
and start to look not just
at um do the same things keep coming up
in pen tests that's
you know obviously something to think
about
but when you get the findings
have you made tickets for each one of
those those findings and have you
actually
remediated all those tickets and if not
why not and you know are they still open
um did did you declare ticket bankruptcy
and you just got rid of it because it
was too hard
that's by the way if you had to declare
a ticket bankruptcy
that is an absolutely like first rate
indicator that you've got
more tech more tech debt than you can
handle um if
you start to see across a number of
different projects that that
new things get abandoned because they
are
really hard because there's all these
blockers fantastic indicator that you've
got
that is blocking your way and you're
gonna have to deal with it
um what do the findings are what they
tell you about the kinds of bugs
that testers continually service they
might be a little bit different but if
they're all of the same
ilk and just to make it really simple
you know if it's cross-site scripting
why is that why does that keep on happen
um and that can go back to a bunch of
different things like testing and
training
it can go back and and by the way if
you're getting back into that and you're
starting to look into code repos then
all these things will end up with
different rabbit holes you know the the
next thing that i wanted
that i'll probably want to dig into and
these things will just pop out at you
right is
well if i'm not looking at my code and
i'm not testing my code am i burying
static credentials in my code
um ask the question uh if you're not
looking for it github is
giving us a bunch of help but a lot of
times you'll find things in there
um and you know
tech debt be gets tech debt so the first
thing you really want to do is find out
what you got and then just keep
following the holes until you're at the
end
it's the same thing with vulnerability
scans right um
look for the patterns they will pop out
um
and and it isn't i don't want to get you
know wrapped around the axle about the
fact you know you've got 1900 criticals
and
they're high and critical and it's
really terrible
i'm actually more interested in whether
you understand whether your people
understand
why you haven't patched because if you
if there is a process where you've just
said okay
you know those are critical but they're
actually just like that that's
not anything that we really care about
um you know tenable thinks it's critical
but we don't think it's critical because
of x y and z that's totally fine that's
that's really good actually um but if
you have accepted it because it hurts
your head
to think about all the different tickets
that you would have to fill
and and answer and and get done and all
the sprints that it would take just to
stop
blocking the thing that's that's
preventing you from
um from from patching those systems
that's
huge in terms of finding technique it's
just not okay right if you're
if you can't write it all up in one
ticket that's a big problem
once you've done that the next step is
to look at
the processes of logging
and disaster recovery your questions
here are really just over and over
can i can i do what i think what's the
difference between what i think i can do
and what i can actually do
and why and and you just have to keep
asking those questions
um the very first thing to do is find
out what you promise other people that
you do
and it's it's never you know it's never
fun but
sit down with the business sit down with
legal sit down take a look at what
you've said you do
um maybe maybe you've promised people
something or even worse you've attested
to something
um and just have a good understanding as
you go in and look at this
about what your promises have been and
then
the questions that you're asking are am
i am i logging when i
think i'm logged i'm really sure um
the best way to do this is through use
cases and
you know i it doesn't matter what you do
for a living you know if you if you have
a an ecommerce site then it's all right
uh customer comes on
they sign in they browse they pick three
items they put them in their shopping
cart
they pay their credit card comes back as
not good they put in another credit card
they pay it's successful we send out the
email
okay show me the logs for that entire
experience what
is missing from that entire experience
and
it's fine whatever it is is missing is
missing you just have to know it um
and and you know if you're if you're in
a bank obviously it's going to be
different every business is different
but what you want to do is find real use
cases that are
simple just dead simple like this is
what we do
every day and run through and see if you
can
find that process in your logs and find
the black holes there's going to be
black holes there's always black holes
but just
find out what they are identify them and
then think about what you want to do
um you know are you looking at dns are
you looking at off logs for everything
can you do it for certain dates and
times if not why not those are the
questions
um however long you're retaining logs
it's not really the retention period
is not really that important unless
you're regulated if you're regulated are
you really doing it for
all of them but give yourself realistic
use cases um
you know if you keep 30 days and sort of
live stuff go back 27 days and see if
you can actually do what i just said
and if not find out why but figure out
what's what's the delta between your
expectation
and the on the ground truth
failover tests are really telling
those are the ones that everybody hates
to do because they're hard and they're
they're terrible and things can go wrong
and it's
you know it's awful but um think you
want to look at failover tests
less being about compliance or your
annual
attestation that you've done and more
about
right you know if you get hit by
ransomware tomorrow what would happen
and what would you what's the situation
that you'd be in and why and start to
to catalog why and the issues of your
last dr
failover test are this is a gold mine
and this is it's a
deep and rich gold mine and this is
going to give you months of work
to try to figure out what is going on
and why it isn't
where you think it should be because it
almost never is um i was just talking to
my friend ian he's at simpress and he
was telling me that reed hastings would
go
through the data centers um you know
watching
netflix on a mobile phone and like just
yanking out arbitrary cables to see what
happens and
i don't care if it's true it's a great
story it's not where any of us
are um but it's a tremendously fantastic
place to be in terms of resilience
and if you can get there with resilience
or if you can think about that's where
you want to get with resilience those
questions i just
just put out there right that's that's
how you get there
you never take anything for granted and
you always understand
the reality of where you are and that's
really critical
when the customer sends you the
spreadsheet because their procurement
weasels have
decided that they don't understand what
you have and they send you the million
questions spreadsheet from hell
it's a good thing it's a better thing if
you do this yourself
and so if you haven't gone to vsa the
vendor security alliance and downloaded
their i think it's like 100 questions
they have a full and a core i think the
core is like 80 questions in full maybe
120 questions
it's all the questions that anybody who
wants to deal with third party risk is
going to be asking
and they're very very straightforward
and most of the answers are binary do
you do this yes or no do you do this yes
or no
if you don't do it are there any
compensated controls if so list them
going through that and taking the time
to do it it should it should
the fastest i've seen it is six hours uh
the longest i've seen or
normally what i get is like it takes
people about a week to go through this
these questions are because they're
critical importance to your customers
they're they're of critical importance
to you
it's a really great reality check it's
also really good
to have it on hand to give to business
development people sales people
and say the first time you have a
meeting with your customer and
it looks like they're asking you for a
trial or it might go to contract
give them this give them our spreadsheet
give them our
documentation about our security let let
their procurement weasel see that we
actually care about this
and now they're going to look at you a
little bit differently and they're not
going to feel like they have to
you know eke everything out from you um
so
there's there's no real prize that's
that's about as much of a prize as that
you'll get for filling in those
spreadsheets
there's a huge penalty for being wrong
right because that question that i asked
about
ssl or tls that's going to be in there
and and so answering that spreadsheet
will give you a lot of the answers and
it's just
fantastic for finding tech debt
just doing the steps that i've said and
you know i only have half an hour so
just doing those steps will take
most people four to six months to
actually flesh those out if you've never
done this before
um i know that many of these might seem
obvious and if you're in a sophisticated
organization
you know you've probably done a lot of
these things um i have been in
very large organizations that haven't
done this i've been in very small
organizations and have right
um when i've gone into incident response
uh over the years
almost every single incident response
i've ever done was
a a byproduct of of some
kind of tech debt that i just that would
have been called by
those things and again it goes from very
very large companies to very small
companies
it doesn't make a difference it's this
stuff is hard so people don't like doing
it
um i want to touch a little bit on cloud
configuration because
um it's a vague concept everybody's
multi-cloud i don't know what your cloud
is some people do lift and shift some
people are
completely infrastructure as code we are
wherever you are in there
um you're still probably going to have
some technical debt
um the most important question just just
from a
a theoretical perspective is how are
your whatever it is that you're running
how are they configured
uh and are you doing it against a
baseline and
what's the reality on the ground so
however many you have
um and is it meeting the expectations of
your baseline
and you and if you haven't done this
first thing monday go out and get either
scout suite which is free or cloud
conformity which
got bought by train micro uh get the i
think they have a free trial
run it just run it and it will dump it
out really really fast it is the most
eye-opening breathtaking thing you'll
ever see because it'll show you any
everything in your cloud environment
and how it's configured and it'll be set
it'll it'll sit there and tell you you
know
41 of your snapshots are not encrypted
and you know a bunch of your instances
are not encrypted and they're sending
things out or your firewalls are
misconfigured
whatever it is you'll get a list and uh
with cloud conformity it's actually done
in low medium and high and critical
which is really really nice um so that's
that's something really to do to get a
sense of where you are
in in your cloud configuration find out
that's the most important and then dig
in i will leave you on this
it's this is like new york city it'll be
a nice city when it's done
tech debt is not done you're it's not a
project that you do once and you're done
unless you're really unlucky it's a
process and it absolutely has to be so
there are
literally there's two ways to deal with
it
you do it constantly alongside
development
and collaboratively with all of
of dev or you don't do that and you
accumulate
so much tech debt that you
get to a point where you have to stop
everything
and shut down for a month or two months
or three months
and go back and fix everything and just
refactor everything and then you'll go
and do it constantly as an effort
alongside dev there is no third option
these are your only options so it's and
i will say this like people say well
that's really hard
or that's really expensive or i don't
have the resources to do that
it will never be easier or cheaper
to do this than it is today every day
you wait
it gets harder and more expensive to do
it you should start today
one thing that i'm doing now and we're
just starting this so i can't really
tell you if it's successful i'm really
hopeful that it will be
uh is thinking about embeds and forward
deployment um
what we're doing is getting forward
deployment of developers from every
squad into security
so they will come and they'll come to
our stand-ups and
uh ultimately we'll send our security
engineers into their standups
once a week it's not it's nothing
serious but but what we end up with is
we we start to to get collaboration we
start to get evangelism on both sides
and we start to really get to know some
of the people who are making decisions
um i would i would recommend that you
get people who think it's cool who think
that it's interesting who think that
it's fun if you just
if you force somebody and frog marks
them into our stand up it's not really
going to be very
very productive but there is at least
one person in every squad who's going to
be
uh who's going to think that this is
neato that's the guy you want
um and another thing is embedding we're
making
ssre security site reliability engineer
um
if we think about since i said where
infrastructure is code if we think about
our security not from the standpoint of
guys and hoodies but like
guys in um
guys who are looking for resilience guys
we're looking to make things work then
it really comes down to security by
virtue of
excellent configuration that's really
what the bottom line is
and the only way that i think that that
can really work is if we are
hand in glove with sre because they're
they are where the rubber meets the road
and in some ways so are we so having a
security site reliability engineer
somebody who
who stands up with sre but but is really
mainly focusing on the parts of our
infrastructure that are
directly related to security we're
thinking that that's going to be a very
successful role
i am now uh done with my presentation no
i'm not
sorry um i i thought of this because
you actually when the military did this
um they
found that reporters they got better
reporting
um because the reporters actually had
skin in the game now sometimes that that
meant that the the reporters were
throwing it in on the side of the
military they were biased towards the
military but a lot of times what it
really meant was that the reporters had
a better contextual understanding of
what was happening
that's what i'm seeking here i think
it's really really clever
as a way as a way to work so here's my
contact information
and i will be available right now if
anybody has any questions for me right
now
all right does anybody have anything to
enter into the q a window
if not feel free to reach out to nick
with any questions we're going to be
posting the recordings and the slides
here in the next couple days
uh on corncon.tv and
uh i'll let you go nick i appreciate you
coming and and giving a great
presentation today thank you
thanks very much take care