all right so obviously this is very
last-minute I have no slides or anything
like that but so I guess I guess I'll
start off with kind of giving you the
idea of what I wanted to accomplish I
wanted to do this for question sorry
James Bauer just a normal InfoSec nerd
and I got an interest in a lot of I
basically have done stuff I started
doing network penetration testing and
then I moved on to app sec and then I
started to do some threat hunting for a
while and now I actually have a role
doing uh for intelligence but one of the
things I've always kind of wanted to do
was because I find myself going back and
doing small kind of pen test here or
there or just being curious about you
know infrastructure for maybe like a
situ or something so the idea that I
always kind of had was I would do all
these kind of things that would be
manual kind of over and over again you
know kind of like the same methodologies
and I'd be running through my you know
my list of the things I want to
accomplish like you know you have an IP
I might want to look up like das
information associated with it or on
like the the web application side doing
like simple recon so kind of where I
wanted to go I was always kind of
looking for a way of you know how can I
find how can I build something that will
go ahead and you know I can simply just
give it a domain or an IP and it'll just
take it off from there whether or not
that's you know and but I wanted
something that I could would automate
that and then scale it out you know
essentially indefinitely and then so it
wasn't actually until and I tried you
know taking Python and you know doing
all that stuff but the problem was it's
like I'm using you know multiple tools
that are written in everything from bash
to Python to Ruby and the output is
totally different you know some is gonna
be just you know XML other parts are
just gonna be JSON some of it's just you
know flat text and combining that all
together and I just couldn't come up
with I was always just scratching my
head trying to I'd get like a little bit
further but I just couldn't really get
anything going
so really wasn't until about six months
ago that I was looking into more stuff
and I came I kind of stumbled across
like DevOps stuff
and so I was looking at a few DevOps
solutions and what I actually ended up
settling on was get lab pipelines so I
don't know if you guys are familiar with
how pipelines work it's essentially
pretty easy I mean you're setting up you
it's really just a ya know file but
you're you're kind of it's all based off
of containers which at the time for a
security perspective I really wasn't too
big into containers you know you know
particularly like docker containers I
had used them before but I never really
had like a great purpose for him so I
was like alright whatever but then I was
like as I started looking into the
DevOps IDE which is pretty much all
based off of containers for the most
part it started to get kind of cool so
the first thing I was the first thing I
kind of wrote for a pipeline was is like
okay I create let's say you create a
JIRA ticket or you do a commit in a
github repo like whatever you have that
can essentially kick off like a web hook
it's pretty much all you need so when I
first started I basically we created a
text file in a github repo and I would
put in a domain name and from there
essentially what I had to do was like
the web hook would go to goes to
essentially like gitlab and that kicks
off my trying to think how I should
structure this when you do like the
pipeline you can break the pipeline down
into what's called stages and those
stages can be they can either run in
parallel or they can run you know the
one right after another and so depending
on what you want so like the first thing
I would do is I would have I created a
container with just a docker container
that had masking in it and I basically
have my you know gitlab pipeline what it
starts off it basically like takes that
domain name and just does a dig on it to
get the IP address from it it takes that
because you guys are familiar with masks
an you know he needs an IP feeds that IP
into it mask and takes that jumps out
that information into XML which is then
actually fed into
nmap and so these are happening you know
one right after the other and then but
the interesting thing is is that what I
did was when masscan gets the IP and it
starts doing its TCP port scan across
you know all 65,000 I also have unicorn
doing the UDP scan at the same time in
parallel because what happens is these
are actually different containers and so
there are actually different IPS and so
the cool thing about this type of
solution is and what I got really
excited about is is that depending on
how you set this up I never have to be
worried about being blocked because eat
the the pool of IPs is is huge and
there's tons of ways that you can set
that up I mean just depending on how you
know you want to build your
infrastructure you can essentially have
it where you could literally kick off
container that will be spun up in you
know as a small little you know ec2 and
every time you do a job it's gonna kick
up a new one and you know so that's
gonna be a different IP so you never
have to worry about you know being you
know scanning from home or so or
scanning from your you know a VPS that
you have or a box that you have on a
network somewhere which for me is kind
of cool but they also the good thing
about it is is that the way they accuse
work in pipelines you know that's so
let's just say that we you know our
whole thing that we want to do is we
just want to basically do mask and to do
all the IP you know to do all the ports
feed the open ports to end map and an
end map but do a more intensive scan on
each port you know looking for the
particular service and sending back that
information to you or storing it and so
the cool thing about using pipelines and
this is where the the DevOps stuff
really starts to become cool is is that
the way that the queues work is I could
have I could give it one domain or I
could give it you know I could give it
an entire you know wild-card subdomain I
can say hey you know whatever here's
4,000 yahoo.com domains and for each one
it will kick off a separate pipeline and
those are all running it those are all
running in parallel
not like obviously not all thousands of
it depends on your resources but you
could have anywhere from you know one to
ten or twenty running at the same time
and then what happens at least in my
pipelines is as a period after nmap does
like a more specific scan on those
individual ports it then takes that data
drops it into it it sends it on forward
but at the same time it sends the that
data to an s3 bucket and then I actually
have Splunk go out to s3 every few
minutes and grab that and import it into
my into my dashboard so once it does
that in at the same time I hope you guys
can see that the like the the scaling is
amazing but then also being able to do
so many things in parallel you know that
you just could not you'd always be doing
those things linearly you know when
you're manually doing it which is what
I've just done for so many years so you
know then after it basically gets those
ports it basically just reps that out
and says like hey are there any you know
HTTP ports if there are I say oh well
hey send it over to another pipeline I
have seating over here just waiting and
send a web hook over there with some
information containing you know the the
domain and the port and have what web
grab just back some basic information
about it and then what web basically
does that takes it puts it out as I have
it going to JSON which again goes to the
s3 bucket which goes back into Splunk
which also then says like hey if web
comes back with certain things then go
ahead and kick off you know go Buster on
that and start doing directory
brute-forcing and then take that
information set it to another s3 bucket
that would go back in this block and so
what's really cool is is that it just
starts to you know you put in one piece
of information and then you're literally
looking at a dashboard that just starts
to populate like crazy with all this
information and like I said depending on
your resources and budget I mean you can
talk about you know you're talking about
scanning potentially you know dozens of
domains or eyepiece all at the same time
all in parallel and that information but
it and it just doesn't stop from there
so essentially what happens after go
Buster finds particularly and then what
I what I've also
done was I have jobs that run because
you can have pipelines are run on
schedules so for instance there are a
few people on the Internet to have
really great word list and so what I
have it do is on the schedule I say hey
every night at midnight go to their
github repos download their latest word
list for whatever plus mine and my
private github repo that I personally
like combine it all together you know do
a unique sort and then basically store
it in this github repo and it does that
on schedule every single night well like
when go Buster basically kicks off the
first part of that code that I have in
that yellow file basically says hey go
Buster go here and download my personal
word list from my private github repo
that's the collective of all the stuff
that I really like and use that when
you're doing your directory
brute-forcing and so the the part of the
schedules is pretty cool as well because
it's like you can constantly keep all
this stuff updated because I don't know
about you guys but you know things like
word lists and stuff are I usually
update them right before I need them and
I'm going out looking for like is there
anything newer or cooler or what or you
know like someone will have something in
you know and it's the same thing for
payloads I have a schedule that does
payloads for like cross-site scripting
because people will come out with
something like somebody put something on
Twitter the other day which was like a
really nice wofe bypass for cross-site
scripting and I was like oh that's
really great so I you know added that to
my own personal like little list and so
again like palos dictionaries you can
run these pipelines that just have it go
out and get all this stuff put it
together combined it's sort of do
everything you want put it up to your
own personal you know then upload it to
your repo and so whenever you call jobs
or you're doing anything it's like you
note that it's always up to date and
then of course if you want to get more
complicated with it you can go ahead and
take you know you can have notifications
so basically sometimes I'll have
notifications either I go directly to
like a slack channel that says like hey
you know the word list you know update
it or whatever and then there's also
like emails in case something happens
like hey it failed for we know whatever
particular reason but the what's really
interesting is and this is why
he's like kind of you know that's why I
was kind of really curious about this
talk about how people are getting like
further than that because the the use
cases get like incredibly complicated
when you go past you know just like
directory brute force and like when you
start to make actual logic even just
actually like when you start to do tons
and like thousands of IPs just doing
nmap stuff used and then you try to keep
that's why it's Blanc is great and I'm
I'm a big blunt guy because I like I
like being able to throw in just tons of
different data and different data
sources and have Splunk be able to go
ahead and handle it regardless of if the
data changes over time and I don't have
to worry about it but let me see if I
can think of a real good way of so the
the potential is pretty amazing to me
especially on the and I haven't met too
many people that are you know a lot
especially on the like the dev it's a
cop stuff it's more like they're you
know we have this code you know we're a
software company we do a commit it
basically will kick off you know a some
tool of a scanner that's going to scan
that for you know could be you know HP
whatever and it's gonna look for you
know vulnerabilities in this code and if
it does it'll fail the job or if it
doesn't it will go ahead and push it and
but being able to take that and actually
start to do like a lot more technically
difficult things beyond just like I'm
trying to give you another example like
so the way that I also have it is like I
basically have a I have this all split
up which is kind of like happening all
in the background in parallel like I
said which is you know if it finds an
interesting you know like let's say I do
a brute force directory and it comes
across you know something like a you
know you know my PHP admin I can
basically have that kicked off to
basically fine you know the version of
that and then it basically uses like the
API for something like vulner sore you
know there's a lot of those now they're
basically like hey here's the version
that it was crept out is that vulnerable
if it is okay great
cuz like and I guess this is kind of
where I should have started but like I
said it wasn't prepared for this but so
because my thing is it's like I'm not
trying to basically make this
completely you know autonomous you know
hack all things kind of thing what I'm
really trying to do is if I only have my
I finally having allotted amount of time
like let's say that I have like two
hours tonight to hack or do whatever
like I want to be as efficient as
possible I want to basically look at a
list or have alerts that basically like
hey spend your time on these URLs or
these domains or these IPS in this
software right here and this is where
and so that's currently where that's
where my machine learning is going to
start coming in is is that I'm gonna
have these basically quantified entries
of this is where I should spend my time
at and then as I'm doing manual things
on there all like let's say that I'm
running a burp suite and I'm doing other
stuff that's all going back into Splunk
and then as I actually begin to find
vulnerabilities that are not false
positives and are actually executable
that data feeds back into it to make the
decision of like well hey here's a
cross-site scripting vulnerability that
actually works in here
what other don't you know out of all the
domains that we've looked at what other
URLs are similar to that and let's try
that same payload in those ones and
record the results out of it and so it's
kind of like imagine if you had all of
your pen tests from all of the years
like every tool that you ran in a
central location and every time you do
or learn something cool it can literally
go back and try it on all the previous
stuff that closely matches what you're
looking at like what you're working on
right now so that's kind of like the
idea of it's like it's really just about
like efficiency it's not about it I
really wanted to have just the highest
probability of you know I'm gonna be
able to spend two minutes and say yes
this is vulnerable to this this is right
here great move on to the next one move
on to the next one rather than having to
spend so much time you know just working
on recon and scanning and stuff like
that and sitting there waiting for tools
to to run so uh yeah I guess that's it
so ok thanks a lot appreciate