let me get this situated can you hear me
okay
all right if you can't see any of the
screenshot just yell like raise your
hand I may not see you but raise your
hand some of them are a little dark so I
can zoom in afterwards of all of these
slides will be on a github repository
and I'll sure that out but they're all
in markdown so it's pretty easy to kind
of follow along we're gonna go through a
lot of data and a lot of information in
a short period of time so all right so
today I'm gonna talk about PI EWS and
ews has exchange web services and this
is a Python package that I wrote that
actually allows you to interact with
exchange web services on Linux Mac
Windows pretty much any system that you
want so before well a little bit about
me yeah that is actually being is it
hello okay so a little bit about me so I
have a lot of background in digital
forensics and it's a response as well as
blue team and I like to automate
anything that I can at all I mean it
doesn't really matter what if I have to
do it twice I'm gonna write a script for
it I also like to call myself an open
sorcerer right no one get it okay I
thought that was funny I also like to
take long walks down binary trees and
yeah you can read them all right so a
little bit about exchange in general but
first before we before we dive in we
need to understand yeah as a rose
talking about this is rest versus soap
anyone familiar with api's you use them
on a daily basis right you should if you
haven't man if you don't I have like a
whole series coming out here soon that
all all sure as well but
rest is representable state transfer got
any more of those rest api's you know
and it's really all about HTTP methods
so you your your sending an HDTV or
request to either get data post data put
data which is like a modify it as well
as delete but all this is all in from
like the client side and it affords it
onto some DB or some system in the
background and then it just returns all
the data and JSON JSON is really nice
and easy to use and it's pretty
straightforward to kind of parse so-so
is a little bit different and if you
can't read that it's as a soap request
thank you sorry this is a rest room now
okay so it's the simple simple object
access protocol it really it's just XML
that is a good representation of XML so
overall in a soap API is actually as
these kind of forms or shapes and it's
all based on a wrapped envelope and
inside that envelope you actually have a
header and you have metadata about that
header attributes then you have a body
element and that body element is
actually going to contain the methods so
instead of actually doing an HTTP GET
post update you're always gonna post and
the body is gonna tell it whether you
want to get data or not and then each
body and method has elements and
variables and then all that in it again
it's post only so you're now going to
submit get requests you're now gonna
sort of put you're always gonna send a
post so let's dive into a little bit
about exchange so exchange is complex if
you have ever anyone here manage
exchange or have managed it or use it
anyone have Outlook so exchange is all
about the clients connecting it and they
connected over different protocols so
whether you're actually sending it over
pop or IMAP there's a whole other
protocol but clients like your mobile
device your outlook client your Mac or
all send it over AWS and they'll
actually connect to an iis server which
then proxies back to the actual HTTP
service of soap the soap
EWS actually connects to the mailbox
database so this is like a wrapper for
accessing the the mailboxes all the data
we turn those mailboxes and you have
some delivery queues and some other
stuff so here's an example with actually
you know you can see I think it's almost
all modern or smartphone outlook for Mac
I look for Windows I'll use EWS
underneath the hood newer phones like if
you have Outlook the outlook app for
office 365 it will connect over arrest
but they still use soap as well but it
connects to EWS as well as some other
services like RPC das so on and so forth
but the basis of everything is around
EWS so you have HTTP soap request that
actually gets sent to an iis web server
that's running on that exchange and that
actually forwards it off and tries to do
lookups and access actual mailbox data
so basically everything that you can do
in exchange and I be like Adam email
delete an email add a calendar item add
a group to an IM or an IM group remove
people everything is done through EWS so
you have access to everything that you
can do in the UI of exchange and war
through EWS Soaker
these are the the most common ways to
actually connect so use dotnet or
c-sharp and it uses the EWS managed API
that's completely different if you ever
see that this is a dll that you have on
windows systems that they just wrap EWS
in to make it easily accessible
same with powershell powershell and
c-sharp j/s node you can access it
through rest but it's only for office
365
not on on-premise exchange all does not
have rest at this time again PowerShell
with Microsoft Exchange Web Services DLL
so a compiled DLL that only runs on
Windows systems and then you get
PowerShell remote you can actually
remote into Exchange Online to view
office 365 you can actually remote to a
jump box or the actual exchange server
please don't ever do that but you can
actually do that and run those commands
and then you also have PowerShell you
know just the general exchange
Commandments you know get mailbox or
whatever Zak it is all of those are
actually using that DLL underneath them
so what about Nix users anyone ever see
this xkcd article I love this I have it
framed because it's amazing it's kind of
yeah let me see if I can stretch that up
there you go because no I'm not
interested in updating my antivirus I
run Linux and he gets on his dirt bike
and runs off pretty funny I have a weird
since humor I think so PI EWS is why
Linux users so this is a framework that
I built the in Python that works in
Python 2 & 3 but it wraps the entire PWS
into an easily digestible package and
we'll go through all of those kind of
functions that they currently support
and how you can extend and allow it to
do whatever the heck you want the
repository is there I'll have the links
at the end
but there's also the documentation it's
pretty thorough so to install it let's
zoom in so install it it's literally its
own pie pie so if you're not familiar
with Python or package management it's
it's underused pip which is their
built-in package manager
pip stands for Python ad there's like
for meetings like Python is Python or
pip is Python I think is what this means
for ya so you can install it on Windows
or Linux and run this
it's just pip install PI - uws and where
you can clone the repository and to use
it you first actually have to create a
user configuration so everything with
piety but PI ews is with this user
configuration object so this will
connect to all the service in points
that you can actually perform actions
with so we have a couple different
options when it comes to actually
authenticating to exchange we have to
import the library PI EWS import user
config but before you really kind of go
down that road make sure you have these
access rights as ediscovery has anyone
ever used ediscovery permissions in
exchange okay
so e discovery is what digital forensics
or ir people or your compliance
administrator will actually put a hold
on your mailbox and you will never know
and that means you think you're deleting
an email and you're not there holding it
and you to do that you actually have to
have these rights in exchange and they
can do automated searching they can do
it but it's all through this crabby UI
that is so slow and crashes like half
the time if you have a mailbox of 20,000
or more it will crash within 24 hours
and it'll probably take about 48 hours
to run that's how crappy it is so but
these endpoints are there which is great
so you need to have even ediscovery
rights
compliance administrator ediscovery
manager so on and so forth but once you
actually have the
then you can provide a username and
password there's no OAuth or any other
authentication mechanism besides your
name user human password with EWS so
it's not the most secure but it's there
and it will be there for a long time so
if you don't have those rights but you
still want to search then you can
actually impersonate so if you are given
the impersonation right for a specific
mailbox then you can actually search as
that user so you're going to personate
that user and you can do this all
through AWS as well so depending on your
situation
it's better to have any discovery rights
because it's very restricted that the
rights aren't they won't get budged I
guess but impersonation rights are
possible and so you need the application
of personation or the actual right on
the ad object is called MS exchange
impersonation once you do that you then
you know create an impersonation object
by importing that part of the library
and then you just pass it into your user
configuration so from now on when you
make any other calls using that object
you actually impersonate based on that
users rights now this is the deep dive
can you see that okay a lot of this is
like comments up there so you don't
really but to impersonate you just need
the SMTP address there's a couple you
can use the SID of that account but you
can actually use SMTP primary SMTP a
principal name etc but if you can see at
the bottom here this is where we get
into the fund ready it is that looks
funky right
IMed so this is it a soap header that
we'll add to our soap request and it
indicates I want to connect using the
SID or using this SMTP address and you
have to actually put that when you
actually post the data to your so
service
so the other thing is that anyone here
familiar with autodiscover anyone ever
seen that right you connect to exchange
and it says hey this is your auto it'll
automatically discover everything it's
awesome
all right so pi EWS actually supports
this Auto discover but Microsoft's
recommendations are a little weird they
actually don't have an official way of
doing this they actually say okay you
can use in what they call an SCP a what
is the code yeah
so we basically correct create these
URLs that you see those for based on
your user name and password or based on
your your email address and you can also
use what they call an SCP which is a
service control point that actually is
registered in your Active Directory or a
juror or Active Directory and it points
to your exchange servers so if you have
40 exchange servers in 10 racks or
whatever it will actually make a record
in active directory called SCP but you
can actually just you know pass in that
user configuration and we're
automatically by default try
autodiscover so we'll generate these
URLs and actually you know try to
authenticate and we'll provide every
single version so there's nine versions
total of exchange and so we'll just
attempt each one so you can think five
times nine we're going to try every
single one of those throughout the time
so here's an example and maybe me so
we're actually going to auto-discovery
and we're printing out you know the
user's email address and then we we
identified the EWS URL and their
specific exchange version and this is
the actual soap response that you get
when you make that call and it says hey
I know who you are and I'm gonna print
all the attributes about that user here
pretty
they're pretty deep but the overall
structure of our request is this massive
let's see if we can find the the actual
XML and I'll zoom in here in a bit but
we have the envelope and then we have
the actual request header and in there
where say we have to specify the exact
version of exchange as well as what
action we want to take here we're
actually going to get user settings and
then we can also provide the connecting
sit if we wanted to impersonate but the
real magic is all in the body and this
is saying in the get user settings
message we actually are requesting these
settings we're requesting the internal
uws URL the the external the user DN so
on and so forth and so you can request
whatever attributes that you want or you
can just get them all pretty
straightforward but if you don't want to
use out a discovery and you're already
kind of know or if you want to use
autodiscover
and you can actually provide a very
specific you know that you're on office
365 then you can do that as well
you can you can limit the scope a little
bit but the other versions are office
365 exchange 2019 16 all the way to
exchange 2010 I can support 2007 but how
dare you ever beyond 2007 if you are
please don't tell me run so the size
autodiscover you know if you actually
know your exchange URL you could provide
that as well so a direct connection so
if you have you know your email
addresses or funky soup comm or whatever
I'm just making things up but you did
but your actual you exchange version is
going to be soup funky so we'll never
guess it right then you can actually
pass in that I that for your realm and
again you can pass in if you wanted to
actually impersonate
so all these objects that I'm talking
about are the user config and all we're
doing is setting up the how do we gonna
authenticate what are the reasons are we
gonna do autodiscover are we actually
going to impersonate or do we already
have rights if we do we're good to go
but all of those object that object
however we create it will actually be
passed into what they call a service
endpoint and currently these are the
ones that we actually support so we can
delete items we can get their mailbox
rules you can get all the searchable
mailboxes meaning whatever I'm
indicating with these are all the
mailboxes that I have rights to actually
search cool thing is that they'll
actually expand all distribution groups
found and automatically add them to a
list you can resolve the names and you
can actually search those mailboxes and
this is where the the GC stuff gets
alright so these are three situations
anyone recognize these and I can read
them off but as a security professional
when someone within my or get reports a
malicious message I want to find in
search all mailboxes for a similar
message right that's pretty a pretty
straightforward use case but as a
security professional
I want to active the hunt for malicious
messages so you have a list of my OCS
you have specific attachments you know
that certain keywords are in the body of
the message certain email sender or so
on and so forth this is where you're
actually you know going on hunting for
actual delicious messages in your
environment by the way don't click on
[Music]
as a security professional though I want
to find all the emails received today
that have any URL or any attachment and
I want to I want to know all of this
right that's pretty cool we'll show you
how you can do all of this so the
typical workflow that you'll receive
right for that first kind of use case
and a few others is will actually
retrieve all the emails and we can
actually we need to find all the
mailboxes within our org too and you can
specify a very specific set of users or
everyone in your work just kinds of
depends on your scope but then we want
to actually search all those mailboxes
and we need to provide a specific search
term that we want to provide and we can
actually delete all of those messages or
very specific ones and there's a couple
different options when we talk about the
so get searchable mailboxes this is all
about finding again all of those
messages in like I said you can actually
expand any found groups and they're all
contained within those mailboxes but
when you return each mailbox we can
actually search an ad based on this
reference ID this is the mailbox ID this
is a unique identifier for that
attribute in the mailbox database and if
you wanted to filter a list you can add
logic there as well but for the scope
it's mostly everyone wants to search
everything can you see that or is that
pretty small every time I like zoom in
it jumps that's still pretty small but
here we're actually just getting all
those mailboxes and it's returning their
IDs as well as some attributes like what
is there a some TV address what is their
name their display name so on and so
forth those are just basic attributes
for that for that item and so we can
actually go through and filter those if
we wanted to all right so the search
mailbox so we now if we actually have
our list of messages that we want or
mailboxes that we want to search we need
to back actually perform a search and
the one reason why pi e WS is extremely
powerful is it is faster than anything
else out there
you wouldn't think for this old protocol
of soap sidenote soap is actually a
standard by an RFC standard rest is not
at all it never has been never will be
but soap is and it has been for many
years but so give another that single or
a list of mailboxes of those reference
IDs we can actually perform our search
so we just need to import the from PI
EWS search mailboxes give it a list and
you can see here this subject account
that is actually our search term so I'm
looking at the subject for any emails
that have the word account on them
pretty basic we need to pass in that
user configuration object that we
created however we want to actually
authenticate and then our reference IDs
so whatever mailboxes we want to
actually search and then when we
actually print those we can we can
return a whole bunch of properties on
each message that it actually found so
this search query is all about it's a
hand or logic so we're gonna actually
run a search here and right now we're
printing out the body of the message
this is just the raw like HTML ramp we
have a conversation and index the
created time culture date time received
a time sent display info flags has
attachments importance I'm skipping some
it's draft his reason is submitted
last modified this is the raw mime
content and it's all based 64 this is
the entire message we have the OWI link
we have sender sensitivity set value
unique ash this is the base64 decoded
ROM I'm content with full headers with
full message body with with everything
that if you open that dot message or dot
EML this is what you would see inside of
it and we have some other attributes but
the entire list and I'm gonna kind of
slowly scroll there's many there's like
30 that you can choose and so you have
access to everything about that message
and if you wanted to get the attachment
so you can do that as well so to be a
link we have sensitivity sort value a
couple of side notes here for some of
these attributes is that the like flag
what did the user actually flag that
message why is that important all right
I mean that is you know that the user
interacted with it or someone did in
that mailbox and that they flagged it
for review so they looked at it they may
have thought it was suspicious who knows
you got to ask the user importance was
it sent at high priority importance it's
hugely important especially when you're
defending against BC and other type of
attacks and you want to know are they
trying to trick you into thinking this
is really really important message you
need to act now we also have again the
mind content was it read does the user
actually read the message or or not that
means that they open it and they took a
gander and two recipients and some other
match words but there's a lot of
attributes in here that are extremely
useful so the search syntax is all based
on exchanges with a call AQS and it's
advanced query syntax
and it's pretty straightforward but the
sightseeing the the subject an
accountant or example on the previous
side you can actually search a lot more
options so you could specify if you
wanted a very specific from user or
almost Raj you can just add a star to it
so some domain you know a star at some
domain to the CC field BCC any
participants the subject and you can
actually see that the product and
product development those two in that
bracket means any any of those two words
and not a full screen if you wanted the
full string it would be actually you
have to put it in quotes in your body
content attachment and you can actually
search for attachment names or
extensions you add any other kind of
there are some other properties but you
know also what filter by win-win like
time and time range and you have these
and or or not connectors woo we're gonna
page goon technical difficulties once it
[Music]
serve at this will I do everything in
markdown because it's fun oh man
we're just gonna go old-school so the
actual you know some attributes that you
can run is that this is one subject
account and body has the word clique in
it and it was received today right
pretty pretty straightforward search but
you can also say okay let's do subject
an account body click and the attachment
had the word fax in it somewhere
doesn't matter where it's at it had fax
you'd also specify a GUID from mr. robot
or from me and subject write any of
those keywords in the subject raised
upgrade warning expire so on and so
forth you can do subject office 365 but
not from the IT department again body
HTTP star is there any HTTPS or HTTP
links or FTP or whatever you want in the
body of the message or attachment star
any message with an attachment doesn't
really matter what it is we don't care
we just want them all so in that case
that's our third use case of you know
finding all messages that have URLs and
our attachments that have been sent
today pretty straightforward to do with
EWS
we can also specifically you know delete
messages and each deleted message has a
couple different options there is the
the heart delete that means it's not
recoverable and it is permanently
removed from the mailbox database you
can soft elite which means if you have
the yeah what do they call it then yeah
dumpster enabled you can actually
recover that message and you're
recovered messages I think is what it's
called or or whatever but you
actually recover or you can just move to
the deleted items pretty straightforward
to actually use this we just need to
provide a very specific message ID to
our delete item object or class and we
can actually provide a user config and
then we just print out the response and
it says hey we deleted this message
pretty straightforward so that's only if
you really need to actually delete a
message I'm actually gonna go back here
and look at these the this property
again so to actually print out some of
these and this is one thing I skipped
over you just access anyone familiar
with dicks or dictionaries or lists and
Python anyone from there when python
yeah I I just started on Python last
year or the beginning of this year
before it was all power show and I I
love it like it so if you haven't
checked it out I actually did another
talk at besides Springfield they did a
comparison between PowerShell and Python
side-by-side comparison so yeah if
you're interested in that let me know
but you access it by just you know
accessing that object and that
dictionary name of message ID or sender
or any of these other like mine content
so on and so forth alright so what are
the the great things here let's see if
we can actually expand this is that you
can add new endpoints really extremely
easy with pi uws i've made it in such a
way where it's it's extensible like last
night I actually realized that it wasn't
getting the entire mime content and
added that in like 10 minutes by by a
little Google and and some code but
overall how the actual service in point
you need to inherit a service in point
which is how it's going to actually do
all the soap requests in the backend
and you just just need to actually
provide what your soap message is gonna
look like and in here again we're
actually you know forming our our
message and we have the XML signifier
for every anyone here to use XML hate it
love it I mean it sucks but it's what we
get and you actually have the soap
envelope and in each one of these you
can actually specify what the syntax is
for that soap here you can see where it
says XML NS that's the namespace of this
XML document and a namespace is just
like what you see in C sharp PowerShell
whatever it almost every language is
they have namespaces of what you can
access based on their that definition
and there's just like the basic XML
namespace but we know that it's a soap
native space we can also provide the XML
s in SN T and this is the types so
Microsoft everything is based on a WSDL
file it's a Windows verb it's a web
service description language I think is
what it stands for Canada acronyms are
horrible but but it's a definition of
how although all the services that you
have available and what you what you
actually have access to and each one of
those actually has inherited types and
those types are the actual functions
that you can perform and they're
extremely detailed you also have the
different messages and those are
typically the message attributes that
you'll find the body as well as any
metadata around it and which a
closed-off or header or I'm sorry we're
starting a header there and we actually
provide a specific version of exchange
so whether office 365 and which was
actually exchange 2016 we have exchange
2019 2010 so on and so forth but we need
to provide that in the header to say hey
I'm trying to
okay here's my grids this is the version
that I know if you don't provide one
that is a valuable or a known attribute
to that exchange server it'll just say I
don't know what the hell you're talking
about
and then that a person a shoe editor as
well and here is where the fun stuff is
with the body elements and this is where
you can actually get very specific
details I'm gonna switch over here don't
look so this approach talking about with
the get item this is actually getting
all the attributes over me all attribute
or a mail message again we have our very
basic soap we have our header but the
body is where we actually define what we
want to do and here we're saying okay
get item here's the the type or the
namespace and the type that we want to
use and the item shape is going to be
okay we want to get all the properties
we want to include the mime content we
want to have the best body type there's
a couple different options for HTML or
plain text but if you use best it'll
just try to figure it out it's weird
that they actually he's best I don't
know of any other property that actually
it's like Oh best is work that works
additional properties is where we would
actually find the property tag and
anyone ever seen exchange msox3104t
is literally 900 pages of just yeah you
have a headache after reading him think
that they're they're massive but they
explain every single property type and
what those attributes and here we're
saying this is a property type of string
but the property tag is this hex code I
don't know what the hell that means I
just know that it points to the message
pointer of all message headers in RAW
format atom I don't really care why it
does that but it doesn't and then we
provide the actual message ID so this is
the very specific message ID you can't
all you can't store message IDs because
they change their there they're always
different I have to get moved if that
email gets updated if something else in
the mailbox needs that ID and just kind
of you know messes things around
especially if you move a message it's
automatically gonna be a new message ID
and all that so if a user touches it or
replies to it it's gonna have a new
message ID so you need to make sure that
you know if you're looking through this
you need to build in some logic to
actually you know use the same or find a
way to actually find that message and
not rely on the message ID you found
yesterday and then we're actually just
formatting the body with our variables
and we close that ourself header and
we're good to go
there's also anyone here just office 365
so I wanted to point out that there's
this other tool that I wrote it's
actually called graph fish gr Lucy this
will actually show the demo gods so this
is called graph fish so I'm horrible at
names I'm not a marketing person but I
looked at it because it's Microsoft
graph 2 API
and fish so I just kind of combined the
two words clever but but this is a graph
fish and this actually does the same
exact thing but using Microsoft graphs
API and but it requires oweth to
authentication if anyone's familiar with
OAuth 2 but it will actually let you
search delete move messages get inbox
rules while that all through Microsoft
graph API which is a rest-based based
API which you don't have to provide all
the soap and all that other stuff but
only for office 365 doesn't work for
exchange online or exchange on-premise
current future features again being
create a new search and the search
actually does it a little bit different
instead of with a search and pipe or in
soap and regular exchange it actually
will create a folder you can move
messages that you found in that search
but with graph it will actually create a
hidden folder in the users mailbox that
will contain all messages found and all
so once you actually create a search any
message is found after you've created
that search will be added to that folder
as well which is extremely powerful if
you have like generic rules then that
you want anyone with any PDF that has
fact
Senate or whatever you wanted to move to
this hidden folder user can't access it
but if you have rights again which you
delegate through is your active
directory you can actually hold those
messages and just keep a repository
across all your mailboxes pretty
powerful
you can delete that search which will
basically release those messages from
that folder you can actually delete a
message you can list all the mailbox
rules across your organization and this
is again super fast pretty easy to use
as well I this package is pretty
straightforward but the other thing is
you can actually move messages to a
different folder so not just deleted
items so if you just had like a phishing
hidden folder and your users mailbox you
can move those as well all right I'm
actually done a little early sorry for
the demo delays but here's the actual
documentation so pi uws is on the read
that read the docs that i oh it's pretty
detailed and then it explains like all
the class attributes and there's also
the repository which you can just go for
core you know clown however you want to
do it I wrote a blog about it for work
at swimlane it's swimlane calm slash
blogs I also did one for graph fish as
well and I have like a five-part series
series explaining a lot too if you're
ever interested because it's unmet eat
it's a mess
also a personal blog let's automate it
and on Twitter I'm MS administrator and
that means Microsoft administrator not
miss