hello I'm Steven or the title of my talk
today is reversing the reversing of the
tri station protocol it's mostly a story
about me learning about industrial
control systems ICS security which I had
you know a couple months ago know
nothing about and three T's in
particular triton try connects and try
station I'll explain what all of those
are shortly a little bit about me Steve
Miller I am a researcher at fire I slash
mandiant I used to do host-based
forensics now I do network stuff and
dabbling in ICS stuff I live in the u.s.
in central New York New York is a big
state I live in a little tiny town right
in the middle I've been playing the
battlefield 5 open beta know if anybody
plays that but my grandma thinks I play
video games for a living so it's
suitable and that's a picture of my
motorcycle BMW f800r yes when I was
riding through Northern California about
a year ago I am on fire eyes advanced
practices team we have a cool logo it's
an eagle shooting to desert Eagles so we
think we have a pretty cool logo anyway
that's a little bit about me this is a
just to get you into why I became
interested in security it wasn't this
but when I was traveling I was living in
Australia and my mom received a letter
of mail from the US Office of Personnel
Management and it was about a year after
the OPM hack went public they decided to
send mail to everyone which was nice of
them but the US OPM is manages the
records of all the government employees
so my entire security clearance and
everything I ever the government ever
knew about
is now owned by China you could probably
look it up on the dark web if you're
interested but anyway that's kind of why
I'm into security and for the next about
20 minutes I'll talk to you about Triton
which is a malware framework try konnex
which is an ICS PLC programmable logic
controller and tri station which until
recently was a undocumented proprietary
network protocol so this is really just
my interest in these subjects and I kind
of poked at them for a little bit and I
hope you will learn something I'm not
sure what you all have different
backgrounds in different interests I'm
not sure what you'll take away from this
but we will see
so Triton sometime around December 2017
I think fireEye released a blog about
the Triton malware framework it was
written in Python it was found at a
customer site in the Middle East it was
compiled in PI 2 Exe
and it was designed to interact with try
connects controllers this is the first
of its kind we live in an industry where
everything's like a total GameChanger
but I think this was this is kind of a
game changer so it's designed to
interact with try connect controllers
this is what a try connects controller
looks like there's like three of
everything triple redundancy controller
Tricon very clever but it's a plc and
this is a particular type of PLC which
is designed for safety systems so it
connects to sensors and if the sensors
see something they have logic that turns
a gizmo or a motor or a widget and you
know controls some industrial process
for safety reasons so that things like
oil refineries don't explode which is
you know I think we can all see some
value in that
so petrochemicals oil and gas power
pharmaceuticals anybody with a
industrial plant that wants to
have safety systems in a plant you might
have hundreds of these literally
hundreds and they could be all over the
place and they are networked of course
try station is for the purpose of this
talk will be the in protocol the binary
protocol that allows a PC running some
software to communicate with the Tricon
or try connects controller so everybody
got their try contract connects try
Triton try station we got a lot of
buzzwords here so I will try not to mix
them up so sometime earlier this year it
was like probably May somebody was like
you should look at the Triton malware I
was like wow that's cool
I didn't have anything to do with it
when my company responded in the
incident response and somebody was like
it uses a proprietary protocol I was
like what is proprietary mean I don't
know it there's like it's undocumented
and I'm awesome undocumented things are
my favorite things so I thought to
myself I became really like curious how
does this malware framework use the
protocol what does the protocol look
like I asked around nobody had any pcap
which is my worst case scenario there's
no pee cap so the big questions for me
and this was you know part of my job
right was to you know what the hell what
the hell's going on how does the try
station protocol work what does it look
like to what extent did the malware
authors reverse-engineer the protocol
yeah how did they do that and how did
they build a malware family that spoke
with this protocol and then to what
extent did they implement the protocol
network stack into the malware these are
important questions because if what I've
observed in looking at a lot of malware
is that when you implement a protocol
you have a very good chance of doing it
wrong and if they did it wrong if there
was any sort of error in their
implementation then we would be able to
detect that right that was a kind of
yes and that is true for a lot of other
protocols I was kind of hoping it was
true for this one so I wanted to poke at
that this is kind of what it looked like
in practice I stole this from the fire I
blog but you have a PC over there and
that's running some sort of you know
special software the try log exe is the
PI to exe compiled malware and inside of
that it has a bunch of scripts and it
talks to the controller and the purpose
of the malware was to communicate with
the controller the Tricon controller so
I had to talk to it had to reprogram it
or manipulate it and ultimately exploit
it and the purpose of this was to inject
a custom binary backdoor the controller
does not run Windows it does not run
Linux it has its own custom Orca texture
and firmware I'm not qualified to talk
about any of that so I decided to kind
of focus on the the networking protocol
so try logged exe to break it down a
little bit anything that's written in
Python and compiled with PI THC can
actually be uncompelled when you compile
you have the option of specifying some
zip files but you know developers and
malware authors being lazy like me used
all the defaults so all the scripts were
you know just compiled there and inside
of the main executable stored inside a
zip file called library dot zip and this
is what when we uncle pile it kind of
breaks out pyc is a Python bytecode but
that's actually pretty easy to
uncompelled to break down the the plain
text scripts so untidy Exe a nun
compiled six these are on github you can
go download them and do this yourself so
this is kind of what the logic looked
like so back in this one I looked at all
these so it's like two hundred and forty
Python scripts in here most of them are
just default Python stuff
support different types functions but
there were a bunch of them that
obviously seemed very specific to the
malware and specific to the
implementation of tri station so these
are the ones evil evil stuff dot PI
that's the main routine the malware that
was what was going to inject the
controller with the backdoor and then we
had all these other things but you could
see it's a very nice cascading logic
evil stuff imports TS hi
TSI imports TS base and so forth and so
all the way down that chain was where I
decided to start looking at the plain
text content of these scripts so I'm
looking for the implementation of try
station when you look and you want to
document an undocumented protocol there
a couple things you gotta look for you
want to look for magic values you want
to look for counters you want to look
for timestamps padding there are a lot
of different things but when you want to
document a protocol you want to look for
the structures the major structures and
there's a lot of other things but those
are kind of the basics that you go into
and this all turned out to be much
easier than I thought
so when I opened up see names
TSC names pi what was apparent it was
that it was just basically a bunch of a
race with string text and these string
texts were attached to kind of integer
values and you get the sense that it
serves as a library for something else
but then when I want to go look at where
these arrays are referenced and some of
the other scripts we kind of have to
dive into some of the functions so we
can see that this is an array called TS
c st and that t sc/st is referenced as a
it's a dictionary result so used as a
library of strings for TCM result and so
that was the function i wanted to go
look at next so we look at that and
that's actually in a different python
file and right here we're already seeing
that they're breaking down the try
station protocol in some way when we
have a bunch of UDP data coming back we
see that the
first four bytes are broken into type
and size so pretty pretty
straightforward and then the last two
bytes are a CRC so it's right there it's
really easy we didn't have to do any
reverse engineering we did it the easy
way
so that that's pretty cool and if you
want to document a protocol you can do
it in the you know kind of C structure I
call this the the pretentious way I just
kind of penciled it down on a napkin and
then what I wanted to do is is dive in a
little bit further and then going into
TS result we also saw that the next
bytes after those first four bytes
this is packet type five and in that
first array I showed you type five was a
command reply so just for command reply
packets we now have these additional
fields broken out in the data so one two
three four five six seven eight nine ten
more bytes are unpacked there so now our
understanding of the of the protocol has
grown a little bit and we can document
that and just based on the variables in
the in the code we could get the sense
that you know this stands for something
or the other you know size and checksum
or unknown you know anybody label
anything unknown probably probably not
maybe but anyway so this is the command
type structure and that's all well and
good Oh what was that
oh and okay the commands and then so
that particular byte was referencing a
different array in that first python
file and it has all these you know all
these like different about all these
different strings for what the command
reply is so that's really cool we're
seeing that the malware authors had a
pretty good understanding of this
protocol they have all these message
types yadda yadda yadda right so I
wanted to validate
that the structure of tri station that
they they knew was actually true I want
to see if they made any errors or
anything so obviously I googled arounds
my ask people for p-cad nobody had pcap
for me it was a sad day I found a paper
by the coordinated science laboratory at
the University of Illinois
urbana-champaign that's a mouthful this
is a really cool paper it was behind a
paywall I did Google long enough and
found like one of those PDF reproduction
sites where they try to sell it to you
for a dollar but you can take the screen
cap of all the all this stuff and I read
the paper and there were a couple of
things that these researchers did one
the Tri Station protocol which is
undocumented is similar to a documented
protocol by the same protocol author
called tsaa and the researchers used one
protocol to make inferences about how
the other protocol worked and they also
were able and they did that and then
they did some testing of their own and
they documented the Tri station basic
structure and so this is what it was
like
which validated a lot of what I saw and
the Triton framework and what do you do
with a protocol structure like once
you've got it first thing that I want to
do is is dissect some pcap but I didn't
have any pcap but they printed out some
bytes in their paper so I kind of
noticed I was like okay if that's a real
try station packet from their
experiments and I also noticed that we
were talking about command Type O 5 and
I noticed that everything before the o 5
was actually probably UDP header data so
I tried to OCR it but when you take a
screencap of this you get a lot of oohs
instead of zeros in an OCR so I typed
all of this into a hex editor very
slowly I know it's terrible if I'm
embarrassed that I had to do that but I
did and I stripped off the the UDP
header bytes and then I used
some cyber chef to print it out to like
a fancy text file and then I used text
to pcap to generate my own pcap and then
I went on a journey of opening it and
Wireshark there's no parser no parser
existed for this publicly no dissector I
created a really terrible Lua dissector
mess up on my github it's awful but you
can go look at that and dissect that
pcap and then about a month later the
news Oh me guys really showed me up and
they produced a totally badass dissector
for this protocol I recommend you you
you go check it out but that's one thing
you could do once you understand the
structure of a protocol this is that
same key cap that I typed out but
dissected by the Nozomi try station
dissectors as you can see it breaks down
the protocol into you know the channels
you know the data length this is the
results of a custom protocol they put
the data length twice I don't know why I
probably wouldn't do that if I were
designing one anyway there are a lot of
a lot of things about this protocol that
are weird
but we also have to remember it was
written probably like 20 years ago what
else can you do with a protocol
structure you can observe the traffic in
real time if you know kind of what it
looks like I tend to favor snort because
all my sensors use snort and it's really
easy language to kind of write in for
packet matching so if we knew that value
one type you know the first two bytes if
the type was one the connection request
and we also know that the you know the
probably the smallest size packet is six
bytes we can kind of guess that type
size and a CRC and because we don't know
CRC we could pretty much calculate that
CRC 16 and then we could write a very
simple snort rule that does a Content
match for that looking for UDP traffic
so what this would do if you push this
out to a sensor it would you know do
TEKT you know if there were a tri
station connection requests on UDP on
the default port and snort rules are
really good for identifying traffic that
you want to observe BSI in Germany they
did some awesome research on the Tri
Station protocol this guy Thomas Schmidt
led to charge in creating a snort rule
set for you know intrusion detection
systems it's free it's accessible to
everyone and again you know before a
couple months ago nobody had any
visibility into this traffic so this is
like a super awesome research project
that actually delivered some value to
anybody who owns Tricon controllers this
is what that snort step looks like and
it depends on a bunch of configuration
stuff but it's out there you can google
for it it's pretty straightforward um so
how was triton developed now that we
kind of like looked at all that stuff
when we D compiled the Python bytecode
on compile six kind of gives us a guess
at the compile time I'm not sure a
hundred percent how accurate this is I
assume it's accurate but there are about
two hundred and two hundred or so
Python bytecode files in librarianship
and we my boss was my boss now he
tweeted out this picture I tried to make
a graph and I was like you know what he
already did it so but this is what the
timestamps on the Python byte code
inside library dot zip kind told us and
what's interesting is you can see that
there are a bunch that were in June 2016
and that what we see were a couple of
edits later in 2016 of November and then
2017 we started to see some more stuff
in 20 August 2017 is when we started to
see the last modified times of the Tri
Station implementation so we know that
this malware framework they spent like a
good solid year on it maybe more I would
guess more and we can tell that from the
Python metadata
there are some discrepancies in the
Triton malware code and I became really
fixated for no good reason on this there
were a couple of misspellings in that
first library of arrays that made me
very curious and there also a bunch of
strings in there and I was googling for
them everywhere and I couldn't find any
trace of any of this
there were most of the train code is Mad
Decent yo it's pretty solid has nice
cascading logic there's good annotations
everything the variables seem well named
so like to look at that library of weird
misspellings and then the rest of the
code seems to be pretty solid it was
really inconsistent there's nothing that
drives me more insane inconsistencies so
like here's an example of some of the
some of the arrays that I was talking
about you know Alec a label here's one
that just driving crazy is just a case
inconsistency
these are small things that I've become
very fixated on and I'm wondering why
these arrays are so different from the
rest of the code reponse there were
probably a bunch of these and I don't
know there were probably maybe 10 or 12
misspellings so I'm like googling around
for these and I couldn't find anything
so obviously I have access to virustotal
intelligence does anybody use virustotal
intelligence a couple people there's a
reason that I go to virustotal
intelligence and we heard from the virus
bay guys earlier is that a lot of people
a lot of products will submit files
willy-nilly to different types of things
and I basically look at this as a file
repository for both good and bad it
stores everything I'm expecting that a
bunch of people submitted some stuff and
I want to try to look in there and see
if there's anything that relates to the
Triton malware inside a virus total so I
go on a retro hunt and a retro hunt has
its limitations because it only goes
back on files submitted within the last
six months so I came up with a big fat
zero
which is an extremely disappointing
result for me because I wanted to find
something and to work around the
limitations of virustotal we have a tool
called awesome possum awesome possum is
an internal tool we take all virustotal
submissions and we index all the
metadata because we can search back
forever and we throw in a big log a big
log and elasticsearch and all that but
one of the interesting things an awesome
possum is that I can search by
submission names back for ever all time
so anyway I got a sense in documentation
that I would be searching for TS 1131
exe and awesome possum
that is awesome presented a bunch of
results for me and we can track the
submitters and see all that stuff so
that was pretty cool so I went on this
kind of hunt to look for source files
and first thing I did is try to run it
in a VM and I obviously see that I'm
missing all these kind of dependencies
and I did this for a while I just like
kept looking an awesome possum to see
files submitted with that name I
download them and I try to reconstruct
this package eventually I went to
dependency Walker and got a huge list of
DLLs that I thought were related to the
legitimate try station software so all
along we had this narrative that the
Triton malware was built with
understanding based on reversing
legitimate try connects software and
that was a narrative and nobody had any
any proof of that and I wanted to go
find some proof so anyway a bunch of
things that I didn't have and I was able
to finally get to a file called lag evn
DOL that had a parent file data 1 dot
cab that had a parent file and now I'm
into some like juicy data right
try log V 4.1 and if you remember the
original Triton malware file mimicked
trilogue it was named try log dot exe
presumably this is related to the Tricon
controller type of software so anyway I
opened this up
I start poking around in dataone cab and
I look through all these dll's and
there's obviously one I'm interested in
the communications so this one really
stood out to me calm anything calm DLL
named very cleverly
but you could see it's a DLL that
supports Tricon communications interface
this is a written in Visual C++ and
started dumping string tables and I'm
starting to see things that are pretty
familiar to me you know all of these
like okay that looks like kind of like a
thing and then here we go is like we can
see the inconsistencies all these same
strings and if you looked I probably
spent a hundred hours looking at the Tri
station or the Tri Triton strings and
the moment I was able to look at this I
was like okay 100% know that all of
these strings in this string table came
from a legitimate try connects DLL so I
was like pretty happy with this result
and everything everything is in there
and it just goes it goes backwards start
download chain start download change all
that all the typos are in there and so
then I went and collected a bunch of
DLLs and I could see that where was it
try comm DLL they had these C++ string
tables in there all of these dated back
like decades right so I'm looking at
some library it was written ages ago
never updated never maybe not Qaid to
well I don't know and I was trying to
figure out exactly what software package
was probably reverse engineered I don't
think I had any really good results from
this but there is an older version of a
Tricon controller called a trident and
these both have you know different
branches of software that we're
supporting the communications for these
controllers in any case there's a couple
of things that are still interesting
about this is that
this is from the Triton malware code
there were some annotations in the in
the Triton Python scripts that were
above what they had reverse engineered
from the Tricon libraries and you can
see that like these unknown values in
this string here in particular are
pretty different from you know the angle
brackets kind of characterization for
maybe unused integers in particular
places so it became apparent to me that
the Triton malware developers although
they reverse engineered some of the
legitimate software they still had to do
a lot of experimentation to figure out
the network protocol and we know that
because not only did their malware
framework take so long to develop they
had a really hard time using it in the
wild and they ultimately got caught
before they were able to do too much
damage so you know again six months ago
or a year ago we only had like the
suspicion that they reverse engineered
something we're able to finally provide
some kind of conclusive evidence to
support that um so that was kind of a
win for me and then a year ago we had no
the world had no visibility into the Tri
Station protocol was a black box
protocol undocumented we have some
documentation of it
the nozomi guys probably know more than
anybody about this protocol we know that
they had a hard time using it in the
wild because when we went to the
engineering workstations that was
launching the Triton malware we could
see from the user Journal which is also
our USM Journal also called the change
journal I think Universal sequence
number this is an NTFS feature that logs
file names and change time but we could
see they uploaded like a hundred
different versions of the Tri and
malware and they also uploaded like a
hundred different other scripts and a
hundred different other utilities they
had a hard ass time getting there and
then making this work and I think the
expiry
imitation that they were doing
ultimately led to the outcome which is
they caused one of the Tricon
controllers to enter a failed state that
fails state which is monitored by the
industrial control systems monitoring
software induced a shutdown of this
entire industrial process and as you can
imagine if you run a big industrial
plant you're like losing a million
dollars a minute
every time your industrial processes go
down so that's what got them caught and
last but not least it's really important
to acknowledge all of the public
research that was done on this I think I
read everything for the most part
there's a couple of good fireEye blogs
on this the zombie networks guys did a
lot of research they purchased the
controller on eBay and reverse
engineered some of the try station
software as well I think doing may be
exactly what the Triton malware
developers did dragos guys have a lot of
good stuff Accenture has an awesome
white paper on this the guy behind
midnight blue labs ics-cert CSL at
University of Illinois there's a github
repo with all of the malware and all the
decompiled scripts and then all of the
resources and all of this is important
because ICS if you think everything is
broken IT you got to you got to check
out ICS it's it's a different world
everything's like 20 years old and so a
lot of that's just hanging out there and
if you're interested in ICS there's no
better time than right now to get into
ICS security and with that I will take
any questions I think we got like a
minute left
yeah thank you like you said you have
one minute left so there's at least one
question right don't ask me who it was
because I can't say