– Welcome back to part two Somebody brought up this picture This picture is in no way representative of hackers It just isn’t Hackers are people who look just like all of you If they were walking down the street, you’d have no idea who they were They don’t wear hoodies, they don’t sit in front of green screens, that’s the media And that’s why I use this picture So that we can have a conversation, because at the end we’re gonna have a much deeper conversation about what hacking really is and about our response to it That’s just somebody’s artist conception because it looks dramatic But it is in no way representative of what hackers are So part two For those of you who missed part one, shame on you These are the things we covered You can go back and look at the video and you can get the context behind them Now, the last one generates a lot of controversy, and let me just go ahead and tell you, this is not bug bounty hunting This is gaming the system Writing a minivan by putting vulnerabilities into your code is not bug bounty hunting So the art of war Sun Tzu said that all battles are won or lost before they’re ever fought, and that’s absolutely true It’s in the planning, it’s in the intentions One of the things that drives me nuts is every time I hear a CEO And I heard a podcast the other day with the CEO of a company called TalkTalk which got penetrated multiple times It’s in England Saying, the bad guys only have to get lucky once We have to get lucky every time Which sounds really good, sounds really intellectual, but it is total garbage It’s not about luck If you are depending on luck to protect your systems, you are going to be sadly disappointed when you find out you don’t have any luck In fact, any luck you have is just bad If anybody’s got good luck, it’s them What this is about is about intention It’s about our planning It’s about the way we go about constructing software in a way that makes it more resistant to people attacking it So let’s start with static analysis tools Let me say from the outset, I love Core Guidelines, CppCheck, Clang Tidy I use them all the time, probably every day But I tried just as a test to go through and see which ones of the pieces of code that we saw in the first half would they catch And with the exception of one, which is the variadic function, which is probably the least offensive that I have in there I had to actually go turn that on That’s actually a specific rule in Clang’s static analyzer So the reason why these don’t work very well for security is they’re not security focused These are the kinds of things that’s for correctness of your code Which correctness for your code is the foundation of writing vulnerability free code, but it’s not the end of it So if you’re relying just on these to protect you, they’re not going to work So there is a series of static analyzers that are specific to security So it’s a lot like, in fact, I just had a conversation with a gentleman during the break where we were talking about the internals of this These are much more sophisticated They are going and looking for, again, patterns But the problem is that, one, they can produce a lot of false positives, because the bugs that we tend to write are not exactly easy for a static analysis to find So what happens when we get a lot of false positives? We turn it off There are tools out there One is called Coverity So these are commercial tools, Coverity, Sonar PVS Studio is upstairs, so you might want to stop by and talk to them about what they offer They do not catch everything They won’t catch, in fact, some of the, a lot of the issues that we went through in the first half, they won’t catch CERT also has a thread safety analysis piece that’s a part of Clang which is very good There’s also some experimental Rose Checkers is kind of hard to use You have to use their VM and it only works with their compiler Some of these other ones are really, really, really experimental Part of the problem with static analyzers is they’re only as good as their rule set So they’re specific and then they have heuristics built on top of that It’s kind of like virus scans Virus scanners look for patterns and then they’ve got heuristics, but in this case, they’re not going to catch everything Because a lot of that behavior that makes our code insecure is beyond their ability So for example, environmental settings, third party libraries, things that they really can’t see into So these work for are you using string copy?

That’s a pretty easy one for a static analyzer to get So now we have, and somebody brought up dynamic analysis tools So there are some very good ones out there What these do is these find vulnerabilities in running code So what you’re looking for is you’re going to run these when your code is fairly well along It’s going to go out, it’s going to find out whether you are overwriting memory It’s going to find out whether or not you hit undefined behavior, whether or not there’s threading models built into it So these are the kinds of things that you need to have running code for them to work on And they’re quite good So the place, though, where they really shine, at least the places where I’ve seen them really shine, is in your performance, stress, and scalability So you’re going to get to a point where you’ve got a pretty good code base You’re now doing some deep testing You are pressuring the system up to its limits to see how it behaves, to see whether you can break it, and that’s where you’re going to find things like threading issues We can all get past, in fact, we’ll deal with some of these later You can get past threading issues just in your functionality tester and in your regression testing, but it’s only when you pressure that system up that you find it So fuzz testing, somebody else asked about fuzz testing Fuzz testing is, remember we talked about interfaces So I have an interface and I’m taking a certain amount of data into that interface, certain types I want to then go and see what happens when I put nonsensical data into it And then I want to go and put a different type of nonsensical data, so I’m constantly rotating that data, changing the data to see if I can trip over things like buffer overflows, things where you have just simply taken a value, slammed it right into an enumeration, and then you wind up having a failure So the best ones that I found are OSS-Fuzz, libFuzzer, and American Fuzzy Lop Now, one of these found the heart bleed vulnerability and I think I watched Chandler do this one time on a video where he actually used fuzz testing to expose the heart bleed vulnerability So the heart bleed vulnerability was just in the It was a mem copy overflow It’s an unrestrained mem copy inside the TLS heartbeat inside of open SSL Most of these, though, if you want to use them require you to be open source So the best ones want to help the open source community So I don’t have price ranges on things like that, but they are very good One of the problems with fuzz testing is it can be rather a challenge to deploy it It’s not simply a matter of okay, here’s my IPC interface, please go and do your work What you have to do is do it almost at the class level If you want to see somebody who did a very good presentation, go back to C++ now early this year, and there is a, Marshall did a whole series on how fuzz testing works And you’ll see the complexities of it It’s almost like TDD where you don’t really just get to give it an interface and it figures it out It has to, you have to program it to do this One of the things that’s nice about these is they use a lot of genetic algorithms and heuristics to be able to learn as they go on your interface So it’s very good for going back and doing this over and over again as you’ve got code that is getting more and more mature So this is where the situational awareness from the last talk comes in If one of the parts of the situation it learns is I’m not testing for bounds checking, I’m not testing to see if I fit within a range And this is a good place to go and test for those kind of situations in an automated feature and fashion So penetration testing A lot of companies have penetration testers come in What this really is is this is where you’re actively looking for vulnerabilities If you go and you watch a penetration tester work, they know the patterns And they’ll just walk those patterns They’re gonna look at security They’re gonna look at can I send stuff to you and you’re gonna process it when you really shouldn’t Have you got everything patched? So penetration testing is a way of looking at your system from the outside and seeing can I get into that system The problem with penetration system and testing is that it requires a significant level of security experience You have to know what to look for You need to know the patterns, because those are the first things you go after And some of the tools we use are the Metasploit Framework, which is a very good, it’s an open source framework which we go out and every time there’s a new exploit, let’s go plug it into the system so we can go test for that in our environment And then Nmap is also very good because one of the things that I want to do, if you set up a network, one of the first things they’ll do is turn off ping from the outside And the reason why is you don’t want people mapping your network from outside your firewall Nmap allows you to go in and look for attack surfaces, look for things that they can go after

So these are very good penetration testing tools So a couple of other options If you don’t want to get into penetration testing yourself, one is you can go to Black Hat and you can find a company to do it, which is a pretty good suggestion Defcon, they’re not quite so much The other thing is let the hackers help And I want to qualify this in that what I’m suggesting that you do is you set up your system so that it can be attacked so when you watch people attack your system, when you have to go and do a response investigation to find out, so you do your incident response, you’ll find out how they got in, so here’s the thing is Don’t put it inside your firewalls or on any network segment that you do That would be bad because if they get in there, then they’re in your network Don’t use your real data That’s happened before where people went and we call them honeypots They set up a honeypot except the problem was they used actual data and then somebody breached it Imagine that And then they had the data So what you’re looking for here is I want to go take my system, put it in a vulnerable state outside of my firewalls I want to let people hack it And you will be stunned at how quickly people will attack your system It is breathtaking the speed that people will go after it And the really creative ways they’ll figure out ways in and where they’re coming from We all tend to think that it’s somehow foreign nationals that are wanting to come after systems It’s not, I mean, you’re gonna find lots of US IP addresses are coming from American hackers So this works Penetration is for something that you tentatively, something you do sort of late in the cycle when you have a full system So the best practice here All testing is asymmetrical There is no silver bullet There’s not one single thing that if you do this thing I can say, okay, we’re good We’ve taken care of it It takes all of these things and some of the things that we’re gonna be talking about here in just a minute So has anybody ever done threat hunting or we call it threat modeling too? Anybody, so a couple of people Okay, so the answer is you all do it Every one of us We do threat modeling every time or threat hunting every time we drive down the road There’s a car approaching an intersection You’re looking at that car Is that a threat? Is that person going to come through the intersection and crash into me? If you’ve ever gone to a city you’ve never been in, a large metro like New York, it’s after dark, you’re walking around downtown, then somebody’s coming towards you, you are threat modeling that person Is that a good guy or bad guy? So we do this instinctively, we just don’t think about it And things like darkness, isolation, insecurity, vulnerability Any time you’re operating in a situation where it’s unknown and you have these kinds of things going on, you are threat modeling So the reason why we do threat hunting is because it’s no longer enough to just let the automated tools take care of it And we’ll go through a little bit of an exercise here and I’ll show you why it’s no longer enough But we need to understand the nature of those particular threats So in order to understand this, I want to talk about intrusion kill chains for a minute So these come from the military The military has basically a way they go about prosecuting a war You find, fix, and follow your enemy So I’m gonna find him, I’m gonna keep an eye on him, I’m gonna follow them where they go, try and keep them in a defined position, then I’m going to come up with how am I going to attack my enemy, and then I’m going to actually do the attack, go back, assess it, and lather, rise, repeat That if you were to take the prosecution of a military campaign, that’s basically what those steps come down to And the same thing happens with hackers So you have a threat actor The first thing they’re going to do is they’re going to do reconnaissance Now, that could be reconnaissance where I’m probing your external interfaces I’m looking at your firewalls I’m trying to see what the reflex is of it But I will tell you right now that’s actually really old school Because the problem is you will get alerted to that There is actually a better way to do this And that is doing reconnaissance on all of you So let’s say I want to go and I penetrate and these group of people here are people who work for the same company, and I want to penetrate that company One of the easiest ways for me to do it is to get somebody to click on a link But in order to get them to click on a link, I need to send them a link in a way that they’ll click on it So I go out and I go to LinkedIn And I find that you all work for the same company So there’s a couple things I’m looking for What are the relationships? I don’t want one degree of separation I don’t want the guy, I don’t want to send an email from the person you’re sitting next to, because I know the person I sit next to and I know how they sign off their emails But the person down in another group, which is a couple of degrees of separation away, I don’t really know him

I may know of them, but I don’t know how they sign off their emails So when I want to sign that email from them, I don’t want you to know that it’s not them So if your name is David and you always sign your name David whatever when it’s somebody you don’t know, that makes sense But when it’s your friends, it may be Dave So if one of your colleagues that you work really closely with winds up getting an email from David whatever, they’re going, why is he giving me his full name? So this is the thing where when you do reconnaissance, you have to understand the people And we’ll see a case study where this happened So I want to know who you work with Who you work with that’s sort of a couple of degrees of separation I want to know your email addresses A lot of companies have email addresses that follow some standard First initial, first name, last name So if I know three of those email addresses, I can figure the rest out And then if I want to test that, I just run it, send an email to their server, and if it’s wrong, it’ll bounce back an error So now I can reconnaissance the people But the other thing I want to look for is I want to look for keywords If you’re using a Windows 8 EC and you’re using Exchange and you’re using Outlook, I’d like to know that because there’s entire classes of vulnerabilities that I don’t want to use against you I know exactly the ones And so in your, when you do, for example, in the profile that you put onto LinkedIn, he’ll give me some key words, he’ll give me some key words, she’ll give me some key words, and I put them all together So if I know you’re using Apache, Apache Struts, and I see somebody mentioning Linux, it’s probably that your web server is a Linux web server Why, because 90% of all the Apache installs are on Linux web servers So now I can focus on the kinds of exploits that I’m gonna use it And that’s the second step, weaponization What’s the zero day vulnerability that I can use against your system by just knowing something about it? So when you go look at your LinkedIn profiles and you look at the LinkedIn profiles of your colleagues, how much data are you giving me as an outsider wanting to attack you so that it not only helps me do a better job of reconnaissance, but it also allows me to pick the right weaponization Delivery, delivery can be anything from I send you a USB stick Hey, there’s this great USB stick We’re sending out free trials Vladimir Putin did that one time So they had the G20 summit and they sent, he sent swag bags You think we’re the only ones who get swag bags He sent swag bags with all the other leaders except there was a USB stick in there The USB stick was loaded with malware It was wonderful, it was really brazen I thought it was awesome But that’s what happens They’ve run study after study They go on college campus, they drop a bunch of USB sticks, somebody picks it up What is the first thing they do? They go plug it in Well, I was just trying to figure out who it belonged to Okay, it doesn’t matter, you just infected your computer So that can be web, it can be I give you some great deal, I give you an email Anything that allows me to get you to click on a link you shouldn’t or puts something into your system you shouldn’t So exploitation, that’s when the actual vulnerability goes into your system Installation is where now that I’m on that box, I want to move laterally within the system So I’m going to be installing a bunch of other software that you don’t know about and that you don’t want on your system Then command and control, that’s the point at which I now have full control of your system So I’ve got some sort of conduit back to my command and control server Now I’m in full control of your systems And if you get to step seven, which is action and objectives, you’re owned They’re in the system, you’re done They’re going to be running the things against your system that they want to and there’s very little you’re gonna, you probably will not even know that they’re there So Project Aurora That was in December and January of 2009 and 2010 So they picked that time because there’s minimal staffs to defend the networks They ran exactly the exploit that I outlined They went, they looked at the staff, they crafted an email that even if you’re in InfoSec would have looked perfectly legitimate They sent that email out Somebody clicks on the link The first thing that happened is it had a zero day exploit for Internet Explorer, got outside the sandbox, installed itself into their environment Then it opened up an SSL connection back to a command and control server, downloaded another dozen pieces of malware, installed that, and began moving laterally within their systems And what they were going for specifically, Google was a little more than just the source code, but they were primarily interested in source code, which was if I can get Google source code, if I can get Adobe or this source code for Adobe Flash, Adobe Reader, any of these products, now I can have got the source code and I can go find zero day exploits in the source code which is what I prefer And that’s exactly what happened We found a lot of zero day exploits cropping up right after that

What’s interesting, and this is why we think it’s a nation state, and attribution is where we go and we attribute it to a company or a country or even a company They burned a lot of zero day exploits doing this This was a massive attack It’s the kind of attack you typically see in the military, which is why this was so stunning is these were all civilian companies They were getting hit along with defense contractors and we had not seen that So threat modeling is a response to kill chains The first one we have to talk about is the scope If you make the scope too wide, you’re gonna get buried in information If you make it too narrow, you’re gonna miss things So one of the things I suggest to people is you do it as an iterative process You start small and work your way out or go deeper into your systems Because what you wanna do is you wanna catch the easy stuff first and then you wanna begin moving out and catch the harder stuff that hackers will probably find for you Model creation What we care about is where data moves and where it pools You’d be surprised the number of times when people go through threat modeling sessions and they find out, it’s like, I didn’t realize we were keeping that data You want to know where the data starts, where it’s consumed It should not just appear out of the ether You want to have, you want to know where all that data goes, and you’re gonna focus on the data flow Then threat identification is we’re gonna talk about trust boundaries here in a minute, but what you want to do is you want to know where are the threats And there’s several good models We’ll talk about STRIDE here in just a minute and go through a short exercise with that But what you want to know is you want to know where we’re holding data you don’t need, who’s got access to it So this is information in your threat modeling that you wanna go and think about and that’s probably the strength of threat modeling is when I have to threat model my system, you’d be surprised the number of people where you’ve got very large groups of developers This group worked on this area, that group worked on that area Now that they’re all coming together, you’re having these moments of, well, I didn’t realize the system worked that way because that’s not a part of the system that they are used to seeing Threat classification How easy is the exploit? How much damage can I do? What can I get to from this? And the mitigation planning Is it already mitigated in some other way? Is there an inexpensive way to do this? So you’re asking what, basically you’re prioritizing at that point Does this go into our backlog if you’re in agile or does it go into for the next PI or is this something that becomes our technical debt if it’s low enough And then finally, validation You want to go through and follow up and make sure that you actually fix the problem and that’s usually with some sort of testing So STRIDE is just an acronym These are not all the things, but it’s probably the best place to start So spoofing, which is can I masquerade as another person? Can I get you to give me data because you think I’m one person when I’m not? Tampering is just sabotage Pretty straightforward Repudiation is can I act without evidence of action? That’s where your logging comes in Can I go and do something on your system that you will then not know is going to happen? And the exploit that we’re gonna run here in a little bit, that’s exactly the problem is that you can run that exploit and you’re not gonna see it Information disclosure is just the exfiltration of the data Can I get to your data and get it off the box? Denial of service, can I render it inoperable? And then elevation of privilege is gaining elevated access So trust boundaries Trust boundaries are how we define risk Remember at the very beginning, I said, okay, I have a trust boundary, my perimeter security Everything outside is untrusted Everything inside is trusted And that’s how we assign risk We want to know whether or not I’m going from one trust level to another And we have the concept I mentioned earlier of zero trust environments And we’ll see how that affects the way we think about this So trust boundaries occur when I cross the level of trust I go from the outside to the inside I go from operating as a non privileged account to now operating as a privileged account That’s a trust boundary So here we have just sort of a remote security system Let’s say it’s a remote DVR This could be on anything It could be on a drone It could be in a car, could be in a factory, could be in a secure environment In the center here, we have sort of the working parts You’ve got the AV processes, then you’ve got some data and configuration and you can upload that information to something down here on the bottom, which would be like an on premises server Or you can come over here to the right and you’re

uploading it into the cloud and then you can access what’s on the cloud through a web server You would probably also want some sort of command and control for the on premises So you’ll need some way to view data, some way to interact with it And then you would want some command and control here on the device itself How do I configure it, how do I go and look at data maybe in real time? So we have a reasonably complex system And this is a good place to start without getting too deeply into it So now the question is, where are the trust boundaries in this system? – [Participant] The web server – Yeah, web servers Where else? So minimally, our trust boundaries are anywhere we cross a physical boundary So our mobile down here could be a cell phone in one case It also could be a UHF VHF transceiver in another case So I’ve got a drone flying I want to be able to configure that from a distance You have web browsers that can run command and control against the, to actually turn it off, turn it on, those kinds of things You have a web browser that can connect into what you have up here in the cloud You’ve got data moving to another server somewhere on premises So this could be something as simple as the drone lands and uploads all of its data Any others? Yeah, great So around those data stores Do I implicitly trust the AV processes? No, because these, we’re talking about zero trust at this point We don’t implicitly trust the AV process We may be doing encryption at rest and that’s enough to trust the process, because if the process cannot authenticate in order to do the encryption at rest, that’s where I’ve tested to make sure I have verified the data and I validated who was sending it to me The same thing with the data being flowed out We’re talking about exfiltration, the data being flowed out of that server So we need to make sure that the data is talking to the right server So as you go through examples like this and you say okay, where are the trust boundaries, what you have to ask is where am I moving data? And then there’s one at the top That’s the command and control So if I’m somebody who wants to penetrate a system, one of the things I might want to do is just turn the system off And keep turning it off so that it doesn’t operate Maybe I’m trying to sneak into a plant So I can now go turn that system off Now there’s no record Your repudiation is now done There’s a way for me to get in and you have no way of proving that I was in So let’s look at information disclosure We’re just gonna do one of these for your day Normally what you do is you go through all of these You’d also come up with what creative ways you can go and you can look at ways to penetrate this system, get data out of those So information disclosure Where am I worried about information disclosure? It’s easy, just call it out Well, where’s the information? It’s the data stores So I need to make sure that when data is being accessed from those data stores that I’m authenticating the person trying to get that information So in this case, you’ve got data stores that are being accessed by a web server here It’s being accessed by a download program there and this is being accessed by But what about the static content? Static content is data too And static content is used to interact with the data stores at almost every level through the browser and then through the website (participant speaking off mic) Well, so the question is, are you partitioning your data stores or are they just big blobs? The question more is am I securing them? Am I making it so that somebody cannot come in and replace my static content with somebody else’s static content, with their static content, basically, which would give them access to my system And then there’s the web servers themselves These are the things in three instances they have direct connections to the data So what have I done? We mentioned, okay, I know you’re using Apache, Apache Struts, and you’ve got a Linux environment So now I have an entire set of vulnerabilities that I may possibly be able to use against them and I would use it against these if I know they’re Windows versus Linux

So those become the key So when you’re going through information, you have to think about every link in the chain that talks to that data That is, the data itself and how it’s accessed, the web servers which are accessing that data and in those cases, in three cases it’s the web servers, and then the static content itself Because if I can get in and put in different static content, now all of a sudden I get a different reflex out of that web server So the takeaways Zero trust environments are becoming the norm Because if they can get into your environment, you have no trust levels anymore Whether it’s trust levels that are going across somebody being able to get into your system versus somebody being able to run a privilege escalation attack, we now are sort of resigned to the fact that we don’t have any trust environments left Exfiltration is now becoming as important as infiltration So it used to be that we, our entire emphasis was keeping them out, and we figured out, yeah, we lost All it takes is, and it goes back to the story I started the first talk with One human being went someplace they weren’t supposed to and someone was able to get into the environment So we’re now focusing on, okay, let’s assume they’re in the environment How do we keep them from getting our data out of the environment and into their hands? Again, it goes back to complexity I talk about this all the time In the same way that metal under fatigue, all the energy will translate to the part of greatest weakness, complexity is the weakness You may get rid of all the vulnerabilities around it, but when you get to complex pieces of the product and you begin to discover emergent behavior, that is the time at which you say, okay, we’ve got something that’s too complex We need to simplify it And it’s easy with SQL Plus to get into a bath of complexity There’s just so many ways you can do it And then you don’t realize that you’ve done it until it’s too late Threat modeling, again, it exposes the emergent behavior, but it also allows everybody to know the system from end to end And in many environments that I work in, I just worked on this piece But there was a whole nother piece of the product that I had no visibility into When you do threat modeling, include your entire team Get everybody involved, because what you want is you want everybody thinking about it And especially you want new eyes So here’s where we use what Not all these get used immediately So threat modeling gets started really about the time when you start hardening your architecture and you’ve got a pretty good handle that, okay, we know what we have And I broke this actually up into I made the comment the other day that the threat modeling really doesn’t sort of fit into an agile paradigm because agile paradigms will favor working systems over documentation Well, the problem is that threat modeling is an inherently documentation intensive process So people tend not to want to put it in there But you can break it up in certain ways So you get to the beginning of your PI Your first few iterations are going to be architecture That’s the time when you wanna begin thinking about doing threat modeling, because now you’re looking at, okay, how well is our architecture going to be able to stand up? It also gives you the ability to go back and refresh your architecture You may have made changes at the end of your last iteration that affected the architecture and you need to true up the documents and make sure that they’re actually accurate and valid for the product that you’re releasing So static analysis starts really from the first line of code, dynamic analysis starts when you have code that is pretty well along The problem with running dynamic analysis when you have a wildly changing code base and you’re making lots of changes to it is you invalidate what you ran before Now you don’t know anything That prior data doesn’t really tell you anything And then starting with your fuzz testing begins when you have enough interfaces that you feel are fairly stable and you’re gonna begin actually testing the code behind it Fuzz testing, it’s best, in the same way as TDD, it’s best to start fuzz testing at a time when you’re developing the interface, because you’re doing them along at the same time, and that way you don’t have to take this massive amount of interfaces that you have or classes that you’ve got and now you’re going to go and you’re going to test it and you’ve got to write all of this other infrastructure to be able to do fuzz testing It makes more sense to do them together And then penetration testing That starts when you complete a story or you complete a feature You’ve gotten something where you can actually go start doing some penetration testing Again, if you’re changing it a lot, you invalidate the testing So you want to get to the point where you feel like you’ve got a pretty good handle on a product and the product is pretty solid So a couple of things

One of the things we always talk about is we want to build in layers So we have the principle of least permissions And this actually is a pretty good source of lots of bugs I’m running in a minimal position or permissions I need to go and escalate my permissions because I need to do something I’m normally not allowed to do But in that code, I have an error Or I throw an exception And because I’m bailing out of that, I’m not resetting those permissions So now I’m running at a much higher level of permissions than I should, which allows people to exploit the fact that you’re running in an escalated level of permissions And again, back to complexity Bob was talking about this one of his talks It affects understandability It is really hard to reason about complex code So the simpler you can write your code, the more likely you’re going to put code out on the street that does not have these kinds of vulnerabilities in it, because they’re obvious at that point Another place is logging When I want to penetrate a system, the first thing I’m going to do is I’m going to go grab all the logs I can get Because you would be surprised how much we put in the logs We’re like chatty Cathy sometimes when we do our logging, especially in the embedded world, because the first thing that we do in the embedded world is well, we’re gonna go get, what are all the registers, where’s all the memory And I’m like, great, tell me, that’s wonderful I need all of that so now I know where to go after your system So one of the things that, and we did this in the first talk where we went and looked at the memory, just because you’re seg faulting does not mean that you should just bail out and say, oh well, the memory is corrupted If the memory is corrupted in this specific way, that tells you something So capture that And I don’t know about anybody else who digs through core files, but that is not my favorite thing to do Core files are really hard to get through Log security We don’t really secure our logs, although we really should secure our logs We don’t encrypt them because we can’t There’s no way you can encrypt your logs You’re gonna wind up paying a huge performance penalty for the fact that you’re encrypting the logs But what you have to do is be careful about what you put into it Don’t put usernames It may be convenient, but it tells me something about the system It tells me about what permissions I might need And our logs wind up being a treasure trove of information for people who are looking to go after our system Audit trails When you’re logging Okay, so when we write a piece of code, what is the first thing we do? Well, we’re writing out tons and tons and tons of information Maybe we have six or seven levels But we don’t ever really think about, okay, what if somebody has penetrated this system? What are the markers? What are the things I need to write to the log to tell me somebody has done something in the system that they shouldn’t? Our logs tend to focus solely on bugs, solely on this process ran that process ran this function ran followed by that function Instead, we can use them as audit trails So if you treat logging as a way of knowing who’s doing what and when and you can obfuscate the user that’s doing it so that you can go back and do it, but it makes it harder for somebody else to Use your audit, use your logging as audit trails or have a separate log if you have to So if you’re dealing with, for example, you’re dealing with evidence, there’s the logs that I capture that tell me how my system is running and then there’s the logs that I put out that are the audit trail for that evidence to prove that that evidence was not manipulated along the way So audit trails are a really good way of dealing with repudiation So the next best practice is secure systems begin with secure designs If you aren’t designing a secure system, you can make all the right choices as far as your software’s concerned and the code you’re using, but if you’re not starting with a secure design, you’ve defeated all of that So it takes both We talked a little bit about code reviews here before I don’t know about you, but most of the code reviews I’ve been, it’s correctness, your line links are too long, I don’t like your variable names, this is hard to understand We rarely go and look at it as oh, you’ve got a security vulnerability Mainly because we don’t think that way But one of the things we do because we don’t want to irritate and alienate our colleagues is we’re not willing to be ruthless I mean, it is what it is You’re either gonna find a bug in a code review or they’re gonna find it for you So if you’re looking at people who are Or a hacker will find it for you If you’re looking at code that is overly complex, if you’re looking for code that you don’t really understand it, don’t just do a plus one on the review and send it on Go back and ask for clarification Go back and make sure that that’s actually the code you wanted And the other thing I got into is don’t do plus ones if you’re using Gerrit Don’t do plus ones on incrementals

Nd we’ll talk about that here in just a minute why that’s a bad thing Because I’ve had cases where somebody actually introduced a bug or a vulnerability but I plus oned because I was just looking at the change they made here I wasn’t looking at the totality of the changes And then legacy code Legacy code, and I mean, we’ll be honest, does anybody here love digging through legacy code? Yeah, me neither I’m not gonna go so far as to say I’d rather have a root canal, but I’m pretty close to not wanting that The problem is that they often have our worst vulnerabilities They’re the code that has gone back into antiquity Nobody’s looked at the code Nobody’s looked at it for a security review It is simply a matter that that now becomes your vulnerability that you don’t know is there So ruthlessness really is a virtue when it comes to doing a code review We talked a little bit about open source libraries in the first half I prefer libraries that have a continual security check Because if somebody adds a breaking change or something that adds a vulnerability, it doesn’t matter that it was code reviewed by a security company six months before They just put it in today It’s in my code So you have to be very careful about the code that you’re using One of the mitigation ways you can mitigate that is security wrappers So if you have some things misbehaving and it’s going to throw an exception, catch the exception and deal with it Don’t let it just bubble up to the top of the call stack If you have a library that you suspect isn’t really Maybe you fuzz test the library and find out it doesn’t handle all these things You’re gonna have to build a wrapper around that library that handles the data sanitization for it Because it really is true You’re only as strong as your weakest third party library Their vulnerabilities become yours That’s just the way of it So let’s talk about Does anybody have any questions before I go on, because I think I didn’t wait I’ve got a bit, so let’s do a couple questions (participant speaking off mic) The number one, period, is you didn’t validate and you didn’t verify And I find that all the time in legacy code I’m just taking the data that you give me and I’m operating against it So the question was, what is the most common vulnerability I find in legacy code and that’s it (participant speaking off mic) Okay, so the question is, what do I mean beyond incremental commit So in Gerrit, you can do a commit and you look at it and then you come back and there’s some comments by it and someone comes and does a fix for it and they check in a patch Well, in Gerrit, you can go and you can look at just the patch Did they really fix the problem that I said they had? The problem is if I don’t look at it in the context of the whole commit, I’ve now just plus oned something just because they answered my complaint about their code But maybe they broke something else and I didn’t look So that’s it That’s what I mean (participant speaking off mic) Okay, so the question has to do with a library that might be reading a PNG file and how do you sanitize that Feed it a garbage PNG file and see what happens If it’s gonna choke on it, you’re gonna have to verify that that’s a legitimate PNG file before you send it onto the library if you don’t have the source code If you have the source code, you can go fix it Did I get your question? Okay Oh, you didn’t get my answer? (participant speaking off mic) So the question is, do you have to write your own parser If you’re using a library that you think is suspect and is going to choke on a PNG file, the easiest thing in the world for me to do as a hacker is just feed you garbage and watch your system go down So yes, you may very well be at that point or find a better library that’s actually been tested for security and actually does its job correctly And that’s not always And I know that’s not a very satisfying answer because sometimes that’s the only library you’ve got It’s something very specific You can’t really do it Then you have to go back to the code authors You have to make a tough decision about whether or not I want to use it Or then yes, you have to do a front end that is going to protect yourself from that library (participant speaking off mic) Okay, so Chandler says he disagrees about being ruthless, which I understand, about in code reviews Which I completely understand, because you have to work with these people I’m not talking about ruthlessness as in that I found a bug and I’m gonna make you look like a total idiot That is not ruthlessness That’s being a jerk

Ruthlessness is I’m going to look at this from the standpoint of somebody who wants to do you harm That’s ruthlessness So I’m glad he brought that up, because that’s not I’m not telling you about making your colleagues look bad, because then they’ll turn around and make you look bad and you are gonna wreck your team This is about the people who will come after your code do not care about you or your feelings So you have to look at the code as I don’t care about my feelings either Because you need to find the things out there I got about 10 minutes left on that Because I want to get through this particular exploit You want to make sure that you are not just leaving it at, I don’t like your line length, I don’t understand your variable naming, the kinds of things we tend to see a lot in code reviews DirtyCOW, anybody ever heard of it? Okay, a few people So DirtyCOW was a vulnerability in the Linux memory management system Every system out there, so it was introduced in 2007 They didn’t find it again until 2016 and get rid of it It was one of those things where they fixed it one time, it was broken by another fix, and this goes back to the whole thing of looking holistically as you’re checking in code Does it break other things? So it was fixed, it was broken, and then it was fixed again, but it stayed out there for nine years until somebody caught it We do know that it was actively exploited by pretty much all of the evidence that’s out there So this is a vulnerability in the Copy on Write, which is where the name comes from But every operating system has a Copy on Write system in it In this case, by exploiting this, you can write to protected files And it is actually fairly simple to do it So this is the Copy on Write system for Linux I have my file over here in the middle When I read from it, I don’t read the whole file into memory, I just read the segment, the chunk that I want to read So the program is going to read it When I write it, because this is a read only file that I can’t write to this protected system file, it’s going to do a Copy on Write And that’s the vulnerability So we use a function called mem map, which makes it read only So we open it read only We can read it and open it and read write simply because we don’t have permissions to do that We give this private setting, which is what enables the Copy on Write And then you have a file descriptor So here is on the far left is a piece of code We’re gonna open it, for those of you who are not Linux developers, process self mem, that’s just your memory file That’s accessing your memory for your process We have a seek, because we’re gonna go to a specific place, and then we have a write So we’re gonna write this out But there’s also another function called madvise And madvise takes the file and basically tells the operating system, I don’t need you You can dump my changes So if you look over here on the right, I will write to the file, which sets the dirty bit Then I can come in and issue the madvise, in which case the operating system throws it away And then when I go to write to it again, the system first has to reread it, why, because I threw it away So it has to go reread the file back in Or at least that section of file What does that sound like the vulnerability is going to wind up being? Exactly, it’s raised condition, it’s threading So I have two threads I do a write, I do madvise, I do a write, I do madvise, I do it very slowly Everybody’s happy with each other We do it But then what happens when I just start slamming madvise in there and I’m just doing it over and over and over again? In the case of this vulnerability, I can trick the system into writing to the actual file, the one that I’m not supposed to Remember, the system is running as root So it can write to it, it’s just I’m not supposed to So a couple of the files, targets we’d like to go after, ETC password So for strangely enough, the ETC password file doesn’t have any passwords It’s just a list of the people who are on the system So I created a little account calLed Mad Cow And the first, the 1,001 is just the user ID, the 1,002 is the group So if I could go in, because I can’t write that, if I can go in and just replace that 1,001 with all zeroes and make myself root, suddenly I can log in as root The other thing we can do is if you notice these, we talked about this a little bit earlier, this ping, ping six, and SU These are all where the set flag is set, which means that when I run these, I’m running as root

So if I can go in and just simply rewrite ping to handle the same interface but it does something different the next time somebody runs it, all of a sudden they’re doing something that they shouldn’t in the system in the context of root So here’s out code It’s actually fairly simple We’re going to basically just open the password file We’re going to map to it so that we can map to a particular section We’re going to find a position, which I want to replace that 1,001 So I’m gonna figure out where that begins Then I’m gonna spin up two threads One is the madvise and the read The code for madvise and read is on the right hand side, so we have it just opens and then it just sits there and it runs forever It just goes and seeks and writes, goes and seeks and writes And then the madvise just keeps hammering that, telling the system, nope, throw it away Nope, throw it away And that’s where the vulnerability winds up being So let’s go look at how this works So on the right hand, yeah, it’s your right hand side If you look down here at the bottom, you’ll see my Mad Cow account, it’s got 1,002 1,001, 1,002 So that means I’m just a regular user And we’ll run this out of the pane on the left So let me, I can log in as DirtyCOW Do what? Oh, sorry, it’s Mad Cow, not DirtyCOW And I’m just a regular user So I should not be able to get to the shadow file, because I’m just a regular user Access denied, as it should be So if I run And this is just gonna sit there and run and run and run and run Now all of a sudden that 1,001 has been replaced by all zeroes So we’re done here And now I’m root So this is a, from an operating system, it’s a horrifying bug And now I can access the shadow file So this is one of those bugs you’re not going to find in It’ll be hard to find it in the code review, but you’re not gonna find it with static analyzers So as far as countermeasures, it’s pretty much all just bad news This code had been reviewed and reviewed and reviewed from changes It still made it in There’s no logging So repudiation is still intact I didn’t do that because you have no logging that proves that I did And then other than a code change, there’s nothing in the operating system This is the operating system, so there’s nothing there watching it to protect it It’s not like we can rely on ASLR to give us some protection You might find this on a dynamic analysis The problem is you’ll have to know what to look for The way I think this probably was discovered by somebody outside is I’ve got the code, I’m gonna go look, I say, wait a minute, this doesn’t look exactly right I wonder if I can exploit that And then I come up with a way of doing it, test it, and suddenly, I’ve got it So this is where having the source code becomes very, very important So the last best practice, be committed Like I said in the first talk, it drives me nuts when I hear CEOs talk about how well, we have to get lucky every time The hackers only have to get lucky once If you are trusting that luck is going to fix it for you, it is not, because the luck will run against you It’s always going to run against you There’s hundreds of companies out there where the luck ran against them But here’s the thing I own the software That’s why I picked the title Your first line is the last line of defense I own the software I own the network, I own the infrastructure, I own everything that goes The only thing that the hackers have that I don’t have is they get to choose the time of the engagement, that’s it I have everything else and that’s why I did this picture It’s a completely ridiculous caricature These people are not demigods They’re not geniuses We can do a whole talk based on all the times when they’ve messed up and put things out in the wild and made it easy for us to stop what they were doing But the bottom line is that when it comes to my software, when it comes to my systems, I’m gonna choose to either be the predator or the prey

And I choose to be the predator Because they’re in my backyard and I have no intentions of just letting them walk in and waltz all over the things that I’ve created and we let them get away with it And that’s one thing I want us as a community to begin doing is stop thinking of this in terms of a problem we can’t solve It’s solvable It just takes training It takes a different mindset So it may not be a fair exchange, but you’re either going to be the predator or the prey You’re either the hunter or the hunted It’s that simple in this environment But the good news is you have all the controls And so this is where it starts So if you’re interested in security, we have a lot of security talks We have more security talks here than we’ve ever had So we have a panel coming up Thursday So if I don’t get to all your questions, I’ll be happy to answer them after or come to the security panel Besides myself, we have Patricia Aas, which I found out this week that you pronounce her name that way because it rhymes with awesome Matt Miller from Microsoft actually does this for Windows He actually goes in and runs penetrations against Windows to see if he can break the system and get into places he shouldn’t And then we have Michael Wong from Code Play, who’s been a part of the standards committee He’s part of several standards committees for a long time, which gives him an insider’s view And so Eva Conti, who’s sitting in the back, will be our moderator But there’s other talks So you’ve had two from me today Patricia will be doing two this week So she has Tuesday and Wednesday You have Chandler Carruth over here who is going to do what I think is gonna be an outstanding talk on Friday on specter and meltdown And then there will be a panel after that So you have plenty of opportunities here to engage people who are in this industry who work in this to find out, okay, what are the things we can do in order to make our code very safe And with that, I will take two and a half minutes for the questions There was somebody over here, yeah, you had a question – [Participant] Yeah, so you told us that bad design is bad security (mumbling) Or what’s your opinion? – Since I do the vast majority of my software in C++, I think C++ is an outstanding language just in general As far as security is concerned, C++ I think is a very secure language And let me qualify that statement If you know what you’re doing If you know where the sharp edges are If you’re testing One of the things that we love about C++ is that C++ gives this incredible power Well, that incredible power lets you do a lot of things that you probably shouldn’t do We have a standards committee that is not just changing the language overnight They’re taking their time They’re going through these things, be it SG12, which is constantly looking at can we make the language safer? So I don’t really have a problem with C++ as a safe language The problem comes with how we wind up using the language Even string copy, if you use it correctly, there’s nothing unsafe about it It’s just that we don’t So rather than looking at the language and saying, well, is the language safe, if we were to strip out all the sharp edges on C++, we get Java And you still wouldn’t be done with all the vulnerabilities, because there’s lots of vulnerabilities even in Java So I would not look at C++ and say we don’t wanna use the language because it’s got sharp edges Every language has them You just have to be careful (participant speaking off mic) Okay, so what’s my comment about security through obscurity as a valid mechanism? If security through obscurity was a valid mechanism, it wouldn’t be so easy to get past it Most of what we do with obscurity is I hide a file Well, that’s fine, but if I know you’ve hidden it or if I find out you’ve hidden it, your obscurity is gone Security in obscurity has its place, but it is a weak form of security A better form of security is validating and verifying the data that you’re using and the people that you’re talking to So I don’t spend a lot of time doing security through obscurity, simply because I think it’s a weak model for security Is that it? Oh, yeah, right here (participant speaking off mic) The place I would start would be threat modeling

If you start, you say, so the question is, where would you start? I’ve mentioned a lot of tools Which ones would you start with? Coverity is not cheap Most of the static analyzers are fairly expensive So I would start with the places where you get the most knowledge Static analyzers are an automated tool that when you run it, you’re going to wind up getting some result, maybe not the result you expect, but you’re not going to really learn a lot So I would start with the places where you as a team will learn Threat modeling, code reviews Go figure out what the patterns are There’s tons of information out there You can have people in the training What are the patterns? Talk to pen testers They’ll tell you where all the patterns of the mistakes are So what you want to do is, especially if you’re not talking about going and starting at system, the ones that give you the most value are things like code reviews when you understand the kinds of patterns that you’re seeing, things that you basically get I love dynamic testing tools because they’re inexpensive I can implement them almost immediately Fuzz testing has, it’s reasonably inexpensive, but it has a really high startup cost So I would start with the pieces that you see on this list that are the pieces that are either free or things that give you them And for me, it’s gonna be threat modeling, it’s going to be code reviews It’s getting my head wrapped around a problem that I’m not using to dealing with – [Participant] A lot of the static analyzers are (mumbling) – Yeah, so the, in fact, we are out of time So the people who need to leave, you can go ahead and please feel free, because I know everybody has a place to go So the comment was is that a lot of static analyzers are going back to CERT and getting checked That’s actually a great place to look If you want to learn the patterns of the mistakes we make, CERT, MITRE are very good place to go Go to Patricia’s talk tomorrow I think that’s the one where she’s doing CNC vulnerabilities Okay, thank you all very much I’m gonna (applause)

You Want To Have Your Favorite Car?

We have a big list of modern & classic cars in both used and new categories.