>>  SO, I’m going talk to you about highjacking build pipeline. I’m Kyle and this is Greg. We both came from Rackspace. All right. When I talk about build pipelines I’m talking about your source control from GitHub, GitLab GitBucket, wherever you are storing their code, depending what site you are on. Continuous integration, are you running on (?), Kravitz, do you just run it on your local box? What actually is  — where are your tests running as well as upstream sources? Like NPM, by pi ruby, GEMS, all of those. The reason I care about this is I work with a lot of open source software where we set up builds for that software and you know anybody can contribute, anybody can push patches in, so what sort of vector is that? What can you do with that? What can you leave? Kind of the precursor here is all need secrets, you might need secrets for testing if you are deploying to production, database credentials, OF ced, might be cloud credentials, GitHub tokens, Twitter tokens, the actual SSH keys to get onto boxes. So need to manage these secrets in some way. Not everybody does this appropriately. Credentials get leaked. Like if you do a simple search on GitHub for AKIAJ you’ll find Amazon credentials instantly. I’m throwing 1071 results just finding for that stream. Right? Run infrastructure or do whatever, keep going. I think this is a problem with how people treat Git. (Pause) Like you start up a new project run GitApp, everything goes in there. If you have any secrets encoded into the repository itself it’s there for the world to see next time you get pushed. This even happens to people who know what they are doing. Rich Milgo gave a talk at BlackHat on working with the cloud, and to do security appropriately, set up the DEV environment appropriately as well as lock down your instances and how to handle keys. When he ended up looking at the account that got highjacked for him, what did he find running there? Right, so they didn’t actually care about his running infrastructure at all, they just wanted someplace to go mine. They pulled up really big, really, really big instances and just started spending his money. If you want to read more about that, go to bit leave/mobile, he does a good article on what happened, what he did forensically to figure out what they had done, tried to go back through cloud trail to figure out what they had access to and what else they tried to get access to. It’s not just in the GitHub credentials either, this is an Android file, APK. If you look here in the middle, there’s there cedes so every client that downloaded the app has access to their Amazon (Laughter) So what fun can we do with cloud credentials? I can spin servers, take down their servers, redistribute DNS and load balance exercise, maybe you have another node you want to put into the mix. I can remount volumes, if you are on Amazon you have your cloud server and it has a EVS volume on it Disconnect that, connect it to a separate like you’ve got their server and their EBS volumes connected to their own. Have your SSH and make you have access on their box. For open stack Nova you can append your own key to the main key they use when they spin up new servers so that every time the authorized key file gets dropped you get access to the new server. So how do we find these keys to begin with? Like I was saying before for Amazon, just search, find a list of keys, there it is. For open stack and the open stack writers it’s a little harder because they actually just use a hex decimal D so you have to look at what it would look like in the code. OS passion word would be if they had an environment variable or in some config file. The results are really big here but it could have come from a library. Same thing with Bianchi, looking at 3,000 results. And then API key, 305,000 results. Again, that’s like could be the libraries, too, doesn’t necessarily mean it’s the actual API key. We have

to do some amount of triage. For GitHub they only let you search for 1,000  — you only get 1,000 results back so you to have break it up. So like one day we took the Rackspace EPI key and we were looking for one good Python, we can do it with or without language. GitHub is their own search setup so you can break out as many things as you want and also look down particular paths and languages We get 4 20 for Python and 8,303 for the rest. When I was breaking these out I had to go through pretty much all languages and break them into their own discrete begins Paragraph bins. Right. Keep sweating. Couldn’t we just let people know when they fuck up? Like those are all there, their e-mails are attached from GitHub. So we created a project get get sec nanny. It searches repositories for the same security oop. He mail the original committer and the owner of the project, let them know how to revoke keys and also how to panic! (Laughter) The responses were very flip. They were like wow, thank you, how did you find these? You’re such a savior and, this is only a testing project. Is it your testing credit card? (Laughter) So far I found about 265 keys, I’m clearly going after one provider I want to protect There is certainly more out there to find. One of the biggest offenders for this were rails developers, config initializer secret RV, it’s got usually with cred, there are rails itself and this one is infamous, there’s a port out there that will never get accepted that just adds it to the get ignore so it doesn’t happen to people in the future The comment from rails was, they’re not gonna do it in there because people shouldn’t be doing it that way. (Laughter) You know  — (Laughter)   — great! So they’re supposed to use environment variables, that’s the way you are supposed to develop and deploy an app Anyway… (Laughter) So you have to be  — we have to do it the right way if we’re doing a security checker. This isn’t my security checker I just noticed this last week so Danny Greenfield here he got the message back. You can’t read that enhanced. There we go! All right. So they said they have confirmed this possibility by manual inspection and then there’s the secret. I don’t think they inspected that. So that’s it for secrets. But what if you actually need secrets for testing? It’s not like you can live in a vacuum where your code never touches anything. You are not gonna build any real web infrastructure with that. One opportune nivt  — I’ll give — (Inaudible) — in a second, travesty i I great for open source, just do a get push, hits the web hook and then sends out tasks to run the build and let you know if you have success Lot less control than Jenkins but super simple. This is the definition for a Travis yammel, you sign up language, here we are doing Python, 2.7, we have, before the install you have steps as well as the installation and then what script will run. They thv neat thing called encrypted secrets, though. Like you put secret on the Web out in the open, so I was curious if I could use it All this does is sets up an environment variable that can be used later. So could we leak those deC ed secrets? Like where do they come out? How do I get the secrets out? Say we open a request and we’ll try to echo off their environment variable has the secret. Oh, nothing! What! So Travis, the keys used for encryption and decryption are tied to the repository. If you fork a project and add it to Travis it will have a different pair of keys than the original What that is saying is if you make a repo it gets one key, someone forks your repo they get a separate key. They could run their own secret stuff they are running off their own fork to do testing, but then only the stuff that is on master gets tested

with the original keys. Nice So let’s say we emergency that port request, secret message then comes in. So drink. Right, so you are only  — only variable is cloud review and the fact you to have trust Travis At some point Travis is decrypting secrets so you have no idea. But those guys drink good coffee and live in Berlin so I assume they’re great. Now we’ll talk about Jenkins and I’ll pass over to Greg (Applause) >> Thanks, Kyle. So for those of y’all who are not familiar with Jenkins or you’re not  — Jenkins is a continuous integration and deployment suite used for not only functional testing but also deployment to production. Even if you are not a Jenkins user this is something you have never heard of before, why do we care about Jenkins? Well Jenkins is the road to production. If there is a general since server at the end of that pipeline is either a repository that multiple servers are using, or a production server with data worth ceiling sealing. The normal interaction is hipster developer pushes code to GitHub  — (Laughter)   — Jenkins is notified via web hook that there is some code ready for testing. What we want to do is masquerade as hipster developer so we can push code to production servers or repositories and compromise them. So let’s say that hipster developer makes an oops. He either commits his cloud credentials which have no off token or he commits his GitHub token directly to his code repository. We’re kind of off to the races, right? So if there is a repository at the end of that pipeline, all we do is propagate vulnerable code we can scan for and then pop every single server  — yeah, and the mic  — (Laughter)   — so you can pop every single server that is using that repository. Can y’all hear this? >>  Yeah! >> No shit! Okay. (Laughter) So what’s the other option, right? The other option is instead of a repository there’s a server with information we want to steal and sell. So one thing you can do is propagate things for persistence, think reverse shells, the other options is to try to extract the secrets that Jenkins has out of the build, so there’s two different ways to manage environment variables in Jenkins. Either you throw them directly in configuration or you’re using the credentials plug-in but at run time they all boil down to actual environment variables. So both plug-ins could possibly be leaked if you could sneak something like this into the code. So the goal of this script is to print the environment variables and then phone them home to a server. No problem. So that’s kind of cool but that will only give you one server. You know what’s really cool is compromising multiple servers. So how do we do that? We target the Jenkins install directly. So for my testing, I was using the Jenkins database library or Jenkins user management for authentication, Jenkins is like widely configurable so you might not run into this unless you use that exact authorization to replicate this. I was curious about how Jenkins decided to manage its users and it boils down to this file called config.xml and here is what it looks like. You’ll see that there is a user name, a hash’d API token and a hash password So what caught my attention is they are using JV script. So I did more digging. A lot moreing All the way down to this file called Hudson private security realm. And wall you’ll see is how Jenkins is actually encoding and stories passwords. There’s

nothing really wrong with this except that it’s really to duplicate. So using this three-line Java code, we can create hashes that are passable to Jenkins. But so now we have to figure out a way of how to actually pass those hashes and get them installed on Jenkins So what if we have something like this in our build, the goal of this code is to locate the config files for all users and attempt to replace all hashes to a password of our choice. Let’s find out. (Pause) So over here is my Jenkins server. To show I’m not cheating, I need a password, anyone. Admin? That would be a good guess (Laughter) So “admin” clearly doesn’t work, right? What I do is go over to my Java code (Pause) We get back a secret path. So then we go over to the compromise repository. And we throw this in our code. So if we go back to the Jenkins instance, we will see our project is now building or it’s queued to build and it’s building. Building >>  — (Inaudible) — (Laughter) >>  So part of the reason that it’s going through this right now is in the script there’s a command to restart because you have to refresh the file. And boom! We can then  — (Cheers and applause) >>  So from there it’s really simple, right? All you have to do is go change the voice script and propagate them through. The other thing I wrote just so  — thankfully it worked, too, I’m really scared of live demos, the other thing our script did is phoned home all the users that Jenkins has so I wouldn’t even have to know user name, Admin It’s right there for me to see Yes. Maybe. Not in this room They kick you out. But so there is a catch to this technique and that’s this config file is only stored on the master node of Jenkins. So if builds are allowed to run concurrently that’s really good news for us All you have to do is add a sleep function to your malicious code to put the nodes to sleep, and back on (?). If you are not allowed to run concurrent builds you’ll have to get creative. One option is to just keep committing and that’s because we know these people we have compromised are developers and they will be executing other builds. The other thing you can try is with the compromised credentials, commit other repositories and push other repositories to try to trigger other bills on Jenkins but what if there aren’t any oops, right? What if we really want to compromise this deployment system because at the end there are 100,000 servers that are consuming the repository or maybe you have like a vendetta against someone and you just really want to get into their production boxes. Well, one thing that can help you out is

that the low-hanging fruit in this scenario is that they’re doing automatic floor request building which basically means they are taking all the core requests they get and they are building them before they decide to push them to master or merge them into the repository. From a security perspective that sounds really stupid, right. Why would you ever run code that you have no idea what it does. Well the nice thing about Jenkins is if you were to do this, it would tell us first it was functional and, two, how the performance compared to previous versions of the code. There is definitely an argument to be made for doing this. I’m glad I don’t develop because I would probably do this. The other option, though, is that you hit the gate. And you get this notification from Jenkins asking for one of the Admins to verify the file. So if you are not going to give up, and no one likes to give up, you essentially have two options (Pause) (Silence) The web hook we originally talked about, right, Jenkins is notified there is code that has to be a web hook from GitHub and it’s always or it’s by default configured to be at GitHub-web hook on the Jenkins server. How the process is kicked off is a simple post request. If you can forage that post request, you can beat the gate and cause a trigger and compromise the Jenkins server and all things attached to it The worst-case scenario, right, it let’s say there is no continuous integration, there is no deployment system, it’s just a repository you want to compromise. Sneak in your code, hope no one sees it and it might propagate. So real quick, I’m going to do the quickest overview ever on how to secure a Jenkins server. So disable anonymous access, date your deploys, use random port, disable executors on master, change web hook from default URL, and it’s that easy (Applause) >>  So we can take questions. If you have questions, come up to the mic >>  This is kind of tall. I noticed that Jenkins failed. I have a suggestion. There is a URL you can hit on the Jenkins master that will make it reload all the files without restarting so no one would be able to tell you just did that. >>  You have to be authorized for that That’s my question. You back off! (Laughter) You’re absolutely right. There is a reload URL in the Jenkins server but there’s two problems with that. A, you have to be authorized and, B, you have to know where Jenkins server is This is like a blind injection I don’t even have to know where the server is because it will phone home that information to me. >>  Okay, thanks. >> Anyone else? Anyone? Anyone? No I know you. I’m not taking your question. (Laughter) Thank you all so much for coming and feel free to come up and talk to us (Applause)

You Want To Have Your Favorite Car?

We have a big list of modern & classic cars in both used and new categories.