– [AJ] Hello, and welcome to the SANS APAC webcast IR and computer forensics in the virtual realm I am AJ Boyle, and I will be moderating the course today The instructor is Paul Henry Before I turn the time over to him, I would like to cover features of the interface for those who have not attended a SANS APAC webcast before The screen is split up into three main regions, the participants list, the direct message chat window, and the whiteboard, which has already been loaded The participants list has several columns: the first with the little hand indicates someone has raised their hand to ask a question, to raise your hand, click the hand button right below the participants list The next column is a yes/no indicator, you can click the toolbar button with the green check or the red x to indicate a yes or no answer to a question from a moderator or the instructor The next column is the microphone, if this icon has a yellow background, someone is talking The microphone will be blocked, so only one person can speak at a time If you have a turn to speak, you can click the microphone button in the audio section Do note that Illuminate software will buffer speech, so if you experience network delays playback will speed up, causing a chipmunk effect The next column is the chat indicator, anyone who is typing in the messaging entry box will have a yellow background here I will cover how messaging works in a moment The next column with the pin icon is a Whiteboard editing indicator; I have disabled Whiteboard editing for participants The last icon before names is application sharing, which has also been disabled for participants Finally, the list of names you see are your classmates, moderators, and the instructor, who is also a moderator Let me explain how the messaging system works: to enter a message, point your cursor at the entry box below the main window before sending, use the drop-down to select who the message will go to Now, the choices are all participants, only moderators, and only selected participants, you can select names of other students, the instructor, or only moderators to send a message only that person, but do note that moderators will see all messages, both public and private As mentioned, the whiteboard already has the slides loaded, the instructor will advance the slides as he moves through the course material, and it will automatically advance for all students If you experience technical difficulties during the class, please send an email to [email protected] or notify your moderator Thank you for your attention, now let me turn the mic over to your instructor, Paul Henry – [Paul] Hello everyone, my name is Paul Henry with SANS, today’s topic is going to be “IR and Computer Forensics “in the Virtual Realm” You know, according to the numbers from Gartner, 80 percent of those servers that could be virtualized have been virtualized, yet 60 percent are actually not as secure as their physical counterparts, hence, we can count on having incidence response and computer forensic issues within the virtual realm Critically important that people very quickly update their policies and procedures to address providing for IR and forensics within the virtual realm Now Again, just a little background on myself, I’ve been with SANS for a number of years, many of you may be familiar with me, I did quite a bit of work back in my cyber guard days in Southeast Asia For about the last four years I’ve run a forensics practice here in South Florida called Vnet Security, we primarily focus on providing incident response and computer forensics in the virtual realm I’m the board VP of FACCI, the Board USA seat for ISFCC, board member of Ashlar, member of the Miami Electronic Crimes Task Force, a member of HTCIA and HTCC as well I hold a large number of certifications within the industry, so I’ve sat in the seat many of you are sitting in today, I sincerely believe in the value of third-party validation for credentials, et cetera, et cetera, as you can see from my sig And I’ve written quite a few book chapters over my career, including many of those that deal specifically with both security and forensics Now, at SANS, what do I teach? Well, I teach 401 Security Essentials, I teach all the forensics tracks at SANS, as well as some of the fun stuff, such as SEC 553, and of course, the newer, in fact, we just released the new 579 virtualization security around the first of the year, and of course, I’m the author on the VoIP security class for SANS Now SANS is the most overall complete curriculum out there today with respect to

computer forensics, and we typically bring people in with FOR 408, Computer Forensic Investigations, Windows in-depth, they would advance to FOR 508, Advanced Computer Forensic Analysis and Incidence Response, and then you have a decision to make, are you going to move onward with Network Forensics, FOR 558, or perhaps mobile device forensics, FOR 563, you can then advance to FOR 610, Malware Analysis Tools and Techniques, and of course, FOR 526, Advanced File System Recovery and Memory Forensics Now on this slide, it does indicate the website for the Forensics Tracks app SANS, where you can find very valuable information, there’s also a tremendous amount of insight available from industry experts on the SANS forensics blog, I would highly encourage you to take a look Now, if you have in fact taken a SANS class and you do have a portal account, there’s a number of really exceptional free tools available from SANS, such as the SIFT Workstation You know, I have access to pretty much every commercial tool out there, and regularly, I find myself migrating back to the SIFT Workstation from SANS when I have to get something done very quickly, simple as that We also have some really cool digital forensics challenges, if you want to test out your skills, also available on the SANS website, and of course, most of the instructors at SANS do use Twitter, so again, you can follow most of us on Twitter, my Twitter handle would be phenryCISSP Now, moving forward, we’re gonna jump into a little bit of the basics here Looking at virtualization technology today, most of us are primarily familiar with server virtualization, that’s taking that physical server and basically abstracting it into a virtual environment We of course have application virtualization, application virtualization is kind of like sand boxing, a number of vendors have taken this approach, in fact, in the new version of the operating system for Apple that’s coming up here in the July timeframe, they’ve now taken a turn and they’re going to be using sand boxing and/or application virtualization within the Apple platform itself Now on the desktop virtualization, what we’re talking about there primarily are products like Workstation, and of course, Fusion for the Mac On storage virtualization, we’ve really seen this roll out through VMware here in the last couple of product releases You now have the ability to fully abstract storage within VMware, hence you’re able to move that entire storage array using a storage V motion off of the hardware, repair that hardware, if you have to apply firmware patches, et cetera, et cetera, and then simply storage Vmotion it back Now the primarily players today in server virtualization would of course be VMware, they’re the 800-pound gorilla, Citrix, and of course Microsoft Microsoft is putting a major push into virtualization currently, it’s clearly reflected in their pricing If you price out at configuration with literally the same capabilities, now, mind you, you’re gonna have to turn it down a notch for Microsoft, they’re pretty much following VMware today, you’ll find that the pricing is significantly less for the Microsoft platform Now just to get some terminology squared away here, with respect to server virtualization, when we’re referring to a host, we’re talking about the server virtualization platform that you would host virtual machines upon When we’re talking about a guest, we’re talking about virtual machines running on top of a host platform Now, speaking of hypervisors, this would be your virtual machine monitor, or VMM, that’s the software that enables virtualization on the host You have your Type 1 hypervisor, that would be your bare metal hypervisor, it really is somewhat of a self-contained platform You know, VMware today primarily operates in a bare metal configuration, that’s where you get your best performance with VMware A Type 2 hypervisor runs on top of a traditional operating system A good example of a Type 2 hypervisor would be of course, VMware Workstation or VMware Fusion Again, you have the overhead of dealing with the underlying operating system, hence you’ll always get better performance of a Type 1 hypervisor, the bare metal implementation Now in virtualization risks, it all drills back down

to the old target we’ve all seen in the past In the x86 platform privilege levels are as follows: ring 0 is your most privileged level, and it’s where you’re essentially controlling the hardware Ring 3 is where your applications themselves typically function Now, a hypervisor’s job is to present securely that virtual ring 0 to each of your virtual machine guests Now the problem that could potentially arise of course, would be that a compromise of the hypervisor could in fact lead to control in ring 0 Just yesterday US Cert published a bulletin that a guest to host exploit has been discovered within virtualization that could potentially allow a guest to compromise a hypervisor Now, I read through the bulletin myself last night, it does not impact VMware That being said, it looks like the primary risk today would be associated with the Citrix product as well as the Microsoft product, you might want to Google that, again, that would be a cert bulletin on virtualization whereby a guest could be used in the compromise of a host Now within VMware, looking at the most common infrastructure out there today, we’re gonna be looking at version 4, it’s very similar to version 5, mind you, but we’re seeing the largest deployments out there are still four, as people migrate to version 5 You of course have your ESX and your ESXi server itself, you’ll see that as the all-encompassing block surrounding everything You of course have your virtual switches, you have your vCenter management console, you have your vSphere clients that allow you to connect either to vCenter or directly to the or directly to the ESX/ESXi host, you would then have your virtual machines running on top of ESX and/or ESXi, and of course your storage, so that just gives you a very quick overview of the infrastructure associated with the VMware product Now we’re gonna look at a couple of different options available to users today, and how they’re gonna handle their DMZ The picture shown on your screen right now is a partially collapsed DMZ, I’m seeing a lot of deployments utilizing this methodology of configuring their DMZ so they can take advantage of still using their still existing physical security products You’ll note that many DMZ systems are being virtualized today, the network connections are still physically distinct, hence you’re able to connect those network connections to your existing firewall products, IDS, IDP, et cetera This provides the most flexibility with existing network security tools Again, looking carefully at the picture, you’ll notice that we’ve actually separated our servers into individual containers utilizing ESX server, we’re bringing that output out through a physical net, and you’re able to isolate your trust zones using your existing physical security products Now looking at a partially-collapsed DMZ In a partially-collapsed DMZ we’re running multiple servers on a single copy of ESX In the most current VSS from PCI, they indicated that the level of trust available on VS separation within a copy of ESX server is in fact equivalent to a physical separation; hence, your able to run distinctly separate servers on top of a single copy of ESX server Now you’ll note in this drawing, we’re still bringing out the IO to physical interfaces, so even though we’re running multiple servers on top of a single copy of ESX or ESXi, we’re still able to take full advantage of the existing, the physical infrastructure that we may have for our existing security products Now we have a fully-collapsed DMZ In a fully collapsed DMZ you’re really gonna get your money’s worth out of the investment that you’ve made in virtualization Now, you’ll not that we have a single copy of ESX server, we’re running multiply individual servers, taking advantage of the separation provided by ESX, but we’re using virtual firewalls, we’re no longer using our existing physical firewalls Now again, the problem that we have today with respect to fully collapsed DMZs is we have very little history behind today’s virtual firewalls

Now, that’s not to say that they’re not equivalent to yesterday’s physical products, but the bottom line is, we really don’t have the history with them You know, you’ll note that it really seems that firewalls vendors as an example, thought this whole virtualization thing was only gonna be a fad, and very few of them wrote the necessary software to interact with the Vsafe EI to able to apply policy to intra-VM traffic, it’s kind of a problem I mean again, we had Trend Micro, we had Altor, et cetera that had been in the game for quite some time now, but if you go back just one year ago, you could count the number of firewalls that could be deployed within a virtual environment that could actually see and act upon the intra-VM traffic, you could count them on one hand Now it’s changing rapidly, I know that Checkpoint as an example, the market leader in physical firewalls has actually now released a product, it was about six months ago, that can actually be quite effective within the virtual realm So it remains to be seen just how well virtualization firewall vendors are going to perform and how good of a job they’re gonna be able to do, again Six months of a product being out there is not quite enough time to make a determination as to how much security it’s gonna be able to afford, so for myself, I’m not recommending a fully collapsed DMZ yet, I want to see a little bit more operational time, see a little bit more track record and history behind these products before I’m willing to commit them for my clients Now in looking at your overall ESX architecture, it’s kind of interesting You had of course your virtual machine monitor, and you had a service console Now, with an ESX that service console was based upon Red Hat, simply put, you had a full Red Hat console available within the ESX platform Now ESX is being replaced, it’s being replaced by ESXi, it’s a major shift of ESX, there is reportedly no service console at all Well actually, it’s still there, it’s just somewhat unsupported, it has less than a 90 megabit footprint, it can be embedded within the hardware itself, I see a lot of Dells today being shipped with VMware ESXi embedded, or of course, it can be installed using media It of course does support most major vSphere features, and ESXi is free from VMware, where they’re making ESX a little bit difficult to get nowadays VMware is clearly moving to ESXi, that’s where the future is, so again, if you’re running ESX, migration to ESXi is in your future Now shared storage in VMware, all of the cool things that you can do with VMware really requires shared storage, Vmotion, HA, DRS simply cannot be done from local storage, local storage is typically only used for storing things such as templates, ISOs, et cetera, highly likely the VM you wanna get to from an IR or forensics perspective is stored on shared storage I find it somewhat laughable, I’ve seen a number of people come in and perform investigations within the virtual realm, whereby they would go the system administrator and they would ask the system admins specifically which one of those one use servers was running the virtual machine System admin would point to that Dell 1950 on the rack, the law enforcement typically would unplug the machine, image the hard drive, and go back to their lab to analyze it, only to find in fact, the virtual machine does not exist on that hardware platform Again, most virtual machines today do not operate on the local storage of the hardware platform that is running the virtual machine, it does exist on shared storage Being able to actually locate where that virtual machine VMDK folder exists and being able to carve it out is key to doing an effective job in incidence response and forensics There a couple of more gotchas associated with that, and we’ll talk about that as we progress forward You have a number of different types of shared storage available today We have network-attached storage, NAS, typically using NFS, seeing this really beginning to take hold today You of course could have your SAN running on top of Fibre Channel, it in fact is the best-performing solution out there If you’re virtualizing something such as a web server that’s used in eCommerce,

where you’re gonna have a large number of simultaneous connections, Fibre channel will in all likelihood be your first choice I’m seeing a lot of SANS use using iSCSI, still within small business You can actually go out and download a free ISCSI server from the VMware marketplace, install it on top of some older existing hardware that you may have, and do a good job of handling shared storage, but again, you’re not going to be able to handle the higher IO that might be associated with an eCommerce site in comparison to something such as a Fibre channel SAN Now the vast majority of people that I’ve worked with here in South Florida are running their SAN over iSCSI, that being said, on newer deployments today I’m seeing a lot of pickup in the NAS running NFS, I’m running here in my lab myself, a year ago we had issues because very few NAS devices were in fact on the HCL from VMware, but that’s changing rapidly, I just recently switched my own lab to NFS for my NAS, I’m running a ready NAS here, and is working quite well, I have no downtime with it whatsoever, it is much simpler to deploy that iSCSI Now we have to talk about VMware’s VMFS file system This is the virtual machine file system, it’s a journal file system that was created by VMware, it can handle multiple disks, multiple LUNs, it is able to work with up to 256 VMFS volumes per ESX host, that would be Version 3, and again, multiple ESX hosts can concurrently access the very same VMFS volume and files Now typically you would manage VMFS using the VMKFS tools command on the command line All right, now one of the issues you’re gonna have is occasionally you’re gonna have to work with that VMFS file system outside of VMware, as an example there, if you have a client use a third-party to come in and create a DV image of that VMFS file system and present you with either a DV or a EL1 file, you then, as the forensic investigator, would have to mount that to be able to carve the VMDK to get to the files associated with that virtual machine Now there are some great third-party tools out there, VMFS from open source virtual file machine system driver is a great tool, I’ve provided the URL for it here, it’s up on the Google code site, it can work on a command line, or it use it with WebDAV There are also a couple of new Java applications out there that can also work with VMFS Now it’s critical that you understand that there is not a single commercial tool out there today, end case FTK et cetera as an example, that can actually understand VMFS They can all work with the VMDK, but none of them have the ability to carve out a VMDK from VMFS, that really is a limiting function here, it’s been a problem for many, many people Again, it would seem that virtualization with respect to forensic vendors looks like they really dropped the ball again, it would seem that they kind of thought this whole virtualization thing was nothing more than a fad, and again, we see no support from either guidance or from FTK for that underlying VMFS file system Now again, on the vendor support, it simply does not exist, it’s really tough to blame the vendors, though, as VMware never released any standard that they could write code for Now again, while there is no support for VMFS, the VMware or VMDK folder itself, which is the folder that would contain the entire abstraction of the virtual machine is a published standard, and most forensic vendors can in fact now and analyze a VMDK, but you first have to carve it out of the VMFS file system; that can be a little tricky Now, we show a number of different ways to accomplish this within our course itself, we work with primarily, the Java capabilities that was brought out, as well as a couple of binaries that are available today that do allow you to actually mount a VMFS file system and then carve out the respective VMDK Now Encase and of course and VMware,

Encase can in fact analyze a VMDK data file, this is the folder that contains the abstraction, but again, you have to keep in mind that they cannot carve it from the VMDK Now what’s really cool about VMware is that all of the files associated with a specific virtual machine are contained within that VMDK folder What’s even cooler yet is they’ve abstracted the physical hard drive, so effectively, when you make a forensically-sound copy of the VMDK, you’re actually getting that virtual abstraction of the hard drive, which would include both allocated and unallocated space So you’re able to actually then carve out deleted files that would typically be associated unallocated space from within that VMDK, very cool capability, I’ve done it regularly myself Again, I would simply mt VMDK, either using Encase or FTK, and I’m able to recover all of the deleted files from the abstraction of the hard drive, because it does include within the abstraction all of the unallocated space Now there are a number of other products out there, some pretty cool ones, it you’re working on VMFS itself off of a VMware platform, you can download a free copy from Sanbarrow of the product called MOA MOA will allow you as well to work with an actual VMFS outside of a VMware environment so again, another great tool for working with VMFS outside of VMware Now on recovering deleted files within a VMFS file system; this gets a little bit tricky If we look at ESX in version 3.5 update 3, we actually had a really good undelete capability They created a list of all of the blocks where the files were stored, you would simply log into the service console and enter the VMFS undelete command, and you could recover files that had been deleted from version 3.5 of the VMFS file system However, since version 4 from VMware, there is no longer any simple way to undelete a file VMFS no longer keeps a backup block list associated with any specific file Hence, no block list equals no recover capability, this absolutely is a sore point within VMware Again, it’s hard to understand why this capability would have been removed, but we all have to clearly understand that VMware is a high-performance virtualization platform, it is not necessarily an IR or forensics platform Now from a forensic considerations perspective, looking at VMFS, if you’re trying to get at a single virtual machine, it can be a bit of a problem You may have in fact, hundreds if not thousands of virtual machines that could potentially be impacted by simply taking the shared storage VMFS file system offline We also have to consider privacy; again, you could have hundreds of thousands of virtual machines running on that shared storage, if you copy out a copy of the VMFS file system, are you going to have possession of intellectual property belonging to other clients that are not really within the scope of the search warrant? Probability of recovery of any deleted materials is low, so why not focus on the specific VMDK for the virtual machine that really is the focus of the investigation that you’re doing? Now in the VMware VMDK the simplest of terms, it’s your container for all the abstracted files associated with a specific virtual machine Now again, the word “abstracted” is really key here, as the VMDK actually in some respects, emulates a physical hard disk So again, you’re gonna get both allocated and non allocated space for the abstracted hard disk in your copy of the VMDK Now VMDK file types, there’s a number of different file types available when you create your VMDK You have a zero thick, eager zeroed thick, thick, as well as thin Now a thin would be your default on NFS volumes in your VMware workstation product itself, some of these disks formats will actually overwrite the disk itself with zeroes as they construct your virtualized disk Others in fact will overwrite with a zero only as they’re actually creating a specific place

for the file within that file system structure, so again, a careful thing to consider there is that in some cases, as you’re creating an abstraction of the hard disk within VMware, it’s actually overwriting all unallocated space as you create the disk, you’re gonna gain a little bit there with respect to performance, but it’s going to take longer to create the disk The alternate of course, is to only overwrite with zeros as you’re actually creating disk that you’re going to be using Now in that light you’re going to pay a performance hit, but you’ll be able to very quickly create that disk, because you’re not allocating the entire disk instantly, you’re only allocating space as you need it I primarily see that methodology being used in production today, people tend to over commit disk space so they will run in a manner whereby they’re only creating the actual disk space as they need it Now you’ll also have a couple of different modes of operation available with VMDKs You have persistent mode, which is in fact is the default mode, whereby the VMDK behaves just like a standard physical disk, as you make changes, they are instantly written to the disk We have an alternate mode of operation called nonpersistent mode, once set as nonpersistent, changes are written essentially to RAM, they’re never written to the hard disk You’ll find that operations such as a kiosk, et cetera, would be great application of nonpersistent mode, users could walk up to the console, take care of whatever business they have to take care of, and essentially, nothing they have done is written tot eh hard dis Hey, if you’ve got kids, nonpersistent mode’s another great mode of operation you might wanna consider for them I have five children myself, they like playing games such as WOW et cetera, and again, they’re constantly downloading those cheat codes, and the malware that comes with them, so they’re operating in non-persistent mode in my house, when they download that stuff and they include malware, it’s never written to the hard disk, so essentially, each time they log into their VMDK in VMware, they’re logging into a clean machine and I’m not wasting my weekends cleaning up PCs infected with malware So again, you might find that within your own organization, the use of nonpersistent mode may in fact be a benefit, and could in fact be usable within your environment Again, I primarily see it used in terms of a kiosk-like operation, you may want to consider that let’s say, as a mode of operation you would use in your office lobby, where you’re allowing vendors, et cetera, that are visiting your company to check email, et cetera, again, it does not save any changes that are made to the system, nothing is written to the hard drive Now let’s go ahead and create a virtual machine called SANS and see what kind of files does in fact generate We of course are going to have a SANS.vmx, that’s our actual VM configuration file, you can actually take a look at VMX file using a text editor, it really is kind of cool, you’re able to see the full configuration simply looking at it with a text file You have your SANS.vmdk, this essentially is the data associated with the abstraction of the hard drive itself Now again, you can view the VMDK with the text editor, and it will show you exactly what you would think you see in a configuration file for a hard disk, it will talk about clusters, sectors, and everything that we’re primarily familiar with in describing a hard disk Now, we would also generate a SANS-flat.vmdk, that’s the actual binary file that represents the abstraction of the hard disk, so no, you would not be able to view that with a text editor It also creates a SNAS.nvram file, which would be the virtual machine’s BIOS file, you can view that with a text editor, and of course, again, what you would expect to see in a BIOS file, such as boot from CD, et cetera, et cetera, you will find within this BIOS file We have a number of different SANS startup log files that are in fact created as well, that would be associated with that individual virtual machine We get a SANS.vswp, which would represent the virtual machine’s swap file, we would get a SANS.vmsn, or vmsd, depending on whether or not we are able to run under a workstation or

run under the bare metal hypervisor Now I’ve got a question here, I’m gonna go ahead and answer that real quickly, been paying attention to the slide deck, not looking at chat, I assume you’re gonna hold questions till after, but we’re gonna go ahead and jump on these now So the question is, is nonpersistent hardened? No, absolutely not, it is not hardened, it does give the benefit in fact, of not writing changes to a hard disk, but it really is not hardening That’s one of the issues I see in virtualization today is so many people are mistakenly assuming that simply because they’re running on top of a hyper visor, they don’t need to harden that guest, you absolutely are skipping a critically important step if you’re not hardening the guest So again, would running in nonpersistent mode replace going through and either using DSS dig or CIS hardening guide on the guest OS, absolutely not, you still have to harden that operating system I’ll go ahead and jump back to our slides here So again, you would get the SANS.vmsn or the vmsd, depends on whether or not you’re running on top of a bare metal hypervisor or on top of work station, it’s actually kind of cool, it’s your virtual machine snapshot metadata, we of course have our SANS delta files, this would be a real-time snapshot write file, you know, when you create a snapshot, you’re basically telling the system, hey no longer write this to the original abstracted hard drive, create a new hard drive The original abstracted hard drive remains static, and you’re writing only the new we’ll say delta.vmdk representation of the hard drive Now we have another file here that is absolutely worth mentioning, that would be the SANS-***.vmss; this is actually a snapshot of memory that’s created when you suspend the virtual machine I really like imaging suspended virtual machines, I get a static disk that I can in fact image, hence my hashers are gonna match, and on top of that, I actually get a complete image of RAM at the moment I suspended that disk, I’m really big today on actually performing a full forensic analysis of RAM We’re seeing time and time again today where the bad guys are inserting malware into a running process such as a DLL Now if you follow the traditional IR mantra, when you have a compromised machine you pull the plug and you image the hard drive, you may have in fact wiped out all of your evidence, again, even using metasploit, I can insert my malware into a running process If you’re not capturing an image of RAM, you’re not capturing your evidence So I think again, it’s really really cool that by suspending the virtual machine I can also get the copy of RAM, and I gotta tell you, I can do the same thing with the snapshot, but it’s an option, you have to turn on that option as you’re creating that snapshot, but it absolutely the way to go, you gotta get that copy of RAM, you gotta look at what processes were running and in fact, you may find that you had malware existing within a running process in RAM, and without that image of ram, you would completely miss it So let’s image the VMDK, well, most of the old-school forensics guys out there are used to using things like DD, everybody should remember DD here on the event Again, with DD you’re making a bit by bit copy of an entire hard disk, and we’ve all got used to its use over the years, DD was the only way to really perform a solid, sound forensic copy job of a hard disk, you had to make that bit by bit copy Again, today we’re talking about everything being abstracted within a VMDK container, so all of the allocated and unallocated space is available there So again, is it really necessary to use DD? In fact, I say it is not, again, I can run an MD5 against the original VMDK, and then simply copy it out using any number of different tools, such as FastSCT from Veeam, an SCT on the command line, and even within VMware, I can write out that VMDK using the GUI to a a new destination and then I simply run an MD5 or a SHA-1 for that matter, against the copy Bottom line is, if the MD5 or the SHA-1 of the original matches the MD5 of the copy, we have a forensically sound copy Now, I do give examples in the 579 class,

we do let the old-timers like myself run DD to make their copy, but again, it really is unnecessary We have an abstraction of a hard disk within the VMDK, so again, simply making a forensically sound copy of that VMDK folder itself is a forensically sound method, again, it’s a method I recommend myself You can use DD if like I said, you’re an old-timer and you’re stuck on it, we do give examples of it, but again, it’s become a bit too time-consuming Again, if you’re using the tool such as FASTSCP from Veeam, which we give a copy of within our course, it essentially is well over 10 times faster than running DD over SSH et cetera, so absolutely a time-saver, it can make the difference in hours when it comes to imaging an individual VMDK versus DD Now, let’s look at some VMDK state considerations from a forensics perspective You know, a VMDK can be copied in any one of the following states Well, it could be simply be running, turned on, but that’s really useless from a forensic perspective, because you’re never going to be able to get a hash that matches Again, if the machine is on, it’s gonna be constantly changing that hard disk, so you’re never gonna be able to verify that in fact, the copy that you made is forensically sound, because when you create the original hash and then attempt to copy it out, it’s constantly changing When you make your final hash and compare the two, it’s simply not going to match, excuse me We could simply turn it off, that’s been what’s been traditional in the forensic imaging in the past, however, if you’re turning it off, you are in fact impacting production Many organizations simply don’t want you taking their system offline, so we have to find an alternative to turning it off Well, we could suspend the virtual machine, now again, as I noted earlier, you also get a copy of what was memory, and that was really a bonus for you, and of course, you could snapshot the virtual machine, create the image of the now unchanging disk and let the machine run along with the new file that it created So again, referring to the snapshot, it really is nice as the VMDK is no longer actually changing, it’s writing data to that new hard drive that it did create, and you’re also gonna get that image of RAM when you created the snapshot of that moment in time So for myself, a vast majority of IR and forensics I do, it would be against a snap shotted virtual machine, and truly is the best way in my opinion, again, you’re able to get that forensically sound copy, the hard drive itself is no longer changing, so you’re able to verify the hash, plus you’re gonna get a bonus of getting that RAM image at the time the snapshot was created, so it really is the best of all worlds, truly is today, so again, in my practice, and the vast majority of time we’re performing IR in the virtual realm, we’re performing that incidence response against a snap shotted VM Now some consideration regarding snapshots, again, snapshots are a very useful tool for creating an image without having to shut down, but there are some drawbacks to snapshots The problem with snapshots, simply put, is people tend to get snapshot happy; it’s not uncommon to find instances where a given virtual machine may have dozens of snapshots, and that makes life very difficult in performing a full analysis, because effectively you have to revert each individual snapshot, create an image, and then perform an analysis and then move on to the next snapshot If you simply reverted all snapshots all at once, you could in fact be overriding evidence within that virtual machine Now when you create a snapshot, you’re basically telling VMware, okay, I’m going to create a new disk for this VM, so from now on, you only write to this new disk, and no longer write to the original disk, that’s the beauty of it, and again, after you snapshot, you create your actual image of the original disk, you’re not creating it of the new disk, you’re letting that new disk go ahead and accept the changes So again, you’re static on the original disk, so it’s going to be forensically sound Now, this is great if you’re only going to create a single snapshot, but gets very complex again with multiple snapshots There is not tool available today that can fully analyze across multiple snapshots,

so again, the workaround is to restore and analyze each one sequentially, that is a tremendous amount of work, I tell my clients, take my proposal for this incident response and simply multiply the overall cost times the number of snapshots, it truly is a full multiple Again, you’re gonna have to restore each individual snapshot and perform a full analysis, that gets very very expensive Now, snapshots are in fact an asset when they’re properly managed, the graphic shown here would depict a good use of a snapshot You’ll note we’re running Windows XP, we had our snapshot that was created, we have our Day One snapshot as well, very easy to perform an analysis on this given implementation Now let’s contrast that with something a little bit different This is a snapshot happy configuration, this is simply way too many snapshots, I mean look at this graphic, where would you even begin in performing a thorough forensic analysis in an environment where you have not only a high count of snapshots, but so many branched off? Now again, if you simply came in and restored everything back to a single image, you would absolutely be potentially overriding evidence The only way to approach something this bad would be to simply revert snapshots individually, create new images, and perform a full analysis on that snap shotted disk itself Again, it could become very very costly, we really have to reign in the number of snapshots that people are creating Again, I refer to the term as being “snapshot-happy”, you simply keep creating snapshots at will There’s really, in my opinion, very little need to have more than one, or perhaps two snapshots within any environment Again, in the graphic that’s show here, that would be an absolute nightmare, that would be the type of IR or forensics job I’d happily let my competition have Now, more on snapshot considerations, again, you have to consider that when you consolidate your snapshots, you’re effectively sequentially applying each snapshot and the respective changes contained within that snapshot to the original VMDK You have to do it in the right sequence, and again, you have to actually image the VMDK as you are in fact restoring individual snapshots or you do pose yourself some risks in potentially overriding evidence by actually restoring more than one snapshot without creating that new image of the drive itself Now, valuable evidence could of course in fact, potentially be found literally be found in between snapshot events, therefore of course it’s important to analyze the VDMK after each individual snapshot is restored, and again, that’s where the real labor comes into play here Again, if you didn’t have a clue as to how many snapshots existed before you submitted your proposal, you could find yourself absolutely underwater with respect to that IR or forensics job you had put that bid in Be very, very careful that you understand how many snapshots you’re gonna have to deal with As I said, you’re gonna have to restore each one individually, create a new image of the VMDK and perform a full analysis with respect to each individual snapshot, so be very very careful there Now more on snapshot considerations, in my opinion, a thorough forensic analysis of course would require that the original VMDK and each individual restored snapshot should be analyzed, that’s a major time sink Well, in products such as Shadow Analyser for Windows, it does have this really cool capability of being able to analyze across shadow files in a Windows environment This is the kind of tool we absolutely need within a virtual realm, but simply does not exist today I’ve reached out to the authors of Shadow Analyser, and I’ve expressed my opinion that we really need this capability being brought into snapshots within the virtual realm Hopefully they’ll get started on a product that can meet that need in the very near future Now again, all forensic product vendors need to take note that we need something like Shadow Analyser to reduce the burden of performing the forensic analysis an environment where we have multiple snapshots Now we’re gonna jump into a bit of a mini lab, you’re not gonna be able to get any real hands-on with this, but I’ll walk you through it myself So here’s our situation: the client needs a forensically sound copy of a potentially compromised virtual machine

The client can provide access to their vCenter admin console, and can provide local console access, bu the client did note that the ESXi console remotely is currently not enabled, so we’re not gonna go in through SSH directly, we’ll have to turn a few things on to be able to perform our IR in this particular case So VM itself is a database server running on top of Linux, files are stored locally on hardware, and there are currently no snapshots of the virtual machine itself You know, the client again, has no issue with a reasonable amount of downtime, after normal working hours, so again, it might be in our best interest to work either from a snapshot or actually suspending the VM for a moment to get the image to prevent having to work after normal working hours So our proposed process, we’re going to first enable the remote console in ESXi, we’re then going to simply quickly suspend the VM using the vCenter Client, we’re gonna navigate to the storage medium using the remote console, we’re going to hash the respective VMDK and the memory snapshot, the VMSS We’re gonna copy the VDMK and the VMSS to a removable temporary NAS device, using DD just to keep it within the scope for the old-timers like me out there, we’re gonna hash the copies, and we’re then gonna verify the hashes actually match the original If they in fact do match, we do have a forensically sound copy, and we will then resume the virtual machine and get that client right back into production So here we simply go in through the ESXi console locally on the machine, and we’re going to find the support mode has in fact been disabled, so we’re going to have to enable that We go back into the GUI, one problem we have today within the ESXi is you can get to the GUI, but no root password is typically required, it comes by default with no root password, so the first thing we have to do is set up a password so that we can come into the SSH Once that’s taken care of, we simply go in and we enable local technical support mode It’s done here as shown on the GUI, you would simply select Enable Remote Tech Support over SSH Again, we’re able to then get to our underlying console, again, with ESXi, they’re running a busy box console with ESX, where we’re used to working within a Red Hat enabled console Now the key there is both of them do provide the tools you need to perform ESX, I should say, to perform IR, so again, with ESX and ESXi you will find some ends such as MD5 some NDD, the basic tools that you need to perform an incident response Now here we’re simply connecting up a NAS box, what I’ve done here is again, using the ESX CFG-NAS command, I’ve established a connection to a remote NAS box and I was able to hang on the wire at IP address Now as I’ve connected a NAS up, I have a great destination to store my image in of that virtual machine So here we’re back in virtual center itself, and I’m able to simply right-click on the individual virtual machine and suspend that VM With the virtual machine suspended, I go back to my console and I simply seek out the location of my files So again, I first pinged my storage device to make sure I had connectivity, I then navigated down through the VMFS file system to Volumes, within Volumes it listed out the virtual machines that are actually available to me, and I was able to go then down and change directories to the individual VMDK folder that is associated with the virtual machine that we want to image Now once I know where my virtual machine is, I change to that directory, you’ll note it’s a rather long string leading to the directory, but within the directory here we have our files We have our Linux Red Hat 6.2 VMDK, our VMSD, our VMX file, et cetera, et cetera, all the pertinent files with respect to that virtual machine located within the VMDK folder, which again, is a container for the abstracted virtual machine So now we have some command line kung-fu, we have to actually DD out a copy of the respective files that are pertinent to our investigation

In this slide we’re showing the DD command, the IF of course equals would be the input file, so we’re listing the full path to the flat.vmdk file, that of course will be the actual binary file that represents the abstraction of the hard disk, our output file is noted here in the command line, is essentially pointing to the NAS that we connected, so again, keep this, this is kind of a reference for yourself in using DV to copy out an individual file In this slide we’re simply showing the same thing, but this time we’re actually copying out the .VMSS file, this would be the actual image of RAM that was created at the exact moment that we suspended that virtual machine Again, a great reference for you here on the command line, and again, DD is in fact included both in the ESX and the ESXi So again, makes kind of a trivial way to copy out a file, you know you’re getting that bit by bit copy but again, there are easier and faster ways to do it Again, I included DD in here, just for the old timers like myself, so that you would have something shown that is something somewhat familiar to how you’ve done it in the past Now here we’re actually then creating out a copy of the files on our NAS to an alternate store so we can then work with the files So again, you can simply use the copy command to move the files around just as well as you could a DD command Now here we simply show the actual files that I copied out to my network storage device, I simply connected to it in a Windows environment, so you can see that in fact we got our VMDK and we got our suspended RAM image, and I also created MD5 hashes of the files themselves Now, here we’re simply comparing the MD5s from before the copy and after the copy, you’ll note that the MD5s actually match, so in any court, in any court in the land, I should say, as long as the MD5 of the original file matches the MD5 of the copy, you do have a forensically sound copy Now, we go through about four different methodologies in doing this in the 579 class, we of course use DD, we also use FAST SCP and we use SCP over SSH, we go through the process of working with a suspended virtual machine, and snapshot of virtual machine, as well as an offline virtual machine, just to give users the experience of doing it in the common ways that they’re going to have to out there in the wild Again, we try to cover each one of them within the class to give you a little bit more hands-on experience, as again, we want you to leave the class with the ability to handle this in the real world So in summary here, virtualization changes many things, and it of course does change how we actually respond for both IR and forensics You know, imaging the entire VMFS is not always an option, and is simply not necessary to analyze a specific virtual machine I find it somewhat laughable that here in the USA we’re constantly reading stories about law enforcement that was called in because an individual virtual machine may have been serving up malware, and they actually take down the entire shared storage array, impacting hundreds, if not thousands of virtual machines for days on end We had one care recently where law enforcement actually came in and physically seized the shared storage server so they could take it back to their lab environment to image it They were literally down for well over a week, there is absolutely no need to do that, you can easily carve out the VMDK associated with and individual virtual machine, and you can have no impact on the production, not only of that virtual machine, but all the other virtual machines that might be running on the shared storage that is associated with it So continuing on here, snapshots can of course both be a friend or a foe Again, if you walk into a environment where they manage snapshots correctly, you’re typically never gonna have more than one or two It’s where snapshots get out of control, as I said, the user gets snapshot happy, that it can create some serious issues Again, from a time perspective, take the time that you would allocate it for both the imaging and the analysis and simply multiply times the number of snapshots that are made on that machine, it truly is a labor intensive process

to do it right Again, you have to literally restore individual snapshots, create new images, and then fully analyze that image or you risk in fact, overriding the potential evidence Now, the VMDK is a complete abstraction of a server and can be imaged in various states For myself, I like suspending the VM for imaging, you get RAM as well, and of course, we can’t forget that you that you can also snapshot that VM, you’re gonna get the static drive as well as the RAM itself Now, my recommendation for you is you really need to get some of that priceless hands-on experience with VMware on a lab machine before ever trying this in a custodian environment You can run workstation on pretty much any Windows machine today, and on top of Workstation, you can load up a copy of ESXi and then put multiple virtual machines on top of ESXi and you can do that all on your laptop, again, you don’t want the first time you’re having to deal with a virtual machine IR response to be in a running environment, you really want to get that hands-on well beforehand, so again, my recommendation is using a Windows PC, run workstation, load a copy of ESXi, and multiple virtual machines and work at copying out individual VMDKs For you MAC users out there, you can do the very same thing with Fusion Again, in our classes we typically have the students running a Windows environment, but we always have a student bring in a Mac, and as long as you’ve got at least eight gig of RAM, that’s typically not an issue Be aware that Fusion does not have the network configuration capability of workstation, so it’s a little bit more work for the student to handle that I see a question here, “How are swap spaces “on VMs managed?” Well, that’s a very good question Bottom line is that within ESXi, you have a swap file associated with it Within the virtual machine you have a separate swap file that’s associated with the individual virtual machine Now the actual swap space associated with an individual virtual machine is a component of the VMDK Nothing associated with an individual running virtual machine is stored outside of the VMDK, so again, the evidence that you seek is generally always found within the individual VMDK, most all data associated with that virtual machine is written within the VMDK folder itself, and we typically do not have issues with trying to grab the underlying copy of ESXi or ESX swap file off of the machine itself, our investigation is focusing on an individual virtual machine Now mind you, if your investigation is broader in scope, you’re not focusing on an individual virtual machine, it’s a major change in what we’re talking about here Now you’re gonna have to get copies of pretty much everything, you’re gonna need to get a copy of the Oracle or MySQL database associated of course, with vCenter, you’re gonna have to get copy of the respective swap space for the vertical machine that’s running within ESX or ESXi, and you’re gonna have to grab copies of any templates that may associated with the running virtual machine that might be just sitting out there on shared storage, that’s a totally different scenario Again, I thought I’d keep it kind of simple for you guys in this, and here in the analysis that we just walked through, we were targeting an individual virtual machine that may in fact have been compromised, so we were limiting the scope to an individual virtual machine It truly does expand the scope, when you’re getting outside of an individual virtual machine All right, and I’ll get back on topic here, so that pretty much wraps it up for today’s presentation, wanted to call a couple of events into the session here real quick, let you know what’s going on We of course do have SOS Singapore 2012, that’s the 15th through the 20th of October, a number of great courses, we have SANS Security 401, a great course written by Dr. Eric Cole, I’ve taught that a number of times myself, we have Security 503, Intrusion Detection In-Depth, Security 560, Network Pen Testing and Ethical Hacking, and we have yours truly, I’ll be teaching Security 579, six days of intensive hands-on with respect to virtualization and private Cloud security, offense and defense And of course, we have Forensics 508, the new version, it’s Advanced Computer Forensic Analysis and Incident Response, truly is a very cool course;

its’ been recently upgraded to include a full APT, I teach that myself, love working through the lab on that We also have SANS Bangalore 2012, 29th of October through the 3rd of November, there you’ll find again, Security 401, SANS Security Essentials bootcamp-style, Security 542, Web App Pen Testing and Ethical Hacking, and of course, Security 560, Network Penetration Testing and Ethical Hacking So that wraps those up for getting Suresh’s information out there, and the two events upcoming in Southeast Asia That does wrap it up for me today, I trust you found this presentation both interesting and informative, I thank everyone for coming along, if you have questions I will hang out for a little bit, again, happy to answer them If you prefer, you can reach out to me, at [email protected], I do actually answer emails, so feel free to reach out, be happy to chat with you about any specific IR or forensic questions you may have within the virtual realm Again, I thank everyone for attending, and we really are good to go here, so again, I’ll hang out for a minute now Oh, did have another question, “Does 508 have prerequisite as 579?” Actually, I recommend people take 408 to get their essentials covered before taking 508, 579 could really be considered to be a bolt-on to either 408 or 508 In 408 you’re primarily focusing on Windows; in 508 you’re primarily focusing on Linux, and they do cover the essentials in both 579 we’re really focusing only on the virtualization environment, so again, it’s not really a prerequisite, but it is somewhat recommended that you have the essentials covered first Any other questions out there, folks? Well actually, with respect to ESXi as a mandated standard, it’s not really a mandated standard, however, ESX is going to be end of life, ESX is going away, VMware is adamantly pushing people to ESXi If you’re still running ESX, the writing is already on the wall that it will be an unsupported product from VMware Again, they took away that free copy of ESX, it’s no longer free, eventually you will find that perhaps ESX will no longer work within vCenter long term Again, you really do need to migrate to ESXi in any environment I mean, again, they’re end of lifing ESX Again, I’ll hang out here for a few minutes, if anybody has questions Yeah, I hear that on the “get well” Bit of a cold here, that’s what I get for traveling to Denver at 97 degrees Fahrenheit, and then traveling to Canada, 30 degrees in the rain That’s dedication to the cause, folks Sure, go ahead and ask your question Shiv, it looks like you might be having

problems with the GUI there, and formulating your question Feel free to email me, at [email protected], and I’ll be happy to answer your question regarding security levels, again, Shiv, I did not see your full question come through, so I really can’t answer it So again, reach out to me on email at [email protected] and we’ll be good Okay Suresh, time for me to sign off here, friend All right, thank you very much folks, I’m gonna go ahead and sign off for now, I enjoyed it, hope you did as well

You Want To Have Your Favorite Car?

We have a big list of modern & classic cars in both used and new categories.