Automating OpenIOC with Splunk

This year a colleague of mine, @trakzon, and I talked about integrating MIR, OpenIOC and other network data sources such as BRO and Palo Alto Network Firewalls logs with Splunk at MIRCon. We never got around to publishing the code until now.   Here is the code to automate the searching of OpenIOC format IOCs through Splunk’s API:

https://gist.github.com/irctrakz/4742762

As always we welcome feedback and features. 

Auto-Generating OpenIOCs

Wow, it has been two years since last post, and this post will not be mind blowing. For the past couple of years I have been busy building an incident response team and SOC so I have not had a lot of time write much worth posting. I finally got some free time this last week and created a quick and dirty script to create OpenIOC format IOCs from unstructured data. This was a pain point in my teams day to day operations since we use OpenIOC format IOCs for just about every part of our automation to look for known indicators, be in in Splunk or MIR. The script is not complicated but saves a ton of time when you have to import a long list of IOCs (MD5’s, IP’s or domain names). To execute the script all you need is a basic install of python, no other libraries. I’ll walk through an example of its use below. First step is to find a source for IOCs. In a perfect world all IOCs would be in a structured format already but most of the time they are not. In this example I will be creating IOC’s from a a FireEye blog post. If you want to play along here is the site: http://blog.fireeye.com/research/2013/02/operation-beebus.html Just copy the contents the blog post to a file. Once you have the contents copied over execute the script as below, “blobofIOCs” is the copied content:

This will create an OpenIOC format IOC in the same directory:

From here you aren’t done it does most of the work but you have to edit some of the property files and check for BAD IOCs. You can load the IOC up in Mandiant IOC editor located here:

http://www.mandiant.com/resources/download/ioc-editor/

When parsed it will look similar to the figure below:

“ioc_creator.py” will generate domain, IP, MD5 hash, registry and file path IOCs. Be sure to change the name, add groups, and change the description as it defaults to BulkImport . The IOC will still load into MIR if you don’t change this but you want to change this data for context if it ever hits. The IOC also has some bad auto-generated IOCs. The “File Full Path contains “\Windows,” which is a bad IOC, as well as the single registry path contains.  Delete these BAD IOCs out. “ioc_creator.py” defaults IOC term logic to the following for each IOC term type:

• FileItem/FullPath – contains
•  RegistryPath – contains
• Network/DNS – contains
• PortItem/remoteIP – is
• FileItem/Md5sum - is

The auto-generation will catch some very high false positive IOCs so make sure to use IOC lint and double check what has been auto-generated for you.  In the end my IOC was cleaned up to look like the following figure:

Please let me know of any bugs or features needed. The script can be found here:

ioc_creator.py

Good Forensics, RE, Pen Testing, Network RE Training at Hole in the Wall Prices

If your budget for training is tight this year you might want to check out a training event I will be helping out with this year called Tracer FIRE III. The link for the site is below:

http://csr.lanl.gov/tf/

I will be co-teaching the forensics class and the other instructors are experts in their area as well. The class I will be teaching is focused on live intrusion forensics so you can leave our write blockers at home.

Forensic Crash Dump Analysis

I attended and spoke at Mandiant's MIRCon last week. It was a really good conference, not even counting that it was free. I have uploaded by slides from my talk and they can be downloaded here:

Forensic Crash Dump Analysis

I probably skipped over a lot of the details and did not leave people enough time to write down all the good registry and windbg tidbits. I am looking forward to the conference next year. I am working on getting the MIR scripts and shell scripts released and will be posted on this blog when they are released.

Good Malware Blogs and Job Posting

I just got done with my morning blog roll reading and wanted to link to a couple of good entries. The first was by Zynamics talking about creating better malware signatures:

http://blog.zynamics.com/2010/08/13/privatesigs/

It specifically talks about creating signatures with their product VxClass. If you have not had a chance to use or see this product I suggest you do. The second blog entry worth re-posting was one from Nick Harbour at Mandiant. His post talks about finding command and control functions in malware, specifically focusing on the COM point of view, here is a link to the post:

http://blog.mandiant.com/archives/1396

Finally I would like to advertise a couple of positions that are open at my current work place. My team is looking for forensicators, incident responders, red teamers and malware analysts. If you are interested in the job apply at the link below:

http://www.hr.lanl.gov/JobListing/SingleJobAd.aspx?JobNumber=217724

$MFT Parsing EnScript

There is a considerable amount of forensic goodness in the $MFT on NTFS partitioned disks. What is a $MFT? Well the $MFT is the master file table on NTFS partitions that is a kind of database that keeps track of all the files on the partition including its location and metadata about the file. I am not going to delve into the depths of the format of NTFS because it has already been explained in numerous books like File System Forensics by Brain Carrier. What I am going to do is quickly summarize the "goodness" available in the $MFT and how you can extract this data with a EnScript I have authored.

The $MFT contains an entry for every file and directory on a partition including itself. Important metadata with in a $MFT are the name of the file, inode number, standard information attribute, file name attributes and data attribute. The size of a $MFT record is 1024 bytes. Below is an example of a $MFT record:

The header of a $MFT record is "FILE0". A $MFT record entry can contain the contents of a file if the size of the file fits into the allotted size for data attribute. Below is a $MFT record broken down into what I think are its important parts.

The standard information attribute (SIA) will contain the file times most people are used to seeing on the file system such as created, last written, last accessed and last modified. These times can also be easily changed by the attacker, the "worst" forensic problem ever until now because now you will know about the file information attribute (FIA). The FIA stores dates associated with the file's name and parent directory. These dates cannot be altered using Windows API calls like the SIA can. The meta data kept in the file information attribute consists of the file name creation date, file name modified date, file name last written and file name last accessed. Comparing the SIA to the FIA can detect the dreaded timestomping but be aware of the number of times this happens on a non-compromised or purposely altered system so stick to time windows when doing this type of analysis. To find all these times you are going to have to parse them by hand, wait no you don't there's a script for that below:

MFTParser.EnScript

The EnScript I created, inspired by Keith Gould, will parse the important information mentioned above for you and provide you with a tab separated text file you can open up in Excel or parse with your favorite awk command. Don't worry I am not forgetting about the $MFT slack section mentioned above in the important parts, I will go over it in the next post.

Page Files

I have not posted in a long time....seems to be a bit of a pattern but I should be posting more in the next couple of months with some new EnScripts. Before those posts though I wanted to point people to a great blog on why acquiring page files the same time as memory is a more complex problem than you would think.

http://blog.mandiant.com/archives/1102

Enjoy

HP MediaSmart and Apple Support

This post is off topic but I thought it would be interesting to a large audience. I have had a HP MediaSmart 470 home server for about a year. I have enjoyed it was a file server for my home network and have been able to stream music and video to my PS3 seamlessly from it. It also serves as a nice print sharing server as well. I have been able to easily upgrade the RAM on the system from 512MB to 2GB and easily add a few TB's to its storage capacity. I have not used the other features of it such as photo sharing or page hosting. One feature I had not used at all was the system back up feature. Most of my home systems are Apples so I never had the need for a windows backup feature. About six months after I bought my HP MediaSmart, HP announced the feature on their new MediaSmarts would support Apple Time Machine backups. I was excited but disappointed with this new feature because I just bought mine and wondered if I should go get a new one just to have the Time Machine back up feature. I could not justify this purchase so I searched the forums and web for the support of a MediaSmart backward compatibility for about three months with no avail. About a week ago I stumbled across the MediaSmart support page and noticed a link to the download of the MediaSmart client support for OS X. The software for the client can be downloaded below followed by the installation instruction link.

OS X Client

Installation Intructions

Installing this client is pretty straight forward. By default with the older HP MediaSmart server the Time Machine backup feature won't work. It will complain about not having a "MAC" folder on the server. This is easily fixed by creating a "MAC" folder in the root share folder of the MediaSmart server. The folder layout should look like the figure below:



Once this folder is created you can open up the Time Machine preferences on your Mac and backup to the HP MediaSmart Server. I have two Mac systems backing up to the MediaSmart and it has worked well so far. I would recommend the HP MediaSmart to anyone looking for a fileshare, printshare, or backup solution for their home network.

Cool Tools and DoD Cybercrime Conference

For the past two weeks I have been doing web application assessments. During this process I have had to try different tools to get the job done. During the trial and error process I can across some really cool tools, they are probably not new but new to me. The first is Charles proxy. This proxy is great for doing binary web application assessments allowing you to easily view the binary data being passed. A screen shot of Charles in action is below in Figure 1.

You can get the Charles proxy here.

Since I was looking at flash applications I needed analyze the swf file. While searching for a decent decomplier I found FileInsight by Secure Computing. Get FileInsight here:

This tools is great. It is a hex editor with collaborative reverse engineering in mind. Besides being a hex editor it allows you to do a number of actions I found to be useful:

-Breaks out PE header.
-Understands OLE2 structures.
-Ability to decode using multiple.
-Bookmark capability.
-Plug-ins for strings and an anomaly chart.

This will be replacing my other free and paid for hex editors, it is great cause it is free. It also looks to have a scripting capability that looks like it will come in handy.

On another note I spoke at the Defense Cyber Crime Conference last week. I addressed forensic issues on the Advanced Persistent Threat (APT). The talk is titled "Complex APT Forensics: Answering the Difficult Questions". You can get the sides and the customs Template and EnScripts from the following link:

APT M-unition Pack

Extract OLE Objects with OLExtract

Recently I was challenged to see if I could extract files from Microsoft Office documents without a file header or footer. The files being put into the Microsoft documents were text files, configuration files and source code files. I do not know why these files were being put into office documents except maybe to hide the files. The files were not obfuscated in the documents and could be extracted manually. Extracting these files manually works for a small set of these documents but for a large set, manual extraction was not feasible. The location of these documents were on EnCase images so it made extracting them easier for me with an EnScript. The files in the office documents were kept as OLE objects. To view OLE objects in an office document within EnCase you can right click on the document and select "View File Structure". This action is shown in Figure 1 below.


Figure 1 - Viewing Office Document "File Structure".

This also becomes a manual process viewing each document and then seeing what OLE objects are located in the document. EnCase recently came out with a way to automate the "View File Structure" process. The automated process is done with the File Mounter EnScript. This EnScript allows you to view the embedded file structure of many files such as Thumbs.db, zip archives, and Office documents (YES). Executing the process to automatically mount these types of files looks like Figure 2 below.


Figure 2 - Mounting Office Documents with File Mounter EnScript.

Once all the Office documents have been mounted you can extract all the OLE objects that are non-picture files with my OLExtract EnScript. Get it below.

OLExtract

This EnScript looks for all the OLENative Entries in the mounted documents and then extracts them to your EnCase default export folder. The EnScript will separate each file extracted in relation to which document it was extracted from and files with the same name will be incremented. The results of the execution of OLExtract will look similar to Figure 3 below.


Figure 3 - Results of OLExtract

OLExtract will ignore jpegs stored as OLE objects because they are stored differently then other types of files stored in the document. You can extract jpegs easily from Office documents using file carving.

MemScript

Just wanted to cross post about a blog I recently posted on MANDIANT's blog site. I will not go into the details and let you read it but the topic is about MemScript. MemScript is an EnScript that integrates EnCase, Memoryze and Audit Viewer. Check out the post here.

MemScript Blog at M-unition

It has almost been a year since my last post but I hope to keep it more up to date if my work schedule allows for it. While out and about I noticed some bugs in my current EnScripts that I fixed. The first deals with the Prefetch Analysis EnScript. If the file executed did not have an extension of .exe the bookmark for showing the location of the executable would be mangled. This bug is now fixed. You can get the updated version below.

Prefetch Folder Analysis

I am hoping to clean up some of the EnScripts I made for actual work use and get them out during the holidays. For convience sake I have also attached the MemScript to this post, get it below.

MemScript

Windows Memory Analysis

Recently I got the chance to use EnCase Enterprise's ability to acquire physical memory over the network. This is a great feature and it is nice to see that Guidance is starting to catch up with other products that already offered this feature namely Mandiant's Intelligent Response and Techopathway's Pro Discover. I had made an EnScript to parse these data dumps for current and exited processes from XP machines last summer and I have finally got the OK to release it to the public. This EnScript will take a physical memory dump, be it done with either of the aforementioned products or with good old dd and makes bookmarks of the processes data. The bookmarks will list the process id, parent process id, process start time, and process end time. If the process has exited it will display a (X) by the bookmark folder name. I based the XP off the DFRWS memory image and created a Win2K EnScript before the XP one so it is listed here to just in case there are still 2K users out there. The XP EnScript has the option to export the results to an excel file. I will be trying to add future capability to this EnScript so stay tuned and give me feedback on what you would like to see added to it.

Windows Memory Analysis (XP)
Windows Memory Analysis (2K)

Luhn Algorithm EnScript

After seeing a long list of posts in the forums on credit card finder module I wanted to post an EnScript I made up for some recent cases with an overwhelming amount of credit card hits with the EnCase credit card finder module. For those of you that do not know what this algorithm is I would explain it here but it is already on Wikipedia so look there. This EnScript reduces the amount of credit card number hits dramatically, from tens of thousands to hundreds. The basics of this EnScript are it looks at search hits already created by the module and then creates bookmarks of valid credit cards. Quick note is that you have to have the search hits you want to have analyzed blue checked. Have fun.

Credit Hit Analysis

PEvil Carver

Alright it has been a while but here is a EnScript I talked about before in some forums and I have gotten some more questions about it so I am going to post it to let all have access to it.

So you might ask "What is the PEvil Carver EnScript"? Hopefully what this EnScript will do for you is find that evil program that is on your machine but you just can not find it through traditional forensic analysis, i.e. anti-forensic techniques are not allowing you to find it. This script takes an intelligent approach to finding PE type files (.dll, .exe, ocx, to name a few) from unallocated clusters (but you can search other unstructured data types) and does not just look for the "MZ" signature. This intelligent approach starts with finding that good old "MZ" signature but goes further to find the "PE" signature that follows it after a variable amount of space specified in the Image_Dos_Header. If these two signatures are present this script will start bookmarking and carving the PE file based on the data that is stored within the PE file structure. It will also look on the local system for files that already have the same hash and not export it if it is on the local system to cut down on false hits. The PE files are exported to an EnCase LEF file that can be added to a case and then mounted with VFS and then scanned with AV or other malware finding tools such as Mandiant's Red Curtian. The bookmarks also contain a plethora of data on the PE file itself. It parses out the Image_DOS_Header, Image_Optional_Header, Image_File_Header, and each sections Image_Section_Header. Looking at the data within these headers can help you identify if the PE file found is evil or not, the data here is much like PEiD. The hash and size of the file is also included in the bookmarks and a search can be done on the size or hash of the PE file to see if it is evil or not based on previously known indicators. Other features are that you can limit the size of the PE file to be exported, in initial testing PE files of significant size (1-2GB) were found in unallocated space and traditionally evil PE files are probably not of that size and what to keep a low profile. There is also an option to limit the size of PE files exported, as mentioned already must "Evil" programs will not want to have a big presence so limiting the size of these will produce better results. Some recent tests done with this EnScript reveal you can extract full PE's from memory dumps that match NSRL hashes and files have been identified that are known evil program from unstructured data found with Red Curtain and AV. Here is the EnScript, enjoy it and please as always give feedback. Thanks to Nick Harbour for the suggestion to make this script and just as a side not I will always provide the source code to how these EnScripts work, it is important to all forensic professionals to know what the tools they use do and not just point and click and get answers but know how they got those answers.

PEvil Carver