Bro IDS + Python == success!

I recently learned about the Bro IDS project, and I think it's really cool! The only problem is that I didn't want to have to learn their special language to process network data. I'm more used to Python's powerful tools to do processing like this. So I wrote BroScanner, which essentially streams from Bro logfiles and reads the CSV into an easily parseable python dict. It can also pipe out the data as JSON for other programs to analyze.

Today, I want to show you how to use BroScanner by creating some small passive IDS scripts with it. First, you need to download and configure Bro. Then you'll need BroScanner and its dependencies, see the aforementioned Github repo. Once Bro is up and running, you can run main.py to extract json from one of the logfiles. Assuming Mac or Linux, you can list the logfiles with ls /usr/local/bro/spool/bro. 

$ ls /usr/local/bro/spool/bro

app_stats.log dns.log       files.log     notice.log    ssl.log       stdout.log    x509.log    conn.log      dpd.log       http.log      software.log  stderr.log    weird.log

(This isn't a complete list of the files, they are created as needed by bro)

$ python main.py software.log

{"host_p": null, "name": "Unspecified WebKit", "unparsed_version": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Spotify/1.0.20.94 Safari/537.36", "version.minor": 36, "version.minor3": null, "ts": 1450646720.519703, "host": "10.0.1.19", "version.minor2": null, "version.addl": null, "software_type": "HTTP::BROWSER", "version.major": 537}

(don't include the path in any calls to BroScanner, it is added automatically)

$ cat /usr/local/bro/spool/bro/software.log 

#separator \x09

#set_separator ,

#empty_field (empty)

#unset_field -

#path software

#open 2015-12-20-14-25-21

#fields ts host host_p software_type name version.major version.minor version.minor2 version.minor3 version.addl unparsed_version

#types time addr port enum string count count count count string string

1450646720.519703 10.0.1.19 - HTTP::BROWSER Unspecified WebKit 537 36 - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Spotify/1.0.20.94 Safari/537.36

As you can see, this json is a lot more readable than the default CSV-type logfiles. It can be piped to any program that can decode JSON, but it can also be read directly inside your Python file as a dict. This makes it really easy to create your own intrusion detection programs.

Virus scanning

For example, I wrote this simple script to check for prohibited mime types and report their hash digests:

## simple file type detection.

## if i had unlimited api access, i would send every single md5 to something like VirusTotal

forbidden_types = ['text/x-bash', 'text/scriptlet', 'application/x-opc+zip', 'application/com', 'application/x-msmetafile', 'application/x-shellscript', 'text/x-sh', 'text/x-csh', 'application/x-com', 'application/x-helpfile', 'application/hta', 'application/x-bat', 'application/x-php', 'application/x-winexe', 'application/x-msdownload', 'text/x-javascript', 'application/x-msdos-program', 'application/bat', 'application/x-winhelp', 'application/vnd.ms-powerpoint', 'text/x-perl', 'application/x-javascript', 'application/x-ms-shortcut', 'application/vnd.msexcel', 'application/x-msdos-windows', 'text/x-python', 'application/x-download', 'text/javascript', 'text/x-php', 'application/exe', 'application/x-exe', 'application/x-winhlp', 'application/msword', 'application/zip']

import bparser

for i in bparser.parseentries('files.log'):

if i['mime_type'] in forbidden_types: ## scan all of these types

print

print "{} downloaded a file from {} via {}".format(i['rx_hosts'],i['tx_hosts'],i['source'])

print "Filename: {}  length: {}  mime type: {}".format(i['filename'],i['total_bytes'],i['mime_type'])

print "MD5: {}  SHA1: {}".format(i['md5'],i['sha1'])

This script could be implemented in a workplace, to monitor employee downloads and possible viruses. If a virus is detected, one could implement the appropriate security measures on the affected machine (as we know its IP address). As stated in the comment, one could implement a check for each hash to be sent to a virus database. Rate-limiting would be a problem though, so in a workplace setting, local virus databases would be more suitable.

Authorized devices only

Another example of a use of this is scanning for unauthorized devices on the network. Since each device has its own unique MAC address, only specific addresses could be allowed.

$ python main.py dhcp.log

{"lease_time": 86400.0, "uid": "CO9XE21x6UMR8CZvZ2", "id.orig_p": 68, "id.resp_h": "10.0.1.1", "ts": 1450648870.748178, "id.orig_h": "10.0.1.19", "id.resp_p": 67, "mac": "a8:bb:cf:07:92:50", "trans_id": 3937219451, "assigned_ip": "10.0.1.19"}

Mac addresses start with a code that identifies the manufacturer of the network interface card. One could write a script that whitelists certain MAC addresses or manufacturers. Every day, each host has to register with DHCP services, so there is little chance of escaping this detection. Additionally, once an unauthorized host is detected, a managed router could be used to sinkhole that host from any network communication.

One small problem with using only the manufacturer ID to whitelist hosts is the fact that MAC addresses can be spoofed. However, if there is a small list of full MAC addresses whitelisted, this problem virtually disappears.

If a company decided to disallow all mobile phones from using the protected corporate network, they could scan for MAC addresses that contained the manufacturer ID of popular smartphone brands. Since MAC address spoofing is practically nonexistent on phones, this would be highly effective.

Notifying IT

A final example of a use for BroScanner is notifying admins of portscans and other possible intrusions that Bro detects. I discovered a logfile that contains this information, and all I needed to do was implement some kind of admin notification.

$ python main.py notice.log

{"uid": null, "id.resp_p": null, "actions": "Notice::ACTION_LOG", "fuid": null, "dropped": false, "remote_location.region": null, "remote_location.country_code": null, "sub": "local", "proto": null, "dst": "10.0.1.1", "ts": 1450650532.338251, "note": "Scan::Port_Scan", "peer_descr": "bro", "msg": "10.0.1.19 scanned at least 15 unique ports of host 10.0.1.1 in 0m0s", "remote_location.city": null, "remote_location.longitude": null, "remote_location.latitude": null, "src": "10.0.1.19", "id.orig_p": null, "id.resp_h": null, "n": null, "id.orig_h": null, "p": null, "file_mime_type": null, "file_desc": null, "suppress_for": 3600.0}

So a simple Python file can notify the appropriate people:

import bparser

import notifiers

for i in bparser.parseentries('notice.log'):

print i

notifiers.notify(i['msg'])

As you can see, there was a portscan detected, even though I ran it with nmap -v -PR -Pn -sV 10.0.1.1 to try to minimize traffic. Bro's builtin threat detection is no slouch.

The notifiers module contains whatever types of notifications you would like to send. I implemented the OSX notification center for a local Bro instance, so OSX users will see a popup notification informing them of the intrusion. Email, SMS, and many other forms of notification could be implemented, giving IT the edge in eliminating threats. As previously stated, network routers could sinkhole these hosts, effectively removing the threat.

With Python, the possibilities are endless! There are many more uses for this program, and I'm interested to see what people come up with.