DIY OSINT Collection with Scumblr
Got OSINT?
Open-source intelligence (OSINT) is data collected from publicly available sources to be used in an intelligence context. With OSINT, you can enhance awareness of cyber security threats toward your landscape, and use this understanding to augment security decisions for your organization.
But isn’t that what expensive Cyber Threat Intelligence feeds are for? Yes, and they provide a lot more information than you may be able to glean during your open source research. Your OSINT collection should be an addition to any information you are already receiving.
There are some great lists of OSINT sources here, and over here. But maybe you don’t have time to manually do the research, or you don’t have a fat wallet for CTI feeds.
Let Scumblr do the work for you…
Scumblr can automate lookups to websites, api’s, social media, forums and google searches for any mention of something you are monitoring.
For instance, you can monitor Pastebin for any paste containing @yourdomain. A positive result would indicate an email from your organization was found in a paste.
You could monitor Twitter accounts and hashtags for announcements of an attack against the financial sector.
What is Scumblr
Scumblr was developed by Netflix and open-sourced. They describe it as “a web application that allows performing periodic syncs of data sources (such as Github repositories and URLs) and performing analysis (such as static analysis, dynamic checks, and metadata collection) on the identified results.”
In my opinion, it works best when paired with Sketchy, another open-sourced tool by Netflix designed to grab screenshots and text from websites.
You can use Scumblr’s Search Providers to automate OSINT collection. A Search Provider is a script to reach out and search a site|forum|api for information you wish to monitor (ie: a mention of your organization on Twitter by a known malicious group)
Scumblr comes with Google, Facebook, Twitter, iTunes Store, Certificate Transparency, Ebay, Google Play, Reddit, RSS Feeds and YouTube Search Providers.
I wrote a few search providers for Pastebin, 4chan, 8ch and Onion (Tor based) Sites.
Setup Scumblr and Sketchy
You can install Scumblr+Sketchy anywhere (an old computer, micro AWS instance, or an ESX server in your basement). The installation is pretty straight forward.
Follow these instructions to install Scumblr.
Then follow these instructions to install Sketchy and enable screen-grabbing.
I have further instructions for the installation and setup of custom search providers here, and setting up the Facebook, Twitter, Google, and YouTube Search Providers.
I’m setup. How to I do the OSINT?
Easy. Create a Task.
- Give that Task a unique name
- Add that Task to Group 1
- Select a search provider from the task type dropdown
- Fill in the query field with what you wish to search
- You can add Tags that will automatically be added to any result this Task finds
- Click Create Task
Enable Screenshots by creating a Task, adding it to group 2, and selecting sketchy task from the task type dropdown.
Automate It
You can use the in-app scheduler or create a cron job detailed here. I use a cron job that runs all tasks every 15 minutes.
All of the tasks in Group 1 will run. Then, if there was a result, the sketchy task in group 2 will capture a screenshot (in text, html, and png).
Notes
The nice part about the Pastebin Search Provider is you don’t have to collect ALL of pastebin. You are only collecting the information you are interested in.
The Onion Website Search uses a custom google search to find any .onion and .link pages that contain your query. If google didn’t crawl the site, it wont have it.
If you are going to set this up in AWS, I recommend switching to https rather than its default http:<yourip>:3000
Scumblr has a built in workflow. You can create status’s like new, investigating, closed.
To prevent Pastebin getting upset with how often you hit their api, you can create 1 pastebin Task and separate everything you want to query with a ‘;’.
TL;DR:
- Automate OSINT collection with Scumblr
- Monitor paste sites for data leaks
- Watch Social Media and boards for potential threats
