Something else I’ve been playing with is the SpiderFoot project for intelligence gathering on targets like xyz.com domain or whatever you would like to point it at.
It gathers a ton of information from search engines, social media, the target domain itself and can also check sources like Virustotal, Shodan Honeypot Checker and others using API keys if you have them.
As you would expect, it gathers information of all types which could be useful.
This is another project that’s super simple to get up and running on a VPS or hardware if you like. I did need to run this on at least a 1GB VM so that SpiderFoot did not run out of memory, but other than that it seems to perform well.
You’ll also find some interesting details out about yourself when pointing at your own domain.
I’ve been experimenting with a few different logging solutions; free Splunk, loggly.com and now Graylog.
Splunk would be my choice were it not for the high price. Free Splunk works great to a point but after 60 days alerting and some other features stop working. And well, logging without alerting is kind of silly.
Loggly.com was actually pretty cool for about $80/month which included alerting, log backup to Amazon S3, secure log transmission and various setup scripts for getting it all working on Apache, Nginx or whatever.
As nice as loggly.com was, I was still looking for something similar to Splunk.
Graylog has been the closest solution so far. There’s a Open Source version which has most of what you need for a personal solution. The Enterprise version adds support, Audit Log and Archiving. The Enterprise versions state around 200GB/day so I’m not exactly sure what the Open Source limit is or if it has one. But for a personal solution, whatever the limit is it’s probably sufficient.
There is also a Graylog marketplace with a crap load of add-ons for integration with Amazon AWS, a DNS resolver filter for reverse lookup on source field, JIRA integration and a whole lot more.
To this point I’ve added a few Streams for picking out info that I’m interested in from the logs, created alerts and email notifications along with a pretty nifty dashboard.
As far as hardware, I have it running on a 2GB droplet at Digital Ocean and it was super simple to setup and get running. Add in https using free certificates from LetsEncrypt and your good to go.
I had no idea this solution even existed and had never heard of it until a little while ago, but so far it has performed great without any issues.