After some prodding from the other half, I decided to install Splunk on our monitoring server. As usual when installing new software on a production server I was prepared for the inevitable headaches. After initially being disappointed there was no Yum install for it, I was reading the install guide to find out what extra modules I was going to need to compile to get it installed. To my pleasant surprise, Splunk required a single file download and a one line install command. After launching the service and accepting the terms and setting the service to launch on boot (and of course opening up the ports on my firewall) I found the install was complete.
Configuring Splunk was just as easy. I simply had to tell it which directory my logs are stored in. Setting up a forwarder on my other servers was just as easy. I found the method that worked was do a full install, configure the directories and forwarding then set it to lightweight forwarding mode. Next thing i knew all my log files were being sent to the monitoring server (and exceeding the daily 500MB limit … you would think that it would skip the limit during initial configuration).