Does this site look plain?

This site uses advanced css techniques

[RPAT logo]

A customer in the web hosting business was experiencing repeated attempts at password guessing for a subscription site on his servers. The source IP addresses were scattered all over the world, and a bit of investigation showed that they were unrelated anonymous web proxies: clearly the perpetrator was trying to avoid detection.

Table of Contents
  1. Overview of RPAT
  2. Identifying "abusers" in realtime
  3. The Netstat Table
  4. SNMP Details
  5. The RPAT Daemon
  6. RPAT Client/Server Communications
  7. Triangulation Algorithm
  8. Setting it up
  9. Potential Pitfalls
  10. Command Line Reference
  11. Download
  12. Legal/Abuse Considerations
  13. Future Directions

He had methods to detect hacked accounts by scanning the web logs, so in practice this wasn't impacting his business too much, but it was an annoyance nevertheless. He asked me to investigate.

In the process we developed a technique which we have not seen described before, and we're writing it up here in the hope that it proves helpful to others. Though proxy abuse for the purposes of attacking a webserver may seem like a narrow problem to solve, we've found unrelated problems which this approach seems to apply to, such as tracking IRC bots to their masters.

Overview of RPAT

The RPAT technique involves three components cooperating to produce a unified output.

Detection of the abuse in realtime, usually from web logs
It is central to the success of RPAT that abuse be detected in as real-time as possible. Even a two-minute delay can be enough to miss the "window" of detection. In our case, we simply tailed the web logs and sniffed out the suspect IP addresses from the signature of access patterns.
Fetching of the anonymous proxy's "netstat" table via SNMP
Once the log-watcher has determined that a request is suspicious, the RPAT deamon makes a series of SNMP (Simple Network Management Protocol) requests to the IP suspected of being an open proxy. These requests are for what amounts to the "netstat" table, the list of all open connections. If we are able to do this in a timely manner, each table contains both a connection from the proxy to the target web server, and from the abuser to the proxy. All netstat tables are logged.
Processing the collected data to "triangulate" to the source
The accumulated netstat data can be quite voluminous, and any individual table won't provide enough information to pinpoint the abuser's IP address even if it's included in the data. But by correlating the TCP connection data from multiple, unrelated proxy servers, we can often "triangulate" to locate the abuser: the same source IP showing up in multiple unrelated netstat tables may very well be the guilty party.

Identifying "abusers" in realtime

The key to RPAT working correctly is to detect the abusive activity immediately, and not all circumstances lend themselves to this. In the case of the web-site hacking that prompted this research project, however, it was.

We found that the abusers were brute-forcing passwords via HTTP "HEAD" requests, each of which failed with a "401" error ("Unauthorized"). By using a HEAD request with authentication information, they could crank through passwords in relatively short order. It also made them very easy to detect from the logs.

... "HEAD /Members/index.html HTTP/1.0" 401 0 "-" "-"

We created a small perl program that did a "tail -F" on the logfile, and when it found a HEAD request that failed with a 401, it sent the responsible IP address over to the RPAT daemon for processing. The -F "followed" the logfile and accounted for log rotation: if the web log was rotated out, tail would detect this and reopen the file to get the new one.

The Netstat Table

All IP stacks maintain an internal table of open connections, and this is normally queried via the netstat command. Under UNIX or Windows NT/2000, the netstat -n command displays this table. The -n parameter supprses the normal IP-to-name DNS lookup, and above we have only shown the TCP connections (netstat also usually shows udp and raw IP connections as well).

$ netstat -n
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0      TIME_WAIT
tcp        0     93       ESTABLISHED
tcp        0      0       ESTABLISHED
tcp        0      0       ESTABLISHED
tcp        0      0           ESTABLISHED

The first connection is an outbound one to the web server, and the rest are all inbound to the mail or secure shell servers. The last one is from a random (made-up) cable modem address to the web server

We will note here that the customary -a parameter is not included. This shows the current connections as well as all "listening" sockets, and this is normally important data. But for our purposes, only established connections are interesting, so the listening endpoints are not.

Most "interesting" connections identified by RPAT will be in a closing state, usually TIME_WAIT. After a TCP/IP connection is closed, the IP stack must keep track of the connection for a time to allow straggling ACK and FIN packets to filter through the network. This is known as the "2MSL Wait", and it's usually between 30 seconds and two minutes. This timer starts the moment the connection is closed, and we must be able to gather the interesting data before this ages out of the connection table.

RFC 793 discusses the 2MSL wait, and a good discussion can be found in TCP/IP Illustrated (Volume 1) by W. Richard Stevens on pg 242.

SNMP Details

It's possible to demonstrate the use of SNMP with standard Linux tools. Initial research was done under Red Hat Linux using the UC Davis SNMP toolkit (ucd-snmp-utils-4.0.1-4): this should be widely available from your favorite depot of RPMs. In particular, snmpwalk is required.

The command "snmpwalk -s public tcpConnTable" can be used to fetch the entire TCP connection table, though in practice this returns more data than is actually useful (as we'll see shortly). We provide the remote IP address, the SNMP community string, and the starting point in the tree, and via a series of GET-NEXT requests, the TCP connection table is fetched one entry at a time to the standard output.

The full OID (Object ID) for the TCP connection table is, but this is more often abbreviated to just tcpConnTable. There are five columns in the table:

Though it's customary to fetch tables in their entirety a column at a time, it turns out that the first column (tcpConnState) encodes everything from all columns into one. This means we need only fetch that column and split it into the constituent parts. This dramatically reduces the network I/O required when fetching the connection table from the proxy.

Fetching a representative table from a random machine shows how much data there is to wade through. The state is the actual return value, and the local IP/port and remote IP/port are encoded into the returned OID.

TCPConnState Decoding
$ snmpwalk -s public tcpConnState
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = established(5)
tcpConnState. = established(5)
tcpConnState. = listen(2)
tcpConnState. = listen(2)
tcpConnState. = established(5)
tcpConnState. = established(5)
tcpConnState. = established(5)
tcpConnState. = finWait1(6)
tcpConnState. = timeWait(11)
tcpConnState. = timeWait(11)
tcpConnState. = finWait2(7)
tcpConnState. = synReceived(4)
tcpConnState. = timeWait(11)
tcpConnState. = established(5)
tcpConnState. = timeWait(11)
tcpConnState. = established(5)
tcpConnState. = finWait1(6)
tcpConnState. = lastAck(9)
tcpConnState. = established(5)
tcpConnState. = finWait2(7)
tcpConnState. = lastAck(9)
tcpConnState. = established(5)
tcpConnState. = established(5)
tcpConnState. = synSent(3)
tcpConnState. = closeWait(8)
tcpConnState. = established(5)
tcpConnState. = synSent(3)
tcpConnState. = established(5)
tcpConnState. = synSent(3)
tcpConnState. = synSent(3)

The great bulk of this table can be discarded out of hand by the data-gathering procedure, leaving the remainder for later processing. Some of the rules for processing the full table include:

After filtering with the above rules, the connection table is left as (shown with the "interesting" port in bold):

  Local IP:Port        Remote IP:Port           State
------------------   --------------------    ------------       established    established       finWait1       timeWait       timeWait      finWait2     synReceived    timeWait    established    timeWait         finWait1        lastAck         established      finWait2        lastAck        established      established         synSent         closeWait       established       established       synSent        synSent

Correlation of this data is done later.

The RPAT Daemon

For a single remote proxy server, it's straightforward enough to call the snmpwalk program for the IP in question, but this does not scale well as things get busy. The fetch itself is fully blocking for the whole request, and this can take many seconds (most of which is spent waiting for I/O). For an active run of password hacking, there is no way that sequential snmpwalk requests could keep up with the attempts.

The RPAT daemon is largely just an SNMP engine that accepts "work requests" over a UDP socket, and it "cranks" multiple simultaneous SNMP sessions to all the targets at once. Written in perl, it is a single-threaded event loop that maintains a list of all active sessions while it fetches one TCP connection entry at a time via SNMP GET-NEXT datagrams.

When a work request is received over the control socket, these steps are taken:

  1. If the IP address has an SNMP "session" already in progress, this new request is discarded. There is no need to run multiple sessions to the same remote server.
  2. If the IP address is on the "stop list", it is discarded entirely. This keeps us from repeatedly bothering servers that have been determined to be unavailable.
  3. Otherwise, the IP address is put on the work-to-do list.

The work-to-do list contains an object for each target we're trying to query, and each object has a bit of state attached. This state records the last query made to the proxy's SNMP TCP connection table, and whenever a response is received the appropriate GET-NEXT request is sent. When the "no more data" response is received from the proxy, the entire job is removed from the work-to-do list.

Timeouts and retries are handled automatically, and when too many of these occur, the job is also removed from the work-to-do list.

All proxies that provide no response of any kind are put on the stop list, and they will never be queried again. This stop list can be manipulated from the control channel (see next section).

RPAT Client/Server Communications

The RPAT daemon does the great bulk of the work, but it still must be notified of IP addresses to query. When the abuse-detector finds an entry worthy of investigation, it must notify the daemon: the RPAT client is used for this.

The daemon can listen on a UDP socket for messages that are simply ASCII strings, and the daemon extracts the needed data from it. These message strings come entirely from the command line (usually invokes from a perl or shell script), and the supported commands are:

work IPAddress
Add the given IPAddress to the daemon's work list. This is only a hint: the daemon may discard the requests if they are already in progress or are found on the stop list.
addstop IPAddress
Add the given IPAddress to the stop list. Future requests to query this IP address will be discarded, though current requests in progress are not interrupted.
delstop IPAddress
Delete the given IPAddress from the stop list. Future requests to query this IP address will be processed as usual.
Gracefully shutdown the daemon after all work in progress has completed.
Hard stop the daemon right now: work in progress is discarded.

For instance, if the abuse-detector finds an IP address to check, it runs the command

rpatc --dest=10.1..3:1234 work

Here, the --dest parameter tells the client where the RPAT daemon can be found - both IP address and port -- and the underlined portion is the literal message sent to the RPAT daemon.

Because UDP datagrams are used for client/server communications, message delivery is not guaranteed. In circumstances where the client and the server are both on the same machine, localhost (the default) should be used to minimize the network resources used, but for machines that are "close" to each other topologically, even regular UDP should be reliable enough.

If the abuse frequency is so low that the loss of a single work request can jeopardize the results, there probably is probably not enough data to work with in the first place.

The RPAT daemon never replies or acknowledges these messages.

Triangulation Algorithm

Once the daemon has gathered sufficient data reflecting diversity of abused intermediate proxies, the output logs can be processed in an attempt to "triangulate" to find the IP address of the abuser. By considering all connections to the proxy servers, the common source IP addresses may very well show up.

For the sake of discussion, we define these terms:

The goal is to find the Remote that is attacking us, and this will show up as the same IP address connecting to multiple unrelated proxy servers. This requires that we filter out a lot of extraneous data (indeed, most of the data are extraneous).

While scanning the data one line at a time, we look for reasons to eliminate entries that won't contribute to finding the abuser. Though some filtering has already been performed by the RPAT daemon (throwing away data that could not possibly be useful), we prefer for it not to be so aggressive: when in doubt, save the data. It's much easier to disregard data later found not to be useful than it is to regenerate data discarded too soon.

Discard private IPs
If either the Proxy or Remote IP address is in a range that could not be part of the public internet, discard the whole entry. This includes the RFC1918 private ranges, localhost, multicast, and link-local ranges.
Discard "neighbor" proxy access
We presume that many of the proxy connections are legitmate users of the server in question, and we wish to discard as many of those as possible. If the Proxy and Remote addresses are in the same class C block, we presume it's a "neighbor". We may expand this definition of "neighbor" to include larger spaces (/20, for instance) if we find that it prunes the data substantially.
"Fold" neighboring proxy servers into one
If the triangulation program finds two proxy servers in the same class C block, it can "fold" them into one so that all such accesses are seen to be from the same machine. This allows a network of insecure proxy servers to be seen as one.
Discard Remote servers
If the Remote TCP port number is less than 1024, it's likely an outbound connection from the Proxy to a Remote service (mail, web, ftp, etc.), and these won't help us triangulate to the abuser. It may make sense to note remote web servers to see who else this abuser is attacking, but this is not part of the current project. We discard outbound connections to other proxy servers as well.
Keep Proxy connections
We have identified several TCP ports that are typically used for proxy servers: 1080, 8000 and 8080 (may be expanded), and we keep any inbound connections from Remotes to these ports.

Once an SNMP entry line is to be kept, it must be filed: perl's associative arrays are ideal for this. Each Remote IP address is the key to a large hash, and this itself contains a list of all proxies that were used by it.

The entire RPAT daemon output log is processed by triangulate, and a tally is kept of each time a Remote had a connection to a Proxy. The result is sorted by the number of proxies used per Remote (showing the "busy" Remotes first), and then listing all the proxies found for each one.


        PROXY     55
        PROXY    63
        PROXY    121
        PROXY      40
        PROXY     99
        PROXY    90
        PROXY     50
        PROXY    300
        PROXY     70
        PROXY     65

        PROXY     50
        PROXY    10
        PROXY      16
        PROXY      12
        PROXY       11
        PROXY    167
        PROXY     21
        PROXY    12
        PROXY     18
        PROXY        14

        PROXY     13
        PROXY    36
        PROXY     69
        PROXY      15
        PROXY     46
        PROXY    43
        PROXY     77
        PROXY     83
        PROXY     36
        PROXY     45

The --minproxy and --minhits parameters can be given to the triangulate program to limit the output. For instance, --minhits=10 discards any proxy/count pair for a remote where the count is less than 10: this tosses out proxies with very few uses that may be no more than distractions. The --minproxy=5 won't show any remotes that have less than five proxy servers associated (this factor is considered after --minhits exclusions).

Adding --namelookup will try to look up the inverse DNS name of each IP address in the output display. This adds substantially to processing time because so many inverse DNS servers are misconfigured, but triangulate does caching of names to minimize resolver traffic.

Setting it up

The RPAT daemon itself can run on any UNIX/Linux machine that has network connectivity and a working perl installation. In realtime mode - the only really intersting way to use his program -- a UDP port must be chosen for the daemon to listen on and accept work requests. We typically use 1234 for no good reason. Only root can bind to ports less than 1024.

To launch the server:

$ rpatd --workport=1234 > rpatd.out &
$ tail -f rpatd.out

This runs the server to the rpatd.out logfile, which will include both debugging and recorded SNMP information. We "tail" the logfile to watch what it's doing.

Then, the abuse must be monitored, and this is highly dependent on the particular application in question. When abuse is detected that might be from a remote proxy, the rpatc command must be run to send the work IPAddress command to the daemon. Though it would be possible to build the socket communication into the abuse detector directly, for our volumes we've typically just called the system() command.

Though not an official part of the release, the Download section includes a link to the rpatwatch program. This was our ad-hoc program used to monitor an Apache web lot for a particular kind of traffic, and when suspicious activity was seen, it notified the daemon. This perl program may be tuned to local needs. Be sure to set up the --dest parameter properly to reflect the IP address and port of the RPAT daemon.

Once the daemon has been running for a time, it's possible to run a triangulation procedure on that file even while the daemon is still running. The triangulate program knows to ignore the debugging information, considering only the "real" SNMP data.

$ triangulate < rpatd.out > abusers.txt

The resulting abusers.txt file contains the list of potential abusers in descending order of which ones used the widest range of proxies.

Potential Pitfalls

We have identified some potential pitfalls that may arise while trying to use this sytem.

No SNMP servers available
It should be obvious that if the hijacked proxies are either not running SNMP servers, have non-public community strings, or are otherwise secured against outside access that the RPAT system will get positivley nowhere. There is not much that can be done about this.
Not enough data collected to triangulate
The triangulation algorithm depends on having enough data available to separate the innocuous access from the abusive access. If the frequency of "abuse" is too low, then we won't get a sharp "hit" on the abuser's IP address.
Abusive use might not be easy to detect
In the scenario that prompted this project, system access that constituted "abuse" was very easy to define, so we could be fairly confident that our SNMP probes were to insecure machines. Not all scenarios are so well defined, and in some cases it may be necessary to err on the side of caution when applying the "abuse" label or not. Being overly cautious won't permit collection of enough data to triangulate properly.
RPAT will be called "abusive" itself
Since the RPAT daemon is making outgoing probes to the machines that are accessing us, it's possible that the probes themselves will be called "abusive". We feel this is unlikely, as the RPAT daemon will not probe machines that have turned out to be unreachable. But it's possible, and this factor should be clearly understood before undertaking this kind of research.
RPAT won't be realtime enough
The TCP connection table only keeps connections around for a very short time - perhaps as small as a minute - and this timer starts when the final connection is closed. This gives the log-checker and the RPAT daemon very little time to fully query the hijacked proxy to fetch all this data. Though rpatd may start fetching the data right away, the entire table must be processed withing the 2MSL time period defined by the TCP/IP RFC. This time can be as short as 30 seconds, and for very busy proxy servers it may be impossible to query the whole table in this time period.
There may be multiple abusers
RPAT relies, to some extent, on a "clear abuser" bubbling to the top of the list, standing out in contrast to the rest of the activity. If there are multiple abusers, then this won't be nearly so clear. This is even more of a pitfall if the same database of open proxies is circulating widely among the abuser community.
Abuse may not have enough proxy diversity
RPAT cannot triangulate without data from unrelated proxy servers, so if the abuser changes proxy servers only infrequently (say, at the same time his IP address changes), then RPAT won't be able to triangulate the source.
Abusers may use multi-stage proxies
If an abuser hijackes numerous machines on (say) DSL or cable modem networks, and installs his own Trojan horse applications, he may be able to use multi-stage proxies to defeat RPAT. From his home network, he could hit the hijacked machines and from there go through the anonymous proxies. In this case, RPAT would point not to the abuser but to the hijacked machines.
RPAT misses non-standard proxy ports
The triangulate program assumes that proxy access is made via TCP ports 1080, 8000 and 8080. To the extent that abusers are using other ports (which may be common ports), they will not be considered.

Command Line Reference

There are three programs delivered with the RPAT system.

rpatd - RPAT Daemon

This is the main daemon that talks to the network making SNMP queries to remote servers, and it requires a source of work. This is done by either passing a list of IP addresses on the command line or by using the --workport parameter to specify a command listener port. The latter method is the only way to achieve "realtime" triangulation, but the IP-address-list method can be used for testing.

$ rpatd [options] [IPAddress ...]
Show a brief help listing to the standard error stream.
Show a bit more debugging (mainly for developer use).
Wait up to S seconds (default=5) for each UDP datagram to return.
Each request is sent up to R times (default=3). Each attempt waits up to --timeout seconds, and when the final try times out, the entire SNMP session is presumed to have failed.
Send no more than N SNMP packets per second. This is a rate limiter designed to reduce data loss when used over slow or error-prone links. Default = unlimited.
Bind to and listen on UDP port P for work requests from the RPAT client rpatc. Only root can bind to UDP ports less than 1024. Default = don't listen.
Keep all SNMP information received from the Proxy. By default, information that we couldn't possibly use is quetly discarded, but this can be suppressed if the concept of "couldn't be interesting" changes over time.

rpatc - RPAT Client

This small program is used to send a message to the RPAT daemon, and the general form is:

$ rpatc [options] message here
Show a brief help listing to the standard error stream.
Specify the IP address and UDP port number for the RPAT deamon. Default =

The "message" is a literal text string, and rpatc does not inspect the message at all - it is merely passed unchanged to the daemon. See the RPAT Client/Server Communications section for details on the supported messages.

triangulate - postprocess RPAT daemon logs

This perl program scans the logs produced by the RPAT daemon and winnows out the interesting IP addresses

$ triangulate [options] < logfile
Show a brief help listing to the standard error stream.
Attempt to look up a DNS name for each IP address displayed
Treat all proxy servers in the same class C block as one proxy.
When reporting remotes, ignore proxies that have less than N hits. Typical values are 5 or 10, and this trims out incidental access that does not really contribute to triangulation.
Do not report remotes that use less than N proxies. These are not likely to be the real abusers.


This is an ongoing research project, and from time to time I update the web site with the latest. I provide an overall bundle, plus individual files for download. Internally for development we use separate main files and helper perl modules, but for release we use an "inliner" tool that bundles all the required local modules into a single file: thus the unusual constitution of some of the modules, especially the rpatd daemon.

All of it is in the public domain. Feedback welcome. Enjoy.

Legal/Abuse Considerations

Nearly every time this technique is described, somebody points out that we are effectively "hacking" the anonymous proxies to fetch the SNMP data, and this is a point well taken. Whether a large web company could employ this in large scale is one question, but small-time or occasional use is another story.

Since any SNMP failures - for whatever reason - cause the RPAT daemon to put the IP address on the stop list, sites without open SNMP will see no more than three attempts. There is no kind of "banging away" on sites that do not respond to us. And any sites that do have open SNMP are unlikely to notice that we are making these requests. Our experience has shown that these sites often have open NETBIOS as well, so readonly access to SNMP data seems very minor compared with the damage that a malicious user could inflict.

It's been argued that since SNMP is open to the public, that it must indeed be a public service, so we can fetch the data without fear of having this activity interpreted as a criminal act.

But we are very clear that this document contains zero legal advice.

Future Directions

During this process we have come across quite a few ideas that might make this system a bit more full-featured.

Creation of generic SNMP engine
The RPAT daemon was designed to be a much more generic "UDP engine" that can crank through multiple simultaneous "sessions". It should be possible to create other SNMP services with plug-in modules (say, a brute-force password tool) with this as a base.
Update - We'll note that when we created this, we were not aware of the NET-SNMP async perl library interface but have since found that it's almost certainly exactly what we need. Building our own SNMP engine was a useful learning experience, it was not really a great use of our time.
Creation of a mod_perl module to detect abuse
There is more information available in a web request than is saved in the logs, so by creating a mod_perl Apache module, it may be possible to inspect the request with a bit more detail than would be available from the logs.
This would also provide even faster access to the proxy's IP address that is forwarded to the RPAT daemon. We may even be able to hold open the TCP connection for a while to help keep the proxy user around: this may allow the RPAT daemon to collect more information about the abuser before it expires from the netstat cache.
Secure the client/server communications
The UDP datagrams sent from the client to the server are entirely in cleartext, and no provisions have been made for protection against malicious interference. Were the bad guy to discover the RPAT daemon, he could send fake work requests to the UDP socket. UDP source IP addresses can be easily spoofed.
For our research purposes, this was not a consideration, but should this system be deployed in a larger scale setting, it would be important to add some kind of security mechanism. For instance, even adding of a secret key to the work request (say, at the end of the packet) would allow the client and server to agree on the key and disregard requests that did not provide it.
Hack the proxy logs directly
The triangulation approach described here requires collection of a fair amount of data to get any kind of certainty on where the attacker is. But the same lack of attention to security that allows server admins all over the world to leave their proxies open to abuse keep them from securing them in other ways. We have found that it's easily as common for an anonymous proxy to allow open NETBIOS as to allow SNMP, and in many cases it's possible to simply attach to the C: share and read the proxy logs directly. This is clearly more invasive than the SNMP approach, but (as with SNMP), it's a safe bet that access to NETBIOS will not be detected.
This approach, which requires compromise of only a single open proxy server, allows for certainty in knowing who used the proxy to attack the target web site.