To detect botnets in my way of greylisting, you would need:
- A UNIX-like host
Linux, FreeBSD, NetBSD, or Solaris will all do. You don't need high-end hardware. Doorstop:) with single Pentium-III CPU, 64MB RAM, and 5GB hard disk should be plenty enough. Cheap VPS is even better, because it is easy to install and configure, saves on power and network bills, and are both environment and pocket friendly.
- Heavily spammed domains with no active mailbox
We are going to trace back to the origin of spam to find botnets, so you need to possess some domains which regularly get lots of spam (something like 5K spam every day will do). No spam, no bots. To prevent collateral damage to existing mailboxes and simplify the system, my detection is designed for domains no longer in use, which I call them as "trap domains" below. You need to modify the system yourself when applying it to active domains.
- Identify bots with greylisting
Any hosts trying to send mail to trap domains is all a bit problematic, but I want to focus on botnets, which seldom could pass greylisting. Mail servers (or abused open relays) which do retry sending mail will get a response like "no such user" for each recipient, and the corresponding sessions will be eliminated when compiling the resulting IP list of botnets.
- Log full mail headers for trap domains
Most abuse contacts want you to include at least full mail headers when reporting mail related abuses, because it is easier for them to explain what happened to their clients. So for every mail destined for trap domains, and not originating from known mail servers, I keep its full mail header in log files for later notifications.
- Install Exim with SQLite support.
Exim is the SMTP server I am most familiar with, and its powerful ACL is a tremendous help for my detection. I knew from the start that I would need to query the data collected by greylisting a lot, so I based the greylisting I use on the Simple Greylist for Exim, which gives me the ability to use SQL without maintaining a full blown database server.
If you want to build Exim from source under Debian (like what I did), at least you need to make sure you have the development file for SQLite ready and change the makefile for Exim accordingly. Install the necessary library with the following command:
apt-get install libsqlite3-dev
Enable SQLite support from the Local/Makefile (inside the Exim source archive) by uncommenting:
and including the path for SQLite3's library file:
The instructions above are probably not enough for you to build the Exim binary. If you have not done so before, please consult the documents inside the Exim source archive or the Exim specification (perhaps the best documentation I have ever read for open source software).
- Create the greylisting database.
The Simple Greylisting for Exim uses two database tables to keep track of greylisting entries and known resenders, respectively. But for the purpose of detecting botnets, I am mainly interested in expired greylisting entries, which represent IP addresses without reasonable retry behaviors. So I add another table for expired entries to the database.
The SQLite script for creating the greylisting database is as follows:
CREATE TABLE expired (
CREATE TABLE greylist (
CREATE TABLE resenders (
PRIMARY KEY (host, helo) );
CREATE INDEX expired_time on expired (time);
CREATE INDEX greylist_time on greylist (time);
Under Debian, you need to install the command line interface for SQLite 3:
apt-get install sqlite3
suppose that you save the script in a file named "create_greylisting_db.sql," and want to put the database file named "greylist.db" under the path /var/spool/exim/db, the following command will create the greylisting database:
- Modify the Exim configuration.
Create the data file, /usr/exim/grey_domains (if you change the paths of data files, remember to adjust relevant settings below accordingly), for the trap domains, with one domain per line, like:
Though mail bodies of any incoming mail will be discarded, a catch-all alias for each trap domain is still needed, so the aliases file /usr/exim/domain_aliases would contain lines like:
Now it's time to integrating greylisting with Exim. Add trap domains to your local domains by adding reference to the file /usr/exim/grey_domains:
domainlist local_domains = @ : /usr/exim/grey_domains
Add the following lines to the main configuration setting:
acl_smtp_helo = acl_check_helo
The settings above specify the name of the ACL for SMTP HELO, and the filename of our SQLite database. For ACL configuration, add the following (ACL for SMTP HELO) immediately under the line "begin acl:"
warn !hosts = +relay_from_hosts
!condition = $acl_c_will_retry
dnslists = list.dnswl.org
log_message = $sender_host_address whitelisted in \
set acl_c_will_retry = yes
The warn clause above looks up the sending host of every inbound connection via list.dnswl.org, which tells us if the connection comes from a known mail server. Variable acl_c_will_retry is set to "yes" for known mail servers.
The ACL for SMTP RCPT is inserted after the line "acl_check_rcpt:", as shown below:
deny domains = /usr/exim/grey_domains
condition = $acl_c_will_retry
message = no such user
The deny clause rejects each recipient from known mail servers, hoping to prevent them from delivering mail here again in the future.
The ACL for SMTP DATA is much longer, as shown below, and should be appended right after the line "acl_check_data:".
The ACL here is similiar to the Simple Greylisting for Exim mentioned above. The 2nd warn clause saves a copy of mail header in the log file. The number "900" in this ACL specifies the minumum time (in this case, 900 seconds = 15 minutes) between the first connection attempt and the retry considered effective, so bots which retry immediately after the failed delivery attempt will still be blocked by greylisting.
From real world experience, I know that not every host which does retry is a healthy, well-functioning mail server. So the table "resenders" is only used to eliminate retried sessions from the final result, and is not for hosts to bypass greylisting.
To satisfy Exim's routing requirement, a domain_aliases router referencing /usr/exim/domain_aliases is inserted right after the line "begin routers":
- Set up crontab to collect expired entries.
A scripte, /usr/exim/bin/purge_greylist.sh, is executed regularly from crontab, to move expired entries to the database table "expired." The script is as follows:
THREE_DAYS_AGO=`expr $NOW - 259200`
sqlite3 /var/spool/exim/db/greylist.db <<EOF
delete from expired where time < $THREE_DAYS_AGO;
delete from greylist
where exists (select * from resenders
where greylist.host = resenders.host
and greylist.helo = resenders.helo
and greylist.time = resenders.time);
insert into expired (id, time, host, helo)
select id, time, host, helo from greylist
where ($NOW - time) > 28800;
delete from greylist where ($NOW - time) > 28800;
Entries expired more than 3 days ago will be removed from the table "expired". Retried entries are removed from the table "greylist." Entries without being retried for 28800 seconds (= 8 hours) are considered expired, which are very likely to be malware-infected computers, and are moved from table "greylist" to table "expired."
- Point the MXes of trap domains to your detection system.
This is the last step. Change your DNS setting to make the host you are setting to be the MXes for the trap domains. Restart your Exim process. Sit back and relax, your system is now working.
The minimum time gap (900) and expiration time (28800) mentioned above are all changable. If you are going to summarize botnet's IP for the previous day, remember to give them enough time to expire. That is, you should not summarize the collected data until 28800 seconds (the expiration time) has passed since 00:00.
I set out to write down a step-by-step procedures, to make it easy for everyone to help detect botnets. But now I realize that writing documents, let alone good ones, is too hard for me. And for someone with no prior experience with Exim, the description might be impossible to understand.
This is certainly not the only way to detect botnets with greylisting. I just hope that sharing what I have done, can give you some ideas to develop your own version of botnet detectors.