Setting up sendmail on a firewall, Part 1

Unix Insider –

One of the greatest features of sendmail is the extreme flexibility it provides the administrator. This is also one of its worst features, the reason being that it is so often misconfigured. Usually, a system administrator inherits a sendmail configuration file that he or she is afraid to touch, lest it should break. Over time, this file becomes hopelessly confusing, contradictory, and redundant.

I began writing this column with the intention of providing a simple, straightforward, "sendmail for dummies" approach. I ended up appreciating how Bryan Costales must have felt when he wrote his handbook, Sendmail for O'Reilly & Associates: Nothing is simple in sendmail. I also learned to appreciate just how powerful and flexible sendmail really is. It's worth learning. It's also worth a few columns. I'll devote this month's column to discussing some of the background and new features in sendmail 8.9.3 as well as how to build the source code. Next month we'll look into the configuration file. If there is enough interest, I will add a third column for special situations and testing techniques.

Sendmail on the firewall?

Like most security admins, I've always been told it's a bad idea to run sendmail on the firewall. It's generally considered better to run something like smap instead. Indeed, I ran smap myself until I discovered a problem I couldn't fix with smap. The problem was that company email was forwarded from an internal mail gateway system to the firewall. The firewall could only rewrite the external header to say the mail came from "" There was still an internal header that showed the mail was routed through "," complete with the mailgate internal IP address. I couldn't have the inside machine masquerade as because there were other internal mail gateways it had to communicate with. (They couldn't all be But because sendmail is extremely flexible, open source software, it's possible to add functionality to strip out the inside header. Also, many of the security risks with sendmail have to do with the fact that it's generally run setuid to root. It isn't necessary to run it this way on a firewall because there are no direct mail users on the firewall. For added security, I run sendmail in a

<font face="Courier">chroot</font>
ed cell with the program mailer using the sendmail restricted shell,
<font face="Courier">smrsh</font>


There are certain resources you should have available before you start working with sendmail. This column isn't going to attempt to cover all aspects of sendmail. For that, I direct you to Bryan Costales's Sendmail, Second Edition. Aside from being an excellent resource, at more than 1,000 pages this book can also double as a child's booster seat!

You'll also need the latest copy of the sendmail source code (sendmail 8.9.3 as of this writing), available from Verify the PGP signature! Some very useful links are included with the latest information.

Although it should go without saying, you need to be able to build the code using

<font face="Courier">make</font>
<font face="Courier">gcc</font>
<font face="Courier">cc</font>
and have your environment variables set appropriately. It is strongly recommended that you use a system other than the firewall to compile and test the code. Remember that any system you use has to be running the same release of the operating system and have a compatible architecture. The firewall shouldn't have a development environment installed. If for some reason you have to use the firewall to compile, at least keep the development environment on a spare drive that you can unmount when you don't need it. Life with sendmail is much easier if you have the GNU utilities already installed because it is assumed in the makefile that
<font face="Courier">groff</font>
is installed. You can get the GNU utilities (
<font face="Courier">gcc, gunzip, groff,</font>
to name a few) from

For the purposes of verifying the source (and not just for sendmail), you should have a copy of PGP (Pretty Good Privacy) installed (version 2.6.3 or higher). You can download a personal copy or buy one from Network Associates.

New features in sendmail 8.9.3

Sendmail 8.9x contains several new features that will probably cause your mail to break unless you configure them properly. These features mostly help filter spam and prevent your site from being used to relay someone else's mail. You have to decide how to configure them based on your site's policy. For my own network, I can rather tightly define from whom I'll accept mail. However, an ISP cannot be so selective. Please note that I haven't personally configured mail for an ISP, so I can only speculate on the special concerns of ISPs. Because ISPs have to service the general public, I would imagine they have legal issues to consider before blocking mail from a particular site.

Relaying is denied by default in sendmail 8.9.x. While this might sound desirable, this feature will probably cause your mail to stop working because it will refuse to relay mail from your own internal systems. There are several ways to fix this. The easiest is to use the feature,

<font face="Courier">relay_entire_domain</font>
. Refer to for a full description of these new features.

Another major change to sendmail is that it is now picky about the permissions on the directories it uses. You can turn this off by using the DontBlameSendmail option. If you do, don't blame sendmail if you have security problems. I recommend fixing the directory permissions.


Using the default options, you can simply create a file called

<font face="Courier">/etc/mail/relay-domains</font>
that contains the names of the systems for which you want to accept relaying. For example, a domain known as "" and also known as "" would have the following:

<font face="Courier">
  • Advantage: If you only need to relay mail from a few systems, this is probably the simplest solution and the one I recommend for most installations. It prevents you from acting as a relayer, while allowing your mail to go through.

  • Disadvantage: You have to keep the file updated, and you may have other requirements. If you're an ISP, you may have to allow relaying from quite a large number of domains. Also, every time you update this file you have to restart sendmail (
    <font face="Courier">kill -1 [sendmail_pid]</font>


This feature allows relaying from anywhere (not recommended).

  • Advantage: You don't have to worry about mail being denied.

  • Disadvantage: Anyone can use your system as a mail relay. Use of this feature defeats the antirelay features of sendmail and is definitely not recommended.


This feature allows any host within your domain as defined by the

<font face="Courier">m</font>
class (
<font face="Courier">$=m</font>
) to relay through your system. By default, this would be
<font face="Courier">*</font>

  • Advantage: You don't have to worry about mail within your domain being denied. Systems outside your domain can't use your system to relay mail unless they're defined in the relay-domains file or in the access database. If you only want to relay hosts in your own domain, you can use this instead of the file

    <font face="Courier">/etc/mail/relay-domains</font>

  • Disadvantage: You may not want to allow other organizations in your company to route mail through your system. Also, if you're known by several domain names (popular with companies seeking product recognition), this won't buy you much.


This allows relaying based on the individual host name, not the domain name.

  • Advantage: Fine-tunes the permission for relaying by requiring a fully qualified host name rather than just the domain name.

  • Disadvantage: Requires you to specify in either the file
    <font face="Courier">/etc/mail/relay-domains</font>
    or the access database the host name of the system you're permitting to relay. For example, if I use this, my
    <font face="Courier">/etc/mail/relay-domains</font>
    file would have:
    <font face="Courier"><BR><BR><BR> 

Mail blocking

Here are the basics of sendmail's antispam features.


By default, if the sender's domain cannot be resolved in DNS, the mail is rejected. For example:

<font face="Courier">
MAIL FROM: <wkeys@nonexistent> 501 <wkeys@nonexistent>... Sender
domain must exist


<font face="Courier">accept_unresolvable_domains</font>
overrides this and accepts mail from any domain or IP address.

  • Advantage: You may have to accept mail from IP addresses if the senders don't have their headers rewritten to come from a registered domain.

  • Disadvantage: You can be spammed from anywhere. You may be better off listing known sites with this problem in the access database.


To use the access database feature, your system must support at least one map type such as NDBM (standard on most commercial systems such as Solaris) or the Berkeley database (Berkeley DB) 2.X. If you want to use Berkeley DB, you can obtain it from Sleepycat Software. If you install the Berkeley version, make sure you build sendmail with the

<font face="Courier">NEWDB</font>
flag (and include the Berkeley DB libraries and include files).


<font face="Courier">access_db</font>
feature causes sendmail to look in a database map file (by default
<font face="Courier">/etc/mail/access.db</font>
) and decide whether to accept or reject mail from a particular user or site. You can even send a custom error message. This feature can also be used to control relaying permissions.

  • Advantage: Really allows you to fine-tune who you will accept mail from. For example, I may not want to accept mail from domains that can't be resolved, but I want to make an exception for a particular domain or address. If yours is a large site, you may want the greater flexibility this can give you -- it's rather like a firewall rule base.

  • Disadvantage: You have to build the database file and keep it updated. It can become rather complex -- it's rather like a firewall rule base. The more complex you make it, the harder it is to maintain.


By default, if the sender's address isn't fully qualified, sendmail will refuse the connection. For example:

<font face="Courier">
mail from: <wkeys>
553 <wkeys>... Domain name required

Use of this feature overrides the default so the connection will be accepted.

  • Advantage: I recommend always using fully qualified addresses. However, on an internal mail gateway, you may not be able to control how the other local systems send you mail, and this will allow you to accept mail with unqualified sender addresses.

  • Disadvantage: You lose some ability to track where mail is coming from. Don't use this on a firewall.


This allows you to block incoming mail to accounts you don't want to receive mail by listing the account in the access database. The mail appears to be accepted but is actually dumped to

<font face="Courier">/dev/null</font>

  • Advantage: You may want to use this for "nobody" or "guest" accounts.

  • Disadvantage: You have to set up the access database. This isn't much of a disadvantage, but it is another step.


If spam is a major problem for your site, you'll be interested in the Realtime Blackhole List. A list of known spam hosts is maintained at The

<font face="Courier">rbl</font>
feature causes sendmail to check with (or another RBL server if you specify) and blocks mail if the host is on that list.

  • Advantage: You don't have to worry about maintaining your own list of spam hosts.

  • Disadvantage: You're trusting that the RBL server is accurate. This may also cause a delay in accepting a connection while the RBL server is contacted.


If a host has an MX record that points to your site, this feature accepts and relays mail for them.

  • Advantage: You don't have to add hosts to the access database if they have an MX record that points to you.

  • Disadvantage: This seems like a lazy way out. It's probably quicker to look up systems in the access database than check for MX records. This will allow a third party to relay through you without your permission.


This allows your users to relay mail though you if the domain portion of the sender address is a local host.

  • Advantage: If you have users that belong in your domain but are routed through another domain, this will allow their mail to be relayed.

  • Disadvantage: It's very easy to fake the sender address, and using this opens you up to spammers. I don't recommend it.

Building the sendmail binary

The first step of course is to download

<font face="Courier">sendmail.8.9.3.tar.gz</font>
<font face="Courier">sendmail.8.9.3.tar.sig</font>

1 2 Page 1
ITWorld DealPost: The best in tech deals and discounts.
Shop Tech Products at Amazon