This has been a rough week. It all started when I decided that I needed to lock down administrative access to our production network. Unfortunately, all didn't go according to plan.
My company's hosted application lives on approximately 50 servers that reside in a data center on the other side of town. Our production environment is considered critical; therefore, administrative access to these machines must be strictly controlled.
However, prior to my arrival at the company, everyone from the remote sales offices to the corporate marketing department had direct administrative access to the production environment. This was unsatisfactory, as only about 25 employees need access to this environment. I had no doubt that unrestricted access would eventually lead to problems -- a very scary and potentially career-ending situation for any security manager.
I contemplated a variety of methods to control administrative access before settling on this one: I would funnel all administrative access to the production servers through a single point. That point would be the "gateway," a highly secured server that would reside on its own segregated network. I built two Unix servers and called them Gateway 1 and Gateway 2. Gateway 1 would be the primary server, while Gateway 2 would function as a spare. Then I locked the servers down, stripping them of unnecessary services such as Telnet, file transfer and e-mail.
This Week's Glossary
SecurID token: A token is a handheld authentication device that displays a one-time password for access to system resources. With some systems, the server gives you a "challenge" number that you enter into the token, which in turn produces a "response" that you type into the network. Other types of tokens, such as that from SecurID, display a number that changes every 30 or 60 seconds and is synchronized with a special server. The SecurID token can also be used with a personal identification number to provide an additional level of authentication.
The site for the Internet Engineering Task Force, which organizes working groups that develop the standards for determining how the Internet works. The groups consist of engineers, operators, vendors and researchers who write requests for comment that dictate how certain technology, applications and protocols should be built and configured.
The Computer Security Institute's Web site gathers statistics on all types of computer crimes in its annual survey.
Some standard Web server ports are reserved for special uses. This link is handy when you're trying to determine which service runs on which port. It divides port numbers into three groups: well-known ports (0 to 1023), registered ports (1024 to 49151) and dynamic or private ports (49152 to 65535).
RSA Security's registration and download page for the Ace/Agent for Linux.
Visit this link to download Idled, Mike Crider's free utility that tracks idle user accounts and automatically logs them out. You can also use it to restrict multiple log-ins under the same account or to block accounts.
Next I installed the Ace/Agent for SecurID token-based authentication from RSA Security Inc. in Bedford, Mass. SecurID tokens provide two-factor authentication. In other words, after users provide a valid user identification and password to a system, they must then input an additional level of authentication that consists of a personal identification number followed by the number displayed on the SecurID token. The displayed number changes every 60 seconds and is tied to a central server I've installed on a protected secure network, rightfully dubbed SecNet. In addition, I installed the latest commercially supported version of Secure Shell (SSH) for encrypted administration.
Then I installed a little freeware utility called Idled (pronounced idle-dee). Idled is really cool. One of my fears is that an administrator will get access to Gateway 1 and then leave for lunch or for the night without logging out of the system. If the user isn't using a password-protected screen saver, then it would be easy for someone, such as a cleaning person, contractor or disgruntled employee, to walk up to that person's desktop and access the production environment. Idled tracks idle sessions and times them out after a specified interval.
I then had one of the network engineers configure the firewall so the gateways would be the only servers that could access the production environment. Prior to the firewall configuration, I contacted each department to discuss their requirements for access to that network. For example, the operations center needs to monitor the production network. Engineers need to push code updates to certain production machines. Database administrators need to access certain ports for database administration. Our application has a special administrative tool for setting up and administering customer accounts. And the list goes on. I spent about 150 hours working on this, until all the necessary access to the production network was addressed. Or so I thought.
We applied the firewall access rules during our change control window on Thursday evening. I spent Friday morning, the weekend and now all this week addressing the access issues that folks either didn't think about or didn't address during the meetings I held with the different departments.
What Went Wrong
For certain applications, it's important to know which ports need to be allowed through the firewall for a certain function. Ports are channels by which services on the Internet operate. For example, SSH listens on Port 22. In order for SSH to connect to a server, the firewall must allow communication through Port 22. Web access needs Port 80 opened. Sendmail, which is used for e-mail, runs on Port 25.
Each standard service runs on a specified port, and there are 65,535 available ports on the Internet. The first 1,024 are reserved for the standard services, while the rest are generally up for grabs. Custom and third-party applications can be written to run on almost any port. If you're interested in this, you can read the Internet Engineering Task Force's request for proposals about Internet assigned numbers.
Anyway, the rush of e-mails, pages and phone calls that started after we implemented the new system came because some people lost access to some functions on the network. These people were accessing ports that weren't being allowed through the firewall. The firewall had blocked specific workstations from accessing certain servers on the production network.
In order to address the access problem, we had to decide between two options. The first was to call the firewall vendor and ask which ports needed to be opened on the firewall for the applications to function properly. The other option was to watch traffic on the firewall and capture the dropped packets to determine which ports the firewall was blocking. We chose the latter.
By watching traffic at the firewall for dropped packets, we were able to see which ports were blocked. We then applied the appropriate rules to the firewall to allow that traffic to pass and had the end users attempt to access the applications again. We had to go back and forth a few times before we got it right. I don't know if this was the most efficient solution, but it worked.
Administrative access to critical infrastructure is one of the most neglected areas of security. Most companies concentrate on the external threat of someone hacking their Web page or getting some sort of unauthorized access. The Computer Security Institute in San Francisco, which collects data on computer crimes in an annual survey, claims that the threat of an employee hacking from inside a company is much greater than the external threat. I have to agree. For our company, internal threats such as a disgruntled employee, corporate espionage or hacking for personal gain are much more of a threat than some college hacker. With this issue out of the way, I'll be tackling access control. It'll be my next focus regarding our critical servers.
This story, "Server lockdown locks out end users" was originally published by Computerworld.