We applied the firewall access rules during our change control window on Thursday evening. I spent Friday morning, the weekend and now all this week addressing the access issues that folks either didn't think about or didn't address during the meetings I held with the different departments.
What Went Wrong
For certain applications, it's important to know which ports need to be allowed through the firewall for a certain function. Ports are channels by which services on the Internet operate. For example, SSH listens on Port 22. In order for SSH to connect to a server, the firewall must allow communication through Port 22. Web access needs Port 80 opened. Sendmail, which is used for e-mail, runs on Port 25.
Each standard service runs on a specified port, and there are 65,535 available ports on the Internet. The first 1,024 are reserved for the standard services, while the rest are generally up for grabs. Custom and third-party applications can be written to run on almost any port. If you're interested in this, you can read the Internet Engineering Task Force's request for proposals about Internet assigned numbers.
Anyway, the rush of e-mails, pages and phone calls that started after we implemented the new system came because some people lost access to some functions on the network. These people were accessing ports that weren't being allowed through the firewall. The firewall had blocked specific workstations from accessing certain servers on the production network.
In order to address the access problem, we had to decide between two options. The first was to call the firewall vendor and ask which ports needed to be opened on the firewall for the applications to function properly. The other option was to watch traffic on the firewall and capture the dropped packets to determine which ports the firewall was blocking. We chose the latter.
By watching traffic at the firewall for dropped packets, we were able to see which ports were blocked. We then applied the appropriate rules to the firewall to allow that traffic to pass and had the end users attempt to access the applications again. We had to go back and forth a few times before we got it right. I don't know if this was the most efficient solution, but it worked.