There's no excuse. For the cost of a single server, you can build a monster IT test lab. A cheap, dual-CPU, 12-core AMD Istanbul-based 1U server can run several dozen virtual machines in a test scenario for about $1,500. Using VMware Server on Linux or VMware ESXi, you can avoid software licensing fees, while maintaining a perfectly valid platform for testing anything, from software upgrades to new packages, new operating systems, or even network architectures.
Combine a virtualized server lab with tools such as GNS3, and you can build and test just about any planned network or system infrastructure you want. There's no easier way to determine where resource bottlenecks reside than in a test bed, and if that test bed is as easily constructed as it is in a virtual lab, there's no reason not to find them. Moreover, with a virtual lab, you can find the sweet spot for certain servers, including how much RAM and CPU resources they'll need to function under expected (and unexpected) loads, thereby ensuring you waste fewer resources.
Performance tip No. 5: Watch everything
Network and system monitoring is the granddaddy of bottleneck diagnostics. When users complain that the network is slow, the network usually has nothing to do with it. But unless you have the facilities to show exactly where the problem resides, you're left hunting around in the dark for the solution.
Whether you prefer proprietary or open source tools, there's a myriad of options available to monitor everything from network latency and throughput to RAM and CPU utilization, to SAN performance and disk queue lengths -- you name it.
If it exists, it can be monitored. If it can be monitored, it can be graphed. And if it can be graphed, there's a very good chance that a simple perusal of the resulting graph can lead you in the right direction, greatly accelerating the problem-detection portion of any troubleshooting effort.
And when implementing network monitoring, be sure to leave no stone unturned. Monitor the CPU utilization of your routers and switches; watch the error rates on Ethernet interfaces; have your routers and switches log to central syslog servers and implement some form of logfile analysis to alert you when there are reports of anything from IP conflicts to circuits going down. Careful, conscientious implementation and tweaking of your monitoring framework will save enormous amounts of time and energy, especially when it counts the most.
Performance tip No. 6: Know your apps