October 08, 2008, 12:30 PM — I did a blog post in May 2008 with a suggestion to Cisco about their test question database. I had just passed my CCIE recertification written test and had used the preparation tools offered by NetMaster Class (Bruce Caslow, #3139 and Val Pavlichenko, #3765). I continued to see a high incidence of repeat questions on the Cisco written test. My rough guess was that the test database was about 400 questions, which was verified by someone at Cisco. This small database makes it easy for brain-dump organizations to track question changes and to have a market for their offerings.
My suggestion is to increase the question database to over 1000 questions. I think that's a low number. Even better would be a test database of over 2000 questions or perhaps even 3000 questions. Why not allow the people in the industry to submit questions (and maybe even answers) to Cisco to help populate the test database? Guidelines could be published and Cisco would then only have to validate the questions and answers prior to including them in the test database.
With that quantity of questions, memorizing the test questions and answers becomes much more difficult. You really have to understand the technology and be able to properly answer the questions. Interestingly, one of the folks at Netcordia was just going through airplane pilot and commercial rating training. He said that the FAA database is approximately 1000 questions and that they are required to publish the questions and answers. While it is possible to nearly memorize (for most of us) all 1000 questions, I was curious if that large a corpus would help someone better understand the topic. When I asked this person about whether it helped or hindered his understanding of the topic, he indicated that it helped. Training companies could then cover the technology in ways that facilitate answering the test questions (I'm not suggesting teaching to the test – teach the technology thoroughly enough to allow someone to successfully pass the test).
Cisco could contract with a few of the training companies to generate questions and answers. Each contractor could generate 100 or 200 questions and only have visibility into a portion of the full database. These people could then participate in the certification themselves.
One of the problems I have with the current test database is that the Cisco employees who create the questions are having problems finding questions that test understanding of a technology. Once a good question is created, it doesn't take long for the brain-dump organizations to learn the question and publish the right answer. So the Cisco folks then engage in a game in which they try to come up with new questions. Unfortunately, there are only so many questions you can ask about a given topic before you've exhausted the set of good questions. The Cisco folks are then tempted to start asking questions about technology trivia. The result are often questions that don't demonstrate my understanding of how a given technology operates.
Let me give a concrete example that I've seen repeatedly in various forms over the years. The question requires me to identify the international standard that implements some protocol, so an example would be something like:
Q: ISDN is implemented by which of the following standards?
A: A – X.25
B – E.164
C – T1S1.5
D – Q.921
E – Q.931
How does this question test my understanding of ISDN and my qualification as a CCIE? My employer is not really interested in whether I know the standard designation. What matters is whether I know when to use ISDN, how to implement it in real networks, and how to troubleshoot it.
That brings up another problem that I have with many of the test questions. They're often ambiguous. In the above example, Q.921 describes LAPD, the link layer protocol used by ISDN, while Q.931 describes ISDN's connection protocol. If you're familiar with the protocol standards, you may be uncertain which one the question author intended, since they're both about ISDN. Publishing the questions and having people comment on them in a public forum would provide a way for Cisco to identify ambiguous questions and improve them.
I've also been disturbed by question ambiguity. The question author has one concept in mind and doesn't think of alternatives that may be based on real-world experience.
Here's an example:
Q: Which of the following protocols would you use to implement connectivity between sites in a retail network?
A: A – Frame Relay
B – MPLS
C – ATM
D – DSL
E – Internet VPN
There's not enough information there for me to answer the question with any level of assurance. It's a retail network, which most often is a hub and spoke design. The question says “connectivity between sites,†but isn't clear on which sites. The author had something in mind, but I can't tell what. Is it hub and spoke, or is it full mesh? MPLS would be my preference for full mesh and one of the other protocols for hub and spoke. Even for hub and spoke, there's not enough information to make a clear determination of which technology should be used. Using the network consultant's typical answer: “It Depends.†I find these types of questions too simplistic to be useful and I'm left guessing at what the test author intended.
How does this improved test question database get used? I recommend a test engine that knows which questions a candidate has seen in prior tests and selects a new set of questions on the same topics for each time the test is taken. A candidate would have to take the test 10 times to see a full set of questions. Each test would differ significantly from other tests. Repeatability of questions would go down as the test database grows larger than 1000 questions.
What do you think would help improve the Cisco written exams? -Terry
