Sun's flawed tests delay Java certification process
Some Java application server vendors say they have been hampered by hundreds of flaws in test programs that Sun provides to make sure these products use Java correctly and work with each other.
Some vendors gripe about wasting weeks struggling with unexpected bugs, while others report only minor problems. The products being tested form the foundation of enterprise Web commerce sites.
Sun engineers patched an unknown number of the almost 6,000 test programs, and 771 went on an "exclusion list," meaning vendors don't have to run these tests to win the coveted "Java-compatible" certification for their application servers.
Rollouts not hurt
While the problem has ruffled feathers, the vendors and industry experts say it should not hurt product rollouts.
"[The Java 2 Enterprise Edition specification] is a big animal," says Randy Heffner, an analyst with Giga Information Group. "It's a major standard that can be [interpreted] in different ways. It's normal that vendors will have trouble implementing it."
But the glitches are frustrating for Sun, which released Java 2 Enterprise Edition (J2EE) with great fanfare one year ago. J2EE specifies behaviors, services and APIs as part of a programming model for Java applications on servers.
By all accounts Sun has responded quickly in making fixes to some tests, excluding others, and keeping the test process moving along. Sun engineers are putting the finishing touches on the next version of the J2EE specification, and this time Sun plans a beta test for the new code and the modified programs to test it.
ART Technology Group, BEA, Hitachi and iPlanet are shipping application servers certified as having passed the test suite. Iona, Silverstream and Sybase have announced their products have passed the tests but are not yet shipping.
The test tempest is likely to make enterprise users wary.
"People claim they're J2EE-certified, but I'm going to [certify their products] through my own evaluation testing," says John Amos, director of financial systems at McKessonHBOC, a San Francisco drug distributor, and a BEA user. "There's a difference between marketing and reality, and it's a question of where the two shall meet."
To ensure that a Java component will run the same on any application server, Sun engineers wrote two sets of Java code.
One is their own version of the J2EE APIs and services -- called the reference implementation -- for which the source code is available to each licensee.
The second is a group of 5,824 tests, called the compatibility test suite, which a vendor uses to check that its product will correctly store an Enterprise JavaBean, make a call to a database through the Java Database Connectivity interface, and so on.
A 'fiasco' for some
But as vendors began testing their products, they found not only problems with their own code, as intended, but also bugs in the reference implementation and the tests.
"It's a fiasco," says Robert Bickel, executive vice president for Bluestone Software, a Philadelphia application server vendor.
"We wasted three weeks on just a couple of issues," Bickel says.
According to Sun, one issue had to do with transactions that involve two or more databases, a feature that is an opptional part of the current J2EE Version 1.2 specification. Initially, there were no software drivers available for these distributed transactions, says Karen Tegan, Sun's software engineering manager for J2EE. Bluestone added distributed transactions to its product and then found that Sun's tests could not verify key parts of this type of transaction.
"Now there are drivers available, and we're working on a patch for these tests," Tegan says.
In general, Tegan says, if flawed tests are found that have a major impact on application portability or affected vital areas, engineers scramble to write and release repairs, often in a just a few days. Otherwise, flawed tests are put on the exclusion list. All J2EE licensees can ignore tests on this list.
Other vendors reported minor and manageable problems, and praised Sun's responsiveness.
"I think anyone who expects that any company can produce this amount of code [in the reference implementation and test suite] that's bug free, is dreaming," says Loren Corbridge, a product manager at Sybase.
"We did find some flaws in the tests. We reported them, and most of them were patched," Corbridge says.