How to choose the right application development technologies

Soft on the outside, hard on the inside ...

Most application development tasks these days require the careful stitching
together of a variety of different technology pieces. At a software level, there
has never been a wider range of programming languages, data manipulation languages
and knowledge capture languages to choose from. Adding to that, there has never
been a wider range of application environments in which to embed new systems.

Example of the former range from the obvious candidates such as Java, C#, XSLT,
SQL to the more specialized such as Microsoft DTS and Autodesk Autolisp. Examples
of the latter range from Facebook to Microsoft Sharepoint to Eclipse RCP.

Application design can involve dizzying oscillations as a result of the power
of these technologies. Should you start with a C# application and embed bits
of XSLT as required? Or, alternatively, should you start with XSLT and step
out into C# if/when required?

To develop that cool application with the word process functionality, should
you start with the word processor and use its templates/macros/extension features
when required? The alternative is to create your own master program and to embed
the word processor within it?

How to choose?

There are a number of useful rules of thumb. No absolutes I'm afraid. The first
thing I look for is expressive power.

First, it is a very important in my opinion that the top level of your application
"stack" be as flexible and as general-purpose as possible. Take for
example, the not-uncommon problem of transforming XML from one form to another.
An obvious first port of call for this would be XSLT. Now, XSLT is a language
specifically designed for XML transformations. It is not a general purpose programming
language. (Yes, I know that it is Turing Complete. Yes, I also know that XSLT
implementations tend to provide an extensibility framework so that they can
theoretically "do anything". Bear with me for a moment.)

Faced with an XML processing task, do you start with an XSLT script as the
main application layer or do you make the main application layer a Java/C#/Python
application and invoke the XSLT parts from within it?

I generally favor the latter. Applications that do XML transformation have
a habit of developing requirements in areas such as logging, security, progress
metering, user input of variables, user interactions half way through transformations...that
sort of thing. Even if they are not on the table when you start, some of them
will be by the time you try to finish. Mark my words.

Now, to implement these ex-post-facto features, what makes the most sense?
To do them outside of the XSLT environment or to try to achieve them using the
extensibility hooks that XSLT environments tend to provide? This is where you
run up against the "Well, in theory..." arguments. Yes, in theory,
you could do everything from within. It will probably involve some contortions,
debugging may prove an interesting problem, finding people with necessary knowledge
may prove very interesting indeed...

No thanks. I prefer to start with a general purpose - mainstream and boring
toolset - and invoke domain-specific, specialized languages from within. Nine
times out of ten, I find this to be the pragmatic trade off.

That is what I mean by "soft on the outside and hard on the inside."
I like the main driver application environment to be as soft and squishy as
possible. Maximum flexibility. Less malleable - but powerful tools - like XSLT
and SQL - on the inside.

Of course, care is required. I have seen applications that consist of a 3 line
driver program in Java and 1000 lines of a specialist scripting language invoked
from within it. That is unlikely to be a good example of the pattern I'm suggesting.
Equally, the 10,000 lines of general purpose C++ that could be replaced by 100
lines of SQL represents an equally unbalanced example.

Deciding which way to go is becoming undeniable harder. Back in the days when,
say, SQL-like languages were very limited, developers had little choice but
to embed them in, say, Cobol master applications. These days domain languages
are becoming so powerful that they can serve as master application drivers themselves.

Take all due care before you make the decision. It can be a very expensive
proposition to take an application crafted as an A-within-B design and turn
it into a B-within-A design.