Development

Pet Peeves - Unicode

December 8, 2008, 09:54 AM — 

Making it possible to write software that will work in any language in any country, in any culture in the world is an extremely laudable goal. A goal that I wholeheartedly sign up to. We should make sure that the software arts make it possible. It is simultaneously the commercially sensible thing to do and the right thing to do from many human perspectives. A rare alignment of drivers indeed.

At this point, a voice somewhere in the room (is that a pointy haired boss I see out of the corner of my eye?) proclaims the answer to be obvious. It goes like this : "Just use Unicode!".

Sigh.

Where to start? The trouble with explaining why this is not true - not even close to being true - is that it takes time and takes more knowledge of the technical ins-and-outs than most care to carry around in their heads.

Let us start by looking at four main variables in the space : language, country, culture and technology. Language. Let us take that one first. How hard
can that be? Well, what list of languages do you want to target? Please don't say "all of them". There is no such thing as a definitive list of languages and even if there was, the list of languages supported in Unicode changes across various incarnations of Unicode. Oh, and there are languages with unbounded sets of "characters" such as Chinese which literally cannot be fully described in Unicode.

Let us dig out a globe and do a quick tour to find suitable languages to target. Western Europe? Not too bad. Some variations on ASCII but nothing too worrying. Lets head further East. Oops! Greek. Those guys still write left to right but they have a different character for everything! Oops! What is that I see as I fly over the Urals? Cyrillic! Another entirely different set of symbols. Life is becoming more complicated the further we travel from the Western world.

Let us keep going...After all, sometimes problems get easier as you generalize them. Oh dear, when we hit the Middle East we hit languages that drive the wrong way on the page. Right to left! That changes essentially everything user-interface-related in most software. Further east we hit ideographics. The concept of a "letter" has just flew out of the occidental window. Not only that but the text is laid out top to bottom. What to do?

Let us back up the truck and start again. Let's look at the country variable this time. Maybe that is a better starting point? The British speak English right? Well yes and no. There is British English which is rather different from, say, US English and different again from, say, Hinglish. Hmmm. When does a language stop and a vernacular or a patois or a dialect start?
How do these relate to these locale things that computers are fond of setting? Hmmm. The boundaries are very fuzzy and the standards keep changing.

Sign up for ITworld's Daily newsletter
Follow ITworld on Twitter @IT_world

I like it!
Comments

Sounds like someone needs a nap!

Yes, it's really too bad that not everyone in the world speaks the same language, writes with the same script -- heck, it'd be nice if everyone just ate burgers and pizza too.

It is pretty tedious to read someone whine about how difficult and complicated it is to write localized software. Yeah, it's harder to write localized software than non-localized. It's also harder to write software that has any sort of UI than software that doesn't. No matter how hard we try to change this, people tend to want software to work with them instead of vice versa.

Finally, there seems to be some implicit confusion here between "unicode" and localization. I'm assuming the author is actually aware that unicode doesn't govern all aspects of localization -- maybe that's the point of the article (in which case the the title could probably be a little less misleading).
| reply

Hans, "I'm assuming the

Hans,

"I'm assuming the author is actually aware that unicode doesn't govern all aspects of localization -- maybe that's the point of the article..."

Exactly so. The core point of the article is that many who say "just use Unicode" do not know that, which is unfortunate.

regards,
Sean
| reply

Few points

1. Unicode is exactly the answer for many of the things you mention: it displays Cyrillic, Greek, Chinese, etc characters just fine.

2. Your GUI system should know about directions of text. Try to run Gnome in Arabic, and see how the entire user interface seemlessly works.

3. A lot of the cultural/language specific items you mention should be resolved by the translators. What would you like to do, use Google-translate to translate your whole app? You need to have your application strings translated by professionals. Read about how the gettext library works.

4. Read about iconv(), it makes switching between codesets much more straightforward.

5. Printing localized numbers should be easy. I know glibc has done that correctly for at least 3-4 years.

Yes, there are quirks that are hard to get right with localization. But, if you use the proper tools and libraries, not localizing your software is a sign of your lack of professionalism.

P.S. Being a native speaker of Bosnian/Croatian/Serbian, this is the first time I hear about "changing a string of text between uppercase and lowercase changes the length of the string". AFAIK, that's not true for Croatian, but might be true of German ß.
| reply
peer-to-peer

Esther Schindler
If the comments are ugly, the code is ugly

claird
SVG a graphics format for 21st century

pasmith
Take Chrome OS for a test spin

Sandra Henry-Stocker
Solaris Tip: Have Your Files Changed Since Installation?

sjvn
64-bits of protection?

jfruh
Android fragments vs. the iPhone monolith

mikelgan
What Gizmodo missed about the Pro WX Wireless USB disk drive

 

Sidekick: The Good News & the Bad News
Either way you look at it Microsoft Data Center management did not follow standards or best practices in this failure. In which case it makes me wonder more about the outsourcing of corporate data much less personal data.
- mburton325

Join the conversation here

The Daily Tip

The Daily TipQuick, practical advice for IT pros. Made fresh daily.

Hot tips:

Want to cash in on your IT savvy? Send your tip to tips@itworld.com. If we post it, we'll send you a $25 Amazon e-gift card.

Newsletters

Subscribe to ITWORLD TODAY and receive the latest IT news and analysis.

I would like to receive offers via email from ITworld partners.
By clicking submit you agree to the terms and conditions outlined in ITworld's privacy policy.
Featured Sponsor

AISO founders envisioned a Web hosting company that was environmentally friendly. While the company employed energy-efficient innovations like solar panels, its infrastructure produced unacceptable power and cooling requirements. Find out how AISO leveraged AMD technology to overcome their challenge in this case study white paper.

In this whitepaper, Scalar explores the opportunity to change the landscape with respect to mission critical databases built around Oracle. Leveraging technologies such as Linux, high-end commodity processing power and Oracle RAC technology to architect, design, build and maintain database infrastructure that delivers maximum availability, reliability and performance at a fraction of traditional cost.

On a typical day, weather.com, the Web site for The Weather Channel in Atlanta, serves up between 15 million and 20 million page views. But in September 2004, when back-to-back hurricanes ransacked Florida, the peak traffic on one day more than tripled: over 70 million page views by more than 7 million unique visitors. Read the full success story now.

Marketplace