Climbing The Cognitive Wall: Human barriers to progress in IT

As time goes by, I find myself increasingly frustrated with the all-too-binary phrase "hardware and software". I find it much more satisfying - and more representative of reality - to use the tertiary phrase "hardware, software and wetware". The "wetware" bit provides suitably elevated recognition of the vital part that human-power (and human-weakness) plays in making all this IT stuff work (or not work) in the modern world.



Now you may be thinking that I am about to launch into a discussion about team structures and personality profiling and what not. I am not going there. Instead I want to focus on the limits we have between our ears when it comes to that most important of IT skills: reasoning.



Some time ago, I wrote about inevitable illusions and how they impact the IT[1] landscape. More recently, I stumbled upon the Wason selection task[2] which brought the whole subject bubbling back up through my mind. Then, as luck would have it, I was flicking through "The Landscape of Parallel Computing Research: A View From Berkeley"[3]. As a software engineer, that paper is both exhilarating and terrifying in equal measure. "How", I find myself asking myself, "will I ever be able to reason about designing/debugging a desktop PC application that is running on 64 separate processors?!"



There is a school of thought that says that this is just a new skill we are all going to have to pick up. Gee. I wonder. This "skill" is on a whole other level from, say, learning to program a relational database. Using the most prevalent models for dealing with parallelism, massive CPU power corresponds to rampant, mind-bending non-determinism in the software that runs on top. Humans, by and large, have difficulty reasoning in the face of rampant non-determinism. It seems to me that this new "skill" is like asking people to rid their brains of susceptibility to phenomena like Wason Selection. Or to re-wire their brains so that their eyes see the St. Louise arch as square rather than rectangular[4]. That is a big ask.



To make matters worse, only a small number of folk can be expected to put their hands up and say "This is nuts. We need to deal with massive parallelism some other way.". Instead, other critical aspects of human behavior will come in to play. The most important one being complexity management through complexity movement[5]. Simply put, many developers will find ways of reducing the complexity they have dealt with by moving the complexity somewhere else. Best case, this results in an overly complex - but functional - system. Worst case, the complexity just gets moved around endlessly until finally, somebody somewhere - typically armed with a costing spreadsheet and a project plan - brings the movement cycle to an end.



Hardware continues to advance apace. All the talk about CPU speed walls and RAM walls and storage walls and transistor density walls needs to be supplemented with serious discussion about the cognitive walls we face. There is a great irony here. Computer software has been instrumental in the creation of better and better CAD programs that allow hardware engineers to do their work. In a sense, software built using the simple serial programming model has been vital in the creation of a world that requires a complex parallel programming model. Looked at another way, software has engineered itself into this mess.

I am aware that this all sounds a bit gloomy and negative but I am having trouble seeing any significant breakthroughs for this problem any time soon. Hopefully that is because by-and-large the industry doesn't seem to realize it has a big problem. Maybe when it does, the solution will be obvious. Perhaps this is wishful thinking rather than logical thinking. In my defense, us humans are prone to doing that from time to time.

[1] http://www.itworld.com/Tech/4535/nlsebizillusions031216/index.html


[2] http://en.wikipedia.org/wiki/Wason_selection_task


[3] http://view.eecs.berkeley.edu/wiki/Main_Page


[4] http://dustbunny.physics.indiana.edu/~dzierba/slarch/


[5] http://www.itworld.com/AppDev/nls_ebizcomplex050531/