Domain Dependence

Occasionally I come across a golden nugget that puts things in perspective. Usually it’s a simple and beautiful idea which elicits the “duh yes of course” gut response when it’s first explained but then continues to reverberate and come back into focus again and again. Ideas such as this one articulated by Nicholas Nassim Taleb.

“Some people can understand an idea in one domain, say, medicine, and fail to recognize it in another, say, socioeconomic life. Or they get it in the classroom, but not in the more complicated texture of the street. Humans somehow fail to recognize situations outside the contexts in which they usually learn about them.” – N. N. Taleb

He refers to this as the domain dependence of our mind. He puts forth a few examples, including that of the business man who pays the hotel porter to carry his luggage upstairs, then heads for the gym to lift weights – mimicking the natural action of lifting a suitcase. Simply put – we put blinders on and fail to recognize familiar patterns in unfamiliar situations.

It explains a lot of the frustration that comes about in process improvements and agile transformations. At least for those of us that like to think and speak in analogies, Taleb’s words ring true. We are simply too accustomed to thinking about things in our familiar ways to see the simple truth staring at us from the other side of the mirror. It gets worse as our expertise deepens. Expert knowledge is deep, and therefore becomes more and more domain-specific. The groove wears deeper and obscures our view of the world. It gets harder and harder to look outside our own domain for solutions to the problems we are working on.

But for innovation that’s exactly where we need to look – outside of our own domain.

We experience the pattern clearly when we discuss Lean/Agile concepts with someone that has a firm waterfall view of the world. Most people can understand and agree with the concepts, but many also have clear convictions that those same ideas simply don’t fit in their area.

There are a couple of mental shifts that have to be made in order to recognize foreign ideas and implement them locally. First you have to take a particular situation and understand the general idea and principle behind it. That takes some abstract thinking abilities which some people are better at than others. Nonetheless you need to extract the general principle at play. Then, that same general idea needs to be applied in a different way in the new domain. The application may look different but the principle behind it is the same.

For example the idea that an apple will fall from the tree to the earth is well-known and accepted in the domain of everyday things; The idea that the earth actually pulls the apple towards the planet’s center until it tears loose from the tree is harder to imagine.

At a different level, we have almost no trouble learning about the gravitational pull of planets. Once we understand the principle of gravity explained using planets as the domain we can start to see the commonality. The only difference between the moon and the apple here is scale – it’s just that you can’t see the similarity unless you shift your viewpoint significantly. Once we look at them both from outer space, it’s a bit easier to visualize the apple as a really tiny moon. Now the apple doesn’t fall down any more. It gets pulled inwards towards earth’s center.

Or for a more obvious example – since you are reading this blog – consider the difficulty of seeing the translation of Lean Manufacturing methods to Product Development. The Software crowd are starting to get it, but the Hardware/FPGA folks? “That won’t work in our special domain!” (BTW to me that’s not a problem at all – it’s an opportunity to be the first HW/FPGA developers to outpace the competition). Check out www.agilesoc.com and @nosnhojn as an example of pioneering work here.

Domain dependence, then, is one hurdle that we repeatedly have to cross. That is especially true for Lean/Agile transformation because the Lean principles (which are fundamental to Agile methodology) are so counter-intuitive to “common” knowledge. Lean/Agile principles are really sensitive to domain-dependence because of that.

But there is a bit of good news: if you can shift the viewpoint and get to the principle behind the practice, then this can be your detour around domain dependence. Talking about general principles without being weighed down by “common knowledge” will open the door. Once you have developed an understanding of how general principles work (like gravity) then it is a bit easier to move forward since rejecting a fundamental truth just because it looks different is, in a word, irrational. Of course calling someone irrational would not be a rational thing to do in most situations, so find another way to make the point. Patience helps; most of the time you just have to wait for the gears to turn and the point will make itself.

So, domain dependence is a real obstacle and it’s usually not recognized as a factor when we gauge resistance to new ideas. Remembering that simply shifting the viewpoint and focusing on the general principles at play is one way to reduce the impact of domain-dependence. Plotting that course is a tough job which falls to you – the change agent.

Advertisements

Conway’s Law meets Lean/Agile

“Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.” – Melvin Conway, 1968

You’ve probably heard of Conway’s Law in the context of system architecture and design. The architectural blueprint of a system tends to mirror the layout of the group that built it. Conway noted in his paper that the concept has much broader scope than just software systems, as it can be applied to any type of “system”. It’s a sociological observation, not a technical one.

If you overlay your company’s organization chart on top of your development process diagram you can see that Conway’s Law applies in a different kind of “system”. We can state a corollary of Conway’s Law as

“Organizations which design processes are constrained to produce a process flow which is a copy of the communication structures between departments”

Image

Try it. Map out your own organization and see how well it matches the phases of your development process. You should get a close enough match between departments and process phases to convince yourself that it’s not a coincidence. Functional departments are formed around technical domain expertise (e.g. Software, Hardware, Test), and the development process phases align with those boundaries. It doesn’t matter which came first – the process or the organization. Whoever framed up the process probably didn’t set out to create this match, it’s just a natural consequence of Conway’s Law.

So if the development process and the organizational structure are so closely matched, then what happens when we introduce Lean/Agile which has a fundamentally different process flow?

You already know the answer – friction and discomfort. Departments are typically preoccupied with their own internal and local efficiency. Lean/Agile is concerned with the overall end-to-end global process efficiency. The two rarely align. Departmental sub-optimizations are in direct conflict with the Lean/Agile flow.

So… Agile means a new organization structure?

The “straightforward” remedy to fix this conflict is to re-draw the organizational lines to line up with cross-functional teams rather than departmental domain expertise. Unless you are ready to push the reset button on many things that currently aren’t broken, this is not a great option. In an entrenched organization it will be a very upsetting and difficult thing to do. If you have the power, guts and the ability to actually do this, it might be the way to go. But for the rest of us (the 99% majority) we will have to be a bit more pragmatic about it and try to preserve the things that actually work, not drive good people away.

Good people should not be threatened by Lean/Agile

Some level of change is necessary when you move to Lean/Agile, and the new system is threatening to the old system. Sure. But remember if Lean/Agile is threatening to the organizational structure, there is no reason it should be threatening to the organizational managers. These are good people who have brought the company to where it is today, and if they can transition into a Lean mindset along with the company then that’s a big win: we can retain a lot of valuable experience and knowledge which simply can’t be found in a new hire.

Most of the people I have worked with over the years have the ability to re-tool and engage with the Lean/Agile model. Those who can’t or won’t tend to have a stressful time and it is not fair to ask them to work in a way that creates discomfort to them. That would violate Lean’s pillar of “respect for the individual”. Find other meaningful positions for them if at all possible, but for the good of everybody involved: don’t put them in an authority position (or even support position) on an Agile project. Here they will do more damage than good, and stress themselves out in the process.

Department heads are excellent domain experts

I find that in an R&D organization the department head typically has a lot of knowledge and experience in their technical domain. These folks are great resources! Thinking about departments as “centers of excellence” who provide skilled staff to the cross-functional teams instead of compartmentalized workshops is perhaps a way to reconcile the need for functional organization with cross-functional Agile teams. In that model there is still room for a “software organization”, a “test organization”, a “verification organization” and so on.

Image

The last thing we want to do is to sacrifice the continuity of technical domain knowledge simply in order to form cross-functional project teams. Someone needs to move the individual disciplines forward, identify and close skills gaps, provide coaching, keep up with technological advances, ensure the right tools are in place and so on. In such a setup there is a meaningful place for the departmental manager. Department members may be assigned to cross-functional Agile teams (even for the long term) but they always have a home base and an identity based on their technical domain skill set, and they have somewhere to go for help and advice in their discipline.

It can be a win-win to preserve the organizational structure and have the departments contribute their value-add to the Lean/Agile process. Most people will do good work if you let them. It’s better to join forces with department heads in established organizations. They are great allies, and as you jointly start to deliver Lean/Agile projects the sun shines on everyone involved.

At some point the atmosphere may become more relaxed and the option of drawing organizational boundaries closer to the cross-functional team structure may become more attractive. That’s a much better position to re-organize than at the start of a Lean/Agile transformation – but it requires patience.