I love the Boeing 777. I have flown more than a few miles over the years, and for the majority of them it has been the 777 that carried me and millions of other passengers safely across oceans and continents. For most of us the flying experience is judged by the quality of the food and in-flight entertainment options, and whether the flight is on-time or not. We don’t pay much attention to what airplane we’re in and just want to get to where we need to go, but somehow I have grown fond of the 777. It’s like an old reliable friend now, so when I see the 777 on my itinerary I don’t mind the flying part so much.
But this blog isn’t about someone’s creepy love story with an airplane – although that could be interesting enough. It’s about Product Development.
How do you develop an airplane like the 777? It turns out the story of the 777 development program is even more interesting than the plane itself. The 777 program included elements that were both Iterative and even Agile. I enjoyed learning about the program from various sources and found it hard to reduce it all into a single blog post. There is a lot of information out there if you are interested, including a really good 5-part PBS documentary “21st Century Jet – the Building of the 777”. See the bottom of this blog post for some good places to start.
The 777 Development Program
In the late 80’s Boeing was trying to decide on how to fill the product line gap between the 767 and 747. One option was to evolve the 767 design but in the end it was determined that a completely new aircraft was needed. It’s a big investment – an estimated $5BN for the 777. Even though Boeing was in a strong financial position in 1990, a $5BN expenditure would sink any company if it wasn’t successful.
It’s tempting to avoid any kind of risk when you’re in that position but Boeing broke many barriers in this next-generation aircraft design, both in engineering and manufacturing. The 777 was Boeing’s first “fly-by-wire” (computer-assisted control) aircraft, had the first fully computerized cabin, and was also the first airplane ever designed using 3D CAD systems. Seems impossible to understand how it could be done any other way today, but before the 777 airplanes were designed with 2D drawings and it wouldn’t be until the first prototype was built that form and fit could be tested.
The old tried and true approach had its share of familiar problems: in Boeing’s previous development effort, on the 767, an estimated 13,000 individual design changes (large, small and tiny) were made to the door assemblies at various stages in the development process. The 777 would be even worse if something didn’t change.
10,000 engineers were involved in the 777 program spread across the world in the US, Europe, Japan and Australia. Talk about coordination nightmares and opportunity for error. Large-scale project, indeed.
The 777 program was incredibly successful, as evidenced by the timelines and the results. Conceptual design started in January 1990. Manufacturing of the first prototype started in early 1993. The 777 had its first flight on June 12, 1994 and less than a year later (May 1995) the first plane went into service with United Airlines. Boeing estimates that the number of changes and errors was reduced by 80% compared to previous design projects.
Another first for Boeing: their very first airplane delivery was accepted by United Airlines on the first walkthrough, with only a handful of smaller defects to be noted. That’s beyond remarkable for any complex system, and difficult to comprehend given the safety concerns and reliability requirements that an airplane has to satisfy. A defect-free delivery of the very first airplane on-time as promised speaks volumes about any complex development program. I don’t understand airplanes but I know complex systems don’t come together very easily.
If that wasn’t enough… the very first 777 prototype plane built (W001) was good enough to be updated and eventually put in service by Cathay Pacific in 2000. Not bad for a prototype!
The engineering story of the 777 is also the story of Alan Mulally. The management team was initially led by Phil Condit as the executive in charge of the 777 program and Alan Mulally as the director of engineering. Alan Mulally would later replace Phil Condit as the person overall in charge, and it is Mulally’s management style that formed the framework of the 777 program and the “Working Together” model.
A hint of Agile
The first hint of Lean/Agile mindset can be seen in one of Mulally’s weekly program review meetings. A partial glimpse of what looks like program/meeting rules is barely visible, but you can make out at least some of the principles:
- Plan for zero overtime [Agile: sustainable pace]
- Weekly DBT reviews [Agile: weekly scrums]
- Panic Early [Agile: fail fast]
- Quality and/is Schedule [Agile: Quality is a critical ingredient]
- Make decisions faster [Agile: act and learn fast]
The labels “Agile” and “Scrum” were not yet born when Boeing kicked off development of the 777 in January of 1990. However the management team knew that something had to change. Boeing’s environment had become bureaucratic and department-focused. Specialists in various departments would design their own parts and then it was up to the manufacturing team (the system integrators) to figure out how to make it all come together. It was a “throw-it-over-the-wall” environment where communication and disconnect was a constant problem.
This time, Boeing would work closely with their customers to design the airplane, and would also tear down the walls between departments by organizing their own workforce across discipline boundaries. People that were normally separated by organizations or development phases would now be engaged together and at the same time, talking and collaborating in real-time.
“Working Together” essentially boils down to “cross-functional” teams but it meant more than that. It meant working closely -really closely- with the customer airlines and suppliers. It signaled a radical departure from the bureaucratic project organization of the past and set completely new expectations going forward. Today we would call such a thing an “Agile Transformation”.
“Working Together” foreshadowed Agile, and the model is -unfortunately- still lightyears ahead of the majority of product development teams on the planet.
Although the 777 program was planned and executed in an overall phase-gate process with key milestones, integration points and deadlines, we can see many elements and attitudes which are definitely Lean/Agile-inspired, if not outright Agile. Looking at the 777 program through the lens of the Agile Manifesto we can recognize many familiar concepts.
Individuals and Interactions over Processes and Tools
The Working Together model pulled people from many disciplines together in what we today call cross-functional teams. This was a radical departure from the way things used to work, where design engineers would design their piece in isolation, then throw their designs over the wall to the manufacturing team – waterfall-style. This was the mode of operation in Boeing in 1990, and chances are this is also the mode of operation in your company today if you work on a project of any significant size.
To address the problem of communication, Condit and Mulally organized the program in cross-functional Design-Build Teams (DBT). There were almost 250 DBT’s on the program. DBTs were formed around functional areas of the airplane, for example there was one DBT for the engine, one for the cargo door, one for the passenger door, one for the leading wing edge, one for the trailing wing edge, one for the flaps, one for the rudder etc.
Each DBT was staffed with the right people that could carry a particular design from concept through manufacturing and maintenance. Teams had design, manufacturing, tooling, finance, materials, maintenance, subcontractor representatives and even customer representatives to make sure that every aspect of operation, manufacturing and maintenance was covered.
This gives us a hint as to how we can organize Agile at large scale. You can’t possibly coordinate or co-locate 10,000 engineers, but you can make sure that the individual DBT’s are co-located and then coordinate the DBTs. DBTs were organized in a cascading hierarchy according to how functions of the airplane could be decomposed. For example, there were ten DBTs responsible for the wing’s Trailing edge, including a DBT each for Inboard Flap, Outboard Flap, Aileron, Spoilers and so on.
One potential problem of decomposing the project into DBTs is that individual teams lose track of the whole design. Weekly DBT reviews helped counteract that, but Boeing realized that the teams always need to be bonded together by a higher-level goal. When you are invested and care about something, you look out and make sure your work, and the work of the person next to you, is at the highest level.
To achieve that alignment Mulally did something incredible which I still have a hard time imagining. He pulled together all 10,000 engineers working on the 777 program for all-hands meetings once a quarter. By mid-1993 the 9th (!) all-team meeting convened in Seattle. That’s 9 of those in 30 months…
We can only speculate how much it cost Boeing to bring everyone on the team together once per quarter, but Boeing understood the real economics of product development: total team alignment allows you to reduce errors and disconnects and avoids prolonged delays. The cost of bringing everyone together for 2 days (10,000 x 2 days of salary plus travel) is much smaller than the cost of a one-month delay to the program (10,000 x 30 days of salary plus downstream fallout).
When making resource vs. time decisions, you always need to consider the burn-rate and cost of delay for the project. The burn-rate of a 10,000-person program is much higher than the cost of these all-team meetings.
Working Software (or planes) over Comprehensive Documentation
It’s all about creating fast feedback loops in order to discover problems early while they are still correctable. Agile is a Fail-Fast model.
Quick iterations and virtual integration
It is of course not practical -yet- to build and test an airplane incrementally in Sprints, but in the early 1990s CAD software was just being introduced. The 777 was the first airplane to be designed almost entirely using 3D CAD software instead of 2D drawings. This allowed Boeing to simulate form and fit quickly instead of building the traditional physical mock-ups which took both time and resources. Now for the first time 3D CAD models could be fit together and verified almost in real-time, before the first prototype was built.
The 777 program built 9 working airplane prototypes, compared to the usual 6 for a traditional airplane development program. Considering that the list price of a 777 would be in the 100M range, the extra 3 prototypes must have put a big dent into the development budget. However if you value fast feedback loops, then the cost of not building the extra 3 airplane prototypes would be even greater.
I don’t know for sure of course, but assuming that 9 prototypes each cost at least $100M, the prototype expenditure for the 777 program was in excess of 20% of the total budget.
Not all subsystems needed a full-scale prototype. To test the maneuverability and visibility of the new flight-deck layout, a flight-deck prototype was built and mounted it on a wheeled frame which allowed them to taxi around the airport to get a feel for the handling, controls and visibility of the new flight-deck design.
Mock-ups and test-beds
If you can test early, then do so – especially for the risky new components.
A prototype of Pratt & Whitney’s new engine was fitted to an old 747 and given a test flight. It’s a costly experiment which was debated internally, but in this case it was justified: a surge (engine back-fire) was experienced on that first flight, and was discovered early enough that Pratt & Whitney could address the problem without delay to the 777 program.
The new Fly-by-Wire system was similarly tested on a 757 airplane first to make sure that everything would work smoothly on the 777.
These added prototypes and test beds resulted in positive improvements for the customers too. The accelerated testing schedule made possible by the additional prototypes made a difference in getting the necessary FAA certifications in record-time, allowing for a much faster customer deployment of the plane.
Customer Collaboration over Contract Negotiation
Boeing needed a lead customer for the 777 and after fierce competition, United Airlines selected Boeing and the 777 for a $22BN deal that effectively launched the 777 program from concept into reality.
United Airlines and Boeing did write a formal contract, but the essential agreement upon which United Airlines would award Boeing their business was the famous “Condit-Guyette” memo. This hand-written note, drawn up in the early morning hours by United Airlines executive James Guyette punctuated several days of competitive negotiations between Airbus, McDonnell-Douglas and Boeing.
Signed by both Boeing and United Airlines executives, it stated that Boeing and United would work together to design a new service-ready airplane.
Thanks due to the PBS documentary flashing the original memo on the screen for a few seconds we can reverse-engineer what it actually said:
The Condit-Guyette memo, transcribed
United + Boeing + Pratt-Whitney
In order to launch on-time a truly great airplane we have a responsibility to work together to design, produce and introduce an airplane that exceeds the expectations of flight crews, cabin crews and maintenance and support teams and ultimately our passengers and shippers.
From day one:
– Best dispatch reliability in the industry
– Greatest customer appeal in the industry
– User friendly and everything works
October 15, 1990
Signed by United Airlines, Pratt-Whitney and Boeing executives
Seems quite reasonable from a customer perspective yet unrealistic from a product development standpoint. What a great way to kick off a project, and it doesn’t get any clearer than that: customer collaboration over everything else.
Responding to Change over Following a Plan
Although the design of the 777 followed a master production plan, we can see a few ways where Boeing and its suppliers expected and responded to change within the boundaries of that plan.
As many as 8 airline customers had full-time representatives sitting alongside Boeing in Seattle, with British Airways peaking at 75 people integrated with Boeing on the 777 program. 1,500 design features were reviewed with the airlines, and changes were made to 300 of them as a result.
As another example, for the first time Pratt & Whitney (the jet engine manufacturer) held several open design reviews where they invited airplane mechanics from customer airlines to critique and provide feedback about the serviceability of their new engine. This helped reduce human error in maintenance, and helped build confidence with the customer airlines that they would have a reliable and serviceable engine on the 777.
There are many other stories of requirement change, such as the rudder team in Australia which had to endure changing requirements twice after rudder manufacturing had started. Those are the big and visible ones. But how would we deal with all the small day-to-day changes?
On a day-to-day basis the DBT’s would deal with new and emerging information, and had put in place fast feedback loops between manufacturing and design engineers. Change requests which normally took weeks to process were handled in a single day with the DBT approach. When you can collaborate quickly, it is also much easier to implement (and undo) changes.
The 777 program is a technical and commercial success, and has garnered numerous innovation awards. Unfortunately Boeing did not push the “Working Together” model across the company. The amount of change, training and continuous attention experienced in the 777 program was quite high, so it was decided to leave it optional for each development program to decide. Of the next 3 development programs, only one implemented the “Working Together” model. Judging from the 787 impressions (numerous delays and early equality issues, fleet groundings), it seems a lot of the 777 lessons have been forgotten or ignored.
Alan Mulally himself later joined Ford in 2006 as President and CEO. There he continued the corporate turnaround with similar management methods. It just goes to show: principles are transferable but practices are not.
References and Resources
You can find out quite a bit about the 777 development program if you just do a bit of searching. Here are some of the links/resources which i came across:
- “21st Century Jet – the Building of the 777” – PBS 5-part documentary about the 777 development program. It’s available onyoutube and most of the pictures above are captured from there
- Philip Condit and the Boeing 777: from Design and Development to Production and Sales
- The book “Working Together – Twelve principles for achieving excellence in managing projects, teams and organizations” by James P. Lewis
- “Working together in the 21st Century” – speech by Phil Condit
- The Boeing 777: Development Life Cycle Follows Artifact