The Fighter Pilot Who Changed Software Development

For those in the Agile Software Development community who have spent any time at all looking into the Lean underpinnings of Agile, the name John Boyd should be familiar.  Boyd was the originator of the OODA - Observe, Orient, Decide, Act - loop that represents a more detailed refinement of Scrum's Inspect and Adapt cycle.  While OODA was created as a strategy for fighting a war, its underlying goal is to provide agility in any situation and to ideally allow a unit to out-think its opponent.

One of Boyd's 'acolytes', Chet Richards, wrote in his book "Certain to Win: The strategy of John Boyd applied to Business":
It is not more command and control that we are after. Instead, we seek to decrease the amount of command and control that we need. We do this by replacing coercive command and control methods with spontaneous, self-disciplined cooperation based on low-level initiative, a commonly understood intent, mutual trust, and implicit understanding and communications.
As with OODA being a much more detailed strategy than Inspect and Adapt, in one paragraph this explanation details why self-organization is desirable and what is required for it to be effective!

If we dig a bit deeper into Boyd's career, though, there is another very interesting parallel to Agile.  He served briefly in Korea during the last couple of months of that conflict flying F-86 Sabres against Soviet-built MiG-15's. By that point in the war, the American pilots had a 14-1 ratio of kills to losses, and a 10-1 ratio overall during the full 3 years of the conflict.  Boyd was curious as to why this was the case when, on paper at least, the MiG-15 was superior to the F-86.  After the war he revisited this conundrum, and realized that the Sabre pilots enjoyed much better visibility owing to the large bubble canopy, and they could switch from offensive to defensive manoeuvres and back more quickly because the Sabre had fully hydraulic controls.  In other words, the Sabre provided the ability to better observe the situation and the agility to react as required more quickly than the MiG.

Boyd went on to become a student at the Air Force's Fighter Weapons School (FWS), the equivalent to the better known U.S. Navy's Top Gun.  It should be noted that at the time - the early to mid-1950's - the Air Force was essentially a bomber force under the doctrine of massive retaliation to an anticipated attack from the Soviet Union.  There was very little effort and fewer resources allocated to developing fighter tactics, and indeed the belief was that the fighter battles of previous wars were a quaint throwback and had become obsolete.  Boyd, however, worked in near obscurity at the FWS, writing a paper entitled "A Proposed Plan for Ftr. Vs. Ftr. Training".  In it he advised that pilots needed to think differently about how they flew, concentrating not only on their current manoeuvre but also on the effects that manoeuvre would have on their speed and how their enemy would react to the manoeuvre.

E-M Theory
This research began to solidify during the early 1960's into Boyd's Energy-Manoeuvrability Theory, which stated that it wasn't just speed or engine power that gave a pilot the advantage over his adversary, but rather that it was his total energy level.  At this time, Boyd met Tom Christie, who was able to work with Boyd and provide access to the computer time required to develop and validate the equations required to prove the theory.  Indeed, Boyd "stole" the computer time, as it was charged against other projects for which Christie used their authorization codes!

In the end, Boyd and Christie developed tables and graphs that normalized for all aircraft and were based on the simple notion of knowing how quickly a pilot could gain energy when applying full power with the throttle. It was on this basis that they were able to prove that all U.S. fighter aircraft of the mid-60's were inferior to those of the Soviets, with a few exceptions in some limited part of the flight envelope.  This and lack of focus on fighter tactics were being proven by the dismal performance of U.S. fighters in Vietnam despite the technological superiority they supposedly enjoyed.

F-15 Eagle
Based on his experience and calculations, Boyd had been advocating for a small, lightweight, highly manoeuvrable fighter as the next generation to replace the Century Series fighters and the F-4 Phantom.  He was making some progress when fate intervened in the form of the large MiG-25 that appeared at the Domodedevo Air Show in July 1967.  It was reputed to have a top speed of nearly Mach 3, and it was believed that the twin tails made it a highly agile dogfighter.  (A sample was flown in 1976 to Japan by a defector, and the true nature of the MiG-25 as a bomber interceptor with limited range and manoeuvrability was exposed.)

The fear that the Soviets had a larger, better fighter prompted the U.S. Air Force to build the large, complex and very expensive F-15 Eagle.  Boyd railed against the size and proposed Mach 3.0 speed of the F-15, arguing that it was unnecessary and would compromise the agility and range of the aircraft.

Enter the "Mafia"
While the F-15 project proceeded, Boyd was able to convince a small group of generals that a smaller, lightweight fighter should be explored in case the F-15 project failed.  At the time (1967), the future success of the F-15 was anything but certain and both the Air Force and Navy were concerned after the F-111 debacle.  So, Boyd and his group were able to obtain funding to perform a design study on the lightweight fighter.  This group became known as the Fighter Mafia, named by Col. Everest Riccioni, and was an ironic play on the Bomber Mafia of the 1920's led by Gen. Billy Mitchell who faced similar challenges when the Air Corps didn't believe that bombers were important or necessary.

This group split the funds for the design study between two companies - General Dynamic for the YF-16 and Northrop for the YF-17 (the "Y" indicates a prototype aircraft).  Meanwhile, the F-15 project was becoming hugely expensive, and the already expensive F-14 of Top Gun fame was suffering from poor performance.  The government wanted to rein in the procurement process, and the lightweight fighter study seemed to be a good place to start.  As a result, in 1970 they received funding to proceed with the development of the two prototype aircraft and to perform a fly-off evaluation of them in order to make the decision based on real data as opposed to projections.

During 1974 the two aircraft were evaluated, and competed in a much larger competition for NATO orders.  In early 1975 the F-16 was declared the winner, and has since gone on to achieve considerable success with nearly 4,500 being delivered since full production started in 1976.

The YF-17 didn't fade into obscurity, though.  It was redesigned to Navy standards to be able to operate from aircraft carriers and became the F/A-18 Hornet.  It, too, was intended to complement a much larger, more expensive aircraft - the F-14 Tomcat.  As of 2011, the F/A-18 has replaced not only the F-14, but all other strike aircraft on the U.S. carriers.

Ironically, Boyd had argued against any "gold-plating" of the lightweight fighter in the form of powerful radar or computer systems, but those systems have indeed been applied with great success to both aircraft.  The increasing miniaturization and decreasing cost of electronics allowed those systems to be used without the exorbitant costs of previous generations of fighters.

So, beyond Boyd's OODA loop, how does this apply to software product development?  Look at the parallels between the push to use small, agile, low-cost fighters in the context of their time.  The doctrine of the Air Force from its creation as a separate service in 1947 until the Vietnam war in the 1960's was that a massive nuclear strike capability was all that stood between the Soviets and an invasion of western Europe.  (Whether that fear was founded or not is beyond the scope of this post.)  This drove the U.S. to focus on aircraft that could deliver the bulky and heavy nuclear weapons of the time, which meant bigger and faster ruled the day.  Similarly, in a defensive role the focus was on stopping the Soviet bombers which mean large interceptors capable of high speeds carrying missiles that could destroy a bomber from a great distance away.  It was assumed that the dogfight had been consigned to the history books, and that technology would rule the skies.

It was the hard lessons of the Vietnam War that proved those assumptions to be mostly incorrect, and indeed the Kennedy administration admitted that the nuclear doctrine served to create more conventional regional conflicts rather than prevent them.

Another aspect is that the military-industrial complex in the U.S. became self-serving with big contracts and big projects resulting in big systems and big aircraft.  These all served to eventually force the need for the lightweight fighter programs that Boyd had advocated.  While the resulting aircraft, the F-16 and F-18, are anything but 'cheap', in the context of the huge F-111, F-14 and F-15 programs of the late 60's they have been unqualified successes.

Now think about the Agile Software Development world:
  • In a world of increasingly large and heavy process, one person and later a small group explore and quantify why agility is more effective
  • These people and their work are dismissed, most often by those with a vested interest in the status quo of large projects with large processes, often in large and very large companies
  • Economic constraints force a second look at their work
  • Their work begins to accumulate acceptance as the assumptions behind large process and large projects become exposed as flawed
  • Their work becomes fully accepted and begins to flourish
  • Others start to add more to their work, despite admonishments to the contrary
  • Those additions act to make the original work more robust and even more accepted
Sound familiar?

Agile as we know it in 2011 has its roots in the 1970's.  However, it was the massive build-up of large processes and supporting tools in the 80's and 90's that lead to the "lightweight process mafia" convening at Snowbird in early 2001 in an effort to change how world built software.  That effort exposed the larger, heavier methods as being the massive retaliation targeted at an enemy that would never come, and the dot-com bust would force the post-Vietnam style soul-searching that allowed Boyd's ideas to reach the mainstream.

Today, I hear people talk about OODA constantly, and how it should be used to truly drive agility into the core of a team and organization.  Indeed, the notion of being able to process the loop faster than your competition is a key component of the nascent Lean Startup movement.

We should also be thankful that John Boyd's drive to determine why agility was important, to quantify it in real terms and to implement it in actual practice in the face of the conventional wisdom of the time was shared by those who attended the discussions at Snowbird in 2001, and those who have refined the practice of the Agile Values and Principles since.


    Thnx. For the article - now I know why agile sucks so much.
    Dave Rooney said…
    @Yahuen, would you care to elaborate?
    Scott Stribrny said…
    This piece does a good job of outlining parallels that are well worth careful consideration by anyone that is trying to better understand how to win in business today... especially where software development is involved. Indeed "It is not more command and control that we are after."

    The continuing uncertainty of the economic environment has made it compulsory for corporations to rethink their approaches to identifying and managing risk. If anyone needs evidence that applied risk management expertise (ala Boyd) pays, please also see my Cutter Consortium Executive Update, "We're Gonna Be in the Hudson"
    Don Gray said…
    I liked how this article “OO-OO-OO!” The Sound of a Broken OODA Loop" ties the OODA loop and how people think/act together.
    Dave Rooney said…

    Thanks for the comment - nice to 'meet' another aviation geek! I also have a couple of articles on the Cutter site that use aviation analogies.

    I believe it's very likely that Capt. Sullenberger had been taught at least the tactics that Boyd wrote about in the 50's, and also quite likely about E-M theory and OODA. I remember reading one article, I believe an interview with Skiles, that said they did have a brief moment of denial (measured in seconds), but then they immediately assessed what had to be done.

    What particularly impressed me was the speed with which Sullenbuger and Skiles made the decision to ditch. I guess that's the "speed of navigating the OODA loop is critical" idea in action. :)
    David Wright said…
    Great post, as I read a lot of history in my "off hours".

    It is just that I am always wary of analogies between physical things (be they jets, bridges, pencils, etc) and software, especially to support a point of view.

    In this story, "big" and "heavy" literally means big and heavy things; even the smaller and lighter things are still pretty big and heavy from most people's viewpoint. Roll an F-18 up beside a Spitfire or Mustang(favorites of mine) and the jet looks big.

    Software has no weight, and size is what disc space is needed. So what you are comparing is the physical jets, with the process of developing software, the old "apples and oranges" problem.

    Analogies are good when they help people understand something better, but not all analogies work that way for everyone either.

    So my comment is not about the validity of Agile, it is about our industry's long-standing inability to really describe/define what software is, independent of all other reference points, because software is really different from jets, bridges, pencils, etc.
    Dave Rooney said…

    The analogy about Boyd's push for a simple lightweight fighter isn't for the software itself, but rather for the process used by the teams to create the software.
    David Wright said…
    "The analogy about Boyd's push for a simple lightweight fighter isn't for the software itself, but rather for the process used by the teams to create the software."

    Exactly. To be comparable, you have to be looking at the actual process for building jets. The two types of jets were very different, but was the process to build them significantly different?

    Or you can indeed look at the software in terms of how 'agile' the software is when it is built. Can it react to change as part of its function? Or do you have to change the software to make it recognize change? So, the method to deliver the software in the first place could be agile or not, but it does not automatically mean the software itself is 'agile'.

    I am not a critic of any methods that work for an organization, however it is becoming clearer to me over time that our real goal is not necessarily more agile software development, but is certainly that we need to deliver more 'agile' software/systems. Think BPM and BRM as examples of how to do that... but that's a different topic.
    Hosh said…
    It's very important to understand that OODA is not about iterating through faster. That's a common misapplication of OODA.
    Dave Rooney said…
    @Hosh, thanks for the comment!

    In Boyd's view, it wasn't good enough to just get through the OODA loop quickly. What was imperative was to process the loop faster than your adversary in order to disrupt theirs and thus gain the advantage.

    This is a very close match to the Build-Measure-Learn loop of The Lean Startup Approach from Eric Ries, in which you are continuously working to improve your time through that cycle. You're trying to reduce the size (but maximize the effect) of your experiments. You're trying to reduce the size of, or at least the effort required to deliver a Minimum Viable Product. And, if you have competition in your market, you are indeed trying to disrupt their own cycle by processing yours faster in order to gain an advantage.

    I agree that raw speed isn't everything, which is also what Boyd argued. You need the ability to change direction quickly based on feedback in the situation.

    That's generally referred to as 'agility'. :)
    Nice post. Learned a lot about the history...