Agile Development and Innovation

I've been talking to some people in the software industry about innovation, and we were comparing my innovation and development processes to agile and waterfall development processes used in the software industry.  At first glance, I thought it was a straightforward comparison; that the innovation process I describe is analogous to the agile development process, and the development process I describe is analogous to the waterfall development process.  After thinking about it more, I realized that it wasn't so simple.

The similarities are that the innovation process and the agile process are both iterative, while the development process and the waterfall process are both sequential.  In fact the development process I describe the waterfall development process used in the software industry are very similar, and work well when a process is known, needs to be repeated, and it's important to optimize it.  As I described many times, the system falls apart when companies try to innovate beyond incremental improvements when using this process.  The problem in the software industry is that the waterfall process is too often used to develop very new products.  As a result, people spend time trying to estimate very detailed steps required to develop something new.  However, if it's completely new, these steps are often unknowable.

While the development and waterfall processes are similar, I realized that the innovation and agile processes are only similar in that they are iterative.  The main difference is that the innovation process as I describe it is used to figure out what new products and services should be developed.  It requires a deep understanding of the market, and the ability to translate market needs into criteria for a successful offering.  On the other hand, the agile development process is an iterative way to develop an offering that has already been defined.  What is not known is the best way to develop the offering, because it is the first time it is being developed.  The future is uncertain, and this process acknowledges that as long as progress is made toward a defined goal, the correct steps will be taken at that time.

This also points out why many companies may fail to develop breakthrough innovations even though they are working with iterative processes while they develop new offerings.  Ultimately, development processes of any type are best used to develop defined offerings.  You can't "develop" your way to a breakthrough.  That can only be done by understanding what motivates the market, and translating that motivation into criteria the development team can work with - whether they iterate it or not.