Go with the Flow
by Jim Johnson
Project managers will always have a place, but maybe not in the development of software. This is a problem I have been thinking about and struggling with over the last few years, and it got me thinking about the history of software development and how project management fits into a historical time line. I have come up with four distinct evolutionary periods. The first period I call the Wild West, which ran from around 1960 to about 1980. The Waterfall Period ran from 1980 to about the year 2000. The Agile Period started around the year 2000, and my prediction is it will end around 2020. We are now seeing the beginning of what I’ll call the Flow Period, which will start in earnest around 2020. My crystal ball does not go past 2020, but I imagine it will last at least 20 years.
The Wild West Period – During this period, I was actually writing code and creating software. I was the business analyst and the project manager—even though I was not called a project manager. I wrote the requirements, the engineering specifications, and the actual machine code. I compiled it and created the test suites (both unit and system), the user documentation, and the user training. And finally, I maintained it. However, most of the software was used to automate basic management functions, such as payroll and accounting. During this period, there were very few project managers. If there was a project manager, he or she generally came up the ranks of the software development community. For the vast majority of business people, software was a complete mystery, and the general population considered it to be magic.
The Waterfall Period – This era ran strong from 1980 until the year 2000. The infamous “dotcom bubble,” from ’95 to ’01, catapulted software development into the stratosphere. Software projects grew larger and much more complex. Many of them became mission-critical, fulfilling orders and building products. As the need for computer services grew, so did the need for more job specialization. Quality assurance became a department. Creating requirements and specifications became a discrete skill. It was clear that projects needed a higher degree of management oversight, adding major cost to the projects. IT management looked to the construction industry to see how it managed creating both skyscrapers and normal housing. Projects were broken down into phases like a waterfall and were documented by a plan. Late in this period, IT management thought that almost every project should have a professional, certified project manager. The result was a steep rise in the job market for that profession.
The Agile Period – When it became clear that the waterfall structured approach, with high overhead costs and long delivery times, was not producing improved results and value, the Agile Period began. Projects were broken down into smaller and smaller deliverables, which produced increased success rates, greater value, and higher-quality software. Teams became self-directed, product owners became closer to the teams, and rapid delivery of products ruled the day. Scrum became the King of Agile, with the project management duties falling to the product owner and Scrum master. Project managers tried in vain to find a place to add value to the agile process. The Project Management Institute created an agile project management certificate, but those duties were really about tracking deliverables and accounting. Advances in technology, such as tools to create and manage small projects (now known as microservices), further reduced the role of the project manager. Scrum was also integrated into DevOps to create a seamless delivery between software development, implementation, and services delivered via the software pipeline.
The Flow Period – A renaissance of the Wild West should begin in earnest around 2020 and will be marked by the development of software in a continuous process pipeline. What Scrum was to agile, DevOps is to Flow. In the Flow Period, there will be no project budgets, project plans, project managers, or Scrum masters. There will be a budget for the pipeline, which is a pure direct cost of the output. There will also be a cost to manage the pipeline, which will reduce the current project overhead cost by as much as 90%. This is done by reducing and eliminating most of the current project management activities. Work will come into the pipeline and out of the pipeline fully usable. Change will happen continuously, but in small increments that will keep everything current, useful, and more acceptable to users.
There is an old joke by the comic Steven Wright that I often quote during my talks. The joke goes like this: “If your parents didn’t have children, then chances are you won't either.” The joke is funny but true, and it applies to projects. If you do not have projects, you will not have any failed or challenged projects. You will not have to estimate the cost of the project. You will not have a project manager assigned to the project. You will not have conflicts over budgets. This, of course, does not mean waterfall projects will go away by 2020. Also, agile projects will be around for a long time. The Wild West will continue to be the Wild West. However, we will see organizations move from project-based software development to a pipeline over the next 20 years or more.
What this does mean is that we will have a whole new cast of people to feed, manage, and produce that pipeline. We will further define Flow and look at what kinds of skills, people and principles will be needed to manage software flow in a future writings.