How to fix your planning process

13 min readApr 28, 2021

or How to get into a cycle of continuous learning and better business outcomes

(This article is 2 of 4 in a series about cultural issues in reliability and software engineering. Be sure to also check out parts 1, 3, and 4)

Do your teams suffer from frequent priority shifts and unknown requirements? Do they frequently have to stop project work to handle other “critical” tasks? Are they ever uncertain that the work they’re doing is important or even necessary? If the answer to any of these questions is “Yes!” you need to fix your planning process (and maybe your organization).

Organizational planning is hard, especially in high-growth businesses with several cross-functional teams. And yet, a frequent complaint is “We spend too much time in planning!” Shockingly, even more frequent the answer is to spend less time planning!

The problem isn’t the amount of time, because adequate planning takes time. The problem is you’re planning for the wrong things, and not communicating adequately throughout the planning or execution cycles. You probably are also missing organizational processes needed to handle planning exceptions.

In this article, I will discuss the most common symptoms of poor planning, their causes, and what to consider as you design or modify your processes.

After reading this article, if you’re interested I have a complete quarterly planning schedule that you can use to achieve better alignment, higher throughput, and better employee satisfaction.

Know your operational model

By Snowded — Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=33783436

The concepts in this article work well for complex problem spaces. Traditional (Capital A) Agile or waterfall planning is probably fine for complicated or simple problem spaces, and the methods outlined here may not work as effectively if you are operating in those spaces. It’s important to note that if your organization lacks specific knowledge or experience with the problems they face — and frequently regardless of that knowledge or experience — they are likely in a complex problem space.

If you’re not sure, you’re probably confused according to the Cynefin framework.

The Symptoms of Poor Planning

Proper planning is critical to business execution. Without it you’re wandering in the dark. Very few businesses can afford the luxury of executing on anything that doesn’t add value to customers’ lives and boost revenue. There are three main ways to discover if your planning cycles are working or not.

1. You have Poor alignment

Poor alignment exists primarily due to information hidden in the planning process. This can be intentional (I chose not to tell you what I don’t know) or unintentional (I don’t know what I don’t know and therefore can’t tell you). But just as frequently it’s neglected (I didn’t realize I needed to tell you). Often when engineers and managers discuss work for the next quarter, they will talk about the specific tasks they want to be done, and frequently they won’t know what needs to be done until they start doing it. The danger in this practice is that proper requirements will not be surfaced, and dependencies will remain hidden until implementation time, which leaves potentially disruptive actions unscoped and unplanned. Unexpected work is highly disruptive and almost always creates negative feelings between teams.

2. You’re making poor progress

Poor progress can be caused by many things, but is usually due to unplanned, underscoped, or interrupt driven work. These three categories add friction to progress:

Unplanned work could have been planned for if all the information was available
Underscoped work is usually due to unknown problems (unknown unknowns) that can’t be foreseen
Interrupt driven work is a known risk that is difficult to accurately predict.

Less frequently, you could be experiencing sandbagging from your staff. If you suspect this is the case, you have cultural problems around motivation. How to address these issues is described below.

3. Your efforts are having poor impact

Even with the best project planning it’s still possible to miss critical business objectives. Project oriented, or even strong matrix, organizations will often prioritize project performance over company performance in the same way someone with a hammer will look at all of their problems as nails. Product oriented organizations will frequently focus on what the customer is asking for rather than what they need and is right for the business. All three tend to ignore non-functional requirements like reliability or security. This perspective will no doubt upset a number of product and program managers, however we will all be able to agree that these perspectives are necessary on the path to meeting business objectives. The danger lies in letting one or the other own the entire process. Over indexing on any one of these disciplines, including software engineering, is going to lead you astray from reaching the actual business objectives.

Plan for Outcomes

“When people are only project-oriented, waste is guaranteed.” — Stacey Barr

Rather than taking an action-driven view of your planning (epics, milestones, tasks), think about the intended business outcome (increase revenue, deploy faster, fewer customer-impacting bugs, etc.). You may be tempted to believe you already are outcome oriented, or it’s easy to make the transition. Many people wallpaper over what they do today and call it a shift, but fail to reach outcome-based thinking. Why is that?

You’re defining Actions not Outcomes

It’s important to know the difference between outcomes and actions. Fortunately the amazing Stacey Barr has written a few words on the topic. She states that while we need both project and outcome based planning, knowing when to use each is important. Waste is inevitable when thinking only about projects.

Your Outcomes are not specific

Broadly defined outcomes like, “Be Excellent” can lead you to measure the wrong things. Strategic objectives are frequently phrased as weasel words because they’re easy to define and make us feel good when we hear them, but are otherwise meaningless. Outcomes need to be specific and achievable, which means thinking beyond weasel words.

To achieve substantial business impact with higher quality and happier outcomes for customers and employees, you have to replace your action oriented goals with outcome oriented goals. (Thank you Stacey Barr, I could not have said it better myself).

Applying those outcomes to teams

Teams should come out of the planning process motivated rather than feeling frustrated that the exercise is a waste of their time, and questioning decisions. In Daniel Pink’s book, “Drive: The Surprising Truth About What Motivates Us” he describes three principles that lead to intrinsic motivation.

Autonomy — the desire to direct our own lives.
Mastery — the desire to get better at things that matter to us.
Purpose — the desire to do something in service of a purpose larger than ourselves.

Planning for outcomes is a great way to increase autonomy and provide individuals with opportunities for mastery. Teams frequently have a mix of experience levels. By allowing partnership between skill levels through the design and implementation phases of an outcome, rather than just assigning tasks, you open the door for two culturally beneficial effects:

You create coaching and leadership opportunities for your more experienced engineers, and
You create growth and visibility opportunities for less experienced engineers.

When a team’s effort is directly tied back to business goals through outcome based planning, the purpose — or “something bigger” — is obvious. Using a project oriented approach increases the risk of cognitive dissonance between work-as-disclosed by management and work-as-done by engineers which is unpleasant and costly, ultimately impacting organizational trust and motivation.

When Pink’s principles combine, teams are powerfully motivated to execute to the best of their ability and create an environment of continual learning. This will lead to better team cohesion and motivation, and better outcomes for the business and its customers through less waste and higher quality.

Building a process that works

Cross-functional planning — meaning planning among many teams where everyone can commit to the results because they have clear expectations and understand the tradeoffs — is not simple. It’s an act of discovery. Remember, we’re in a complex problem space so planning details will be emergent. The best insulation against the unknown unknowns of planning are:

Inclusion — bringing in the most skilled and experienced voices you have to define or clarify requirements, scope out, and estimate the effort.
Alignment — ensure that everyone involved including stakeholders, and contributors have a clear understanding of the desired outcomes, commitments, and expectations.
(lots and lots of) Communication — both proactively (prior to the commitment and implementation) and reactively (after planning, when exceptions arise).

Inclusion

If the only people making planning decisions are project, program, or engineering managers, you’re doing it wrong. Inclusion requires bringing Engineering Leads into the process and interacting directly with stakeholders. They should ask questions to clarify and define requirements, scope out the work required, and provide work estimates. These estimates will be wrong, but the goal is to make them less wrong than if a project manager, engineering manager, or anyone else with an “M” in their title had made the estimate. A positive outcome of this inclusion is, everyone at the table understands approximately how complex or complicated the work is and what the risks are.

Alignment

Alignment is more than conversations between people and a general sense of agreement. Misalignment can happen before, during, and after the planning process. There are three processes required to build and maintain organizational alignment.

Build a common understanding of objectives, scope, and priority. Work with customers, stakeholders, and engineering leads to clarify requirements (knowing it won’t be perfect), scope, and align effort with business objectives. Teams should work together to define measures that establish the objectives agreed to have been met. Progress through the planning period will be tracked by these measures. This is the core of planning, and has a cost in human effort, but when done right builds the trust, understanding, and confidence to handle downstream exceptions (which will come up in complex problem spaces). A variation on this process has engineering managers meeting to define common objectives before meeting with stakeholders and engineering leads to discuss the rest.
Establish a planning priority deconfliction process for disagreements during the planning stages. With finite organizational capacity to execute, different teams working together will frequently have conflicting priorities. Infrastructure teams will often suffer from fan-in of external team priorities and can get into contentious debates due to high demand. They are no less subject to organizational capacity limitations than other teams and are frequently understaffed. The end goal is to have a unified set of priorities the organization can agree upon. Therefore it’s important to handle disagreements and impasses efficiently and reasonably so that everyone can have confidence in the decisions being made. It should be psychologically safe to escalate for a decision. Remember the inclusion principle though: stakeholders should be included in the discussion and allowed to present their case for prioritization. A delegated decider — usually some higher level leader responsible for making business decisions — should press for as much data as necessary in order to make a decision of which effort should take priority. Once the decision is made, everyone should be well enough informed enough of the tradeoffs to commit to the decision.
Establish an execution priority deconfliction process for unknown unknowns or other interrupts that arise during execution. Information that was not part of the initial estimate will arrive in the middle of execution. This new information should cause you to re-evaluate your plan to determine if it’s still valid. If for any reason, your plan is no longer valid, downstream work will be impacted and people will need to know. These issues may trigger reevaluation of business decisions. Be prepared to evaluate new work, scope and size it, make or escalate for a priority call, and “brick in/brick out” efforts based on your organizational capacity. Brick in/Brick out means cutting a project or projects of similar cumulative size to the one you’re replacing it/them with as a priority. This ensures you’re not overcommitting your teams and maintaining their need for autonomy, mastery, and purpose while efficiently determining what’s best for business objectives.

Communication

Every step of the planning process requires transparency and consensus, which means every step of the planning process needs a communication plan. In the early planning stages, the communication plan depends on meetings and conversations to build a collective understanding. In later planning stages, the communication plan depends on data, decisions, and agreement. Those decisions and agreements need to be documented and disseminated to all stakeholders so they can have situational awareness of the decisions. And finally, all stakeholders and contributors need to be aware of any changes to plans should they arise. This communication serves two purposes, a) to inform people of changes in plans and decisions made, and b) it gives them an opportunity to raise concerns and give feedback in the case of unforeseen ripple effects of those decisions.

Putting it all together

Getting the entire cycle right takes time and effort between teams. It takes collaboration. It takes practice. There are specific processes that need to be in place for planning and execution to make sure it goes smoothly even when new and contradictory issues come to light.

In this section, I’ll describe all the steps necessary to have a planning process that generates higher organizational alignment, trustworthy cross functional commitments, and autonomy, mastery, and purpose for engineers. This is not intended to be a one size fits all solution. The principles above should allow you to tailor a process that works for your organization. This example will serve as a fully functional process if adopted — in fact I’ve used this and subtle variant forms of this example at different companies — but it is meant to be an illustrative example.

Understand and publish (or remind everyone of) business objectives prior to the kickoff of the planning cycle. Executive teams should be defining these objectives as part of their annual strategy planning.
Allow time for collaborative alignment between stakeholders on the outcomes they would like to see.
Include engineering tech leads (TLs) in requirements definition with stakeholders. Protip: Encourage them to ask questions and note any areas where there aren’t good answers. Thare lie dragons.
Include TLs again in scoping, and estimation discussions. Protip: Measure their estimates over planning periods and adjust for under/over optimism. (This step is critical as it resolves the “why is it so hard” debates that come up, and managers are more often than not ill equipped to discuss adequately).
Measure requests against organizational capacity, and create a prioritized list (in business priority order) of things that can be committed to versus those that can’t. Call out stretch goals explicitly. Protip: All effort — supply and demand — can be converted to units like “SWE Hours” (hours of software engineering effort) but paint in broad strokes using tools like “T-shirt sizes” that map to specific SWEHour values rather than trying to measure precisely. The buffer this provides will be useful given the natural propensity to make overly optimistic estimates. Protip 2: Examine your organizational incentives that may harm psychological safety to estimate incorrectly or overcommit, otherwise you will wind up incentivizing sandbagging and risk aversion.
Socialize all commitments and stretch goals with all stakeholders — those that will be receiving the teams effort, as well as those that aren’t. Give them opportunity for feedback. Protip: Be prepared to justify the reasoning behind decisions to cut or delay work, but adopt an approach of humility and vulnerability. Transparency and openness will lead to better business decisions, and potentially new ways to achieve more optimal outcomes.
Adjust commitments based on feedback. It’s important to make escalation a psychologically safe exercise with the goal of optimal business outcomes. Protip: Keep escalations objective. When both parties can collaborate on an escalation the business tends to make better informed decisions with better outcomes.
Publish the final set of commitments and stretch goals. After conflicts are resolved and tradeoffs decided, this document will serve as a central reminder to the organization of what work to expect and confidence levels. It will also serve as a baseline for handling downstream exceptions. Protip: In the spirit of transparency and openness, this publication should be open for all employees to view. It should be possible to understand both what is happening as well as why so that readers can connect with the overarching purpose of the efforts taking place.
Handle disruptions to the plan according to your organizational capacity. Newly discovered work is going to happen. Making decisions efficiently while considering business priority and the physics of your organization will help maintain high velocity on things that matter as well as high intrinsic motivation. Protip: Don’t take on new work without examining what teams will not be doing as a result. Disruptions will impact commitments and velocity, so make sure any changes are communicated broadly and clearly.

This example may seem complex and time consuming, but with practice it scales to very large organizations and can be performed efficiently among dependent stakeholders. The net result will be a process that lets your engineers flourish, and drives you efficiently towards your business goals.

Caveats

You may be asking, “Where are the project and product managers in this scheme?” And that’s an excellent question. Projects are things that fall out of the quarterly planning process, not the driver of it. Projects are organized effort to reach objectives, but are not objectives in and of themselves. Driving the process with project managers is the very definition of the tail wagging the dog. Program and Product managers are a bit different as they add value to strategic directions and should be leveraged in establishing company level objectives. Engineering should still get the final say as to how to meet those objectives and how long it will take to meet them.

Conclusion

Much has been written about how the planning cycle can help boost productivity, or how a lack of strategy or planning will impede companies, but not much has been said about the cultural impact of planning processes on engineering teams and organizations. When business alignment and priority deconfliction processes are insufficient or missing, morale and trust suffers, as well as your ability to meet business goals. By following the advice in this article you can and will build a more effective planning and execution process that will lead to better autonomy, mastery, and purpose for contributors, and better faster outcomes for your business.

Thanks to Amy Tobey for helping refine the wording around cognitive dissonance, and Thomas Strömberg for editorial assistance.