Work Planning and managing the cost of change when building digital products (software)

Hello there, it’s been some time since my last article. Well, that’s because lately my head has been filled with lots of writing topics; swimming around and begging to be put to concrete words. However, i have been able to tame my raging mind and write this one first before getting to the others primarily for 2 reasons:

1. This article is a work care letter to the product managers i have worked with or currently work with (but can also be a letter to any product manager or any head/VP of product who is interested).
2. This article is an attempt to expand on the ideas i shared in an earlier article i had written especially about integration costs.

Here’s the excerpt from my article (Decisions that matter broadly) on the definition of Integration costs: 👇🏾👇🏾

… Integration Costs on the other hand are incurred during the merging of related or unrelated software units or artefacts (built by different members of the same or different team(s) of software engineers) which must eventually go together in order to create an even more capable, powerful and valuable software than initially available. When installing a third-party open source library or integrating with a third-party API service, these all come with costs that are not obvious hitherto. Like the latest version of a third-party library may not agree with the current version of ReactJS you have already installed so that can cause a delay as you may need to look for another third-party open source package that does the same task.

The thing about integration costs is that they are harder to determine upfront…

Before proceeding with this article, there are certain terms and concepts you should already be familiar with and i invite you to read up on them prior. They include: Proof-of-Concept, Sprint Planning, Architectural Spikes, Sprint Cadence, Lead Time, Kanban, Scrum, Technical Debt, UML Use-Case Diagrams, UML Sequence Diagram, UML Activity Diagrams (flowcharts), Code Reviews, Pair Programming, Pull Requests

There are a lot of friction (and disagreements) that come up between product (or project) managers and software engineers as far as working together is concerned. From which manner (or formula) is best suited when breaking down user stories or prioritising tasks to how best to allocate story points to a Trello/JIRA task ticket. These friction or disagreements are many and seem to be in a weird sense a part of the job — i think not though 😅.

Move fast or try to!

The preference for a specific way of doing many of the things involved in delivering outcomes is mostly subjective at best so at some point, we just have to agree to disagree and move forward with the better judgement of upper management perhaps. However, i will like to share a few thoughts on a particular disagreement that affects task estimation and therefore the allocation of story points based on the costs associated with changing software which i feel doesn’t have to be a subjective discuss at all.

Whenever a sprint starts, the high-level objective is always to change something (or a many number of things) in the software under construction (or renovation — if you’d like). Change is the only constant thing in life and also in software development too. The many decisions and work that go into the endeavour requires a deep understanding of the nature of change when it comes to software systems. Without this deep understanding by both sides (software engineers and product managers alike) and a willingness to conform to its patterns and peculiarities, i fear that the disagreements will only grow louder and a bit more distasteful.

Change of any kind made on software systems always have an associated cost. This cost is not only paid in the work to be done, it’s also paid in the actions taken or (unfortunately) not taken to ensure that the work is done with the least friction and the most correctness and throughput. When team members don’t communicate enough during the sprint because of how the team is organized (a violation of Mel Conways’ law), when product managers are economical with the details of the user stories or PRD (product requirement documents) they share, when the EM (engineering manager) decides to have the same criteria for hiring for a role or doesn’t consider the archetypes for several roles to be filled based on the strengths and weakness of the current team, when product managers and software engineers are not conversant with the “hidden costs” of building software hastily — which sometimes leads them to unintentionally overload a work cycle (sprint) with an extremely high WIP limit.

The unintentional mistakes everyone makes

Sprint planning is that time of the planning phase/process that makes everyone nervous and for good reason. The question on everybody’s mind is how to correctly estimate a tickets’ time to completion (in other words assign story points). It’s not an easy question to answer. Unfortunately, many team members i have worked with in the past have resorted to playing poker with whole thing — they call it Agile Poker or Planning poker. I for one consider Agile Poker to be a very nice idea (because it does solve the fundamental issue of estimating tasks in isolation) but yet a waste of everyones’ time because it removes focus from what it really important to the job of task estimation — determining in a MOST BENEFICIAL manner, the true total costs associated with the changes to be made to software. It also encourages participant to ask questions that seem good but are incomplete like: what skills will be needed for this ? OR how many people are expected to be involved ? but not more complete questions like: how do we organize so we reduce friction and/or delays ? OR how do we organize communication so we reduce integration costs like work duplication, lack of consistency in the codebase details or merge conflicts ? Finally, it relies on the very thing that no product manager or software engineer should rely on alone — intuition.

Planning poker (or Agile poker as some call it) focuses too much on the transactional costs of the work to be done but not so much also on the integration (hidden) costs. Anticipating roadblocks and planning on how to deal with them is ineffective if you do not know the nature, size and sources of such roadblocks/blockers. Sometimes, the nature of the roadblocks or blockers is the other day-to-day stuff that employees do like take part in interviews or pair program with interns. To catch a thief, you must know the thief well; his habits, his schedule, what places he likes to visit e.t.c. You must have a mental model of what motivates this thief to act the way he does. This is similar to how we should approach blockers. It is important to have an actual checklist of all possible things that can happen (which have happened in the past). There should be less focus on curing (dealing with them as they arise) blockers and more focus on preventing them before they happen. Furthermore, you have to know what to prevent to successfully prevent it.

Next up; There seems to be a manner as to how management in general approaches Sprint cadence with respect to capacity and schedules. Paul Graham of YC fame wrote a mildly popular essay titled: Makers’ schedule, Managers’ schedule. I enjoin every product manager to read this essay. It will help you understand the costs and trade-offs that software engineers and by extension the Sprint incurs when you insist on making them fit into your own managers’ schedule. Cadence could suffer a blow as well as leading to issues of insufficient and inefficient resource utilisation on sprint tasks. As a manager you have to understand that writing software code and writing an email don’t have the same exact mental burden even if they both involve simply typing on a keyboard.

My advice is always try to move key meetings to the morning time as much as is possible and make stand-ups asynchronous rather than synchronous. However, meetings that are meant to resolve blockers should be both time-boxed and synchronous with all hands on deck. Stand-ups (by this i mean — reporting only what you did yesterday an today) can happen for each person at anytime during the day but blockers must be communicated periodically with urgency usually very early in morning, noon and 2 hours before COB.

As a product manager, it’s important to value your working relationship with the software engineers on your product team as much and as highly as you value your relationship with upper management. As much as you hold engineers accountable, you should advocate for them also. They need to know you have their backs and that you treat them as your partners and not your slaves or subordinates. Product managers work a lot with uncertainty and ambiguity and need help from software engineers to reduce this uncertainty. It’s important that you and the engineers can handle both and manage expectations. Being awfully rigid or unreasonable about how you setup scope definition and how you deal with scope realization doesn’t help in delivering value faster or better. On the contrary, it makes things much worse.

It isn’t uncommon for software engineers to lose a bit of steam as they proceed from sprint to sprint racing to deliver one or more features that is vital to boosting sales or the bottom line. Subsequent iterations may rack up technical debt and other liabilities that if not dealt with sooner can lead to lower output and perhaps more bugs. A competent product manager is able to listen to these concerns when they pop up and is able to allocate resources to them instead of trying to dismiss them as flimsy excuses not to do the real work. A competent manager wants to affirm their support for engineers to do their best work always.

Furthermore, it’s super important for both product managers and software engineers to learn a bit about each others language or buzzwords and what they mean for the project/product. Product managers need to be a little literate about the certain technical concepts (like UML use case and UML sequence diagrams) and concerns (like refactoring and coupling) that can shift the attention of software engineers away from building out features. Managers should have a surface idea of how the software system is put together and should prioritise communicating in the language of software engineers and putting yourself in their shoes.

More-so, these could inadvertently promote this endless march from sprint to sprint without any breaks. This process is typically endorsed by upper management and usually leads to a “Ship and Forget” culture in the engineering team (which is bad!). As a result, Product managers leave no room for engineers to properly respond to or receive and process feedback on code that was shipped earlier or refactor the codebase (similar to how any machinery needs maintenance from time to time).

The rollercoaster continues towards annual leave for each member of the team. At some point, Product managers can’t find time to think strategically about the product. Also, over time, the team as a whole become very detached from the product refinement and value creation process. Unfortunately, the team could begin to perhaps only focus on whatever it takes to get their pay-checks and probably game KPIs while at it. Cool downs (similar to what is obtainable with Basecamps’ ShapeUp) periods in between work cycles (Sprints) can help in this regard. It gives the teams time to recharge and take stock as well as plan adequately towards the next work cycle.

A situation where managers just chop up the work into user stories and have software engineers act like a conveyor belt moving tickets rapidly and endlessly from “TODO” till “DONE” isn’t a sustainable one.

Software engineers have their own shortcomings too. They (I inclusive too) seems to enjoy shifting most of the burden of planning and preparing for a work cycle (e.g. sprint) to product managers and seem to only get more involved only after a work cycle (sprint) starts. This is a throughly problematic approach to team work both on a technical level and on a product level. I believe that software engineers should get involved early in the discovery and planning process (after the initial pitch has been agreed on). This helps to provide a lot of context and understanding around the problem which leads to little or no misunderstandings in general on what the solution (software) should be, function and appear. Lastly, it also helps engineers generate effort estimates (story points) faster when asked for such by a product manager.

The unintentional mistakes that product managers make

How many times has a product manager asked “So how long is this going to take?” immediately after reading the user story off of a sprint JIRA ticket ? I bet many times. Well, the problem with this is that 9 times out of 10, the software engineer who answers that question is going to be dead wrong! Why ? Because it’s an impossible ask! There’s no way the software engineer can give the manager a correct answer that fast. The software engineer has to firstly check the current state of the codebase with a checklist of things that could possible take time away from the actual effort to build out the feature. Here’s an practical example of such check list:

1. Do i need to update any pre-installed third-party package dependency ?
2. Do i need to find an install any uninstalled open-sourced third-party package dependency ?
3. Does such an open-sourced third-party package dependency exist or do i need to write code from scratch ?
4. Do i need to rewrite any portion of the codebase to allow for this new feature to be added ?
5. Do i need to study a third-party API documentation to proceed on this new feature ?
6. Do i need to move around the files and folders structure to make it possible to update this existing feature ?
7. Do i need to wait for another feature to be done before i can proceed on this new feature ?
8. Do i know how to implement this feature or do i need to research a little bit with help from someone more experienced ?

It’s not possible for anyone to keep the answers to these questions (above) on check list in their heads all the time. Furthermore, finding the answers to these questions (in the checklist) is the only way to answer the managers’ initial question: “So how long is this going to take?”.

Integration costs !!!!

For over 2–3 decades now, the industry have been doing estimations so wrongly that some practitioners, like Allen Holub insist that estimations should never be done anymore. I agree only partially with him. You see, story points were never made to estimate time (how long a task will take). Rather, story points were created to estimate effort (the amount of effort it will take to complete a task). Sadly, the business side of the product development machinery has simply paid certain privileged persons to find a way to incorrectly and unnaturally extrapolate timelines from only story points to a counter-productive end.

There’s no way to estimate time properly without estimating 2 things first:

• Effort (Based off of transaction costs — skill, throughput, man power)
• Uncertainty (Based off of integration costs — unknowns, headwinds)

In the distant past, no one has really cared for estimating uncertainty perhaps because it takes extra time to do so (but worth it). Everyone has always opted for the shortcut of estimating effort only and using some weird extrapolation method to translate that to an estimation of time. Even with methodologies like Kanban, tracking lead time isn’t enough to estimate a similar task in the future. The problem with estimates (as is today in the industry) is that they don’t show or factor in uncertainty. Also, some category of user stories or tasks cannot be planned for or estimated on at all: e.g. improving software performance, improving latency or wait times for software data network calls, improving software security and so on.

This brings me to the first 2 mistakes product managers make. The first is:

All story points are time estimates

The second one is:

Every user story must immediately end up as a task ticket/card deliverable.

These first two mistakes are made because managers either believe there are no “hidden costs” (uncertainties) inherent to the workflow for deliverables in a work cycle (e.g. sprint) or they are in a hurry to deliver irrespective of whether it’s the correct thing to deliver or not. They assume/believe that every work cycle (sprint) starts on a clean slate and all the work done before doesn’t affect the work to be done now. Again, PMs probably don’t care enough that it’s important to factor it in. You see, 50 story points on one sprint may not be equal to 50 story points on another sprint because of the actual time it takes to complete the story points (given that there are no avoidable blockers). However, it should’t be the same because the work done in one sprint isn’t the same as work done in others too. But, most often, it is assumed to be the same. Story points as currently defined are at worst unreliable and at best inconsistent across multiple work cycles (sprints) as a way to track lead time.

The question now becomes: how best do we deal with uncertainties in the workflow to deliver value ?

Well, by reducing or eliminating them! So, what are the ways we can do this ? Again, we often have to embark on a spike. Also, a feature doesn’t have to be finally/completely built in the manner you have conceived it before putting it in front of a user. Why ? well because you (the manager) are never sure a feature (fully built out the way management have it planned) will appeal to end users and provide business value. It’s much safer to build a little of what you can bet on and show the user and then monitor/watch their reaction (product analytics) and go from there to fully build it out.

Take for instance that we had to migrate the code and the infrastructure setup of our software from AWS to Azure to save costs and no one on the team had worked with Azure before or we had to work to integrate with a third-party API no one on the team had worked with this API before. These two scenarios pose a risk/uncertainty going forward. We can’t just dive into the task and estimate the completion time based on a hunch. We have to use a spike and deliver a Proof-of-Concept (PoC) at the end of the spike. Why ? Because sometimes we don’t and cannot know how long these kinds of tasks will take until we try.

Another scenario; say we are to deliver a brand new feature that ought to please users into giving the business (our company) more money because it solves a real pain point. We have a plan we have put together based on long-running engagement with a good number of existing users. Then, we build with the engineering team what we feel is the solution based of what existing users told the product team. But, weeks after delivery of the brand new feature, the feature isn’t being used by up to 50% of the existing users who said they’ll use them. The time we spent building cannot be gotten back. We could have better spent that time engaging the user with several incremental MVPs of the feature over that same time.

To estimate effort properly, we must estimate uncertainty and to estimate uncertainty, we must either work in small batches with more frequent feedback and conduct experiments (or research efforts) called spikes. Also, we could utilise a checklist (like the one above) to identify all possible integration costs (defined above from my last article) and factor them into our estimate. There really isn’t any other way.

Here’s an article that goes into proper estimation techniques in detail which i highly recommend. Anyways, estimation is NOT the job of product managers alone. Software engineers have to pitch in too. Product managers need to a let engineers participate in product decisions. It is then important for both to collaborate on estimation and planning.

Estimation should have a purpose and without that purpose estimation may not really be necessary or even compelling. Estimation usually has the following purpose:

• To ensure proper and efficient allocation of man-power (engineer) resources to a project or a set of projects
• To prepare a contract (especially in software shops) or agreement of how much as contract will cost (fixed price) and what the scope will be.
• To avoid scope realization as much as possible because it isn’t good for business.

However, even though avoiding scope creep totally is mostly attainable, avoiding scope realization totally isn’t. Sometimes, scope realization is a feature and not a bug. Why ? because we cannot predict the future. Nobody can! So when a client at a software shop wants a piece of software working a specific set of ways by a certain date, that’s completely doable but under one condition. Nothing changes and no surprises pop up. I have always found that finding out why a client wants something as a feature is far important than agreeing to o the feature for them. More often than not, things change. Sometimes, change comes from the outside world or the client. At other times, change comes from the surprises introduced by a technical oversight for the software engineers. I am always of the opinion of going for a flexible charging agreement instead of a fixed cost contract when it comes to building software that is susceptible to change. Estimation helps to keep disputes that may arise from a fixed cost contract in check. It also compliments a flexible charging agreement so the parties can understand which aspects of the project need more time, re-working and money and which don’t. See this article for more.

There’s a school of thought that believes that estimates are guesswork. But, this is false. Yes, estimates start off with a good amount of heuristics and uncertainty. However, it would be not false if and only if there were no ways to manage or reduce the uncertainty over time. If uncertainty was always high over a period of time, then estimates will always be wrong and hence will be tantamount to guesswork. But, that is not the case especially when estimates are furnished correctly (i.e. considering uncertainty).

This brings me to the last 2 mistakes product managers make. Here goes:

Prioritising output KPIs over outcomes KPIs

And the fourth and last mistake:

Output KPIs should signal the end of a conversation or inquiry and not the beginning of one

Isn’t it surprising to find out that across multiple software service/business outfits around the world, between 45%-60% of software features delivered to end users are hardly ever used. This is why focusing solely on output from each completed work cycle (e.g. sprint) from each software engineer is always a very tricky move. You want to focus on outcomes more instead. Martin Fowler puts it very nicely in his article here.

So, what are outcomes and what are outputs ? Outcomes are the qualitative, tangible yet almost unquantifiable improvements made as a consequence of superior software work that add value to the customers and the companys’ bottom-line. Outputs on the other hand are the quantitative, tangible and quantifiable contributions that add value to the company’s results for a set of activities related to software work. Also, outputs don’t directly affect the bottom-line most of the time.

Outputs can be gamed as easily as they can be measured but outcomes can’t. Also, despite the skepticisms, outcomes can be measured better when you focus on the product and monitor it usage over long periods of time. In 2020, Mixpanel produced a product analytics report stating that 68% of software services companies don’t have an operational and mature product analytics function. In fact, this infographic showed that Africa digital project hardly give any attention to product analytics. Here:

Africa has maybe just 2 dots: one in South Africa and the other in Egypt

In fact, another product person in Nigeria conducted a separate survey to ascertain the claim by Mixpanel. Here are the results:

It’s important to monitor the product to figure out what features are actively being used and what is not. Furthermore, as one monitors a product, it takes time to really understand what users are doing with the product and areas where they could be stuck or require better interface interaction. The sales & customer success teams shouldn’t be dictating to a product manager on what exactly users are requesting and expect those exact feature requests included quickly. The product manager needs to dig deeper and understand the motivations for such feature requests. This will enable a more a better outcome overall for both the users, product and the business.

The idea that measuring more of the individual contribution than overall outcome for something as complex and adaptive as software engineering is very counter-productive at best. This is not to say that individual contributions should not be measured but using things like Pull request (PR) count, Code review comments, Code commit count, Tasks completed per sprint are incomplete ways to gauge individual contribution. It is better to look at the impact of the work each member with respect to the entire project and the users of the software. Look at what happened because you hired someone and their work impacted the Customer support/success team as support tickets and complaints reduced. Look at the the turn-around time to fix issues or identify them because someone on the engineering team changed the way things were done.

KPIs like Tasks completed per sprint should only be used as basis for more discussion about the team member dynamics. Is someone on the team taking up most of the workload for producing favourable outcomes ? Is a member of the team bugged own by excessive meetings or hiring interviews ? These will make you a more effective product manager.

Finally, as a product manager, understand that planning (a sprint) is more about the process than about the outcome. Do not fixate on the details of the plan but on the resolutions made from the plan. Also, have alternatives for when the plan goes wrong or gets derailed and discuss that promptly with the team as well.

The unintentional mistakes that software engineers make

How many time have you missed a deadline or said you could add a new software feature or fix a bug in an hour and ended up spending 4 hours on it ? No judgement here! I have goofed the same way myself many times before.

Why do you think this always happens and yet it seems as software engineers, we never learn from it. Are we always doomed to repeat it ? Well, No, we aren’t. This brings me to the first unintentional mistake software engineers make.

Making use of only my intuition for passively estimating or judging the effort most tasks will take

As I explained in my earlier article: Decisions that matter broadly, one of the worst things you can do to your career as a software engineer is to depend solely on your intuition all the time. Intuition all by itself is a very bad guide. You will find out that you often have to pair it with something much more definitive and more assuring.

I am not saying that a few times, intuition isn’t just enough. I am saying don’t make it a habit to apply intuition to every case. If you are mostly unsure about how long a task will take, it’s better to not say anything when asked or say “the minute it’s done, i’ll let you know”.

Furthermore, it usually seems very logical and rational to adopt our lax coding habits (habits we are comfortable with on a side project/gig by ourselves) into our workplace. We feel this is an authentic move since others do it too. We are part of a team at work and everyone is bringing their solo coding habit to the job. So, why can’t I bring mine you say ?

When working with others on a time-bound project, efficiency is important. A team cannot be efficient if everyone is doing as they please.

This brings me to the second mistake i know software engineers make:

It’s perfectly okay to bring my extremely lax solo coding practices to my full-time job as part of a team

There are a good number of reasons why this is bad. But first, let me give an analogy: If a lot of musicians gather in on one stage and start signing as lead singers all at the same time and no one wants to be take turns, how do you think the music performance will turn out ? Again, if a dance crew was auditioning for a dance competition and each member of the crew was (dancing) doing what they each rehearsed on their own (but not as a team), how do you think the crew would fair in the audition ?

Well, i guess you get the idea now. The outcomes of the two analogies above aren’t going to be pleasant. Why ? well, one word: disorganisation!

Every great team should have a (one) rhythm! Every good team should have a (one) beat, especially software teams. I have always maintained that the codebase of a project maintained by one team of engineers should look like it was written by one person. If an outsider takes a look at the software code written by a 5-man team, and it looks like 5 people wrote it, then something is very wrong!

In the teams i work in, i always encourage team buy-ins as they help bring the team around the goal more naturally and have the added effect of encouraging team spirit. It’s dangerous to always use a democracy (waste time reaching a consensus with excessive agency) or dictatorship (reach a consensus faster but without any agency) with software engineering teams. I have found that what works best is to openly discuss the pros and cons of a small limited number of ranked solutions/alternatives and choose the one that works best.

My point here is that the lack of quality in people management and organization adds to the cost of building software. If you organize and manage a team well well, there would be great outcomes (lesser costs), but if you don’t organize and manage them well using things like: tight feedback loops, disciplined autonomy (e.g., pair programming, code reviews, or automated testing) and team buy-ins (as above).

I bet you don’t want a situation where you have to spend extra time just to rework the code written earlier by a teammate in order to implement your own JIRA task. Dead weights (or team members who don’t understand the affects of their actions/inactions on others) in a team can surely make useful members miserable (this has been my personal experience in 4 companies i have worked for). So yes, discussing and communicating more in the team can reduce the unintentional friction. I have worked at teams where weaponized incompetence has been deployed by team mates to avoid work that they otherwise can do very well but choose not to.

In the same vein, you wouldn’t want a situation where there is lack of consistency in the work a team does. For instance, if the JSON HTTP response written by a backend team member for one REST API endpoint looks like this:

{
  "status": true,
  "message": "Dispatch processed successfully",
  "data": {
    "id": 102,
    "connection": "opened",
    "strategy": "S-R-U"
  }
}

And another JSON HTTP response for a different REST API endpoint for the same backend service looks like this 👇🏾👇🏾 (It’s easy to see how this can cause the frontend engineers to write extra unnecessary code to deal with the inconsistency — as both are responses for successful HTTP requests):

{
  "statusText": "success",
  "message": "OK",
  "response": [{
    "id": 1,
    "agent_type": "Controller" 
  }]
}

One way to correct this mistake might be to try and develop good engineering practices even while working solo.

Here’s another mistake software engineers make:

Spending more time in the solution space and less time in the problem space

This is a very common mistake amongst software engineers. We sometimes get distracted by. coming up with a solution quickly without understanding the domain or problem. Most engineers mostly believe that they should only concern themselves with just the means to bring to life what the UI/UX designer or what the client has put down as “requirements”. Logic, syntax, data structures and algorithms is all that is needed to be able to do the job of a software engineer — nothing else is needed. So, they push the work of gathering and understanding the problem to be solved to someone else (most manager). Product managers are there to manage the prioritization of and the ambiguity surrounding work delivery not to understand the problem for you.

This creates a gap in the ability of a software engineer to provide a solution that not only is technically sound and feasible but also operationally feasible in the context of the end users. It limits the ability of software engineers to limit technical debt too.

As an engineer, you cannot hope to build a usable, valuable piece of software without understanding what context the solution should operate in. This is why it is crucial that product managers include software engineers as early as possible during the discovery phase and before a UI designers furnishes the first design draft on FIGMA. It is also crucial that software engineers interpret the (non-technical) user story and create technical task(s) from each story.

However, what i have found to be common is that software teams defer to non-technical people (e.g. UI designer, product manager) when it comes to technical decision making. This manner of working together doesn’t lead to better software solutions eventually. A JIRA task ticket isn’t supposed to say if there should be a 2 buttons and a checkbox on a software screen. It’s not the job of a user story to delve into details of technicalities. In fact, technical teams and team members should own the technical decisions and utilise work breakdown session to reduce the work to tasks and estimate properly.

Nonetheless, there are software engineers who prefer to work as part of an “assembly line” of features and bug fixes. Burnouts and indifference might be common in such a working environment.

By taking an active part in understanding the domain and the problem you are building software for, you can save several wasted hours of coding up a solution that doesn’t fit the problem of the domain.

Thanks for reading!