I don't care if your estimate is drawn from the hip or projected using a state of the art Monte Carlo or machine learning model. It's still an estimate. Still, any number of things that weren't in your model could shift the deadline: people getting sick for a long time, people quitting, people getting promoted out of their critical role, organizational dependencies not delivering on time (other dev teams, legal etc), all of these things have happened to me, and when they do, they can throw off your project for months.
No model can take these things into account, and if it does, it will yield an estimate like "three weeks up to a year" which is useless, and I didn't need your SOTA model to get that answer. Unless you're really only doing cookie cutter stuff, the best form of estimate I've seen used is continuous estimation + being willing to cut features to make it to deadlines with something usable, even if incomplete (build a bike, not half a car). This isn't always possible, but when it is, it saves a lot of headache and makes everything run smoother. But it starts with accepting the fact that you don't know everything from the start.
The article is about forecasting not estimation. That's the point. Don't estimate. Measure.
It usually goes without saying that most forecasts do not include provisions for black swan events. It's generally assumed that going bankrupt or other project externalities will have an impact on the delivery.
I agree with this response. We are normally not asked to predict for situations where something big changes in the team. But I of course acknowledge that these things do happen. When you have a stable team, the numbers that this method yields are also very stable.
My experience is that most project managers take a non-probabilistic approach.
Say you have your usual list of breakdown tasks and assign a time/budget estimate for each in terms of “low”, “most likely”, and “high”. The intuitive answer is to sum up the “most likely” for your total estimate. However, this ignores the probability that a delay in one task affects others.
Instead, if you take into account the covariance relationship between tasks (using historic or simulated data) you often find that “most likely” summation has a quite low probability of being met. For the org that applied this, there was a less than 20% chance we’d meet or best that intuitive estimate. No wonder we were chronically over budget and over schedule!
I've been reading the "Software Estimation: Demystifying the Black Art" from Steve McConnell.
He introduces a distinction that, at least for me, has been instrumental: estimations and plans are different things.
Estimations are honest, based on past performance data and probabilistic on their very natures.
Plans are, on the other hand, built with a target date in mind, taking into account the estimate previously made, desired delivery dates from customers and everything we are so used to.
By planing fulfillment of tasks closer to the estimates, you decrease the risk of the plan failing. You can build a shorter schedule and assume that staff will work overtime, assume more optimistic estimates and so on, but, then, the risk of failure will be higher. Such risk will, of course, never be zero though.
It's a simple distinction, but it has important implications. We don't feel anymore the pressure of making pessimistic, therefore dishonest estimates just out of fear of being pressed to cut the schedule. And also gave us a better argumentative tool to negotiate schedules with our clients.
I think it's also useful for making all the probabilities a bit clearer to project managers. It's like "OK, I know that you need me to commit with a delivery date, but I'm also going to make clear to you that there are some risks involved and I wanna make everybody aware of them"
That’s an important distinction. The way we handled it was by letting managers define their acceptable level of risk and then use the model to define the estimates in that context.
For example, if they were ok with a 60% chance of making or beating a cost estimate, the forecast could be much more aggressive than, say, a management expectation of 90% chance of being on budget
It’s a straightforward enough primer that it can be done in Excel, including simulating the data if necessary.
Even if this type of model is too simple for actual estimation, it’s a useful (and sobering) tool to help managers understand why their intuitive estimates can so often be incorrect.
You can intuit how much “active time” it will take you, personally, to do something. How can you intuit how long a task is going to spend in a queue waiting to be worked on because your team doesn’t have capacity, or another team “down the chain” doesn’t have capacity?
We have queuing theory because people are bad at intuiting the latter, and I don’t even think we're anywhere close to good, as an industry, at intuiting the former.
You can talk about BS (in the context of software) like queuing theory or you can actually write software. I suggest the Mythical man month.
Sometimes I think humans developed language only to be able to pretend doing something:
Best hunter of the tribe kills a mammoth. But he is not verbally talented. Now an army of bureaucrats appear and tell everyone that they were instrumental in slaying the prey by applying some BS methodology. The tribe is gaslighted, the bureaucrats gain importance, influence and economic wealth.
Queuing theory is a branch of mathematics. It is useful, in a software context, for things like predicting server capacity and predicting response times of programs. It is also regularly used to predict things like hospital wait times.
Here is a very good introduction, I hope you can learn something new from it (:
there's no actual use of queuing theory in the article though, it's just mentioned as some sort of irrelevant justification. it's not even a monte carlo simulation, it's a bootstrap. you definitely don't need queuing theory to run a bootstrap
If you build a general ledger application for the 10th time, sure forecasting is fairly straightforward. Nothing I do at my (very large, non tech but highly software driven) employer has ever been done before here. All estimates are treated officially as if they are date time accurate, but changes happen during the lifetime of the project so often you may as well use a random number. I call it a "nod nod wink wink" estimate: every wants it to be accurate, but no one really expects it to mean anything, other than the budget people.
One of my favorite managers required i give him estimates.
I hated it because we both knew the number was bullshit.
On the other hand, having to think about the estimate and give him something, even if at times it was a guess, i still found it beneficial. It meant i focused better, stayed on task, and often delivered on time anyway.
Im not saying everyone needs the accountability rails, but some people excell with this particular helper.
I think that attitude is the right one. Make decisions as if you cannot predict the future and you will be on a better path than one where you think you can.
The way I look at it, you have the most unknowns and risk at the start of a project and the least at the end. As you work on it, you learn about the domain of your problem space and the possibilities and impossibilities of what you can do technically, as a result, those unknowns and risks go down. With that in mind, structure the work you do so as to front-load as much learning as possible to help reduce risk and get a better bearing on how long it will all take.
And yes, be able to cut features if that target date is more important than the ideal product goal you set out with.
Came here to say this. If we can't forecast the economy at the scale of millions of people and trillions of data points you're not going to be able to forecast a company. We should focus on building organisations that are robust enough to withstand challenges that are thrown at them
We CAN forecast a lot of those things. The problem is that people either don't think in terms of cinfiy intervals at all, or they neglect the 2.5-5% on the tails.
> And here’s the brutal question: what good are estimates if they hardly ever align with reality? You could have spent that time on building software.
No one has ever been able to answer this question for me.
In my experience the accuracy of estimates is highly variable. Experienced developers who know each other and the system they're working on well tend to offer more realistic estimates, but even on the most smoothly run teams it's still fundamentally guesswork.
From my perspective it seems like the only real effect of estimating units of work is making developers resent people outside their team. Sure, it gives product people a number they can say when they get harassed about when something will be done, but it's no more accurate than one they could have just made up on their own.
I don't think I see any fundamental difference between estimating and what this author calls forecasting in this respect. It does seem like generating these metrics could consume less time than meeting to make up estimates, but it's not obvious to me that it always would be.
What real value does this add to any business? Any product or business people here? I'm genuinely curious what the purpose of estimating is. It feels like nobody is winning. Product and business people get annoyed by missed "estimates" because what they really want is to know the future, which is impossible. Developers resent being asked to predict the unpredictable. Not to mention I don't know any programmers who prefer talking about doing stuff over actually doing stuff -- all the meetings that come with management styles that include things like estimates feel like an enormous waste of money.
There are two very important reasons for estimation:
1. Coordination with other teams or real-world events. In most decent-sized companies there are many people involved in a product or feature launch: product marketing, brand marketing, support, not to mention other dev teams. Many times these teams will need to broadcast external, hard dates (e.g. "our PR firmed lined up exclusives with journalists on date X") Being substantially late can affect lots of other people. However, importantly, this is not always the case. If you're estimating for the benefit of coordinating teams, be very explicit about which other teams depend on your estimates, and why.
2. To prioritize and decide which features to build. Prioritization is solely about balancing expected benefit with expected cost, so if you're wildly off about the cost, your time may have been better spent building something else. Again, though, if you underestimate by a bit, but in the end come up and say "I wouldn't have really worked on things in a different order anyway" then nothing was really lost. I can definitely think of cases, though, where if I really knew how long something would take at the start I would have chosen to do something else, or at least do it in a different way.
Getting a PR or marketing campaign up must be done in parallel to the dev work.
It can take a few months to find the right journalists in the right publications , get their attention and get their time.
On another note, some product launches are also times with fixed date events (Back to school (media industry), CES (Consumer electronics), any other expo in any other industry). it is key to time the dev deliverables. and if you need to chose between two features to dev, well, you need estimates as in inputs.
Estimates are not perfect, you should not spend 5 man days to plan 2 months of work. Planning can be done with guestimates and best effort and when done right, are super useful.
Given what we’ve seen in the article, that estimates are often off by a factor of 4, what does this really help other teams? It might make them feel good about themselves when the dev team inevitably tells them they’re not going to meet the target date, but that’s not really helpful overall.
Because off by a factor of 4 is still far superior to having no idea whatsoever about what's getting built, when it might be done, and what's potentially slipping. I can't plan anything else very well if I have to wait for enough data to forecast, which itself might not even be accurate.
The estimate isn’t as useful but the process you go through to arrive at it can be. It forces our team to ask questions about high risk aspects of our code - where do we need to get data from? Does this touch <scary legacy service>?
Or at least that’s what I’ve told myself, but admittedly it doesn’t address the central issue of we still don’t know how long this will take. We can usually tell if something is “days” vs “weeks” of work but we still get those terribly wrong sometimes. I’ve found people don’t like it when I say “it’ll get done when it‘s done” though so I’m forced to keep trying.
In practice we just try to write code in a way where most things can be done in “days” of time because the variance in your outcome tends to grow very quickly when the estimate is larger.
It is indeed true that arguing about complexity, while estimating, is useful to spark conversations about the item at hand.
I feel that _estimations_ per se are not a good way to achieve those interesting conversations. You can, for example, analyze those items using the cynefin framework.
The tricky thing about estimations is that, for some people, they end up being a promise that then is used to plan the future leading, in some cases, to an unpleasant reality check.
Well usually a dev team is not working in isolation. And a lot of the time input from one team is output to another.
There are ways to mitigate the “you are blocked by this team until they finish their stuff” organizational problem, for example api contract based dev, but fundamentally you’re still blocked finishing your stuff until someone finishes theirs.
And imagine if hundreds of those teams need to coordinate for a product lunch, you can’t really throw your hands in the air and just let it go to chance.
Or more precisely the companies that are trying “something” would usually outperform the companies that don’t.
And we’ve evolved a lot of ways to mitigate uncertainties. Scrum is all about that btw, make projects and estimations really (really) small so that any wrong estimations do not affect other teams for more than 2 sprints or something like it.
All the software companies that are more than “one team of devs” really do struggle with the fundamental uncertainty of it all, its just the ones that cope with it better are outperforming the ones that don’t. Its kinda fascinating to watch this whole thing evolve over the decades like a natural darwinian selection.
> You could have spent that time on building software.
When people ask "how long will it take" and "I need status" a lot of things bump around in my mind:
- I read a pretty common mistake for new military officers is to spend too much time trying to get perfect strategic information
- I think if I hadn't had to spend so much time, agonizing time, trying to get accurate status, I would have had it for you by now.
- status is not fun, it's like doing your taxes
- there's also a great annoyance: if I am actually good at breaking things down and listing all the pieces to get from A to B, it makes micromanagement of the worst kind possible.
You know, when someone looks at all your line items and CANNOT HELP but cleverly rearrange everything like chess pieces on a chessboard, without being overly concerned for the consequences.
Imo, time spent on estimating would usually be better spent on design review and careful thinking about what the product should actually do. In a project with clear goals and vision, it's usually not hard to see what needs to be done.
Usually the hard choices are what you have to cut. It'll be clear what needs to be done, you just won't have the resources to get there.
Then there are the "researchy" type projects, where you have a demo system and maybe some expert help from corporate research, and your job is to take some flighty technology held together with an unholy mashup of F#, MatLab and a depth camera and turn it into a consumer product, something people will want to buy for a hundred bucks. The first half of the project is figuring out if it's possible, the second half of the project is getting it working, the third half is polishing and shipping.
("Every project can be divided into three equal efforts, each consisting of ninety percent of the work.")
I look back and see that there were some managers I've had who were masterful, but shamefully I can only appreciate their talents from here in the deep future.
There is some level of estimation that we would do even if we were told not to. We always choose what to work on or what to push back on based on some idea of the time it would take. With very rare exceptions, you will usually have some idea of the size of a feature, at least order of magnitude.
Taking it to the absurd, if I asked you to add feature A (say, showing the current date and time in the UI) to your product, I doubt you would say 'it could take me 10 minutes, it could take me 10 years'. You will have some better idea of the size. You will sometimes be wrong about the work needed (oh, it turns out that our hardware clocks are bad and we can't rely on them, we need to change the clocks in the whole line, add another year to my estimate), but that is not going to be the most common occurrence, and so you will usually be roughly right, but only up to something like 4-10x (one day estimate - > 1-10 days; 1 month estimate - > 1-10 months).
The problem is when someone takes all these rough estimates and wants to add them up to divine a release date. Of course, they don't want to go with the honest estimate of '5 months - 4 years', so they tend to average it, and treat the 10x as a 'worse case scenario ', when it is just uncertainty in the estimation. So, they end up with something like '8 months to 1 year', which rarely works out in practice.
In the happy case though, the original estimates still come in handy later when it is time to drop features. 'we' re months out from the deadline, this feature was estimated at 1.5 months, and those 7 other features at 2 weeks each; let's cut the 1.5 month feature, it will not fit anyway', which is a pretty good use of that estimate.
I'm surprised to be the first to write: You estimate because you do contract work for US state/local government, and the paperwork and process is horrible, and so you have to write a contract where you lay out the whole final product up front, and you have to decide what to charge for the whole thing before even starting. That number is going in the contract. It isn't going up.
Fun fact: ISTR at least one locality where, if they found out you started work before the contract was signed, there was some law (or maybe it was a policy, but in any case, no one could break it) forbidding the department from paying for said work.
Once I asked about doing time and materials instead of big design up front. The government department I was working with started saying how that would require them to certify hours, and there was a whole process, and I got the feeling that if I pressed the issue then they'd rather just find someone who could give them a dollar amount up front.
Asking how long a feature will take is like asking how much it will cost to renovate an old house. Nobody can know for sure but you as an expert can give a better estimate than most other people. Nobody just asks a structural engineer to just build a house and that costs don't matter.
Why is that "wrong" estimate so important? Because business people do not see the real cost associated with features. They can only guess how many buttons you need to add to the UI. In order to build good products they need to weigh feature benefits against engineering costs. Even if the cost is wrong, it's about the magnitude compared to other options.
If business people get annoyed by missed estimates they don't understand engineering estimates and how to use them properly.
I've seen that these estimates are inevitably used to judge the performance of teams and individual contributors. If you ask why developers put up with this, my answer is that it's not done openly or transparently. Yet I see it happen, and it is at times acknowledged in hushed tones among insiders. No one's explicitly being promoted or awarded bonuses based on their ability to meet ticket estimates or anything like that. However, a team's failure to achieve predictable burn-downs will be used to justify intervention, and an employee's failure to sustain a certain velocity will be used as part of firing decisions. It's a pulse for management to watch and leverage when needed.
No one can tell the future. Some people try to do it by looking at the past. Investing has a mantra for that, "Past performance is no guarantee of future results."
The sad part is this never changes and never will.
There are situations that are way more predictable than the stock market. We as humans use the past to successfully predict the future all the time. For example we predict that we won't be able to walk through walls not because we understand how atoms bounce off each other but because we experienced not being able to do that.
Software Development is certainly more predictable than the stock market but less predictable than the walking through walls example.
In a stable team, often past performance is a strong indicator of what will happen next. It's not perfect, black swans can occur, but no one is asked to predict if your project will get cancelled for example.
The disdain for estimates in some of the comments here indicate that there is an organizational problem in the way those estimates are solicited, rather than a problem with the idea of estimating work itself.
I used to work for an organization where engineers so universally loathed the idea of estimates that I was convinced that it was just a necessary evil. But a few months ago I joined an organization where the relationship between engineering (which I'm in) and business is not very adversarial. Here, estimates are a tool to answer the question "how much work should we take up in the next sprint without burning ourselves out?". The estimation process is not loved, but it's not hated either.
1) Most of HN is young and has never worked on the business side of an org where they had to do things like lose a customer because the feature they wanted is taking too long to finish or repeatedly push back multiple expensive dependent non-engineering projects because engineering keeps hitting development roadblocks.
2) A lot of developers really hate committing to any kind of estimate because they're really just not that good at estimating for a variety of reasons and getting stuck in a commitment that turns out to be unrealistic really sucks.
3) In my experience a good portion of orgs really are bad at business-engineering interactions and end up in adversarial relationships like you're talking about where engineering is seen as "those guys that refuse to commit to any deadline and suck at communicating status I can plan around" and the business owners are "those guys who don't understand what they're asking for and are constantly trying to micromanage to get blood from a stone". Also in my experience, neither side is entirely innocent in these situations.
> A lot of developers really hate committing to any kind of estimate
In development (as opposed to production) I personally fight against any commitment if it comprises the "what" as well as the "when".
You fix the date of the next release and provide me with a prioritized list of bug fixes and features to work on? Nice, let me start to work!
You think that a specific new feature could improve our sales and you let me drop anything else and call you back as soon as I have any news? Wonderful, I'm already analysing the issue in my head!
Anything else, and I'll let you know that your dreams are not likely to come true.
> lose a customer because the feature they wanted is taking too long to finish
That's sales guy talking.
From my point of view, he is not a customer! A customer is someone who buys the product we currently have ready to sell, or someone who pays for customization.
1) Have you ever worked on the business side? And no, business =/= sales.
2) Except I have seen multiple times when customers paid for customization, then jumped ship (and demanded a refund) because that same customization took too long.
Customization for a customer is an area that needs extreme care. It's very easy to accept that "one little change" to get ink on that sales contract. And at times is a reasonable thing to do. But it can lead to a number of very bad situations. Scenario A, promise too many customizations at the same time you're going to have a bad time trying to deliver on them all. This is frustrating for everyone and is often the result of poor leadership. Pushing sales and revenue targets on one side while ignoring the very real resource constraints on the other. Scenario B, customizations that are customer specific, meaning a customization that isn't likely to be used by other customers due to a quirk in that single customer's workflow to get them to sign. This leads to feature bloat and increases the maintenance burden of the product over time.
There is a reason that product owners need to be able to say, "No" and be good at managing expectations. Otherwise they end up with an unmanageable backlog, broken promises, and sometimes will scapegoat the engineering team. That isn't too say over promising doesn't happen on the engineering side of the equation. But the product owner is the individual ultimately responsible for all feature work promised.
You shouldn't take money for what you can't deliver. If enough customers ask for a customization, add that to your pipeline and you'll have some good news to share when it's ready.
That's nice in theory only. In the real world, dependent projects often have long lead times, which often means foregone revenue for the business because of holdups.
Why not ask that question instead of guestimating individual tasks ?
I think people loathe estimations because they are never the end point, only an input data to derive a result that people don’t want to let them decide directly.
It can be legitimate to not have your engineers decide of the roadmap, but you can’t fault them for feeling abused when you’re not asking the real questions.
Why should you burn yourself out just to do an arbitrary number of tasks, based on wildly inaccurate estimates, just to meet an artificial sprint deadline?
To make the company more profitable at expense of your health.
The whole activity of estimation, planning and reviewing is often just a way to implement micromanagement.
What is the benefit? It pushes engineers to spend 100% of their time on urgent tasks and stop doing any learning, exploration of new ideas, discussing alternatives, socializing.
[Over]committing creates a constant sense of urgency that leads to higher productivity in the short term. Unsurprisingly, it's a source of burnout.
I thought that we’d solved this problem - don’t do estimates but rather adjust what we build to fit the available time. And estimates of what is going to be shipped get more and more accurate as time goes on (because you can ship what you’ve already built), whereas estimates of time remaining can continue to be wildly inaccurate for as long as you like (scope creep etc).
Then the end of the allotted time you have a product. Maybe it’s not good enough. So you either give up or you allot more time to it. But the product should be usable as-is and solve some of your problems.
Problem I have with the above is that it requires developers to be honest and put in consistent effort, but that’s more of a leadership and motivation issue.
I agree, but that's still a form of estimation though, you're just estimating that you'd be able to deliver on your list of features in the time you have. You could be wrong, and then end up with less features, or be forced to spend more time.
i don't think the claim was that we're no longer estimating. the author was attempting to articulate the process of estimation, where initially one promises the world in N amount of time. As N draws near you reduce feature / scope as needed.. and perhaps when the deadline hits and all you got is a int main(void) { printf("hello world"); return 0; } you figure out what you can make due with where you're at. i like this approach and it resonates with a lot of other great comments in this post.
99% of teams that need estimations are not in a phase where they are designing something that is hard to estimate. And those that are usually are prudent enough to not bother R&D with project planning(yet).
99% of engineers are working on things that CAN actually be estimated fairly accurately. New CRUD APIs? New ETLs? New React component? New integration with a product that has a published API? New service? New message format? New protocol? Whatever. Just break it down enough until it’s very clear what it takes to get it done. There will be a few unknowns but you can call them out as risks. Try to eliminate risks first and raise a flag so your manager/PM can adjust his forecast.
The thing is, everyone wants to be “working on the next big thing” so much that they actually believe they are working on something groundbreaking that cannot be estimated. Either that or they are not qualified for the job yet(hence they need to learn something new, which can be unpredictable). They just need to be grounded and it’s a fairly easy journey. They are all probably emulating the symptoms of those in 1% by believing they cannot estimate their work.
> Either that or they are not qualified for the job yet (hence they need to learn something new, which can be unpredictable)
IMO this attitude is pretty prevalent, and extremely harmful to software engineers. Software frameworks, methodologies, and techniques change very fast [1], and if you think that someone who needs to learn something is unqualified for the job, you are going to have an inherently adversarial relation with the engineers on your team.
Learning is and should be a normal and expected part of the day-to-day job of a lot of software engineers today. I think this contributes a fair bit to the problem of estimation.
[1] In contrast, I haven't seen a ton of novel developments in project management since the early 200s, we are still beating the drum of "Do Agile Not Waterfall" almost 20 years later
Yeah, the status-quo for me for several years has been "I don't know" because most things I build end up needing to debug something I don't have an immediate answer for, or construct something new, or talk to a new team, or find a new lib, or change a new setting, or migrate to a new platform. If I'm repeating work I'd done many times before, that's something that should be made more flexible and automated so I don't need to do it any more. (these of course exist, but you're pretty easily replaceable if this is a super-majority of what you do daily)
So my estimates are pretty much always "if it's well-built, documented, and fits what I expect, a week or less. If not, up to 2 months or more, but I should know more in 2 weeks". Getting a more accurate estimate quite often means spending as much or more time investigating than it would take in the optimistic case, so usually the answer becomes "give it a shot and we'll decide again later". Sometimes I have good news, sometimes not.
---
We sit on top of millions of lines of code, changing faster than we can read much less understand. You cannot know it all, nor is it worthwhile to try in the vast majority of cases. Poking into new territory is common, and the ability to do so effectively is a super important skill.
Code organization (part of Architecture) matters a lot here. Especially in large code bases. Software is meant to be “soft” or pliable easily. To do that, one should be able to alter a module without having to understand the whole system. If a small task takes too long, there’s a problem with the software architecture or code organization.
Got it. Better to stick to tried and tested “boring” stuff and actually code small functionality yourself than chasing a thousand line library for trivial stuff.
At work, someone used a 3rd party library to send stats to a server in json format and “standardized” on it. 100s of developers had to now learn this new library and it’s APIs and one bug caused everyone a bit of hassle. It probably made the original dev’s job just a little easy but made it a bit hard for everyone else. These things do happen. The point is to learn from this and make better decisions over time, not give up and say we can’t get better. Trying to provide an estimate forces you to think more carefully and become better over time.
It was not a counter argument to anything per se. Just pointing out that even with episodes of failed estimates, there is reason to continue estimating.
Let me clarify: when I said “they are not qualified for the job” I meant they are not qualified to produce the estimate yet. They can always go back and learn what’s needed and come back with a more educated estimate (still an estimate) or they call it out as a risk item and address it ASAP. More often than not, this is because they are not ramped up (manager’s or team lead’s fault) - why else would you ask a complete newbie, regardless of seniority, to estimate?
> 99% of engineers are working on things that CAN actually be estimated fairly accurately. New CRUD APIs? New ETLs? New React component? New integration with a product that has a published API? New service? New message format? New protocol? Whatever. Just break it down enough until it’s very clear what it takes to get it done.
What I can accurately estimate is how long it will take me to get it done.
What I cannot estimate is time spent waiting for people to finish their stuff, the lies that ‘the API is done’, the going back and forth with the stakeholders when they either change their minds, or I learn that what they want is actually different from the user story.
Let’s say the code base is 1M Loc written by 230 engineers over 20 years. No docs. The epic is to make the whole thing support internationalization. That’s hard to estimate.
That’s a more typical requirement. A system that is built in nice incremental steps eg add an api and can be for its life sounds like a dream.
It’s understandably a big project. But you can still start breaking it down, correct?
Break it down by services first. Then do estimates for say 2 services to get an idea of what it takes to estimate the whole thing. Part of the process is also to communicate the cost to business so that they can see if the feature is worth it or not. (Read “Shape Up” by 37signals on this. Free ebook but you get way more than what you pay).
Can’t comment too much without knowing your codebase but it is engineering. Most tasks are not supposed to be this hard. If they are, then it’s time to start paying back tech debt so you have things under control again.
Tech debt is something that the engineers left undone because of pressure to finish quicker. The example above is not that. It's the fundamental assumptions getting changed mid-project.
The big system was designed with the assumption of single-locale users. This let them move faster, and make the system faster. It could mean schema changes, business process changes, and making hot paths in the code slow in a way that users will not tolerate.
Each service might have it's own level of integration with the single-locale, accounting might be 10 weeks of work, UI 5 weeks, and data processing 1 week. If you pick the wrong system to estimate against, you can easily be off by a factor of 10.
Estimates are definitely doable, but take a lot of time and expertise to do well, not just programming skill, but deep knowledge of the affected systems. An engineer can only estimate accurately for the pieces that they fully understand, and they can only estimate how long it would take them to do it, as the things they have context for aren't very likely to overlap well with other team members.
A dev that's been around for 3 years is likely to estimate lower than someone who's in their first year for many systems. But that thing the new guy implemented from scratch and the 3 year dev hasn't ever touched? Changing that will take much less time for the new guy. (all of this is on average, some people are better at holding context than others)
The book you refer advocates a concept of 'apetite' rather than estimation: "Instead of asking how much time it will take to do some work, we ask: How much time do we want to spend?"
Yeah. The reason I mentioned it is to highlight that the business often needs to know what something will roughly cost and cannot just headfirst five into it. Estimation is a form of communication that addresses this.
The purpose of providing an estimate is to get the person asking off your back, so you can go back to getting work done. Beyond that, there’s little to no value. People ask for estimated because lies are strongly preferred over uncertainty.
Estimate are useful in software businesses for a wide variety of important reasons:
- so you can tell the customer when they can reasonable expect to get something they need
- so you know when you will need to be ready with the next project on the agenda
- do that you will know when you might consider adding resources to a project to make it go faster (yes, this is not always possible, but it sometimes is)
- so that you will know if your funding runway is sufficient for your R&D needs
- so that you can compare two or more different projects when deciding what to work on; time to completion can be an important consideration in these decisions
> For example, the historical throughput of your team can be used to run a simulation of future throughput. Some outcomes will be more likely than others and you can use these numbers to come up with statements like: “41 items or more can be completed in 30 days with a certainty of 85%”.
As usual, there are two base assumptions to this:
- your work environment didn’t and won’t have any significant change for a long time (tools, requirements, team composition etc.)
- your team output is somewhat stable, with no month with 300 tickets and others with only 50 tickets.
In my experience these two assumptions are surprisingly rare to justify in any decently growing company.
In particular as company grows, tools change, organization and processes change, people change. On top of that there is often seasonality.
We can still throw around forecasts, but IMO in these conditions it has no more legitimacy than random guesses.
Significant changes will indeed make your forecast not come true. But mostly people are not asking to predict for situations with significant changes.
Incremental changes will just be incorporated in the forecasts over time. The forecast continually adjusts to new information becoming available.
It's interesting to know that after about ten work items completed the number become pretty stable. So it's easy to reset or adjust the forecast in case of a big change in the team.
I was personally surprised to learn how stable the output of a team is if its composition doesn't change. Tools have some impact on productivity but so big that you have to throw away your predictions. No one is complaining anyway if you over deliver a little.
> people are not asking to predict for situations with significant changes
That’s for coming up changes. Your predictions will also be of poor quality after a significant change as you lose the link with historical data.
In the end there might be only a small window between two changes where they are worth anything.
To be clear I am not saying you shouldn’t make prediction anyway, just that the effort to come with “a system” is not worth it in a lot of situation.
For context, tech industry has one of the highest turnover, a team losing/gaining a member is not some rare event. A new boss coming in to change process or teams isn’t either.
The only projects I can accurately forecast is the one I have already implemented elsewhere. Meaning 90% of it is a rehash of something done elsewhere. Companies do this because... well.. reasons. Anything else novel/unique/innovative by definition can’t be predicted well because if you knew everything from the start, there isn’t any innovation being made. Many of the good companies that exist today pivoted from a broken prior idea or discovered a market while experimenting. This applies to every part of software development.
You've made similar comments throughout this thread. You don't seem to realize that "novel" does not have to mean universally novel. It could be something novel for the individual.
Recently I had one of these "boring" tasks at work. Set up simple CRUD API and hook it up to graphql for the front end to consume. Only the graphql server was implemented in the most strange way I'd ever seen, so figuring that out took longer than I thought it would, oh and devops had completely uprooted the way we connect to databases to increase our security.
The company is far from the last agile company I've ever worked at, but on large, complex teams, what I'm describing is incredibly common. You don't know what you don't know.
It only means you didn’t do your homework(no offense, everyone falls into this trap too and conclude the issue is that estimation itself is not possible ). You can’t give an estimate based on what the task would take at your previous job.
You need to call out what you don’t know(Now, don’t give me Rumsfeld wisdom.. Most software is not that complicated)
The homework is the job. You can only know how long it will take if you know what you’re going to do. And if you know what you’re going to do, why not just go do it? Just typing out the code doesn’t take that long.
It could be because you are working on a brand new area every single “Sprint “(or equivalent). That’s not prudent assignment of resources(the manager should know this).
I have worked in companies where ownership is somewhat static(same few folks work on a given area of the product) and things get done within 10% of estimates (a lot of innovation too).
I have also worked in terms where ownership is fluid and “resources” get assigned to tasks. Nothing is now “estimatable”. Your experience is probably in line with the latter.
Personally - a fair bit. :) At my last job I was hired to build something for internal use that wasn't available to buy anywhere commercially (a year after I started, some things started coming out that were lower quality and more expensive than what we had). Even after it was built, a lot of my work was mostly investigative to attempt to root cause things like nondeterminism, in systems that I didn't build and wasn't responsible for.
At my current job I'm on a team of generalists partially responsible for making sure work from every team is going to be usable by other teams - which can involve everything from just advising to being embedded in the other team for a few weeks/months (to simplify a bit).
Edit: both companies are in the self driving space.
Novel is often a matter of perspective. For example a company with no experience in GraphQL might want to integrate it, they don't know how long it will take because it's novel to them.
There's pretty much always something novel or unique in the way things are put together. I recently had a project that was at first glance a pretty trivial "enter data, verify it, package as XML and send to a backend". Until we realized that the combination of privacy requirements and an industry spec would have required us to do schema validation of the XML in the browser. And apparently the only Javascript library which
does that is machine translated from C, not very stable and doesn't support UTF-8.
O/T but was it XML Schema specifically, or just XML DTDs that you needed to validate against? For the latter case, there's [1] (my proj), plus the option to rewrite XSDs into DTDs (under natural assumptions eg no local element redefinitions) without having to wrestle with Xerces and/or libxml (which hasn't formally completed XSD support anyway) and emscripten. You might also have luck by transpiling Xerces/Java to JavaScript using Google's j2cl+closure-compiler (which works very well if you can swallow bazel and/or isolate the actual j2cl Java app from its bazel wrapper).
Some outcomes will be more likely than others and you can use these numbers to come up with statements like: “41 items or more can be completed in 30 days with a certainty of 85%”.
I did something similar recently:
I started with taking the 20 tickets we closed and plotting story points vs actual time they took. Results were as follows:
- 8s were mostly garbage, but consistently took more than one sprint.
- 1s and 2s were also garbage but always took less than a 5.
- 3s and 5s were fairly consistent, but at the same time could take up to 100% more.
Conclusion: split up all 8s if possible, treat 1s and 2s as 3s.
Second, I removed the outliers and created a distribution of point-per-sprint rates.
The result: 5.6 points per sprint with a standard deviation of 2.5 - pretty bad, but from that I could tell that our chance of delivering more than a given amount of points per person per sprint was(assuming a normal distribution):
85% for 3 points.
60% for 5 points.
17% for 8 points.
4% for 10 points.
Conclusion: per sprint we should either take one 5 point task or two 3 point tasks, but avoid 5 + 3 and never take on a 5 + 5. Also take a 3 if we didn't deliver the previous sprint.
Last point: some outliers were estimated somewhat highly (5), but took much less. One feature they had in common was the word "analysis" in the ticket title.
I guess the lesson here would be to ask: are we really planning on taking a whole sprint(two weeks) to analyse something?
I appreciate new approaches to software estimation. But while this is an interesting approach, it fails to adequately address the primary issue: defining a "unit of work".
If it were possible to predict, with any degree of confidence, that task A (to do) is approximately equal to, or a 1/x fraction of, task B (completed), then the estimation problem would not be nearly so difficult to begin with.
Yet that's what this approach requires.
So cool idea, but not terribly applicable to actual projects in my opinion, unless you're able to break down projects into easily definable and predictable units of work (and yes, I know that there are various frameworks that attempt to do just that, with mixed results).
Unit of work is explicitly vague because it refers to the units you are using in your project, be it user story, requirement, epic, bug etc.
You do not need to know the size of each unit. In fact the method I describe acknowledges that there is variance in the size of each unit.
Some people advocate "same sizing" units of work. I have always thought that was a weird idea because it would imply making small units bigger. How would you make the unit that describes changing the color on a button the same size as integrating with an API? You can't even if you would have perfect knowledge of the amount of work required.
> Throughput is the amount of work that comes out of your project by unit of time. It’s up to you to decide what units make sense for your context. Days, weeks, sprints, stories, bugs, epics – anything goes as long as you’re consistent.
Using days/weeks/sprints as your unit of work for determining throughput seems circular. If you want to know how many weeks of work your team can produce per week then you don't need a monte carlo simulation to tell you that.
Using stories/bugs/epics is flawed too, I think. You can have a fantastic model for your team's throughput in stories per week, but that doesn't tell you anything about when the project is going to be done unless you know how many stories there will be. There are two variables here (throughput and quantity) and you can't get useful information out of the product of them for free.
To see why, imagine that you take only the minimum amount of effort to very roughly divide the project into sensible-seeming chunks. In that case, your throughput in chunks per week will be meaningless (i.e. your model will have a confidence window which is uselessly wide) because the chunk division will have barely any relationship at all to the amount of actual work in each chunk. Now imagine that you're a bit more diligent in your project planning and look in a bit more detail at the work that will be involved. You've done some work to clarify what code will need to be written, and your confidence window will narrow accordingly. Now imagine that you're even more diligent. And so on, and so on. You eventually end up with zero error in your model, but in the process you've completely determined what code will need to be written, and the project is finished! Congratulations, you've invented waterfall. There's no free lunch here as the intro paragraphs of the blog post promise.
I assume that in practice you're stopping at some point in the middle of the extremes of doing nothing and planning out every line of code, and then applying your statistical model at that point, but I think you'll still be thwarted by the (at that point) partial disconnect between the chunk divisions and the actual amount of work in each chunk. We've all experienced innocent-seeming tasks that end up consuming vast amounts of time unexpectedly, and various degrees of this phenomenon is what fundamentally ties the amount of error in any pure "stories per week" model to the amount of effort you spend planning and estimating the stories. No free lunch.
Exactly. I read the article fairly carefully, but the logic seemed circular to me. For a novel project, the problem of how to divide it up into predictable units of work remains. It's always the black swan little subprojects that destroy an estimate, and this approach doesn't help with those unforeseen events. If I could avoid those that reliably, I'd be able to estimate projects with a high degree of accuracy, even without using this approach.
In my experience, only increased subject matter expertise and a solid team with lots of experience can minimize the black swan events, and even then not 100%. They still happen with disheartening frequency.
And as you say, it doesn't buy you much if you want to measure how much work your team does per day/week/month, either, for the reasons you describe.
So what happens in your method when you hit a major unforeseen issue, as frequently happens in a project? If your prior 2 projects went by without a problem, then you'll still be sitting on a bad estimate.
Yes, after time, this should start to even out with your method, but who keeps the same team or even stays in the same job for more than 3-4 major projects? It's fairly unusual these days. Yes, I know there are some senior devs that stay at the same job for 5 or even 10 years, but their team, and even the company's hiring standards, are likely to chance significantly during that time.
That said, it is an intriguing idea, and one that bears more study. I'm not yet convinced but I may try it out.
As a freelancer, I often do not see the problem in the estimate or forecast as such, but in the willingness to accept an honest / realistic one. This of course has a direct impact on my sales position. To make a long story short: Most people want to be lied to as long as time is what is being sold.
There are a lot of responses here that boil down to: “Just use your intuition.” This is a legitimate way to estimate, however, like any estimation technique, it has weaknesses that make it inappropriate for many situations.
What’s most helpful is to abandon the idea that there’s “one true way” of estimating and instead educate yourself about the many estimation techniques that exist, and when and where to use them.
Also keep in mind that estimates are inherently political, that they can be used to guide or to destroy, and that may be the dominant factor in the choice of technique to use.
For a very good introduction to all of these factors (and others), and a survey of numerous estimation techniques, I recommend reading “Software Estimation: Demystifying the Black Art” by Steve McConnell.
> you can just start by writing down start and end dates of each chunk of work in your process
End date is usually easy: "when someone closes the ticket" but maybe "when it was released" is better?
I have trouble determining the start date though. Some ideas: "When the customer reported it", "when the ticket is created", "when someone planned a release date for the ticket", "when the ticket was commit to in a sprint backlog", "when the ticket is assigned", "when the ticket was assigned to the one who finished the ticket", "when someone clicked on 'start progress'".
Each possibility comes with its own kind of noise and some requires additional discipline.
I normally recommend when starting out to take a small segment of your process first and then widen it as you get more experience.
For the start point I usually try to think of these aspects:
- Can work still be cancelled at this part of the process. If so it's probably not the right spot to measure the start date.
- Can work still be reprioritized at this point in the process? If so that will significantly widen the precision of the predictions that the forecast will make.
Also note that setting a the start date at the moment of commitment still gives you valuable insights in how much work you will be a able to next.
For setting the end date I think when the software is released is the most interesting point to measure. As always it depends on context. If you do a yearly release this doesn't make any sense because the granularity of all your predictions will be a year.
In short: don't try to estimate how much time each item takes. Calculate the baseline of past tasks and extrapolate into the future.
It's a good start on the path to more realistic estimations. But it does not yet satisfy what has in my career always been the main purpose of estimates: relative cost analysis.
When colleagues ask me how much time a task will take, responding by "well, tasks take on average 8 days" is not helpful. They need to know which tasks are worth the cost.
How could we come up with estimates that are both evidence-based and relative?
Forecasting time to move an item through dev cycle is best done this way, using historical data.
But if you want to know how long it will take to build "new system we have never done before" then, we are back to the estimation because i have to build a model of what the system will be - and although Incan think Inammaccurate I won't be ...
In then end forecasts and estimations are just able to answer two things - when wont it be ready (ie it's definitely going to take longer than two months) and have you thought of all the foreseeable risks?
Reading though the comments, it strikes me as we are all talking about things at many levels of abstraction. Forecasting should work fine for tasks defined at a fine granularity - we mostly take 5 days when we estimate 3 days. However, they fail fast when you are talking about a 6 month project. There are more variables and dependencies involved.
It also seems like the author is not from US and their definition of stable team might be very different from what I have experienced :)
I used to say the software project would be done two weeks after the last change to the requirements and specifications. No more "just one more thing".
i kinda started thinking about how project estimation and deadlines fit into our process of software development. For example, say you're building some feature and you guesstimate it'll take you 3 weeks to implement, maybe it's non trivial but not too hard. How do you convince yourself, or your stateholders, that spending an extra 2x-3x of time would result in delivery a better product.
in other words, if all software was built with the bare minimum amount of time to ship, i think we'd miss out on a lot of quality and a bunch of features that weren't thought out enough. so there are people that somehow are able to stop this rushed process of development and posit that the solution requires additional thought and planning. They go slower to move faster and knowing how to do this I think is a mark of an excellent engineer.
anther interesting element to this is that when one is granted the additional time, the outcome progresses our field. As an example, someone spent _a lot_ thinking about how to implement a concurrent wait-free list data structure. how are you to convince your PM that we have to delay the "onboarding popup" feature because you'll be spending the next 3 months investigating wait free data structures.
I would love to have a prediction market for the plans at work. I fear management would mostly not appreciate such honest assessments though and nobody is prepared to think in probabilities.
Good estimates come from experienced estimators, and to me that is pretty much “guesstimating”. No one is ever completely correct though, the estimate needs to acknowledge risks and how they may blow out time, cost or quality, then mitigate those risks or manage them.
Unrelated; Why is there no rss feed for this blog? I can't find it. It seems they've interesting topics and I'd like to keep posted when there's new content. Do the expect me to go to their website every day to check?
The thinking here was that work that reaches a cycle time that is higher than 85% of the other work we completed is probably worth some extra attention. Is the person working on it stuck? Is the problem so complex that it might be useful to sit down and have a look at it together?
We all felt it was a good motivator to work together on a regular basis. Or at least step up the amount of communication around that unit of work.
My intuition would be: our "forecasting" solution is never going to be 100% right. Here's an example where it's completely wrong, scrap the "forecast" and get the work done. Don't sleep in the office overnight, don't throw more developers at it; acknowledge no forecasting system ever will get creative work predictions/forecasting correct.
One difference I see is that there is no consideration of the size or complexity of a "story" or unit of work. Scrum and Agile dictate that you spend tons of time and effort assigning points and shuffling points around to measure your velocity, etc. etc. This post seems to say, forget about points? As long as these stories represent a finite amount of work, the size of that work doesn't matter when forecasting how many more chunks of work will get done in the future.
Unfortunately some companies insist in using mam hours for estimations instead of story points then you end up into massaging hours into a virtual unit relating to story point to adequately compensate for the fact that not all developers hours are equal.
I don't care if your estimate is drawn from the hip or projected using a state of the art Monte Carlo or machine learning model. It's still an estimate. Still, any number of things that weren't in your model could shift the deadline: people getting sick for a long time, people quitting, people getting promoted out of their critical role, organizational dependencies not delivering on time (other dev teams, legal etc), all of these things have happened to me, and when they do, they can throw off your project for months.
No model can take these things into account, and if it does, it will yield an estimate like "three weeks up to a year" which is useless, and I didn't need your SOTA model to get that answer. Unless you're really only doing cookie cutter stuff, the best form of estimate I've seen used is continuous estimation + being willing to cut features to make it to deadlines with something usable, even if incomplete (build a bike, not half a car). This isn't always possible, but when it is, it saves a lot of headache and makes everything run smoother. But it starts with accepting the fact that you don't know everything from the start.