Pareto

Have you witnessed that discussion between a software developer and the call centre associate where the developer prefers to have no duplicate entries in the bugs database and the associate points out that the reports come from two different important customers? It is almost as if they are talking past each other.

For the developer, whether an issue is reported by one person or by many persons, the fix still takes the same amount of effort. Since the developer usually needs to answer the question as to when the work will be done, which in turn requires figuring out how many things need to be fixed, removing duplicates is the appropriate thing to do.

For the call centre associate, duplicates are an indication that multiple customers are impacted. In addition, when talking with a specific customer, the associate really needs the record of the exact previous interaction, so that he can form a picture of that customer’s experience. Therefore, the associate must keep a copy of all of the duplicates, with perhaps a link to the «canonical» bug report; usually located in a different database.

There is an additional cost to identifying and grouping together duplicates — first one has to find the previous instance, and that is not a trivial thing to do. In general, to determine that two problems have the same root cause requires a specialized effort similar to that of debugging. In some cases, it is possible to automate the identification — for example, for issues that can be found in system logs, one can develop simple pattern matchers (or complicated ones, but simpler is usually sufficient, thus better).

When I look at this situation I see two kinds of work. One is the primary work of debugging and fixing the root issue. The other is the downstream impact of not yet having addressed the issue. The more times an issue has been reported, the larger the impact, so a first simplified model is one where this impact is directly proportional to the frequency of occurrence. A more sophisticated model takes into account different burdens depending on the source of the issues — starting with differentiating user reported issues versus those found internally.

The frequency of occurrence is not always available. Many, if not most, of the otherwise valuable quality assurance techniques will either not provide information about it, or will provide distorted numbers. The reason is that the frequency depends not just on the code itself, but also on the usage patterns. There are many usage patterns, for example those provided by the various testing scenarios and simulations. Certainly, one can make use of such frequency data, if available. However, the most valuable frequency data are those obtained when the software is used by the final customers.

In early development, it is rare to have frequency data. However, later on, as the software starts to be deployed, be on the lookout for the possibility of obtaining it. Those that can use their own software the same ways as their customers will are at a huge advantage: they can get data quickly, even before their software is released. It is not always possible to be in this desirable situation, but perhaps your beta testers will allow you to gather this technical usage data. Your testing efforts might also provide data, but do recognize that the testing scenarios do distort the usage patterns (that’s partly the point, they need to catch rare events).

There is a certain cost to obtain the frequency data; however, there is already a cost due to having to deal with the presence of the issue (support cost, managing the issues list, etc.) In addition, it may very well be possible to automate its collection (just be aware of concerns like privacy)

Once you have frequency data, it becomes easier to determine what changes will have the largest positive impact on the experience of the users, and give them a correspondingly higher priority.

80-20, 90-10, Pareto, tall bar, long tail what is all that? Does it always work?

Supposedly it is a magic recipe that shows how to get a lot of benefits from little work. Everyone says it, but in your experience does it work? Let’s have a look, focusing on the main items (and ignoring pedantic details.)

Imagine you are at the beach with your children. Watch them as they collect ten pebbles, put them on the table and make all sorts of patterns with them. At one point they put eight of the pebbles in one line, and two on the other. That’s 80% in one line and 20% in the other. Is that the famous 80-20? If they move one pebble from the shorter line to the longer one, giving nine and one, is that 90-10? Well, it is *a* 90-10, but it is not *the* 90-10. If I do 10% of the work (move one pebble), I get 10% of the result — the pebble is in the other line, I don’t see what’s the supposed big benefit here — the result is in direct proportion to the effort; in other words, in this situation, 90% of the benefit requires 90% of the work not 10%.

A lot of our daily experience is like that. When I go to the strawberry pick-your-own, what I have to pay the farmer, her benefit, is in direct proportion to the weight of the berries. At the cinema, the take is in proportion to the size of the audience. It could have been in proportion to the number of minutes, like it is for long distance phone call, but they don’t do it that way… Hey, wait a second… so there is a choice in what is counted as effort.

If I have one hundred issues in my bugs backlog, and I fix ten of them, I’ve obtained 10% of the way to eliminating my backlog. Or if I have one hundred customers to contact, and I have contacted ten of them. I’m 10% done… So the benefit is proportional to the effort. Really? Which benefit are we talking about? Sure, if the benefit is getting tick marks next to all to-do items, that is correct. If the benefit is the percentage of the work to be done, considering each item might take differing amounts of work or time, then, on average I’ll be around 10% done. But what if the benefit has to do with the value gained when looked at from a different point of view? What if the 10% of customers I contact are the only ones that will buy that week? Or the 10% of the bugs I fix are the only ones that customers hit? That 10% of the effort yields 100% of the benefits in those specific cases. Or I could be unlucky, and the 10% of the effort just happens to be put on those items that will give 0% of the benefit.

One lesson here is that there is a caveat with 90-10, or 80-20. Those formulations happen to add up to 100, so it is very natural to want to add them up, but they do represent different things. Remember, usually when we say 90-10, we mean that 90% *of the benefit* is obtained with 10% *of the effort* The first number has to do with the benefit and the second has to do with the effort. In 90-10 and in 80-20, the numbers happen to add to 100. I think this is intended to make it memorable, but I just find it misleading.The reality is that you are adding different thing — it could be 0-10 or 100-10 or anything between. There could be no connection between them, or a proportional connection, or an outsized connection. It depends entirely on what is meant by benefit, and what is meant by effort.

Incidentally, in the story with the children, what if the pebbles are of all different sizes, one of them much bigger than the others, and that the smallest is actually a diamond… makes you think about value vs effort, doesn’t it? And from whose point of view, yours or the children’s?

The key observation is to look at benefit vs effort, and see if one can take advantage of those «leverage» situations where the percent of benefit is much greater than the percent of effort.