The Limits of “I Will Know It When I See It”

Both engineering and entrepreneurship alternate exploration and verification cycles to develop a solution that satisfies a customer’s need. Both of these rely on the scientific method of “observation, hypothesis formation, prediction, and experimentation” to develop and validate testable theories, engineering solutions, and profitable products. Both require that a new configuration or an opportunity be recognized as distinct and worthy of experimentation/validation efforts and that you understand if you have satisfied or failed to satisfy each constraint or requirement.  This is the basis for “I will know it when I see it.”

The Limits of “I Will Know It When I See It”

A key difference between the talented first level contributor and the effective manager, or the talented solo entrepreneur and the effective entrepreneurial CEO, is their ability to delegate. They must be able to orchestrate a shared understanding and common sense of mission around an idea.

Guy Kawasaki makes this point in “The Art of the Start” when he talks about “making meaning” and a team mantra. If you look at Apple’s success, it is because they are able to frame the requirements for a product in a way that everybody on the project can link their activities to the key goals of that product. They don’t have a hundred-page feature list; they do have a mission for their product.

“I will know it when I see it” also applies to engineering a new technology product. A new technology product is born at the intersection of entrepreneurship and engineering.

How do you make the leap from being a solo entrepreneur or a talented engineer to becoming an effective CEO or manager? The move is really one of doing the work, forming a hypothesis and verification, to being able to delegate. There are broadly two kinds of delegation:

1. The first kind of delegation is the ability to get it out your head in some way that you understand:
   • Can I write a program that does some of this?
   • Can I run a Google search?
   • Can I build a spreadsheet?
   • Can I construct a model of what I am trying to accomplish: e.g. a drawing, an analogy, a simulation or animation?
   • Do I know it well enough to define step by step the process that needs to be followed?
2. The second kind of delegation is the ability to form a small team and create a shared mission. If I am working on a small team or a medium-sized team, can we have a kind of two-pizza meeting with five or twelve people, and hash things up, run a whiteboard or a flip chart or some shared collaboration environment that we come to a common sense of mission. Once that happens, we get beyond the “let’s go see if the boss is happy with this” to actually acting around the objective.

On the engineering side, “I will know it when I see it” comes in a number of ways. In verification, as a particular detail, we have gotten very good at generating a whole bunch of tests. But, what we haven’t been good at is figuring out, what is the status of the test; are we getting closer or further away, where are we; are we making progress? “I will know it when I see it” is not really a good navigation method. We like to know where we are, and where we want to go. We also need to have at least a theory of a path that connects from where we are to where we want to go. There are three requirements for navigation: we have to know where we are, we have to know where we want to go, and we have to have some idea of a path that connects where we are to where we want to go.

One exciting startup that helps engineers with this particular problem is Achilles Test Systems.

“Achilles solves the problem that was created by the first generation of automation efforts. When trying to validate something, the first step was to generate a huge number of tests. We help with the fall-out of all of these automated tests by analyzing the results. There is the risk, if we apply computing power naively, to overwhelm the team with a mass of detail. Our tools address this issue of analyzing the mass of detail. There is no substitute for detailed root-cause analyses; we are not taking the engineer out of the loop.  We help the engineer visualize what is going on and allow him to focus on the critical issues,” explained Chris Kappler, CEO of Achilles Test Systems.

Achilles goal is augmentation, to free up the engineer to focus on the tasks where he brings distinctive value.  The question is how to alleviate 80 to 95% of the work that doesn’t require expert engineering judgment and analysis and free up the team to focus on the 5-20% that truly benefits from human root cause analysis.

“The first challenge is to prioritize the team’s efforts to where we deploy human expertise against the mass of detail. We run a classifier to categorize the outputs. In a list of a thousand outputs, we want to know: what are some cases are more likely the benefit from human expertise, what are some cases that are less likely; and where should we focus our engineering talent to do some debugging? The second challenge is how we debug or analyze these class or categories of outputs. If we know that we have cases that are similar, can we do root-cause analysis on a couple of them, and then make an inference about the rest of that population. For example we have three populations of problems: we have errors that are red, errors that are blue, and errors that are purple. In the naive process, we might start at one end of the errors, debug all the red ones, make changes to the design or change the approach, and rerun. A better approach may be to pick three or four representatives of red errors, debug them; three or four representatives of blue errors, debug them; three or four representatives of purple errors, debug them; and then rerun and see if we have actually killed the class or did we not get it right,” continued Kappler.

Whether you want to call it exploration and verification, whether you want to call it effective delegation, effective automation, or the need to blend human expertise with automation; this is a problem that engineering teams and startups wrestle with. At SKMurphy, this is a category of problems we have been worried about ever since we formed. We look for solutions that automate the “I will know it when I see it”.

Update Sept 1-2009: I will be giving a talk based on this post at the SFBAY ACM on Wed Sep-16-2009 at 6:30pm. See this page for details.

See also

• Video and Slides from “Limits of I’ll Know It When I See It” Talk at SFBay ACM