Organizing Your Experiment Log

In Chapter 9 of “Zen and the Art of Motorcycle Maintenance,” Robert Pirsig goes into an extended explanation of the Scientific Method using the metaphor of motorcycle repair. He stresses the value of an experiment log, explaining how to organize it so that you don’t become lost in exploring for solutions to a problem. I have excerpted it below and intermixed some commentary about applying it to early market exploring and debugging new product introduction problems.

Two Kinds of Logic…Combined with Guessing

I talked about hierarchies of thought…the system. Now I want to talk about methods of finding one’s way through these hierarchies…logic.

Two kinds of logic are used, inductive and deductive. Inductive inferences start with observations of the machine and arrive at general conclusions. For example, if the cycle goes over a bump and the engine misfires, and then goes over another bump and the engine misfires, and then goes over another bump and the engine misfires, and then goes over a long smooth stretch of road and there is no misfiring, and then goes over a fourth bump and the engine misfires again, one can logically conclude that the misfiring is caused by the bumps. That is induction: reasoning from particular experiences to general truths.

Deductive inferences do the reverse. They start with general knowledge and predict a specific observation. For example, if, from reading the hierarchy of facts about the machine, the mechanic knows the horn of the cycle is powered exclusively by electricity from the battery, then he can logically infer that if the battery is dead the horn will not work. That is deduction.”
Robert Pirsig in “Zen and the Art of Motorcycle Maintenance“

He leaves out hypotheses which may be the results of deduction, induction, abduction (plausible guessing), intuition or brute force exploration.  Forming and managing business hypotheses is a skill that distinguishes experienced entrepreneurs from novices. The experienced entrepreneur often knows where to look, the novice may see a bewildering array of possibilities but can sometimes select a novel approach overlooked by the expert because experience has biased him away from it.

Real Problems Are a Maze of Twisty Little Passages

“Solution of problems too complicated for common sense to solve is achieved by long strings of mixed inductive and deductive inferences that weave back and forth between the observed machine and the mental hierarchy of the machine found in the manuals. The correct program for this interweaving is formalized as scientific method.”
Robert Pirsig in “Zen and the Art of Motorcycle Maintenance“

I have earlier written that “When Exploring, Keep a Log” because your memory can be overwhelmed by a flood of new data. Pirsig goes into more detail about the importance not only of logging observations–the base line for induction–but also how to organize your problem solving.

Organizing Your Experiment Log

For this you keep a lab notebook. Everything gets written down, formally, so that you know at all times where you are, where you’ve been, where you’re going and where you want to get. In scientific work and electronics technology this is necessary because otherwise the problems get so complex you get lost in them and confused and forget what you know and what you don’t know and have to give up. In cycle maintenance things are not that involved, but when confusion starts it’s a good idea to hold it down by making everything formal and exact. Sometimes just the act of writing down the problems straightens out your head as to what they really are.

The logical statements entered into the notebook are broken down into six categories:

1. statement of the problem,
2. hypotheses as to the cause of the problem,
3. experiments designed to test each hypothesis,
4. predicted results of the experiments,
5. observed results of the experiments and
6. conclusions from the results of the experiments.

Robert Pirsig in “Zen and the Art of Motorcycle Maintenance“

These six types of entries are key to effective experimentation. If you don’t have a clear statement of the problem that the team agrees on this lack of consensus will cause ongoing problems. Most entrepreneurs don’t have problems generating hypotheses but they can often be mistaken in their assumptions, which form the baseline for problem definition and hypotheses.

An experiment is a defined course of action agreed to in advance with outcomes predicted by the team. Learning is often accelerated by the surprise that actual results produce that are at variance with your prediction, so having everyone write down their assumptions speeds learning.

Careful observation of results is also important. If your experiment involves one team member asking questions or making offers or otherwise interacting with people it’s a very good idea to enlist a second–or even third–member of the team to tag along and observe. It’s very hard to observe human interaction carefully when you are part of the interaction.

Writing down immediate conclusions captures your state of mind immediately after the experiment. It’s often a very good idea to give yourself at least a day to sleep on them and see what other associations percolate up.

Embrace Your Ignorance

In Part One of formal scientific method, which is the statement of the problem, the main skill is in stating absolutely no more than you are positive you know. It is much better to enter a statement

“Solve Problem: Why doesn’t cycle work?”

which sounds dumb but is correct, than it is to enter a statement

“Solve Problem: What is wrong with the electrical system?”

when you don’t absolutely know the trouble is in the electrical system. What you should state is:

“Solve Problem: What is wrong with cycle?”
“Hypothesis Number One: The trouble is in the electrical system.”

You think of as many hypotheses as you can, then you design experiments to test them to see which are true and which are false.

This careful approach to the beginning questions keeps you from taking a major wrong turn which might cause you weeks of extra work or can even hang you up completely. Scientific questions often have a surface appearance of dumbness for this reason. They are asked in order to prevent dumb mistakes later on.”

Robert Pirsig in “Zen and the Art of Motorcycle Maintenance“

Confidence that you can solve a problem is always an important element of your team’s ability to persevere. This should not preclude a thorough and frank admission of what you don’t know and need to learn. The best experiments, from the point of view of learning, have a 50-50 chance of proving your prediction correct. Winning”sure thing’ bets and losing long shots does very little to change your state of information. Probe closer to the edges of your understanding.

Real Problems Are a Maze of Little Twisting Passages

“Part Three, that part of formal scientific method called experimentation, is sometimes thought of by romantics as all of science itself because that’s the only part with much visual surface. They see lots of test tubes and bizarre equipment and people running around making discoveries. They do not see the experiment as part of a larger intellectual process and so they often confuse experiments with demonstrations, which look the same.”
Robert Pirsig in “Zen and the Art of Motorcycle Maintenance“

When experiments involve conversation you may not really know what you have learned until you have had a chance to correlate it with several others. People are much more mutable than chemicals or machines and the amount of noise in market entry experimentation is therefore much much higher.

“A man conducting a gee-whiz science show with fifty thousand dollars’ worth of Frankenstein equipment is not doing anything scientific if he knows beforehand what the results of his efforts are going to be. A motorcycle mechanic, on the other hand, who honks the horn to see if the battery works is informally conducting a true scientific experiment. He is testing a hypothesis by putting the question to nature.

The TV scientist who mutters sadly, “The experiment is a failure; we have failed to achieve what we had hoped for,” is suffering mainly from a bad scriptwriter. An experiment is never a failure solely because it fails to achieve predicted results. An experiment is a failure only when it also fails adequately to test the hypothesis in question, when the data it produces don’t prove anything one way or another. Skill at this point consists of using experiments that test only the hypothesis in question, nothing less, nothing more.

If the horn honks, and the mechanic concludes that the whole electrical system is working, he is in deep trouble. He has reached an illogical conclusion. The honking horn only tells him that the battery and horn are working. To design an experiment properly he has to think very rigidly in terms of what directly causes what. This you know from the hierarchy. The horn doesn’t make the cycle go. Neither does the battery, except in a very indirect way. The point at which the electrical system directly causes the engine to fire is at the spark plugs, and if you don’t test here, at the output of the electrical system, you will never really know whether the failure is electrical or not.”
Robert Pirsig in “Zen and the Art of Motorcycle Maintenance“

Your business has to mesh many different elements successfully to be viable. The patience to explore key aspects in isolation, which seems slower, prevents confusion later on. The primary difference between the motorcycle maintenance metaphor and most new market entry problems is that you don’t know that any aspect of your business was viable. Startups are not a turnaround situation where the model used to work and something changed and now you have to determine is it an internal problem or a change in the environment you need to manage to get back on track. Many teams have an an idea for a business that is not viable and are more than a few iterations from one that is because there is more than one thing wrong.

Debug the Vision In Parallel With Current Approach

In the final category, conclusions, skill comes in stating no more than the experiment has proved. It hasn’t proved that when he fixes the electrical system the motorcycle will start. There may be other things wrong. But he does know that the motorcycle isn’t going to run until the electrical system is working and he sets up the next formal question: “Solve problem: what is wrong with the electrical system?”

In a startup it can be hard to tell whether the blueprint is wrong–or not viable–or you have not implemented it effectively.  If you can find ways to simulate the model, to validate it at a conceptual level, this does not mean you have verified that it will work in the “real world” but it should give you practice in challenging key assumptions that support it and key relationships and hypotheses embedded in it.

Don’t Be a Lone Gunman

An untrained observer will see only physical labor and often get the idea that physical labor is mainly what the mechanic does. Actually the physical labor is the smallest and easiest part of what the mechanic does. By far the greatest part of his work is careful observation and precise thinking. That is why mechanics sometimes seem so taciturn and withdrawn when performing tests. They don’t like it when you talk to them because they are concentrating on mental images, hierarchies, and not really looking at you or the physical motorcycle at all. They are using the experiment as part of a program to expand their hierarchy of knowledge of the faulty motorcycle and compare it to the correct hierarchy in their mind. They are looking at underlying form.

Some other aspects of real entrepreneurship that are different from the pure scientific experiment model are that if you have a partially operational business model, for example you have launched a new product but are not profitable, you can solicit hypotheses not just from your team and advisors, but also from your customers, your suppliers, and your channel partners. In particular, Pirsig’s suggestion that you need to do this all on your own without explaining it to others only works on relatively simple problems.

Soliciting feedback more widely from your stakeholders and providing as clear a view of your current model as you can to them not only increases your chances of building a viable business but cuts the time and effort needed to realize it.

Related Blog Posts

• When Exploring, Keep a Log
• Buying a Map vs. Learning To Explore
• Five Questions to Ask Yourself Twice a Year
• J. W. King on The Unwritten Laws of Business
• If You Are Cycling Through Chaos, Keep Pedaling
• Write Down Key Commitments And Questions That Need Answers
• How to Tell When Your Team Has a Workable Plan of Action
• Associating, Pattern Matching, and Sensemaking

“Notebook” image licensed; © BlueBright

A Note on “A Maze of Twisty Little Passages”

An early computer adventure game called “Adventure” or “Colossal Cave Adventure” has two mazes: the “all different” and the “all alike.” In the “all different maze each room has a unique description;

YOU ARE IN A MAZE OF TWISTY LITTLE PASSAGES, ALL DIFFERENT.
YOU ARE IN A LITTLE MAZE OF TWISTING PASSAGES, ALL DIFFERENT.
YOU ARE IN A MAZE OF TWISTING LITTLE PASSAGES, ALL DIFFERENT.
YOU ARE IN A LITTLE MAZE OF TWISTY PASSAGES, ALL DIFFERENT.
YOU ARE IN A TWISTING MAZE OF LITTLE PASSAGES, ALL DIFFERENT.
YOU ARE IN A TWISTING LITTLE MAZE OF PASSAGES, ALL DIFFERENT.
YOU ARE IN A TWISTY LITTLE MAZE OF PASSAGES, ALL DIFFERENT.
YOU ARE IN A TWISTY MAZE OF LITTLE PASSAGES, ALL DIFFERENT.
YOU ARE IN A LITTLE TWISTY MAZE OF PASSAGES, ALL DIFFERENT.
YOU ARE IN A MAZE OF LITTLE TWISTING PASSAGES, ALL DIFFERENT.
YOU ARE IN A MAZE OF LITTLE TWISTY PASSAGES, ALL DIFFERENT.

In the “all alike” maze each room is described as “YOU ARE IN A MAZE OF TWISTY LITTLE PASSAGES, ALL ALIKE.”