Preface

This book teaches the theory and practice of Design of Experiments (DOE) — a systematic method for planning experiments that extracts the maximum information from the minimum number of runs.

It is written for engineers, scientists, and anyone who runs experiments, benchmarks, or tests and wants to do so more efficiently.

DOE is not new. Ronald Fisher developed the foundations in the 1920s and 1930s for agricultural experiments at Rothamsted. George Box, Genichi Taguchi, and others extended it to manufacturing, quality engineering, and process optimization. Yet the technique remains underused, especially in software engineering and technology, where practitioners often default to changing one variable at a time or running exhaustive grid searches.

This guide takes a practical approach. Each chapter introduces the theory you need, then shows you how to apply it. Mathematical derivations are included where they build intuition, but the emphasis is on understanding when and why to use each technique, not on proofs.

Prerequisites: Basic familiarity with mean, variance, and standard deviation. No prior statistics coursework is assumed.

How This Book Is Organized

Part I (Chapters 1–2) lays the conceptual and statistical groundwork. You'll learn why planned experiments outperform ad hoc testing and acquire the vocabulary needed for the rest of the book.

Part II (Chapters 3–6) covers the major design families — full factorials, fractional factorials, screening designs, and response surface designs. Each chapter explains when to use the design, how the mathematics works, and walks through a realistic example.

Part III (Chapters 7–9) focuses on analysis: interpreting Pareto charts, fitting response surface models, diagnosing problems, and making data-driven decisions.

The User Guide at the end is a complete software reference for the DOE Helper Tool, including configuration fields, CLI commands, and step-by-step tutorials.

Chapter 1

The Cost of Unplanned Experiments

Every experiment costs something — time, money, materials, compute cycles. A poorly planned experiment wastes these resources and may lead to wrong conclusions. A well-designed experiment extracts the maximum learning from each run.

Consider a team tuning a database. They have five configuration knobs they suspect affect performance: buffer pool size, I/O capacity, connection limit, query cache, and thread pool size. The one-variable-at-a-time (OVAT) approach would be:

1. Fix all knobs at their defaults.
2. Change buffer pool size. Measure.
3. Reset buffer pool. Change I/O capacity. Measure.
4. Continue for each knob.

This requires at least 10 runs (2 per knob) and only measures each knob's effect in isolation, at the default settings of every other knob. It tells you nothing about how the knobs interact.

A Plackett-Burman screening design tests all five knobs in just 8 runs and estimates every main effect simultaneously. A full factorial on the two most important knobs then takes 4 more runs and reveals their interaction. Twelve runs total, and you know more than OVAT would teach you in fifty.

The Three Principles of DOE

Fisher established three principles that remain the foundation of every well-designed experiment:

Randomization does not eliminate lurking variables — it distributes their effects equally across all treatments, on average. This ensures that the statistical tests remain valid.

The OVAT Trap: A Quantitative Example

Suppose the true relationship between yield and two factors (temperature T and pressure P) is:

where T, P ∈ {-1, +1} (coded levels) and ε ~ N(0, 1).

The OVAT estimate is biased because it was measured at P=-1, where the interaction TP reduces temperature's apparent effect. The factorial design correctly estimates both effects and their interaction, using the same total data.

A Concrete Example: Web Server Tuning

Suppose you are optimizing a web server. Two factors matter: thread count (10 or 100) and cache size (64 MB or 512 MB). The true performance:

OVAT would have left 700 req/sec on the table — a 2x performance improvement, missed entirely because OVAT can't detect interactions.

Effect Sparsity: Why DOE Works

A fundamental observation underpins DOE's practicality: in most real systems, only a small fraction of factors truly matter. This is called the effect sparsity principle (sometimes the Pareto principle of effects). Out of 20 possible factors, typically 3–5 drive 80% or more of the variation.

This means you can afford a two-stage strategy: screen many factors cheaply to find the vital few, then investigate those few in detail. The total cost is far less than testing every factor exhaustively.

When NOT to Use DOE

DOE is not always the right tool. Consider alternatives when:

• You have only one factor. A simple A/B test or one-way comparison is sufficient.
• The response is not measurable. DOE requires a quantifiable outcome. Subjective preferences need special treatment (conjoint analysis, paired comparisons).
• The system is completely unknown. If you have no idea which factors might matter, start with domain-expert brainstorming and a few exploratory runs before committing to a formal design.
• Runs are effectively free. If you can test millions of combinations cheaply (e.g., a fast simulation), a random search or grid search may be simpler.

In most practical engineering situations, however, experiments have a non-trivial cost, and DOE will save you time and resources.

Chapter 2

The Linear Model

DOE assumes the response can be modeled as a linear combination of factor effects:

where β₀ is the grand mean, β_i are main effect coefficients, β_ij are interaction coefficients, and ε is random error.

This model is "linear" in the parameters (β), not necessarily in the factors. Even quadratic terms like X² are linear in the coefficient β_ii.

The Full Model for Three Factors

For 3 factors (A, B, C) with interactions and quadratic terms, the complete model is:

That's 8 terms (including the intercept) for 3 factors — exactly the number of runs in a 2³ factorial. This isn't a coincidence: each run provides one equation, and the design is constructed so that the 8 equations are independent (orthogonal). When we add quadratic terms β₁₁x₁², β₂₂x₂², β₃₃x₃², the model grows to 11 terms, requiring more than 8 runs — this is why response surface designs (CCD, Box-Behnken) need more runs than factorials.

Coded Variables

We convert natural factor values to coded values (-1, +1) using:

This standardizes all factors to the same scale, making effect magnitudes directly comparable. A factor with levels 150°C and 200°C has coded values -1 and +1, where 175°C is zero.

Worked Example: Coding Factor Values

Consider a temperature factor with levels 150°C (low) and 200°C (high):

The center (X̅) is (150 + 200) / 2 = 175, and the half-range is (200 - 150) / 2 = 25. Star points in CCD designs produce coded values beyond ±1, which is how they sample outside the original range.

Estimating Effects

Main Effects

The main effect of factor A is the difference between the average response at the high level and the average response at the low level:

Interaction Effects

The interaction effect of factors A and B measures whether the effect of A depends on the level of B:

If this is zero, A and B act independently. If large, their combined effect is synergistic (positive) or antagonistic (negative).

Percentage Contribution

To compare the relative importance of effects, we compute:

This is what the Pareto chart displays — a ranked bar chart of contributions with a cumulative line.

Confidence Intervals

A 95% confidence interval on a main effect is:

If the interval includes zero, the effect is not statistically significant at the 5% level. The DOE Helper Tool computes these automatically using the t-distribution.

What Affects the Width of Confidence Intervals?

The confidence interval width depends on three things:

Orthogonality and Balance

A design is orthogonal when the columns of the design matrix are uncorrelated. This means each effect estimate is independent of every other — changing one factor's range or removing one effect from the model doesn't change the other estimates.

A design is balanced when each factor level appears equally often. Balance ensures that effect estimates use all available data symmetrically.

Full factorial and Plackett-Burman designs are both orthogonal and balanced. This is what makes them statistically powerful — every run contributes useful information to every effect estimate.

Visualizing Orthogonality

In an orthogonal design, knowing the level of one factor tells you nothing about the level of any other factor. Here's the contrast:

Degrees of Freedom

Degrees of freedom (df) determine how much information your design provides. In a 2^k factorial with N = 2^k runs:

• 1 df for the overall mean
• k df for main effects
• C(k,2) df for two-factor interactions
• Remaining df available for error estimation (if you have replicates)

Without replicates, you must assume higher-order interactions are zero and "pool" their degrees of freedom to estimate error. This is why center-point replicates are valuable — they provide pure-error degrees of freedom without increasing the factorial portion of the design.

Worked Example: Degree of Freedom Budget

For a 2⁴ factorial (16 runs) with 2 replicates (32 total runs):

The 16 error degrees of freedom come from the replicate: each of the 16 factorial points has 2 observations, giving 16 × (2-1) = 16 df for pure error. Without replication (single replicate), you'd need to assume higher-order interactions are negligible and pool their df to estimate error.

The Normal Probability Plot

When you have no replicates (and therefore no independent error estimate), a normal probability plot of effects helps identify which effects are real. Negligible effects cluster along a straight line (they are just noise). Effects that deviate from the line are likely real.

Statistical Power and Sample Size

Power is the probability that your experiment will detect a real effect of a given size. Low power means you might run an experiment and conclude "nothing matters" when, in fact, something does — you just didn't have enough data to see it.

The key insight: power depends on the signal-to-noise ratio. If your factor ranges produce large effects relative to experimental noise, even a small design has high power. If the effects are subtle, you need more replicates. This is why factor range selection (Chapter 1) is so important — wider ranges amplify the signal.

Chapter 3

The 2^k factorial tests every combination of k factors at 2 levels each. It is the most complete design — and the foundation for all others.

The 2^k Factorial

For k factors, each at 2 levels, we need 2^k runs:

Growth is exponential. This is manageable for 2–5 factors, but becomes impractical beyond that — which motivates fractional and screening designs.

Understanding Exponential Growth

The run count grows faster than most people intuit. Here's a comparison of run count vs. information gained:

At k=5, only 15 of the 31 estimable effects are main effects or two-factor interactions — the rest are higher-order interactions that are usually negligible. This is the fundamental insight that motivates fractional designs: you're paying for information you probably don't need.

The Design Matrix

A 2³ full factorial design matrix:

Every column is balanced: equal numbers of -1 and +1. Every pair of columns is orthogonal. This ensures unbiased, independent estimates of all effects.

How to Read a Design Matrix

Each row is one experiment. Each column is one factor. The values -1 and +1 represent the low and high levels of that factor. To compute any interaction column, multiply the corresponding factor columns element-by-element:

Notice that every column (including interaction columns) is balanced (four +1s and four -1s) and orthogonal to every other column. This is the mathematical foundation that makes effect estimation unbiased. The DOE Helper Tool constructs and displays this matrix with doe info and doe generate --dry-run.

When to Use Full Factorials

Worked Example: Packaging Seal Strength

Three factors at 2 levels each: Seal Temperature (A), Pressure (B), Dwell Time (C). Response: seal strength in Newtons.

Main effect of A (Temperature):

Temperature is the dominant factor, increasing seal strength by ~12.85 N on average. The tool computes all effects, confidence intervals, and creates the Pareto chart automatically.

Complete Effect Analysis

Let's compute all effects from the seal strength data:

Interpretation: Temperature (A) dominates at 69.3% contribution. Pressure (B) contributes 19.7%, and Dwell Time (C) accounts for 11.1%. All interaction effects are near zero — the factors act independently. The Pareto chart would show a steep drop after A, confirming that temperature is the primary lever for improving seal strength.

Adding Center Points

A 2^k factorial only tests the extremes. What if the response curves between the extremes? Adding center points — runs at the midpoint of all factor ranges — lets you detect curvature without fitting a full quadratic model.

If the center-point average differs significantly from the average of the factorial points, curvature is present, and you should consider upgrading to a CCD or Box-Behnken design for the next phase.

Detecting Curvature with Center Points

Continuing the seal strength example: suppose we add 3 center-point replicates at Temperature=175°C, Pressure=4 bar, Dwell Time=7.5 sec. The results are: 28.1, 27.4, 28.8 N.

The center mean (28.10) is higher than the factorial mean (24.38), suggesting positive curvature — the response surface bows upward in the middle. The difference (3.72 N) is large enough to warrant investigating a quadratic model using CCD or Box-Behnken.

Blocking a Factorial Design

Sometimes you can't complete all runs under identical conditions. Perhaps the raw material comes in batches, or experiments span multiple days. Blocking partitions the runs into groups (blocks) so that known sources of variation are accounted for.

In a 2^k factorial, you assign the highest-order interaction to the block effect. For a 2³ design split into 2 blocks, assign ABC to blocks: runs where ABC = -1 go in Block 1, and runs where ABC = +1 go in Block 2. The ABC interaction becomes confounded with the block effect, but all main effects and two-factor interactions remain estimable.

Worked Example: Blocking a 2³ Design

We split the 2³ seal strength design into 2 blocks. The generator is ABC (the three-factor interaction). Runs where A×B×C = -1 go in Block 1 (day 1); runs where A×B×C = +1 go in Block 2 (day 2):

Each block is balanced: within each block, every factor has an equal number of -1 and +1 values. This means all main effects (A, B, C) and two-factor interactions (AB, AC, BC) are estimable. Only the ABC interaction is confounded with the block effect — which is usually an acceptable trade-off.

Replicated Factorials

Running the entire factorial more than once gives you a direct estimate of experimental error and increases the precision of all effect estimates. With n replicates of a 2^k design, the standard error of each effect is:

where s is the pooled standard deviation. Doubling the replicates reduces the standard error by a factor of √2 — you get about 41% more precision for twice the runs. This is diminishing returns, so most practitioners use 2–3 replicates rather than many.

Practical Guidance: Running a Full Factorial

Here's how a full factorial experiment typically proceeds with the DOE Helper Tool:

Chapter 4

When full factorials require too many runs, fractional factorials let you trade off complete interaction information for dramatic run-count reduction.

The Problem of Exponential Growth

Full factorial runs grow as 2^k. For 7 factors, that's 128 runs. For 10, it's 1,024. This is often impractical.

But here's the saving insight: in most real systems, high-order interactions (three-factor and above) are negligible. The effect sparsity principle tells us that only a few effects — typically main effects and some two-factor interactions — drive the majority of the variation. A fractional factorial sacrifices information about high-order interactions (which are probably zero anyway) to dramatically reduce the run count.

The 2^k-p Design

A fractional factorial runs 2^k-p experiments, where p is the number of "generators." A half-fraction (p=1) cuts runs by 50%. A quarter-fraction (p=2) cuts by 75%.

Resolution

Resolution describes the severity of the aliasing (confounding):

Resolution in Practice

Here's what each resolution level means concretely:

Generators and Defining Relations

A fractional factorial is constructed by writing k-p factors as a full factorial, then generating the remaining p factors as products of existing columns. For example, in a 2^5-1 design, we might set E = ABCD. This is the generator.

The defining relation is I = ABCDE (multiply both sides by E). It tells you every alias pair. Any effect multiplied by the defining relation gives its alias:

In a Resolution V design like this, main effects are aliased only with four-factor interactions (negligible), and two-factor interactions are aliased with three-factor interactions (usually small). This is why Resolution V designs are nearly as good as full factorials.

Multiple Generators and Word Length

When p > 1 (quarter-fraction or smaller), you have multiple generators, and the defining relation contains multiple "words." For a 2^7-2 design with generators F = ABCD and G = ABDE:

The third word (CEFG) is the "generalized interaction" of the first two generators. The minimum word length in the defining relation determines the resolution. Here, the shortest word has 4 letters, so this is a Resolution IV design.

The DOE Helper Tool automatically computes and displays the alias structure when you use the info command with a fractional factorial design. This tells you exactly which effects are confounded with each other, so you can interpret your results correctly.

Worked Example: 2^4-1 Half-Fraction

Four factors (A, B, C, D). A full factorial needs 16 runs. A half-fraction (p=1) needs only 8. We set D = ABC as the generator, giving defining relation I = ABCD (Resolution IV).

The alias structure: A+BCD, B+ACD, C+ABD, D+ABC, AB+CD, AC+BD, AD+BC. Main effects are clear of two-factor interactions (Resolution IV), but two-factor interactions are aliased in pairs (AB with CD, etc.).

Interpreting Aliased Effects

The alias pairs in our 2^4-1 Resolution IV design are:

Main effects are clean because they're aliased only with three-factor interactions (which are almost always negligible). But if the "AB" effect looks large, you can't tell whether it's really AB, or really CD, or some combination of both. This is where subject-matter expertise comes in — does an interaction between A and B make physical sense?

Foldover: Resolving Aliases

If a fractional factorial reveals ambiguous results (you can't tell if the effect comes from AB or CD), a foldover resolves the ambiguity. A foldover reverses the signs of one or more columns and runs the resulting design as a supplementary experiment.

Worked Example: Full Foldover of a 2^4-1

If our 2^4-1 analysis showed a large "AB+CD" effect and we can't tell which pair is responsible, we run a full foldover (reverse all signs):

Combining the two half-fractions:

The combined 16-run design is equivalent to a full 2⁴ factorial. All aliases are resolved. The cost: you've doubled your runs from 8 to 16, but you only do this when the initial analysis reveals ambiguity worth resolving. In many cases, the initial 8-run fraction is sufficient.

# Fold-over to de-alias effects
doe augment --config config.json --type fold_over

# Add star points for RSM
doe augment --config config.json --type star_points

# Add center points to detect curvature
doe augment --config config.json --type center_points

The augmented script only runs the new experiments — it skips already-completed runs automatically.

Choosing the Right Fraction

The DOE Helper Tool selects minimum-aberration generators automatically, but here's the decision logic:

Fractional Factorial Workflow

A typical fractional factorial study follows this pattern:

1. Start with the right resolution. For initial screening, Resolution III or IV. For characterization, Resolution IV+.
2. Run the fraction. Analyze main effects and aliased interaction estimates.
3. Decide next steps:
   • If main effects are clear and no interactions look significant → done, proceed to optimization.
   • If some aliased interaction pairs look significant → foldover to resolve them.
   • If many factors are insignificant → drop them and run a full factorial or RSM on the survivors.

Chapter 5

When you have many factors and limited budget, Plackett-Burman designs let you screen them all with remarkably few runs.

Plackett-Burman Designs

Plackett-Burman designs are 2-level screening designs that require only N+1 runs for N factors (rounded up to a multiple of 4). They estimate main effects only — all interactions are confounded.

How Plackett-Burman Works

PB designs are constructed from special cyclic generator sequences. For N=12 runs, the first row of the design is a specific sequence of + and - signs. Each subsequent row is formed by shifting the previous row one position to the right (wrapping around). The last row is all minus signs.

The resulting design is orthogonal for main effects but has complex confounding patterns for interactions. Unlike fractional factorials, where aliases follow a clean algebraic structure, PB confounding is "partially confounded" — each two-factor interaction is spread across multiple main effect estimates.

Example: 12-Run Plackett-Burman Matrix

A PB-12 design for up to 11 factors. Here we use 7 real factors (A–G) and 4 dummy columns (d1–d4):

The grey columns (d1–d4) are dummy factors. Their estimated effects measure only noise, providing a reference for judging which real factors are significant.

The Screening-to-Optimization Pipeline

The Three-Stage Pipeline in Detail

Total cost: ~25–35 runs to go from "I have 7+ factors and no idea which matter" to "I have a validated optimum setting." Compare this to a full 2⁷ factorial alone (128 runs) or grid search (thousands of runs).

Worked Example: HPC Cluster Screening

An HPC team suspects that 7 parameters affect job throughput: MPI ranks per node, OpenMP threads, memory allocation policy, I/O scheduler, network buffer size, file striping count, and checkpoint interval. Testing all 128 combinations of a full 2⁷ factorial is impractical — each run takes 20 minutes on a shared cluster.

A Plackett-Burman design requires only 8 runs (2 hours of wall-clock time). The Pareto chart from the analysis reveals that MPI ranks per node and network buffer size together account for 72% of the throughput variation. The other five factors are noise.

Interpreting the Screening Results

The analysis output from the HPC screening might look like this:

The first two factors (MPI ranks and network buffer) together account for 72% of the variation. The remaining five factors each contribute less than 10%. Using the dummy-factor approach, suppose the average dummy effect is 3.0. Only MPI ranks (45.2) and network buffer (30.8) clearly exceed the 2× threshold (6.0), confirming them as the active factors.

Dummy Factors and Error Estimation

If your Plackett-Burman design has more columns than real factors, the leftover columns become dummy factors — factors that don't correspond to any real variable. Their estimated effects should be zero (they're just noise), so they provide an estimate of experimental error.

Practical Rules for Dummy Factor Analysis

If you have no dummy columns (e.g., a PB-8 with exactly 7 real factors), use the normal probability plot instead. The tool generates Pareto charts that visually separate the large effects from the noise floor — the steep initial bars are the active factors, and the remaining short bars are noise.

Definitive Screening Designs

Definitive Screening Designs (DSDs), introduced by Jones and Nachtsheim (2011), are a modern alternative to Plackett-Burman for continuous factors. They require only 2k+1 runs (for k factors), but can estimate:

• All main effects
• All quadratic effects
• Some two-factor interactions (when factors are dropped from the model)

DSDs use three levels per factor (-1, 0, +1), so they can detect curvature that PB designs miss. The trade-off is slightly more runs and the requirement that factors be continuous (not categorical).

Comparing Screening Designs

Screening Workflow with DOE Helper

Chapter 6

When screening has identified the key factors, response surface designs add star points and center replicates to fit quadratic models and find the true optimum.

Central Composite Design (CCD)

A CCD consists of three types of experimental points, each serving a distinct purpose:

1. Factorial points (2^k): the corners of the design cube. These estimate linear effects and interactions — the same information a full factorial provides.
2. Star (axial) points (2k): points along each axis at distance α from center. These provide the additional information needed to estimate quadratic (curvature) terms.
3. Center replicates (c): repeated runs at the center, providing a direct estimate of experimental error and enabling the lack-of-fit test.

Here's what a CCD looks like for 2 factors:

The factorial points (purple) form a square at the corners. The star points (amber) extend outward along each axis. The center points (green) are stacked at the middle. Together, these 13 runs provide enough information to fit all 6 terms of a two-factor quadratic model (intercept, two linear, one interaction, two quadratic).

The Alpha Parameter in CCD

The star-point distance α controls where the axial points sit relative to the factorial cube. This single parameter determines the shape and statistical properties of the entire design:

Box-Behnken Design

Box-Behnken is an alternative RSM design that never tests the corner conditions. Every run has at most two factors at their extreme levels — the remaining factors stay at center. This makes Box-Behnken ideal when the corners represent dangerous, expensive, or physically impossible combinations.

In a 3-factor Box-Behnken, the 12 design points sit at the midpoints of the cube's edges, plus 3 center replicates. No run has all three factors at their extremes simultaneously. Compare this to a CCD, which tests every corner:

Latin Hypercube Sampling

Latin Hypercube Sampling (LHS) is fundamentally different from factorial and RSM designs. Instead of testing specific combinations at discrete levels, LHS generates a space-filling set of points across the full continuous factor range.

For N samples and k factors, LHS divides each factor's range into N equal intervals and places exactly one sample in each interval for every factor:

When to Use LHS vs. Factorial Designs

Quick Reference: Choosing a Design

This is the most important table in the book. Use it to select your design based on what you need to learn:

Worked Example: Chemical Process Optimization

A chemical engineer has identified two critical factors from a screening study: reaction temperature (150–200°C) and catalyst concentration (1–5%). They choose a rotatable CCD (α = 1.414) with 5 center replicates, giving 13 runs total.

Step 1: Check for Curvature

The average of the 4 factorial points is (62+74+78+85)/4 = 74.75%. The average of the 5 center points is (88+87+89+86+88)/5 = 87.6%. The center-point average is 12.85% higher than the factorial average — strong evidence of quadratic curvature. The optimum lies inside the design space, not at the corners.

Step 2: Interpret the Fitted Model

The full quadratic model fitted to all 13 runs yields the prediction equation (see Chapter 8 for complete RSM analysis). The fitted model finds the predicted optimum at approximately 180°C and 3.5% catalyst, with a predicted yield of 89.2%.

Step 3: What the Star Points Reveal

The star points at extreme temperature (runs 5–6) give yields of 66% and 79% — both well below the center points. Similarly, extreme catalyst (runs 7–8) give 58% and 82%. This confirms that both factors have strong downward curvature at the extremes — there really is a peak in the middle.

Sequential Experimentation

RSM designs are most powerful when used sequentially, in stages that build on each other. This avoids the common mistake of running an expensive, comprehensive experiment in a region far from the optimum.

The Three-Phase RSM Strategy

1. Phase 1 — Screen: Run a Plackett-Burman or fractional factorial to identify the 2–4 important factors from the original 8–15 candidates. (Chapter 5)
2. Phase 2 — First-order model + steepest ascent: Run a 2^k factorial with center points on the important factors. If the surface is approximately linear (center ≈ factorial average), use steepest ascent to move toward the optimum. Stop when the response levels off or starts declining.
3. Phase 3 — Second-order model: Center a CCD or Box-Behnken around the best point from Phase 2. Fit the full quadratic model. Locate the precise optimum. Run confirmation experiments.

Chapter 7

Once you've run your experiments, the analysis phase turns raw data into actionable knowledge. The DOE Helper Tool automates this entire process.

The Pareto Chart

A Pareto chart ranks all effects by magnitude and shows their cumulative contribution. It is the single most important chart in DOE analysis — one glance tells you which factors matter and which don't.

Consider a 2³ factorial on web server tuning with factors Thread Count (A), Cache Size (B), and Connection Timeout (C). The analysis produces these effect magnitudes:

The Pareto chart renders this as descending bars with a cumulative line:

Main Effects Plots

A main effects plot shows the average response at each level of each factor. It answers the question: "When I change this factor from low to high, what happens to the response on average?"

Using the same web server example, suppose the mean throughput at each factor level is:

Interaction Plots

Interaction plots are the most important diagnostic in DOE. They show whether the effect of one factor depends on the level of another factor. This is information that OVAT experiments can never provide.

Continuing our server example, here are the mean throughput values for all four AB combinations:

For comparison, here's what different interaction patterns look like and what they mean:

Statistical Significance

The tool computes 95% confidence intervals for each effect. A confidence interval tells you the range within which the true effect likely falls, accounting for experimental noise. If the interval does not include zero, the effect is statistically significant — meaning it's unlikely to be just noise.

Here's how the confidence intervals look for our server tuning example (with 2 replicates, SE = 2.1):

The confidence interval for C (Timeout) spans from -1.6 to 7.2 — it includes zero, so we cannot confidently say Timeout has any real effect. The positive estimate of 2.8 could easily be noise. By contrast, Cache Size's interval [14.0, 22.8] is entirely above zero and far from it, so we're confident it's a real and large effect.

Residual Analysis

After fitting a model, always check the residuals (observed minus predicted values). The model assumes residuals are random, independent, and normally distributed with constant variance. If these assumptions fail, your effect estimates and confidence intervals may be unreliable.

There are four key residual plots to check:

1. Residuals vs. Predicted Values

Plot residuals on the Y-axis against predicted (fitted) values on the X-axis. You want a random scatter with no pattern:

2. Normal Probability Plot of Residuals

Plot residuals against expected normal quantiles. If the model is adequate and the data is well-behaved, points fall along a straight diagonal line. Deviations indicate non-normality:

Multi-Response Analysis

Real experiments often have multiple responses. A database tuning study might measure throughput, latency, and CPU usage. Different responses may have conflicting optima — the settings that maximize throughput might also maximize CPU usage.

Consider a server optimization with two responses:

The best throughput (1180) comes with 88% CPU usage — dangerously close to saturation. You need to find the sweet spot.

Effect Size and Power

Before running an experiment, consider: how large an effect do you need to detect? The power of a design is the probability of detecting a real effect of a given size. Higher power requires more replicates.

Power depends on four things:

As a rough guide: a 2^k design with 2 replicates can detect effects about 2–3 times the noise standard deviation with 80% power. If your effects are likely smaller than that, you need more replicates or a larger design.

The DOE Helper Tool computes power for each factor using the non-central F distribution:

# Check power before running the experiment
doe power --config config.json --sigma 2.0 --delta 5.0

# Or estimate sigma from existing results
doe power --config config.json --delta 5.0 --results-dir results/

If power is below 0.80 for important factors, increase block_count in your config to add replicates, or widen your factor ranges to increase the expected effect size.

Confirmation Runs

After identifying the best settings, always run confirmation experiments at those settings. This is the final validation step before deploying to production. Compare the confirmation results to the model's prediction:

If the confirmation results fall within the model's prediction interval, the model is validated and you can proceed with confidence. If they don't:

• Results consistently higher or lower: The model has a bias. Check for a lurking variable that changed between the experiment and confirmation.
• Results much more variable: The experimental conditions weren't fully reproduced. Document what differed.
• Results completely off: The model is inadequate. Consider additional terms, a wider design, or re-examining your assumptions.

Managing Real-World Experiments

Not all experiments are automated scripts. Physical experiments — lab tests, field measurements, manufacturing trials — require a different workflow. The DOE tool provides three commands specifically for this:

A typical real-world experiment proceeds in stages, often over days or weeks:

1. Before the first run: Generate the design, then export a worksheet. Print it or open it in a spreadsheet.
2. During the experiment: After each run, enter results with record. Use status to see what's left.
3. Mid-experiment analysis: Use analyze --partial to get early insights. This is especially valuable for expensive experiments — if one factor clearly dominates after half the runs, you can make informed decisions about how to proceed.
4. After completion: Run the full analysis, optimization, and report pipeline.

Putting It All Together: The Analysis Workflow

Here is the recommended sequence for analyzing any DOE result:

1. Check the Pareto chart. Which factors are in the vital few? Are there any surprises (unexpected interactions)?
2. Examine main effects plots. What direction does each important factor push the response? How large are the effects in practical units?
3. Study interaction plots. Do the important factors interact? If lines cross, the main effect averages are misleading — optimize the combination, not the individual factors.
4. Verify with confidence intervals. Are the important effects statistically significant? Are the unimportant ones truly negligible?
5. Check residual plots. Is the model adequate? Are the assumptions met? If not, transform the data or upgrade the design.
6. Run confirmation experiments. Does reality match the model's prediction at the recommended settings?

Chapter 8

RSM moves beyond "which factors matter" to "what is the optimal setting?" by fitting a mathematical model to the experimental data.

From Effects to Models

In Chapters 3–7 you learned to estimate effects — the change in response when a factor moves from low to high. Effects tell you the direction of improvement. But they don't tell you what happens at intermediate settings, or exactly where the optimum lies. That's where RSM comes in.

RSM fits a mathematical model to the experimental data, turning your discrete measurements into a continuous prediction equation. For two factors the full quadratic model is:

Each β term captures a specific type of behaviour:

First-Order vs. Second-Order Models

Not every RSM study needs quadratic terms. The two model orders serve different purposes:

Worked Example: Reactor Yield Optimization

A chemical process team has screened 7 factors down to 2: temperature (x₁, 150–200°C) and catalyst concentration (x₂, 1–5%). They run a CCD with 5 center replicates (13 runs total). After fitting the full quadratic model, the optimize command reports:

Let's walk through how to interpret every part of this output.

Reading the Coefficients

Both quadratic coefficients are negative, which means the response surface is dome-shaped — there's a true maximum inside the experimental region. This is the ideal outcome for RSM.

Model Adequacy

Before trusting any prediction, you must verify that the model actually fits the data. The tool reports several diagnostics automatically:

• R² and Adjusted R² — Overall model fit.
• PRESS statistic and Predicted R² — Leave-one-out cross-validation measure that penalizes influential observations. A large gap between R² and Predicted R² indicates overfitting.
• ANOVA table — F-tests and p-values for each term, lack-of-fit test, and Lenth's PSE for unreplicated designs.
• Diagnostic plots — Residuals vs. fitted values (check for patterns), normal probability plot of residuals (check for normality), residuals vs. run order (check for drift), and predicted vs. actual (check for bias). These are generated automatically by doe analyze.

R² and Adjusted R²

In our reactor example, R² = 0.964 means the model explains 96.4% of the variation in yield. Adjusted R² = 0.938 penalizes for the number of terms — still high, confirming that the terms in the model are justified, not just overfitting noise.

ANOVA Table

The analysis of variance breaks the model into individual terms and tests each one:

Finding the Optimum

The optimize command reports three things: the best observed run (the highest actual measurement), the predicted optimum at observed points from the RSM model, and the true surface optimum found by numerically optimizing the fitted polynomial using L-BFGS-B with multiple random restarts.

The predicted optimum is 3.1% better than the best observed run, and it occurs at a setting nobody tested directly. This is the power of RSM — interpolation between experimental points to find a better answer than any individual run.

Method of Steepest Ascent

If your first-order model suggests the optimum is far outside your experimental region (the contour lines are nearly parallel, pointing off the edge of the plot), use the method of steepest ascent before committing to a full CCD.

1. Fit a first-order (linear) model to your factorial data.
2. Compute the gradient direction: the path of steepest increase in the predicted response.
3. Run a few experiments along that path, stepping outward from the center, until the response stops improving.
4. Center a new CCD around the best point found on the path.

The DOE Helper Tool automates steps 1–2 with the --steepest flag:

doe optimize --config config.json --steepest

This generates a table of recommended follow-up experiment points along the gradient direction, with predicted response values at each step.

Contour Plots and Surface Plots

The fitted RSM model can be visualized as a contour plot (2D topographic map) or a surface plot (3D wireframe). Contour plots are more practical for decision-making because you can read exact coordinates from them.

Here is the contour plot for our reactor example, showing predicted yield as a function of temperature and catalyst concentration:

Here's what different contour patterns mean and how to act on them:

Canonical Analysis

Canonical analysis transforms the fitted model into a simpler form that reveals the nature of the response surface at its stationary point. The quadratic RSM model can be written in matrix form as:

where ŷ_s is the predicted response at the stationary point, λ_i are the eigenvalues of the quadratic coefficient matrix B, and w_i are the rotated coordinates. The original matrix form is:

The eigenvalues tell you everything about the shape of the surface at the stationary point:

For our reactor example:

Both eigenvalues are negative, confirming a true maximum. The magnitude ratio |λ₁/λ₂| = 1.63 tells us the surface is about 63% more curved in the temperature direction than in the catalyst direction — meaning yield is more sensitive to temperature deviations from the optimum. This has practical implications: you should control temperature more tightly than catalyst concentration in production.

Ridge Analysis

In roughly 30–40% of real RSM studies, the stationary point falls outside the experimental region or is a saddle point. In either case, the stationary point is not directly useful. Ridge analysis solves this by finding the best predicted response at each distance from the center of the design.

Think of it as drawing concentric circles (or spheres) outward from the center and asking: "What's the highest yield I can achieve on each circle?"

A ridge analysis trace for a saddle-point example might look like:

The Method of Steepest Ascent

When you're far from the optimum, steepest ascent is the most efficient way to move toward it. The idea is simple: use the first-order model's coefficients to determine the direction of fastest improvement, then run a few experiments along that path.

From a first-order model with coefficients β₁ = 6.2 (temperature) and β₂ = 8.1 (catalyst), the steepest ascent path moves in the direction proportional to these coefficients. In natural units, you step along:

Yield peaked at step 3 and declined at step 4 — you've passed the optimum. Center a new CCD around (199°C, 4.8%) to map the peak precisely.

Lack-of-Fit Testing

If you have replicated center points (and you should — always include 3–5), you can formally test whether the quadratic model fits the data. The lack-of-fit test compares two sources of variation:

• Pure error: Variation among replicates at the same settings (estimated from center points). This is "real" noise — the inherent variability of the process.
• Lack of fit: Variation between model predictions and observed means at each unique design point. This is "systematic" error — patterns the model fails to capture.

The F-test asks: "Is the model's prediction error significantly larger than the pure experimental noise?"

Multi-Response Optimization with Desirability

Most real processes have multiple responses to optimize simultaneously. A reactor produces yield, purity, and cost. A server has throughput, latency, and CPU usage. The settings that maximize one response often worsen another.

The desirability function approach converts each response to a 0–1 scale (0 = unacceptable, 1 = ideal), then combines them into a single score:

where d_i is the individual desirability for response i, and w_i is its importance weight.

Desirability Functions by Goal

Continuing the reactor example with three responses:

Using Multi-Objective Optimization in the DOE Helper Tool

The --multi flag on doe optimize implements Derringer-Suich desirability automatically. It fits RSM models for each response, evaluates desirability across the factor space, and reports the best compromise.

To control the trade-off, add weight and bounds to responses in your config:

• weight (default 1.0) — relative importance. A weight of 3 means "this response matters 3× more" in the geometric mean.
• bounds (optional) — [worst, best] values for desirability scaling. For "maximize," the first value is the worst acceptable (d=0) and the second is the ideal (d=1). For "minimize," the first is ideal and the second is the worst. If omitted, bounds are auto-computed from observed data.

The output includes:

• Overall desirability D — the weighted geometric mean across all responses
• Per-response desirability — how well each response meets its target
• Recommended settings — either from observed runs or RSM-predicted optima
• Trade-off summary — how much each response sacrifices compared to its individual best
• Top 3 runs — the best observed factor combinations ranked by D

The Complete RSM Workflow

Bringing together everything in this chapter, here's the recommended sequence:

1. Start with a first-order model. Fit a factorial or PB design. Check center points for curvature. If center-point mean ≈ factorial mean, the surface is flat — use steepest ascent to move toward the optimum.
2. Run steepest ascent. Take 3–6 steps along the gradient. Stop when the response starts declining or leveling off.
3. Center a CCD around the best point. Run a CCD (or Box-Behnken if corners are infeasible). Fit the full quadratic model.
4. Check model adequacy. R² > 0.9? Lack of fit not significant? Residuals look random? If yes, proceed. If no, diagnose and fix.
5. Interpret the contour plot. Concentric ellipses with the optimum inside? Great. Saddle or open contours? Use ridge analysis or expand the region.
6. Run canonical analysis. Confirm the stationary point type. Check eigenvalue magnitudes for sensitivity information.
7. For multiple responses, compute desirability. Set targets and weights in your config, then run doe optimize --config config.json --multi to find the best compromise.
8. Confirm. Run 3–5 experiments at the predicted optimum. Compare to the prediction interval.

Chapter 9

Common issues and how to resolve them.

Common Mistakes Checklist

Before running your experiment, verify you haven't made these common errors:

The DOE Workflow in Practice

Here is the complete workflow, from initial idea to validated optimum:

1. Define the objective. What are you optimizing? What constitutes success? Agree on quantifiable responses.
2. List candidate factors. Brainstorm with domain experts. Include factors you're not sure about — screening will sort them out.
3. Set factor levels. Choose bold but feasible ranges. The levels should be far enough apart that real effects are detectable.
4. Choose the design. Many factors? Screen first (PB). Few factors? Go straight to factorial or CCD.
5. Randomize and run. Follow the generated run order. For automated experiments, use the generated runner script. For manual experiments, use export-worksheet to print a run sheet, status to track progress, and record to enter results. Record everything, including conditions you didn't plan to vary.
6. Analyze. Use --partial to get early insights while experiments are still running. Compute effects, check residuals, identify the vital few factors.
7. Follow up. If screening, run a factorial or RSM on the important factors. If optimizing, run confirmation experiments.
8. Confirm and implement. Validate the optimum with independent runs. Then deploy.

User Guide

This section is the complete software reference for the DOE Helper Tool. It covers installation, configuration, every CLI command, and step-by-step tutorials to put the theory from Chapters 1–9 into practice.

Installation & Setup

The DOE Helper Tool is a Python package that works on any system with Python 3.10 or later.

pip install doehelper

Verify the installation:

doe info

This prints the installed version, available design types, and default configuration values. For full installation details — including virtual-environment setup, shell completion, and upgrading — see the Installation & Setup reference.

Configuration Reference

Every experiment starts with a JSON configuration file that describes your factors, responses, and design choices. Here is a minimal example:

{
  "factors": [
    {"name": "temperature", "levels": ["60", "80"], "type": "continuous"},
    {"name": "pressure", "levels": ["1.0", "2.0"], "type": "continuous"}
  ],
  "responses": [
    {"name": "yield", "optimize": "maximize"}
  ],
  "settings": {
    "operation": "full_factorial"
  }
}

The tool supports 11 design types: full_factorial, fractional_factorial, plackett_burman, latin_hypercube, central_composite, box_behnken, definitive_screening, taguchi, d_optimal, mixture_simplex_lattice, and mixture_simplex_centroid.

The configuration supports many more fields — blocking, center points, custom factor levels, optimization targets, response weights for multi-objective optimization, and more. The complete field-by-field reference is in the Configuration Reference.

CLI Commands

The tool provides a set of CLI commands that map to each phase of the DOE workflow described in Chapter 9:

Every command accepts --help for a full list of options. For detailed usage, examples, and flags for each command, see the CLI Command Reference.

Tutorials

The best way to get comfortable with the tool is to work through a hands-on example. The Tutorials section walks you through complete experiments from configuration to final report, tying together the theory from this book with the software commands above.

For a broader collection of real-world scenarios — from web-server tuning to machine-learning hyperparameter search — explore the Use Cases library.