A comprehensive, hands-on guide to designing, running, and analyzing experiments with the DOE Helper Tool.
DOE Helper is a command-line tool that brings the power of Design of Experiments (DOE) to software engineers, data scientists, researchers, and anyone who needs to systematically optimize a process. Instead of changing one variable at a time (OVAT) or running random guesses, DOE lets you test multiple factors simultaneously in a structured way that reveals not just which factors matter, but how they interact.
Tuning database configurations, optimizing compiler flags, finding the best ML hyperparameters, improving chemical processes, calibrating manufacturing settings — any scenario where you have multiple knobs to turn and want to find the best combination efficiently.
For detailed system requirements and dependency information, see the Quick Start page.
Let's walk through a complete experiment from scratch. We'll optimize a web server's performance by testing three configuration parameters.
Edit the generated config.json to define your factors and responses:
In six commands, you went from a question ("which settings matter?") to a statistically rigorous answer with optimization recommendations and a shareable report. The rest of this guide dives deep into each step.
Every DOE experiment follows the same six-phase lifecycle. The DOE Helper Tool provides commands for each phase.
| Phase | Command(s) | What Happens |
|---|---|---|
| Plan | doe init, doe info, doe power |
Define factors, responses, and choose a design. Verify the plan makes sense and has sufficient power. |
| Design | doe generate |
Create the design matrix and generate a runner script. Use --dry-run to preview first. |
| Execute | bash run_*.sh, doe record, doe status |
Run the generated script or manually record results. Track progress with doe status. |
| Analyze | doe analyze |
Compute ANOVA, effects, generate plots, and identify significant factors. |
| Optimize | doe optimize |
Find the best factor settings using response surface methodology. |
| Report | doe report |
Generate a self-contained HTML report with all results, plots, and recommendations. |
DOE is often iterative. After your first analysis, you may want to augment the design with center or star points, run additional blocks, or switch to a more focused design. The tool supports this workflow with doe augment and doe next-batch.
The configuration file (config.json) is the single source of truth for your experiment. Every command reads from it. Understanding its structure is essential.
A config file has five top-level sections:
Human-readable name and description. Used in reports and status output. Optional but highly recommended.
The variables you are testing. Each factor has a name, levels (at least 2), and an optional type and unit. This is the heart of your experiment design.
What you are measuring. Each response has a name, optimization direction (maximize or minimize), and optional unit, weight, and bounds for multi-objective optimization.
How factors are passed to your test script and how results are collected. Controls argument style and result file format.
Design type, test script path, output directories, blocking, and design-specific options like LHS sample count.
Choosing the right factor type affects which designs are available and how results are analyzed.
| Type | When to Use | Examples | Design Implications |
|---|---|---|---|
continuous |
Numeric values on a scale | Temperature (150–200), Pressure (2–6), Thread count (1–16) | Required for CCD, Box-Behnken, and star points. Enables response surface modeling and interpolation. |
categorical |
Discrete, unordered choices | Algorithm (A, B, C), Cache type (LRU, LFU), Material (steel, aluminum) | Works with all designs. Cannot be interpolated. No star/center points for this factor. |
ordinal |
Ordered categories | Priority (low, medium, high), Compression level (none, fast, best) | Treated as categorical in most designs but ordering is preserved in analysis output. |
For maximize responses, bounds are [worst, best] where worst < best. For minimize responses, bounds are [worst, best] where worst > best. If omitted, bounds are auto-computed from observed data.
Fixed factors are held constant across all runs. They are passed to your test script alongside the varying factors but do not contribute to the design matrix.
Use fixed factors when a parameter must be specified for your test script to run but you are not currently varying it. They appear in reports for reproducibility.
The arg_style setting controls how factor values are passed to your test script:
| Style | Invocation Example | Best For |
|---|---|---|
double-dash |
./test.sh --temperature 200 --pressure 6 --out run_1.json |
Most scripts. Clear, self-documenting, order-independent. |
env |
TEMPERATURE=200 PRESSURE=6 OUT=run_1.json ./test.sh |
Docker containers, Makefiles, scripts that read environment variables. |
positional |
./test.sh 200 6 run_1.json |
Simple scripts. Order must match factor definition order. |
The result_file setting controls the expected output format:
json (default) — Your script writes a JSON file with keys matching response names:
Use doe init to generate a starter config file rather than writing one from scratch:
The right design depends on your goals, the number of factors, and how many runs you can afford. This section helps you make that choice.
Screening (many factors, limited budget): Use Plackett-Burman or Definitive Screening to find the important factors. Optimization (few factors, need the best settings): Use Central Composite or Box-Behnken to model the response surface.
| Design | Factors | Runs (k factors) | Detects Interactions | Detects Curvature | Best For |
|---|---|---|---|---|---|
| Full Factorial | 2–5 | 2k (e.g., 8, 16, 32) | All | No (2-level) | Complete understanding of a small system |
| Fractional Factorial | 4–8 | 2k-1 or 2k-2 | Some (aliased) | No | Moderate screening with fewer runs |
| Plackett-Burman | 4–23 | k+1 (next multiple of 4) | No | No | Fast screening of many factors |
| Definitive Screening | 3–12 | 2k+1 | Some | Yes | Modern screening: fewer runs, detects curvature |
| Central Composite (CCD) | 2–6 | 2k + 2k + nc | All (2FI) | Yes (quadratic) | Full response surface modeling |
| Box-Behnken | 3–7 | Varies (e.g., 15 for 3) | All (2FI) | Yes (quadratic) | RSM without extreme corners |
| Taguchi | 2–15 | L4, L8, L12, L16, L27 | Limited | No | Robust design, signal-to-noise ratios |
| Latin Hypercube (LHS) | Any | User-defined | Via analysis | Via analysis | Space-filling, computer experiments |
| D-Optimal | Any | User-defined | Depends on model | Depends on model | Custom run count, irregular constraints |
| Mixture (Simplex) | 3+ | Varies | Via model | Via model | Components that sum to 1 (formulations) |
DOE typically follows a two-phase approach:
| Phase | Goal | Typical Designs | Result |
|---|---|---|---|
| Phase 1: Screening | Identify the vital few factors from a larger set | Plackett-Burman, Definitive Screening, Fractional Factorial | Narrow from 8–15 factors down to 3–5 important ones |
| Phase 2: Optimization | Find the best settings for the important factors | CCD, Box-Behnken, Full Factorial | Response surface model with optimal settings |
Use this table to estimate how many runs your experiment will require:
| Factors | Full Factorial (2-level) | Fractional (Res IV) | Plackett-Burman | Definitive Screening | CCD | Box-Behnken |
|---|---|---|---|---|---|---|
| 3 | 8 | — | 4 | 7 | 20 | 15 |
| 4 | 16 | 8 | 8 | 9 | 30 | 27 |
| 5 | 32 | 16 | 8 | 11 | 32 | 46 |
| 6 | 64 | 16 | 8 | 13 | 52 | — |
| 7 | 128 | 16 | 8 | 15 | 78 | 62 |
| 8 | 256 | 16 | 12 | 17 | — | — |
| 10 | 1024 | 32 | 12 | 21 | — | — |
| 15 | 32768 | 64 | 16 | 31 | — | — |
When you run doe info --config config.json, the tool reports three efficiency metrics:
| Metric | What It Measures | Good Value | Practical Meaning |
|---|---|---|---|
| D-efficiency | Overall information content of the design | > 90% | Higher means more precise estimates of all effects. An efficiency of 100% means the design is optimal for the given model. |
| A-efficiency | Average precision of effect estimates | > 80% | Focuses on average variance rather than overall determinant. Good complement to D-efficiency. |
| G-efficiency | Worst-case prediction variance | > 50% | Ensures no point in the design space has wildly imprecise predictions. Important for RSM. |
For most experiments, the built-in designs are already highly efficient. Use doe info to check, and only worry about efficiency if you're using D-Optimal or heavily constrained designs.
After doe generate, you have a runner script that executes all runs sequentially. The tool supports both Bash and Python formats.
The generated script creates the results directory, runs each combination in randomized order, and saves results as run_N.json files.
The Python script uses subprocess to call your test script. Useful when you need cross-platform compatibility or want to extend the runner logic.
Your test script is the bridge between the DOE tool and your real experiment. It must follow a simple contract:
arg_style (double-dash, env, or positional)--out <path> specifying where to write the result filedoe recordFor physical experiments (lab work, field tests, manufacturing trials), you don't need a test script. Use doe record to manually enter results:
Simply delete the offending run_N.json file from the results directory and re-run that specific experiment. The runner script or doe record will regenerate it. Use doe status to see which runs are missing.
If you cannot complete all runs, use the --partial flag on doe analyze and doe optimize to work with whatever data you have.
For lab and field work, print a worksheet showing all runs with blank columns for recording results by hand:
Analysis is where DOE pays off. A single command produces a complete statistical analysis of your experiment.
doe analyze ProducesThe ANOVA table is the core output. Here's how to read each column:
| Column | Full Name | What It Tells You |
|---|---|---|
| Source | Source of Variation | The factor or interaction being tested |
| SS | Sum of Squares | Total variation explained by this source. Larger = more influence. |
| DF | Degrees of Freedom | Number of independent comparisons (levels - 1 for main effects) |
| MS | Mean Square | SS / DF. The average variation per degree of freedom. |
| F | F-statistic | MS(factor) / MS(error). Large F = the factor has a real effect. |
| p-value | Probability | Probability that the observed effect is due to chance. p < 0.05 = statistically significant. |
When you have no replicates (block_count = 1) and a saturated design, there is no independent error term. The tool uses Lenth's method (pseudo-standard error) to estimate significance, the same approach used by R's FrF2 package.
Main effect: The average change in response when a factor moves from its low level to its high level. For example, a main effect of +15 for temperature means increasing temperature raises the response by 15 units on average.
Interaction effect: When the effect of one factor depends on the level of another. If temperature has an effect of +20 when pressure is high but only +5 when pressure is low, there is a temperature-pressure interaction.
The Pareto chart ranks all effects (main effects and interactions) from largest to smallest. A cumulative contribution line shows how much of the total variation is explained. Typically, 2–3 factors account for 80% or more of the variation (the Pareto principle).
Focus on the factors above the significance line (dashed red line). These are the "vital few" that truly drive your response. Everything below the line is noise.
The diagnostic panel contains four plots that help you validate the analysis:
| Plot | What to Look For | Problem Sign |
|---|---|---|
| Residuals vs. Fitted | Random scatter around zero | Funnel shape (non-constant variance) or curved pattern (model misfit) |
| Normal Probability (Q-Q) | Points following the diagonal line | Systematic departures indicate non-normal errors |
| Residuals vs. Run Order | No trends over time | Drift or trending suggests an uncontrolled variable changing during the experiment |
| Predicted vs. Actual | Points close to the 45-degree line | Systematic deviation means the model is missing important terms |
Once you understand which factors matter, optimization finds the best settings.
The optimizer does three things:
Response Surface Methodology (RSM) fits a mathematical model to your data, then uses that model to predict the response at any point in the factor space — even points you didn't test. This is what allows the optimizer to find settings between your tested levels.
The tool automatically fits both. A linear model captures main effects and interactions. A quadratic model also captures curvature (the sweet spot). If the quadratic model fits significantly better (higher R-squared), the optimum likely lies in the interior of the factor space rather than at a corner.
When you have conflicting goals (e.g., maximize throughput AND minimize latency), use --multi to find the best compromise:
The desirability function converts each response into a 0–1 scale (0 = worst, 1 = best), then combines them using a weighted geometric mean. The weights in your config control the relative importance of each response.
For sequential experimentation (Phase 1 RSM), use --steepest to generate a table of follow-up experiments along the gradient direction:
Use it when your initial screening design suggests the optimum is outside the tested region. Follow the path of improvement until the response starts declining, then run a new RSM design (CCD or Box-Behnken) centered at the new best point.
Augmentation lets you add runs to an existing design without re-running what you've already completed. This is essential for sequential experimentation.
If your fractional factorial design has aliased (confounded) effects and you need to separate them, a fold-over mirrors all factor levels (swaps high and low). This doubles your run count but breaks the aliasing.
When to use: After a fractional factorial shows significant effects, but you can't tell which aliased term is really driving the response.
Adding axial (star) points to a 2-level factorial design converts it into a Central Composite Design, enabling quadratic response surface modeling.
When to use: When your factorial analysis reveals significant curvature and you need to model the response surface to find the true optimum.
Adds 3 replicate runs at the center of the design space (midpoint of all continuous factors). These runs serve two purposes: detecting curvature and estimating pure error.
When to use: After a 2-level design when you suspect the relationship isn't purely linear. If center points show significantly different results from the factorial average, curvature is present.
Before running an expensive experiment, check whether your design has enough runs to detect the effects you care about:
If power is low, you have several options: (1) increase block_count to add replicates, (2) switch to a design with more runs, (3) accept that you can only detect larger effects. Don't run an underpowered experiment — you'll spend time and resources without being able to draw conclusions.
Use doe next-batch for iterative experimentation where each batch of runs informs the next:
This command analyzes your current results and suggests the most informative next set of experiments to run. Six strategies are available via --strategy (or the adaptive config block): refine focuses near the best region, explore maximizes space coverage, balanced mixes both, model_guided targets where the fitted surface is most uncertain, bayesian uses Gaussian-process expected improvement, and multi_objective optimizes desirability across several responses. Use --state-name to branch alternative follow-up trajectories from the same starting data.
Beyond the core workflow, the CLI includes tools for bootstrapping, simulation, and cross-experiment analysis — see the Technical Reference for every flag:
doe suggest — recommends a design, run count, and adaptive strategy from your factor count, budget, and goal, before you write any config.doe scaffold-config / doe scaffold-test — write annotated starter config.json and test.py/test.sh files so you never begin from a blank page.doe simulate --func module:fn — drive the design straight from a Python function, with no runner script or subprocesses.doe sensitivity — Sobol first-order and total-order indices on the fitted response surface, with optional HTML visualizations.doe compare — before/after session comparison with paired t-test, Cohen's d, and per-factor effect deltas.doe trend — regression across many sessions to detect drift over repeated experiments.doe calibrate — fit a parametric Python simulator's free parameters to observed experimental data.doe archive — bundle a session into a .tar.gz with a SHA-256 manifest for sharing.doe serve — browse sessions and their HTML reports from a localhost server.The report is fully self-contained HTML with all plots embedded as base64 images. No external dependencies. Share via email, Slack, wiki, or any file-sharing tool.
Contents include: design summary, ANOVA tables, effects tables, Pareto charts, main effects plots, normal/half-normal probability plots, diagnostic panels, 3D response surfaces, optimization recommendations, and the full design matrix.
Use the DOE Helper with AI tools like ChatGPT or Claude for:
See the AI Prompts page for ready-to-use prompts.
| Factors | Approach | Rationale |
|---|---|---|
| 2–5 | Full factorial or CCD | Affordable run count. Full interaction analysis. |
| 6–12 | Screen first (PB or DSD), then optimize top 3–5 | Too many runs for full factorial. Screen to find the vital few. |
| 13+ | Plackett-Burman screening, then sequential refinement | Screening is mandatory. Only test the survivors in detail. |
Replication (via block_count) gives you an independent estimate of experimental error, makes significance tests more powerful, and helps you detect if conditions changed between batches. For critical experiments, always use at least 2 blocks.
The mistake: Testing factors one at a time while holding others constant.
Why it's bad: OVAT requires more runs than DOE and completely misses interaction effects. Two factors might be fine individually but terrible (or great) together.
The fix: Use any DOE design. Even a simple full factorial with 3 factors and 2 levels (8 runs) gives you complete information about all main effects AND all interactions.
The mistake: Running experiments in a systematic order (all low values first, then all high values).
Why it's bad: If conditions drift over time (temperature, load, degradation), your results will be confounded with the time trend.
The fix: The DOE Helper randomizes run order automatically. Use --seed for reproducible randomization.
The mistake: Using 5 levels for every factor "to be thorough."
Why it's bad: A 5-factor experiment with 5 levels each = 3125 runs. With 2 levels it's just 32 runs (or 16 with fractional factorial).
The fix: Start with 2 levels per factor. If curvature exists, add center points or switch to CCD. You rarely need more than 3 levels for continuous factors.
The mistake: Declaring a factor important because p < 0.05, even though its actual effect is tiny.
Why it's bad: Statistical significance is not the same as practical significance. A factor that changes throughput by 0.1% may be "significant" with enough data but not worth optimizing.
The fix: Always look at the effect size (the actual magnitude of change), not just the p-value. Use the Pareto chart to see the relative contribution of each factor.
| Situation | Method | Why |
|---|---|---|
| Multiple factors, want interactions | DOE | Designed for this exact problem |
| Single continuous parameter to tune | Grid search or bisection | DOE is overkill for 1 parameter |
| Huge search space, cheap evaluations | Bayesian optimization | Better for 20+ dimensional spaces with fast evaluations |
| Need to understand causal relationships | DOE | Only controlled experimentation proves causation |
| Observational data only | Regression / ML | Can't run experiments, must work with what you have |
doe generate --dry-run and doe info before committing.doe power to verify your design can detect the effects you care about.doe report to generate a permanent record of your experiment.| Error | Cause | Solution |
|---|---|---|
No result files found |
The results directory is empty or doesn't exist | Check that out_directory in config matches where results are saved. Run doe status to verify. |
Factor X requires exactly 2 levels |
Fractional factorial, PB, or CCD requires 2-level factors | Ensure all factors have exactly 2 levels, or switch to a design that supports more. |
Box-Behnken requires 3+ factors |
You have fewer than 3 factors | Use CCD or full factorial for 2-factor experiments instead. |
Response 'X' not found in run_N.json |
The result file is missing a key that matches your response name | Verify that your test script writes JSON with keys exactly matching the name fields in your responses config. |
Duplicate factor names |
Two factors have the same name |
Every factor name must be unique. Check for typos. |
Invalid JSON in config file |
Syntax error in config.json | Check for trailing commas, missing quotes, or mismatched brackets. Use a JSON validator. |
If some runs failed or weren't completed:
The tool validates your config against the selected design type. Common validation issues:
The most common issue is response names in the result file not matching the config. Names are case-sensitive. If your config says "throughput", your result file must use "throughput" — not "Throughput" or "THROUGHPUT".
Verify a result file is valid:
For additional help, see the Quick Start for command reference, Theory for statistical background, or AI Prompts to get AI-assisted explanations of your results.