Summary
This experiment investigates zero trust policy evaluation. Fractional factorial of 5 zero trust parameters for auth latency and security score.
The design varies 5 factors: policy cache ttl (sec), ranging from 10 to 300, context attributes (count), ranging from 3 to 12, risk score weight (weight), ranging from 0.1 to 0.9, session timeout (sec), ranging from 300 to 3600, and mfa frequency (hours), ranging from 1 to 24. The goal is to optimize 2 responses: auth latency ms (ms) (minimize) and security score (score) (maximize). Fixed conditions held constant across all runs include identity provider = okta, policy engine = opa.
A fractional factorial design reduces the number of runs from 32 to 8 by deliberately confounding higher-order interactions. This is ideal for screening — identifying which of the 5 factors matter most before investing in a full study.
Key Findings
For auth latency ms, the most influential factors were mfa frequency (44.5%), session timeout (18.8%), policy cache ttl (13.2%). The best observed value was 3.8 (at policy cache ttl = 300, context attributes = 3, risk score weight = 0.1).
For security score, the most influential factors were session timeout (28.4%), context attributes (26.4%), mfa frequency (17.6%). The best observed value was 99.4 (at policy cache ttl = 300, context attributes = 12, risk score weight = 0.1).
Recommended Next Steps
- Follow up with a response surface design (CCD or Box-Behnken) on the top 3–4 factors to model curvature and find the true optimum.
- Consider whether any fixed factors should be varied in a future study.
- The screening results can guide factor reduction — drop factors contributing less than 5% and re-run with a smaller, more focused design.
Experimental Setup
Factors
| Factor | Low | High | Unit |
|---|---|---|---|
policy_cache_ttl | 10 | 300 | sec |
context_attributes | 3 | 12 | count |
risk_score_weight | 0.1 | 0.9 | weight |
session_timeout | 300 | 3600 | sec |
mfa_frequency | 1 | 24 | hours |
Fixed: identity_provider = okta, policy_engine = opa
Responses
| Response | Direction | Unit |
|---|---|---|
auth_latency_ms | ↓ minimize | ms |
security_score | ↑ maximize | score |
Configuration
use_cases/61_zero_trust_policy_eval/config.json
{
"metadata": {
"name": "Zero Trust Policy Evaluation",
"description": "Fractional factorial of 5 zero trust parameters for auth latency and security score"
},
"factors": [
{
"name": "policy_cache_ttl",
"levels": [
"10",
"300"
],
"type": "continuous",
"unit": "sec"
},
{
"name": "context_attributes",
"levels": [
"3",
"12"
],
"type": "continuous",
"unit": "count"
},
{
"name": "risk_score_weight",
"levels": [
"0.1",
"0.9"
],
"type": "continuous",
"unit": "weight"
},
{
"name": "session_timeout",
"levels": [
"300",
"3600"
],
"type": "continuous",
"unit": "sec"
},
{
"name": "mfa_frequency",
"levels": [
"1",
"24"
],
"type": "continuous",
"unit": "hours"
}
],
"fixed_factors": {
"identity_provider": "okta",
"policy_engine": "opa"
},
"responses": [
{
"name": "auth_latency_ms",
"optimize": "minimize",
"unit": "ms"
},
{
"name": "security_score",
"optimize": "maximize",
"unit": "score"
}
],
"settings": {
"operation": "fractional_factorial",
"test_script": "use_cases/61_zero_trust_policy_eval/sim.sh"
}
}
Experimental Matrix
The Fractional Factorial Design produces 8 runs. Each row is one experiment with specific factor settings.
| Run | policy_cache_ttl | context_attributes | risk_score_weight | session_timeout | mfa_frequency |
|---|---|---|---|---|---|
| 1 | 10 | 12 | 0.9 | 300 | 1 |
| 2 | 300 | 3 | 0.1 | 300 | 1 |
| 3 | 300 | 12 | 0.1 | 3600 | 1 |
| 4 | 300 | 12 | 0.9 | 3600 | 24 |
| 5 | 10 | 12 | 0.1 | 300 | 24 |
| 6 | 300 | 3 | 0.9 | 300 | 24 |
| 7 | 10 | 3 | 0.1 | 3600 | 24 |
| 8 | 10 | 3 | 0.9 | 3600 | 1 |
Step-by-Step Workflow
1
Preview the design
Terminal
$ doe info --config use_cases/61_zero_trust_policy_eval/config.json
2
Generate the runner script
Terminal
$ doe generate --config use_cases/61_zero_trust_policy_eval/config.json \
--output use_cases/61_zero_trust_policy_eval/results/run.sh --seed 42
3
Execute the experiments
Terminal
$ bash use_cases/61_zero_trust_policy_eval/results/run.sh
4
Analyze results
Terminal
$ doe analyze --config use_cases/61_zero_trust_policy_eval/config.json
5
Get optimization recommendations
Terminal
$ doe optimize --config use_cases/61_zero_trust_policy_eval/config.json
6
Multi-objective optimization
With 2 competing responses, use --multi to find the best compromise via Derringer–Suich desirability.
Terminal
$ doe optimize --config use_cases/61_zero_trust_policy_eval/config.json --multi
7
Generate the HTML report
Terminal
$ doe report --config use_cases/61_zero_trust_policy_eval/config.json \
--output use_cases/61_zero_trust_policy_eval/results/report.html
Features Exercised
| Feature | Value |
|---|---|
| Design type | fractional_factorial |
| Factor types | continuous (all 5) |
| Arg style | double-dash |
| Responses | 2 (auth_latency_ms ↓, security_score ↑) |
| Total runs | 8 |
Analysis Results
Generated from actual experiment runs using the DOE Helper Tool.
Response: auth_latency_ms
Top factors: mfa_frequency (44.5%), session_timeout (18.8%), policy_cache_ttl (13.2%).
ANOVA
| Source | DF | SS | MS | F | p-value |
|---|---|---|---|---|---|
| Source | DF | SS | MS | F | p-value |
| policy_cache_ttl | 1 | 98.7013 | 98.7013 | 0.667 | 0.4513 |
| context_attributes | 1 | 93.1612 | 93.1612 | 0.629 | 0.4636 |
| risk_score_weight | 1 | 65.5513 | 65.5513 | 0.443 | 0.5353 |
| session_timeout | 1 | 201.0013 | 201.0013 | 1.358 | 0.2965 |
| mfa_frequency | 1 | 1125.7513 | 1125.7513 | 7.604 | 0.0400 |
| policy_cache_ttl*context_attributes | 1 | 201.0013 | 201.0013 | 1.358 | 0.2965 |
| policy_cache_ttl*risk_score_weight | 1 | 1125.7513 | 1125.7513 | 7.604 | 0.0400 |
| policy_cache_ttl*session_timeout | 1 | 93.1612 | 93.1612 | 0.629 | 0.4636 |
| policy_cache_ttl*mfa_frequency | 1 | 65.5512 | 65.5512 | 0.443 | 0.5353 |
| context_attributes*risk_score_weight | 1 | 67.8613 | 67.8613 | 0.458 | 0.5284 |
| context_attributes*session_timeout | 1 | 98.7013 | 98.7013 | 0.667 | 0.4513 |
| context_attributes*mfa_frequency | 1 | 114.7613 | 114.7613 | 0.775 | 0.4189 |
| risk_score_weight*session_timeout | 1 | 114.7612 | 114.7612 | 0.775 | 0.4189 |
| risk_score_weight*mfa_frequency | 1 | 98.7012 | 98.7012 | 0.667 | 0.4513 |
| session_timeout*mfa_frequency | 1 | 67.8612 | 67.8612 | 0.458 | 0.5284 |
| Error | (Lenth | PSE) | 5 | 740.2594 | 148.0519 |
| Total | 7 | 1766.7887 | 252.3984 |
Pareto Chart
Main Effects Plot
Normal Probability Plot of Effects
Half-Normal Plot of Effects
Model Diagnostics
Response: security_score
Top factors: session_timeout (28.4%), context_attributes (26.4%), mfa_frequency (17.6%).
ANOVA
| Source | DF | SS | MS | F | p-value |
|---|---|---|---|---|---|
| Source | DF | SS | MS | F | p-value |
| policy_cache_ttl | 1 | 54.6013 | 54.6013 | 0.565 | 0.4861 |
| context_attributes | 1 | 145.3513 | 145.3513 | 1.504 | 0.2746 |
| risk_score_weight | 1 | 27.0112 | 27.0112 | 0.280 | 0.6196 |
| session_timeout | 1 | 168.3612 | 168.3612 | 1.743 | 0.2440 |
| mfa_frequency | 1 | 64.4112 | 64.4112 | 0.667 | 0.4513 |
| policy_cache_ttl*context_attributes | 1 | 168.3613 | 168.3613 | 1.743 | 0.2440 |
| policy_cache_ttl*risk_score_weight | 1 | 64.4113 | 64.4113 | 0.667 | 0.4513 |
| policy_cache_ttl*session_timeout | 1 | 145.3513 | 145.3513 | 1.504 | 0.2746 |
| policy_cache_ttl*mfa_frequency | 1 | 27.0113 | 27.0113 | 0.280 | 0.6196 |
| context_attributes*risk_score_weight | 1 | 576.3012 | 576.3012 | 5.965 | 0.0585 |
| context_attributes*session_timeout | 1 | 54.6013 | 54.6013 | 0.565 | 0.4861 |
| context_attributes*mfa_frequency | 1 | 526.5013 | 526.5013 | 5.449 | 0.0668 |
| risk_score_weight*session_timeout | 1 | 526.5013 | 526.5013 | 5.449 | 0.0668 |
| risk_score_weight*mfa_frequency | 1 | 54.6013 | 54.6013 | 0.565 | 0.4861 |
| session_timeout*mfa_frequency | 1 | 576.3013 | 576.3013 | 5.965 | 0.0585 |
| Error | (Lenth | PSE) | 5 | 483.0844 | 96.6169 |
| Total | 7 | 1562.5388 | 223.2198 |
Pareto Chart
Main Effects Plot
Normal Probability Plot of Effects
Half-Normal Plot of Effects
Model Diagnostics
Response Surface Plots
3D surfaces fitted with quadratic RSM. Red dots are observed data points.
auth latency ms context attributes vs mfa frequency
auth latency ms context attributes vs risk score weight
auth latency ms context attributes vs session timeout
auth latency ms policy cache ttl vs context attributes
auth latency ms policy cache ttl vs mfa frequency
auth latency ms policy cache ttl vs risk score weight
auth latency ms policy cache ttl vs session timeout
auth latency ms risk score weight vs mfa frequency
auth latency ms risk score weight vs session timeout
auth latency ms session timeout vs mfa frequency
security score context attributes vs mfa frequency
security score context attributes vs risk score weight
security score context attributes vs session timeout
security score policy cache ttl vs context attributes
security score policy cache ttl vs mfa frequency
security score policy cache ttl vs risk score weight
security score policy cache ttl vs session timeout
security score risk score weight vs mfa frequency
security score risk score weight vs session timeout
security score session timeout vs mfa frequency
Multi-Objective Optimization
When responses compete, Derringer–Suich desirability finds the best compromise. Each response is scaled to a 0–1 desirability, then combined via a weighted geometric mean.
Overall Desirability
D = 0.6991
Per-Response Desirability
| Response | Weight | Desirability | Predicted | Dir |
|---|---|---|---|---|
auth_latency_ms |
1.0 |
0.6582
|
18.90 0.6582 18.90 ms | ↓ |
security_score |
2.0 |
0.7206
|
86.30 0.7206 86.30 score | ↑ |
Recommended Settings
| Factor | Value |
|---|---|
policy_cache_ttl | 288.3 sec |
context_attributes | 11.28 count |
risk_score_weight | 0.1105 weight |
session_timeout | 2136 sec |
mfa_frequency | 23.15 hours |
Source: from RSM model prediction
Trade-off Summary
Sacrifice = how much worse than single-objective best.
| Response | Predicted | Best Observed | Sacrifice |
|---|---|---|---|
security_score | 86.30 | 99.40 | +13.10 |
Top 3 Runs by Desirability
| Run | D | Factor Settings |
|---|---|---|
| #4 | 0.4587 | policy_cache_ttl=300, context_attributes=3, risk_score_weight=0.9, session_timeout=300, mfa_frequency=24 |
| #1 | 0.4535 | policy_cache_ttl=300, context_attributes=12, risk_score_weight=0.9, session_timeout=3600, mfa_frequency=24 |
Model Quality
| Response | R² | Type |
|---|---|---|
security_score | 0.8790 | linear |
Full Multi-Objective Output
doe optimize --multi
============================================================
MULTI-OBJECTIVE OPTIMIZATION
Method: Derringer-Suich Desirability Function
============================================================
Overall desirability: D = 0.6991
Response Weight Desirability Predicted Direction
---------------------------------------------------------------------
auth_latency_ms 1.0 0.6582 18.90 ms ↓
security_score 2.0 0.7206 86.30 score ↑
Recommended settings:
policy_cache_ttl = 288.3 sec
context_attributes = 11.28 count
risk_score_weight = 0.1105 weight
session_timeout = 2136 sec
mfa_frequency = 23.15 hours
(from RSM model prediction)
Trade-off summary:
auth_latency_ms: 18.90 (best observed: 3.80, sacrifice: +15.10)
security_score: 86.30 (best observed: 99.40, sacrifice: +13.10)
Model quality:
auth_latency_ms: R² = 0.8640 (linear)
security_score: R² = 0.8790 (linear)
Top 3 observed runs by overall desirability:
1. Run #8 (D=0.5226): policy_cache_ttl=10, context_attributes=12, risk_score_weight=0.1, session_timeout=300, mfa_frequency=24
2. Run #4 (D=0.4587): policy_cache_ttl=300, context_attributes=3, risk_score_weight=0.9, session_timeout=300, mfa_frequency=24
3. Run #1 (D=0.4535): policy_cache_ttl=300, context_attributes=12, risk_score_weight=0.9, session_timeout=3600, mfa_frequency=24
Full Analysis Output
doe analyze
=== Main Effects: auth_latency_ms ===
Factor Effect Std Error % Contribution
--------------------------------------------------------------
mfa_frequency -23.7250 5.6169 44.5%
session_timeout -10.0250 5.6169 18.8%
policy_cache_ttl 7.0250 5.6169 13.2%
context_attributes -6.8250 5.6169 12.8%
risk_score_weight 5.7250 5.6169 10.7%
=== ANOVA Table: auth_latency_ms ===
Source DF SS MS F p-value
-----------------------------------------------------------------------------
policy_cache_ttl 1 98.7013 98.7013 0.667 0.4513
context_attributes 1 93.1612 93.1612 0.629 0.4636
risk_score_weight 1 65.5513 65.5513 0.443 0.5353
session_timeout 1 201.0013 201.0013 1.358 0.2965
mfa_frequency 1 1125.7513 1125.7513 7.604 0.0400
policy_cache_ttl*context_attributes 1 201.0013 201.0013 1.358 0.2965
policy_cache_ttl*risk_score_weight 1 1125.7513 1125.7513 7.604 0.0400
policy_cache_ttl*session_timeout 1 93.1612 93.1612 0.629 0.4636
policy_cache_ttl*mfa_frequency 1 65.5512 65.5512 0.443 0.5353
context_attributes*risk_score_weight 1 67.8613 67.8613 0.458 0.5284
context_attributes*session_timeout 1 98.7013 98.7013 0.667 0.4513
context_attributes*mfa_frequency 1 114.7613 114.7613 0.775 0.4189
risk_score_weight*session_timeout 1 114.7612 114.7612 0.775 0.4189
risk_score_weight*mfa_frequency 1 98.7012 98.7012 0.667 0.4513
session_timeout*mfa_frequency 1 67.8612 67.8612 0.458 0.5284
Error (Lenth PSE) 5 740.2594 148.0519
Total 7 1766.7887 252.3984
Note: Error estimated using Lenth's pseudo-standard-error (unreplicated design)
=== Interaction Effects: auth_latency_ms ===
Factor A Factor B Interaction % Contribution
------------------------------------------------------------------------
policy_cache_ttl risk_score_weight -23.7250 27.2%
policy_cache_ttl context_attributes 10.0250 11.5%
context_attributes mfa_frequency 7.5750 8.7%
risk_score_weight session_timeout -7.5750 8.7%
context_attributes session_timeout -7.0250 8.1%
risk_score_weight mfa_frequency 7.0250 8.1%
policy_cache_ttl session_timeout 6.8250 7.8%
context_attributes risk_score_weight 5.8250 6.7%
session_timeout mfa_frequency -5.8250 6.7%
policy_cache_ttl mfa_frequency 5.7250 6.6%
=== Summary Statistics: auth_latency_ms ===
policy_cache_ttl:
Level N Mean Std Min Max
------------------------------------------------------------
10 4 26.0750 19.6154 3.8000 50.1000
300 4 33.1000 13.0869 15.8000 47.2000
context_attributes:
Level N Mean Std Min Max
------------------------------------------------------------
12 4 33.0000 18.1540 15.8000 50.1000
3 4 26.1750 15.1099 3.8000 37.0000
risk_score_weight:
Level N Mean Std Min Max
------------------------------------------------------------
0.1 4 26.7250 19.2491 3.8000 47.2000
0.9 4 32.4500 14.0196 15.8000 50.1000
session_timeout:
Level N Mean Std Min Max
------------------------------------------------------------
300 4 34.6000 12.8756 18.9000 50.1000
3600 4 24.5750 18.8719 3.8000 47.2000
mfa_frequency:
Level N Mean Std Min Max
------------------------------------------------------------
1 4 41.4500 8.6927 31.5000 50.1000
24 4 17.7250 11.7523 3.8000 32.4000
=== Main Effects: security_score ===
Factor Effect Std Error % Contribution
--------------------------------------------------------------
session_timeout 9.1750 5.2823 28.4%
context_attributes -8.5250 5.2823 26.4%
mfa_frequency -5.6750 5.2823 17.6%
policy_cache_ttl 5.2250 5.2823 16.2%
risk_score_weight 3.6750 5.2823 11.4%
=== ANOVA Table: security_score ===
Source DF SS MS F p-value
-----------------------------------------------------------------------------
policy_cache_ttl 1 54.6013 54.6013 0.565 0.4861
context_attributes 1 145.3513 145.3513 1.504 0.2746
risk_score_weight 1 27.0112 27.0112 0.280 0.6196
session_timeout 1 168.3612 168.3612 1.743 0.2440
mfa_frequency 1 64.4112 64.4112 0.667 0.4513
policy_cache_ttl*context_attributes 1 168.3613 168.3613 1.743 0.2440
policy_cache_ttl*risk_score_weight 1 64.4113 64.4113 0.667 0.4513
policy_cache_ttl*session_timeout 1 145.3513 145.3513 1.504 0.2746
policy_cache_ttl*mfa_frequency 1 27.0113 27.0113 0.280 0.6196
context_attributes*risk_score_weight 1 576.3012 576.3012 5.965 0.0585
context_attributes*session_timeout 1 54.6013 54.6013 0.565 0.4861
context_attributes*mfa_frequency 1 526.5013 526.5013 5.449 0.0668
risk_score_weight*session_timeout 1 526.5013 526.5013 5.449 0.0668
risk_score_weight*mfa_frequency 1 54.6013 54.6013 0.565 0.4861
session_timeout*mfa_frequency 1 576.3013 576.3013 5.965 0.0585
Error (Lenth PSE) 5 483.0844 96.6169
Total 7 1562.5388 223.2198
Note: Error estimated using Lenth's pseudo-standard-error (unreplicated design)
=== Interaction Effects: security_score ===
Factor A Factor B Interaction % Contribution
------------------------------------------------------------------------
context_attributes risk_score_weight 16.9750 16.3%
session_timeout mfa_frequency -16.9750 16.3%
context_attributes mfa_frequency 16.2250 15.6%
risk_score_weight session_timeout -16.2250 15.6%
policy_cache_ttl context_attributes -9.1750 8.8%
policy_cache_ttl session_timeout 8.5250 8.2%
policy_cache_ttl risk_score_weight -5.6750 5.5%
context_attributes session_timeout -5.2250 5.0%
risk_score_weight mfa_frequency 5.2250 5.0%
policy_cache_ttl mfa_frequency 3.6750 3.5%
=== Summary Statistics: security_score ===
policy_cache_ttl:
Level N Mean Std Min Max
------------------------------------------------------------
10 4 67.7250 5.4286 63.0000 73.1000
300 4 72.9500 21.7526 48.5000 99.4000
context_attributes:
Level N Mean Std Min Max
------------------------------------------------------------
12 4 74.6000 16.9692 63.1000 99.4000
3 4 66.0750 13.5810 48.5000 79.7000
risk_score_weight:
Level N Mean Std Min Max
------------------------------------------------------------
0.1 4 68.5000 21.7119 48.5000 99.4000
0.9 4 72.1750 6.3589 64.2000 79.7000
session_timeout:
Level N Mean Std Min Max
------------------------------------------------------------
300 4 65.7500 13.3490 48.5000 79.7000
3600 4 74.9250 16.9272 63.0000 99.4000
mfa_frequency:
Level N Mean Std Min Max
------------------------------------------------------------
1 4 73.1750 20.8069 48.5000 99.4000
24 4 67.5000 8.1515 63.0000 79.7000
Optimization Recommendations
doe optimize
=== Optimization: auth_latency_ms ===
Direction: minimize
Best observed run: #2
policy_cache_ttl = 300
context_attributes = 3
risk_score_weight = 0.1
session_timeout = 300
mfa_frequency = 1
Value: 3.8
RSM Model (linear, R² = 0.7426, Adj R² = 0.0991):
Coefficients:
intercept +29.5875
policy_cache_ttl +3.5625
context_attributes +6.7875
risk_score_weight +3.6375
session_timeout +8.7125
mfa_frequency +4.0125
Predicted optimum (from linear model, at observed points):
policy_cache_ttl = 300
context_attributes = 12
risk_score_weight = 0.9
session_timeout = 3600
mfa_frequency = 24
Predicted value: 56.3000
Surface optimum (via L-BFGS-B, linear model):
policy_cache_ttl = 10
context_attributes = 3
risk_score_weight = 0.1
session_timeout = 300
mfa_frequency = 1
Predicted value: 2.8750
Model quality: Good fit — general trends are captured, some noise remains.
Factor importance:
1. session_timeout (effect: 17.4, contribution: 32.6%)
2. context_attributes (effect: -13.6, contribution: 25.4%)
3. mfa_frequency (effect: 8.0, contribution: 15.0%)
4. risk_score_weight (effect: 7.3, contribution: 13.6%)
5. policy_cache_ttl (effect: 7.1, contribution: 13.3%)
=== Optimization: security_score ===
Direction: maximize
Best observed run: #1
policy_cache_ttl = 300
context_attributes = 12
risk_score_weight = 0.1
session_timeout = 3600
mfa_frequency = 1
Value: 99.4
RSM Model (linear, R² = 0.7713, Adj R² = 0.1994):
Coefficients:
intercept +70.3375
policy_cache_ttl +6.4625
context_attributes +8.4125
risk_score_weight +1.5625
session_timeout +0.3375
mfa_frequency -5.9625
Predicted optimum (from linear model, at observed points):
policy_cache_ttl = 300
context_attributes = 12
risk_score_weight = 0.1
session_timeout = 3600
mfa_frequency = 1
Predicted value: 89.9500
Surface optimum (via L-BFGS-B, linear model):
policy_cache_ttl = 300
context_attributes = 12
risk_score_weight = 0.9
session_timeout = 3600
mfa_frequency = 1
Predicted value: 93.0750
Model quality: Good fit — general trends are captured, some noise remains.
Factor importance:
1. context_attributes (effect: -16.8, contribution: 37.0%)
2. policy_cache_ttl (effect: 12.9, contribution: 28.4%)
3. mfa_frequency (effect: -11.9, contribution: 26.2%)
4. risk_score_weight (effect: 3.1, contribution: 6.9%)
5. session_timeout (effect: 0.7, contribution: 1.5%)