Summary
This experiment investigates tls handshake optimization. Box-Behnken design for TLS version, cipher choice, and session cache for handshake speed.
The design varies 3 factors: session cache size (entries), ranging from 1000 to 100000, session timeout s (s), ranging from 60 to 86400, and ocsp stapling workers (threads), ranging from 1 to 8. The goal is to optimize 2 responses: handshake ms (ms) (minimize) and resumption rate (%) (maximize). Fixed conditions held constant across all runs include tls version = 1.3, cipher = AES-256-GCM.
A Box-Behnken design was chosen because it efficiently fits quadratic models with 3 continuous factors while avoiding extreme corner combinations — requiring only 15 runs instead of the 8 needed for a full factorial at two levels.
Quadratic response surface models were fitted to capture potential curvature and factor interactions. The RSM contour plots below visualize how pairs of factors jointly affect each response.
Key Findings
For handshake ms, the most influential factors were session timeout s (44.8%), session cache size (36.9%), ocsp stapling workers (18.2%). The best observed value was 32.0 (at session cache size = 1000, session timeout s = 43230, ocsp stapling workers = 1).
For resumption rate, the most influential factors were session timeout s (37.0%), ocsp stapling workers (32.9%), session cache size (30.2%). The best observed value was 85.9 (at session cache size = 1000, session timeout s = 86400, ocsp stapling workers = 4.5).
Recommended Next Steps
- Run confirmation experiments at the predicted optimal settings to validate the model.
- Consider whether any fixed factors should be varied in a future study.
Experimental Setup
Factors
| Factor | Low | High | Unit |
|---|---|---|---|
session_cache_size | 1000 | 100000 | entries |
session_timeout_s | 60 | 86400 | s |
ocsp_stapling_workers | 1 | 8 | threads |
Fixed: tls_version = 1.3, cipher = AES-256-GCM
Responses
| Response | Direction | Unit |
|---|---|---|
handshake_ms | ↓ minimize | ms |
resumption_rate | ↑ maximize | % |
Configuration
use_cases/48_tls_handshake_optimization/config.json
{
"metadata": {
"name": "TLS Handshake Optimization",
"description": "Box-Behnken design for TLS version, cipher choice, and session cache for handshake speed"
},
"factors": [
{
"name": "session_cache_size",
"levels": [
"1000",
"100000"
],
"type": "continuous",
"unit": "entries"
},
{
"name": "session_timeout_s",
"levels": [
"60",
"86400"
],
"type": "continuous",
"unit": "s"
},
{
"name": "ocsp_stapling_workers",
"levels": [
"1",
"8"
],
"type": "continuous",
"unit": "threads"
}
],
"fixed_factors": {
"tls_version": "1.3",
"cipher": "AES-256-GCM"
},
"responses": [
{
"name": "handshake_ms",
"optimize": "minimize",
"unit": "ms"
},
{
"name": "resumption_rate",
"optimize": "maximize",
"unit": "%"
}
],
"settings": {
"operation": "box_behnken",
"test_script": "use_cases/48_tls_handshake_optimization/sim.sh"
}
}
Experimental Matrix
The Box-Behnken Design produces 15 runs. Each row is one experiment with specific factor settings.
| Run | session_cache_size | session_timeout_s | ocsp_stapling_workers |
|---|---|---|---|
| 1 | 50500 | 60 | 1 |
| 2 | 50500 | 43230 | 4.5 |
| 3 | 100000 | 43230 | 8 |
| 4 | 100000 | 43230 | 1 |
| 5 | 50500 | 43230 | 4.5 |
| 6 | 50500 | 43230 | 4.5 |
| 7 | 1000 | 43230 | 8 |
| 8 | 100000 | 60 | 4.5 |
| 9 | 50500 | 60 | 8 |
| 10 | 100000 | 86400 | 4.5 |
| 11 | 1000 | 43230 | 1 |
| 12 | 50500 | 86400 | 8 |
| 13 | 1000 | 60 | 4.5 |
| 14 | 1000 | 86400 | 4.5 |
| 15 | 50500 | 86400 | 1 |
Step-by-Step Workflow
1
Preview the design
Terminal
$ doe info --config use_cases/48_tls_handshake_optimization/config.json
2
Generate the runner script
Terminal
$ doe generate --config use_cases/48_tls_handshake_optimization/config.json \
--output use_cases/48_tls_handshake_optimization/results/run.sh --seed 42
3
Execute the experiments
Terminal
$ bash use_cases/48_tls_handshake_optimization/results/run.sh
4
Analyze results
Terminal
$ doe analyze --config use_cases/48_tls_handshake_optimization/config.json
5
Get optimization recommendations
Terminal
$ doe optimize --config use_cases/48_tls_handshake_optimization/config.json
6
Multi-objective optimization
With 2 competing responses, use --multi to find the best compromise via Derringer–Suich desirability.
Terminal
$ doe optimize --config use_cases/48_tls_handshake_optimization/config.json --multi
7
Generate the HTML report
Terminal
$ doe report --config use_cases/48_tls_handshake_optimization/config.json \
--output use_cases/48_tls_handshake_optimization/results/report.html
Features Exercised
| Feature | Value |
|---|---|
| Design type | box_behnken |
| Factor types | continuous (all 3) |
| Arg style | double-dash |
| Responses | 2 (handshake_ms ↓, resumption_rate ↑) |
| Total runs | 15 |
Analysis Results
Generated from actual experiment runs using the DOE Helper Tool.
Response: handshake_ms
Top factors: session_timeout_s (44.8%), session_cache_size (36.9%), ocsp_stapling_workers (18.2%).
ANOVA
| Source | DF | SS | MS | F | p-value |
|---|---|---|---|---|---|
| Source | DF | SS | MS | F | p-value |
| session_cache_size | 2 | 341.8533 | 170.9266 | 0.459 | 0.6475 |
| session_timeout_s | 2 | 663.3533 | 331.6766 | 0.891 | 0.4474 |
| ocsp_stapling_workers | 2 | 106.1772 | 53.0886 | 0.143 | 0.8693 |
| Lack | of | Fit | 6 | 221.8790 | 36.9798 |
| Pure | Error | 2 | 744.6067 | ||
| Error | 8 | 966.4856 | 372.3033 | ||
| Total | 14 | 2077.8693 | 148.4192 |
Pareto Chart
Main Effects Plot
Normal Probability Plot of Effects
Half-Normal Plot of Effects
Model Diagnostics
Response: resumption_rate
Top factors: session_timeout_s (37.0%), ocsp_stapling_workers (32.9%), session_cache_size (30.2%).
ANOVA
| Source | DF | SS | MS | F | p-value |
|---|---|---|---|---|---|
| Source | DF | SS | MS | F | p-value |
| session_cache_size | 2 | 713.4387 | 356.7193 | 0.406 | 0.6793 |
| session_timeout_s | 2 | 1144.5098 | 572.2549 | 0.651 | 0.5470 |
| ocsp_stapling_workers | 2 | 951.4158 | 475.7079 | 0.541 | 0.6018 |
| Lack | of | Fit | 6 | 905.6624 | 150.9437 |
| Pure | Error | 2 | 1757.3267 | ||
| Error | 8 | 2662.9890 | 878.6633 | ||
| Total | 14 | 5472.3533 | 390.8824 |
Pareto Chart
Main Effects Plot
Normal Probability Plot of Effects
Half-Normal Plot of Effects
Model Diagnostics
Response Surface Plots
3D surfaces fitted with quadratic RSM. Red dots are observed data points.
handshake ms session cache size vs ocsp stapling workers
handshake ms session cache size vs session timeout s
handshake ms session timeout s vs ocsp stapling workers
resumption rate session cache size vs ocsp stapling workers
resumption rate session cache size vs session timeout s
resumption rate session timeout s vs ocsp stapling workers
Multi-Objective Optimization
When responses compete, Derringer–Suich desirability finds the best compromise. Each response is scaled to a 0–1 desirability, then combined via a weighted geometric mean.
Overall Desirability
D = 0.9150
Per-Response Desirability
| Response | Weight | Desirability | Predicted | Dir |
|---|---|---|---|---|
handshake_ms |
1.0 |
0.8586
|
36.60 0.8586 36.60 ms | ↓ |
resumption_rate |
1.5 |
0.9545
|
85.90 0.9545 85.90 % | ↑ |
Recommended Settings
| Factor | Value |
|---|---|
session_cache_size | 100000 entries |
session_timeout_s | 43230 s |
ocsp_stapling_workers | 8 threads |
Source: from observed run #10
Trade-off Summary
Sacrifice = how much worse than single-objective best.
| Response | Predicted | Best Observed | Sacrifice |
|---|---|---|---|
resumption_rate | 85.90 | 85.90 | +0.00 |
Top 3 Runs by Desirability
| Run | D | Factor Settings |
|---|---|---|
| #3 | 0.8623 | session_cache_size=50500, session_timeout_s=43230, ocsp_stapling_workers=4.5 |
| #12 | 0.7816 | session_cache_size=1000, session_timeout_s=86400, ocsp_stapling_workers=4.5 |
Model Quality
| Response | R² | Type |
|---|---|---|
resumption_rate | 0.6081 | quadratic |
Full Multi-Objective Output
doe optimize --multi
============================================================
MULTI-OBJECTIVE OPTIMIZATION
Method: Derringer-Suich Desirability Function
============================================================
Overall desirability: D = 0.9150
Response Weight Desirability Predicted Direction
---------------------------------------------------------------------
handshake_ms 1.0 0.8586 36.60 ms ↓
resumption_rate 1.5 0.9545 85.90 % ↑
Recommended settings:
session_cache_size = 100000 entries
session_timeout_s = 43230 s
ocsp_stapling_workers = 8 threads
(from observed run #10)
Trade-off summary:
handshake_ms: 36.60 (best observed: 32.00, sacrifice: +4.60)
resumption_rate: 85.90 (best observed: 85.90, sacrifice: +0.00)
Model quality:
handshake_ms: R² = 0.6594 (quadratic)
resumption_rate: R² = 0.6081 (quadratic)
Top 3 observed runs by overall desirability:
1. Run #10 (D=0.9150): session_cache_size=100000, session_timeout_s=43230, ocsp_stapling_workers=8
2. Run #3 (D=0.8623): session_cache_size=50500, session_timeout_s=43230, ocsp_stapling_workers=4.5
3. Run #12 (D=0.7816): session_cache_size=1000, session_timeout_s=86400, ocsp_stapling_workers=4.5
Full Analysis Output
doe analyze
=== Main Effects: handshake_ms ===
Factor Effect Std Error % Contribution
--------------------------------------------------------------
session_timeout_s 15.8107 3.1456 44.8%
session_cache_size 13.0250 3.1456 36.9%
ocsp_stapling_workers 6.4321 3.1456 18.2%
=== ANOVA Table: handshake_ms ===
Source DF SS MS F p-value
-----------------------------------------------------------------------------
session_cache_size 2 341.8533 170.9266 0.459 0.6475
session_timeout_s 2 663.3533 331.6766 0.891 0.4474
ocsp_stapling_workers 2 106.1772 53.0886 0.143 0.8693
Lack of Fit 6 221.8790 36.9798 0.099 0.9879
Pure Error 2 744.6067 372.3033
Error 8 966.4856 372.3033
Total 14 2077.8693 148.4192
=== Summary Statistics: handshake_ms ===
session_cache_size:
Level N Mean Std Min Max
------------------------------------------------------------
1000 4 43.0000 9.4717 35.0000 55.7000
100000 4 56.0250 10.0613 46.0000 65.7000
50500 7 48.6857 13.9235 32.0000 75.6000
session_timeout_s:
Level N Mean Std Min Max
------------------------------------------------------------
43230 7 55.6857 13.1931 41.0000 75.6000
60 4 46.9000 7.9603 36.6000 55.7000
86400 4 39.8750 7.7474 32.0000 48.8000
ocsp_stapling_workers:
Level N Mean Std Min Max
------------------------------------------------------------
1 4 49.5250 14.4933 32.0000 65.7000
4.5 7 46.6429 13.6731 35.0000 75.6000
8 4 53.0750 8.5597 43.7000 63.6000
=== Main Effects: resumption_rate ===
Factor Effect Std Error % Contribution
--------------------------------------------------------------
session_timeout_s 20.9893 5.1048 37.0%
ocsp_stapling_workers 18.6750 5.1048 32.9%
session_cache_size 17.1250 5.1048 30.2%
=== ANOVA Table: resumption_rate ===
Source DF SS MS F p-value
-----------------------------------------------------------------------------
session_cache_size 2 713.4387 356.7193 0.406 0.6793
session_timeout_s 2 1144.5098 572.2549 0.651 0.5470
ocsp_stapling_workers 2 951.4158 475.7079 0.541 0.6018
Lack of Fit 6 905.6624 150.9437 0.172 0.9607
Pure Error 2 1757.3267 878.6633
Error 8 2662.9890 878.6633
Total 14 5472.3533 390.8824
=== Summary Statistics: resumption_rate ===
session_cache_size:
Level N Mean Std Min Max
------------------------------------------------------------
1000 4 61.9500 19.1744 40.8000 85.9000
100000 4 44.8250 17.1484 27.2000 63.6000
50500 7 47.5571 21.5009 13.2000 74.0000
session_timeout_s:
Level N Mean Std Min Max
------------------------------------------------------------
43230 7 42.2857 19.4086 13.2000 69.7000
60 4 52.7250 23.9978 33.5000 85.9000
86400 4 63.2750 10.6456 48.7000 74.0000
ocsp_stapling_workers:
Level N Mean Std Min Max
------------------------------------------------------------
1 4 47.2500 21.2629 27.2000 74.0000
4.5 7 58.7000 22.5078 13.2000 85.9000
8 4 40.0250 6.4552 33.8000 48.7000
Optimization Recommendations
doe optimize
=== Optimization: handshake_ms ===
Direction: minimize
Best observed run: #3
session_cache_size = 1000
session_timeout_s = 43230
ocsp_stapling_workers = 1
Value: 32.0
RSM Model (linear, R² = 0.1849, Adj R² = -0.0374):
Coefficients:
intercept +49.1267
session_cache_size +5.5875
session_timeout_s -2.2125
ocsp_stapling_workers +3.4500
RSM Model (quadratic, R² = 0.5929, Adj R² = -0.1399):
Coefficients:
intercept +58.3333
session_cache_size +5.5875
session_timeout_s -2.2125
ocsp_stapling_workers +3.4500
session_cache_size*session_timeout_s +10.0250
session_cache_size*ocsp_stapling_workers +2.1500
session_timeout_s*ocsp_stapling_workers -2.0000
session_cache_size^2 -3.0792
session_timeout_s^2 -4.5792
ocsp_stapling_workers^2 -9.6042
Curvature analysis:
ocsp_stapling_workers coef=-9.6042 concave (has a maximum)
session_timeout_s coef=-4.5792 concave (has a maximum)
session_cache_size coef=-3.0792 concave (has a maximum)
Notable interactions:
session_cache_size*session_timeout_s coef=+10.0250 (synergistic)
session_cache_size*ocsp_stapling_workers coef=+2.1500 (synergistic)
session_timeout_s*ocsp_stapling_workers coef=-2.0000 (antagonistic)
Predicted optimum (from linear model, at observed points):
session_cache_size = 100000
session_timeout_s = 43230
ocsp_stapling_workers = 8
Predicted value: 58.1642
Surface optimum (via L-BFGS-B, linear model):
session_cache_size = 1000
session_timeout_s = 86400
ocsp_stapling_workers = 1
Predicted value: 37.8767
Model quality: Weak fit — consider adding center points or using a different design.
Factor importance:
1. ocsp_stapling_workers (effect: 12.5, contribution: 42.3%)
2. session_cache_size (effect: 11.2, contribution: 37.8%)
3. session_timeout_s (effect: 5.9, contribution: 19.9%)
=== Optimization: resumption_rate ===
Direction: maximize
Best observed run: #10
session_cache_size = 1000
session_timeout_s = 86400
ocsp_stapling_workers = 4.5
Value: 85.9
RSM Model (linear, R² = 0.1530, Adj R² = -0.0780):
Coefficients:
intercept +50.6667
session_cache_size -7.8875
session_timeout_s +6.0875
ocsp_stapling_workers -2.3250
RSM Model (quadratic, R² = 0.6768, Adj R² = 0.0950):
Coefficients:
intercept +34.2333
session_cache_size -7.8875
session_timeout_s +6.0875
ocsp_stapling_workers -2.3250
session_cache_size*session_timeout_s -19.8750
session_cache_size*ocsp_stapling_workers -4.1500
session_timeout_s*ocsp_stapling_workers -4.2500
session_cache_size^2 +8.9958
session_timeout_s^2 +6.9958
ocsp_stapling_workers^2 +14.8208
Curvature analysis:
ocsp_stapling_workers coef=+14.8208 convex (has a minimum)
session_cache_size coef=+8.9958 convex (has a minimum)
session_timeout_s coef=+6.9958 convex (has a minimum)
Notable interactions:
session_cache_size*session_timeout_s coef=-19.8750 (antagonistic)
session_timeout_s*ocsp_stapling_workers coef=-4.2500 (antagonistic)
session_cache_size*ocsp_stapling_workers coef=-4.1500 (antagonistic)
Predicted optimum (from quadratic model, at observed points):
session_cache_size = 1000
session_timeout_s = 86400
ocsp_stapling_workers = 4.5
Predicted value: 84.0750
Surface optimum (via L-BFGS-B, quadratic model):
session_cache_size = 1000
session_timeout_s = 86400
ocsp_stapling_workers = 1
Predicted value: 101.3208
Model quality: Moderate fit — use predictions directionally, not precisely.
Factor importance:
1. ocsp_stapling_workers (effect: 16.0, contribution: 36.4%)
2. session_cache_size (effect: 15.8, contribution: 35.9%)
3. session_timeout_s (effect: 12.2, contribution: 27.7%)