After performing these 4 steps and analyzing the results, we found
that only options o2 and o10 had a significant effect on latency and
throughout across the entire configuration space. These results
surprised ACE+TAO developers since they thought that all 14 run-time
options would contribute substantially to latency and throughput. Our
analysis of the screening design data give the same results. We were
therefore able to get accurate data at 
the cost.
In the second phase of the process we used the information that o2 and o10 are important options to generate all possible (in this case 4) configurations for the binary options o2 and o10. Default values were assigned to the remaining options. We then measured latency and throughput on the benchmark test applications.
Our results showed that the performance distributions obtained from
the main effects set were similar to the ones obtained from the
exhaustive runs at a fraction of the cost. In contrast, randomly
sampled configurations ( i.e., 4 chosen at random) produced very
different data. It would therefore be an unreliable
indicator of performance degradation following system changes.
Table 
shows the percentage of
observations for each performance metric in the entire configuration
space that fall into the range of the observations obtained from
screening and random designs.
Table: Range of Performance Metrics Covered by Screening and Random Design