Scenario 2 assessed the portability and correctness PSAs of ACE+TAO by executing regression tests on each compile-time configuration using the default run-time options ( i.e., the configuration new users encounter upon installation). We used the 96 regression tests that are distributed with ACE+TAO, each containing its own oracle and reporting success or failure on exit. We expanded the configuration model to include options that captured low-level OS and compiler information, e.g., indicating the use of static vs. dynamic libraries, multithreading vs. singlethreading, and inlining vs. non-inlining. Also, some ACE+TAO tests can only run in particular configurations (such as when the multithreading is selected), so we also adding test-specific options to the configuration space.
The new test-specific options contain one option per test,
(
), which indicates if that test 
is runnable in a
given compile-time configuration. We also defined constraints over
these options, e.g., some tests should run only on
configurations that have more than Minimum CORBA features. After
making these changes, the space now had 14 compile time options with
12 constraints and an additional 120 test-specific constraints.
After resolving the constraints, we compiled 2,077 individual tests,
of which 98 did not compile and 1,979 did. Of these 1,979 tests, 152
failed, while 1,827 passed. This process took 
52 hours of
computer time on the Skoll grid available for the experiments.
In several cases, tests failed for the same reason in the same
configurations. For example, CTA analysis showed that
test compilation failed at a given file
for the following option settings (
and POLLER = 0
and CALLBACK = 0). This compilation error stemmed from a previously
undiscovered bug that occurred because certain TAO files
assumed these settings were invalid and thus could not occur.
Using our model-driven DCQA environment and process, we were therefore
able to determine whether the current version of ACE+TAO successfully
completes all regression tests in its default configuration.