Online crash reporting systems, such as the Netscape Quality Feedback Agent and Microsoft XP Error Reporting, gather system state at a central location whenever a fielded system crashes, which simplifies user participation in QA by automating certain aspects of problem reporting. Likewise, many popular open-source projects use distributed regression test suites that end-users can run to evaluate installation success. Well-known examples include GNU GCC, CPAN, Mozilla, the Visualization Toolkit (VTK), and ACE+TAO.
Auto-build scoreboards are a more proactive form of distributed regression test suites that allow software to be built/tested at multiple sites on various hardware, operating system, and compiler platforms. The Mozilla Tinderbox (www.mozilla.org/tinderbox.html) and ACE+TAO Virtual Scoreboard (www.dre.vanderbilt.edu/scoreboard/) are auto-build scoreboards that track end-user build results across various platforms. Bugs are reported via problem tracking systems (such as Bugzilla or Jira), which provides inter-bug dependency recording, advanced reporting capabilities, extensive configurability, and integration with automated software configuration management systems, such as CVS or Clearcase.
Although these prior efforts have provided a good starting point, they have several limitations. For example, they have a limited focus, ignoring PSAs related to QoS and performance issues. Our approach can be customized to address a wide variety of PSAs. Moreover, while these efforts help document and automate portions of the QA process, the decision of what and when to test it is left to users. In contrast, the model-driven DCQA techniques described in this paper enable developers to control key aspects of the QA process, thereby minimizing gaps and inefficiencies. As discussed below, our approach can exploit incremental results and selectively ignore problems discovered earlier, which avoids wasting resources that could otherwise be devoted to identifying other problems.