Managing PSA Challenges with Model-driven Skoll

The cornerstone of Skoll is its formal model of a DCQA process's configuration space, which captures the valid configurations for QA subtasks. A configuration in Skoll is represented formally as a set , , , , where each is a configuration option and is its value, drawn from the allowable settings of . Since in practice not all configurations make sense ( e.g., feature X may not be supported on operating system Y), we define inter-option constraints that limit the setting of one option based on the setting of another. We represent constraints as (), meaning ``if predicate evaluates to TRUE, then predicate must evaluate to TRUE.'' A predicate can be of the form A, , , A | B, or simply , where A, B are predicates, is an option and is one of its allowable settings. A valid configuration is a configuration that violates no inter-option constraints. Skoll uses this configuration space model to plan global QA processes, adapt processes dynamically, and aid in analyzing and interpreting results.

Since the configuration spaces can be quite large, Skoll has an Intelligent Steering Agent (ISA) which controls DCQA processes by deciding which valid configuration to allocate to each incoming Skoll client request. When a client becomes available, the ISA decides which subtask to assign it. To do this, the ISA can consider many factors, including (1) the configuration model, e.g., which characterizes the subtasks that can legally be assigned, (2) the results of previous subtasks, e.g., which captures what tasks have already been done and whether the results were successful, (3) global process goals, e.g., test popular configurations more than rarely used ones or test recently changed features more than heavily than unchanged features, and (4) client characteristics and preferences. The configuration must be compatible with physical realities such as the OS running on the remote machine. Also, client preferences, which are declared in a Skoll client template must be respected. For example, suppose a product runs in normal or superuser mode. A security conscious user might only want configurations in which the mode is normal.

Once a valid configuration is chosen, the ISA packages the corresponding QA subtask implementation into a job configuration, which consists of the code artifacts, configuration parameters, build instructions, and QA-specific code ( e.g., developer-supplied regression/performance tests) associated with a software project. The job configuration is then sent to the requesting Skoll client, which executes the job configuration and returns the results to the ISA so it can learn from the results and adapt the process. For example, if some configurations fail to maintain certain PSAs, developers may want to pinpoint the source of the problems or refocus on other unexplored parts of the configuration space. To do this Skoll process designers can develop customized adaptation strategies that monitor the global process state, analyze it, and use the information to modify future subtask assignments in ways that improve process performance.

Skoll applies various model-driven tools that raise the level of abstraction and reduce the accidental complexity of dealing with Skoll's internal formats. For example, Skoll employs the Benchmarking Generation Modeling Language (BGML) [] that allows developers to (1) visually model interaction scenarios between configuration options and system components using domain-specific building blocks, (2) automate generation of common parts of test code and reuse QA subtask code across configurations, (3) generate control scripts to distribute and execute the experiments across the Skoll grid to monitor performance and functional behavior in a wide range of execution contexts, and (4) enable evaluation of multiple software attributes such as correctness, throughput, latency, jitter, and other criteria.

Since DCQA processes are complex we often need help to interpret and leverage process results. Therefore a wide variety of analysis tools can be plugged into Skoll. One such tool we added to Skoll implements Classification Tree Analysis (CTA) []. CTA's output is a tree-based model that predicts object class assignment based on the values of a subset of object features. As we show in Section , we used CTA to diagnose which specific options and option settings were most likely causing specific PSA test failures. This helped developers quickly identify the root causes of some failures.