DCOM Benchmarking Framework

This proposal describes the design of a benchmarking framework to systematically measure the sources of latency and throughput overhead, priority inversion, and non-determinism in DCOM. The goals of this framework include the following:

Focus on high-performance and real-time features -- The framework will emphasize empirical evaluation of the performance of DCOM.
Support multiple configurations -- All framework benchmarks will be performed for configurations where clients and servants are:
- In the same address space in the same thread;
- In the same address space in different threads;
- In different address spaces on the same processor;
- In different address spaces on different processors.
In addition, all benchmarks will measure different data types, data sizes, and synchronous/asynchronous operation invocations (when DCOM supports asynchrony).

Sources of Overhead, Priority Inversion, and Non-determinism in DCOM

Based on our prior experience benchmarking real-time and non-real-time CORBA ORBs, and based on benchmarks conducted by Microsoft research, we anticipate that the performance overhead, priority inversions, and non-determinism of DCOM will stem from the following:

Network and transport communication protocols -- e.g., flow and congestion control, retransmission strategies, connection management, error handling.
Communication protocol implementations and integration with the OS I/O subsystem and network interfaces -- e.g., data copying, fragmentation, reassembly, context switching, synchronization, checksumming, demultiplexing, marshalling, and demarshalling.
DCOM protocols -- e.g., the DCE RCP protocol.
DCOM implementation and integration with Windows NT endsystem services -- e.g., data copying, fragmentation, reassembly, context switching, synchronization, checksumming, event demultiplexing, timer handling, request demultiplexing, marshaling/demarshaling, framing, error checking, connection management, and concurrency models.
Many of these issues are the same as #2. However, they occur at the user-level rather than at the kernel-level, which may make them easier to solve in a portable manner.

Benchmark Framework Content and Deliverables

To determine the sources of overhead empirically for DCOM, we plan to develop a comprehensive suite of benchmarks designed to test various aspects of DCOM performance. The content and deliverables of the benchmarking framework include the following:

Latency, jitter (latency variance), and throughput performance of operation invocations between 1 client and 1 servant. This test measures system "point-to-point" performance.
DCOM scalability measurements, i.e., what happens to latency, jitter, and throughput as the number of objects in endsystems and distributed systems increases. This test measures the efficiency and predictability demultiplexing operations in DCOM.
Number of QueryInterface operations per second. This test measures the latency required for a client to acquire object references from a server.
Latency, jitter (latency variance), and throughput performance of operation invocations between m clients and n servants. This test measures "aggregate" system performance.
Static and dynamic memory footprint for different configurations. This test also measures the amount of object reference caching.
Degree of priority inversion and non-determinism. This test determines how well DCOM can support end-to-end priority.
Impact of the concurrency model on client and server. This test measures the performance and predictability of DCOM concurrency models, such as single-threaded apartment and multi-threaded apartments.
Measure CPU utilization and processor cache effects for different workloads and different concurrency model configurations.

All tests will run on Windows NT 4 (and the Windows NT 5.0 beta) over ATM, Ethernet, and Fast Ethernet. To support this benchmarking effort, we'll use our high-performance testbed, which consists of 6 quad-CPU 400 Mhz Pentium II PCs containing ATM cards and Fast Ethernet cards.

Our project deliverables will include tested and useable DCOM benchmarking middleware, demonstration applications, technical reports describing the benchmarks' performance on Windows NT platforms, and documentation in HTML format.

Budget

The total cost of the proposed effort is $30,000 for a 12 month period of performance. This budget will be used to fund a graduate student and a percentage of Dr. Schmidt. The participants in this effort include the following personnel:

Faculty member, i.e., Douglas C. Schmidt -- Ph.D., Professor, Washington University.
Graduate assistant, i.e., Darrell Brunsch -- M.S. graduate student, Washington University.