CS333 Lab A-5:
Global Snapshot

Goal of this lab:

Lab Guru:

Questions about this lab may be directed to Ram Sethuraman, ram@cs.wustl.edu, or the instructor.

Introduction:

In many distributed systems, it is useful to capture a "consistent global state" of the system. For example, one might want a consistent picture of the state of the system for debugging purpose, to monitor the execution of the system, or to perform some sort of audit on the system. In this lab, we consider the problem of finding a consistent global state of a distributed system in which processes send messages to each other along FIFO (first-in first-out) channels.

A global state consists of the state of every nodes and every channel in the system, where the state of a channel is the sequence of messages "in-transit," those that have been sent on that channel, but not yet received. We will take a global snapshot of the system by having each node in the system record its own local internal state, as well as the sequence of messages on its incoming channels. However, the question becomes when to have each process take its local snapshot.

If we all processes were synchronized by a single clock, then taking a consistent global snapshot would be trivial. Each process could record its state and that of its incoming channels at an agreed-upon time. However, in most distributed systems, there is not a centralized clock. Communication and computation steps are not synchronized, and messages are needed to trigger each process to take its local snapshot, so the local snapshot taken at each process occurs at different physical times. Furthermore, the problem is even more difficult because we want our snapshots not only to be consistent, but also to be "recent." For example, although the state reported would be consistent, we would not find a global snapshot algorithm acceptable if it always reported the global initial state of the system.

We can define an execution of the system as the sequence of communication events (send and receive events) occurring at every process in the system. Now, consider the global state of the system at the physical instant of time when the global snapshot is requested. Let's call this the "before" state. If we could just capture the "before" state, then we'd be finished, but we can't do this because messages continue to flow through the system even during the process of taking the snapshot, so by the time we notify a process to take a snapshot, it's state may have changed. Next, consider the global state of the system at the physical instant of time when the global snapshot is finished (i.e., after every process has taken its local snapshot). Let's call this the "after" state. Similarly, if our snapshot could just capture the "after" state, then we'd be done. However, we can't do that either because we're not allowed to stop the other messages flowing through the system after we take the local snapshot at a node. Therefore, by the time all nodes take their local snapshots, the global system state may have changed.

So, we define a recent and consistent global state as a global state that "could have occurred" between the "before" state and the "after" state. In other words, consider the sequence S of communication events that occurs in the system between the "before" and the "after" state. A recent and consistent global state X is reachable from the "before" state by some subsequence of the events in C. Furthermore, the "after" state is reachable from the state X by the remaining events in C. Notice that this global state didn't necessarily occur at any point in real time, but it "could have occured" between the "before" and "after" states using the same sequence of communication events.

Global snapshots are useful in a wide variety of distributed applications. One application is in distributed databases, for instance a group of bank branches. Another use is deadlock detection: a global snapshot can be examined to see if there has been any progress made by the algorithm. Termination of a distributed algorithm can detected in the same way.

In this lab, you will consider the problem of computing global snapshots in a distributed bank application. You will implement bank modules that send money to each other at random intervals.

Directions:

Read over the entire assignment before starting.

  1. The Distributed Banking System: First, implement a bank module that will act as one branch of a distributed bank. Basically, these bank branches just send money to each other at unpredictable times. You will create many instances of this module and hook them together to form the distributed banking system. Specifically, the bank module
    1. starts out with some initial balance (amount of money),
    2. has at least one input variable where other bank branches connect to it and send it money,
    3. has some number of output variables (specified on the command line at module startup) so that it can send money to different bank branches,
    4. and randomly picks an output variable and sends out a random amount of its money on that connection to the other branch.

  2. Global Snapshot: Add a global snapshot capability to your bank module, using the global snapshot algorithm of Chandy and Lamport. In the case of the banking system, a global snapshot will consist of the local state of each bank branch (i.e., its balance) and the amount of money in transit on each of the communication channels. Each bank branch will be responsible for reporting its own local state (balance) and the total money in transit on each of its incoming edges.

    The basic strategy of the algorithm is that when a processor initiates a global snapshot, it records its local state and sends a special marker on all of its outgoing channels to signal its neighbors that a snapshot has been initiated and it begins remembering all the messages it receives on its incoming channels. When it gets a marker on an incoming channel from neighbor X, it knows that neighbor X has taken a local snapshot that reflects all of the messages X has sent up to the marker. Therefore, the process receiving the marker records the state of that channel (i.e., the messages it has received between recording its local state and receiving the marker on the channel). After the local state and the states of all channels have been recorded, the local node reports that information. All participants in the global snapshot use the same algorithm, summarized below.

    1. To initiate a global snapshot, a module records its own state (see 2).
    2. Any module recording its own state will send a special marker on all of its output channels
      1. after recording its own state and
      2. before sending anything else on those channels.
    3. The state of a channel is recorded by a process as the sequence of messages received along that channel
      1. after recording its own state and
      2. before receiving a marker on that channel.
    4. Upon receiving a marker along an incoming channel, a module must do the following:
      1. If the module has already recorded its own state, then the module records the state of the channel as the sequence of messages that arrived between the time it recorded its own local state and the time it received the marker.
      2. If the module has not yet recorded its own state, then it records its own state (see 2) and records the state of the channel as empty.
    Any bank branch module can initiate a global snapshot at any time. You can let them do this randomly. Be sure to test your implementation thoroughly. Notice that if the snapshot is correct, the total amount of money in the bank branches and in-transit should equal the original amount in the system.

Further Reading:

The following classic paper describes the global snapshot algorithm in detail.

K. Mani Chandy and Leslie Lamport, Distributed Snapshots: Determining Global States of Distributed Systems. ACM Transactions on Computer Systems, Vol. 3, No. 1, February, 1985, pp. 63-75.

To receive credit for this lab, you should:

  1. Clean up and print out your code. (Don't turn it in, but save it for your code/design review.)

  2. Turn in a Project Evaluation Form near the beginning of class on the day you want to do your demonstration and code/design review. You should be prepared to demonstrate your working application, explain your design and code, and answer questions.