Winckp: a Transparent Checkpointing and Rollback Recovery Tool for Windows NT Applications P. Emerald Chung Woei-Jyh Lee Bell Laboratories Lucent Technologies
[email protected] [email protected]
AT&T Labs
[email protected]
•
Abstract The goal of Winckp is to transparently checkpoint and recover applications on Windows NT. The definition of transparency is no modifications to applications at all, period. There is no need to get source code, or object code. It does not involve compilation, linking or generation of a different executable. We employ window message logging/replaying to recreate states that are otherwise difficult to recover by checkpointing alone. In the paper, we describe the design and implementation of Winckp, and present the challenges and limitations. The software is available for download from http://www.bell-labs.com/projects/swift.
1
Deron Liang Chung-Yih Wang
Yennun Huang
Introduction
Windows NT has become a popular computing platform for building new applications due to the proliferation of software tools and components and the potential lower hardware cost. The telecommunication industry has also started to build fault-tolerant and highly available systems on NT. We have been working on the NT-SwiFT project to understand the fault-tolerance and high availability requirements of applications running on NT. NT-SwiFT currently contains a set of components which add fault-tolerant capabilities to applications on Windows NT and Winckp is one of them. Winckp supports five major functions. • Checkpoint: store the state of an application process in persistent storage. • Rollback: roll back the state to the last checkpoint in the same process. • Recover: restore the last checkpoint in a new process.
•
Institute of Information Academia Sinica
[email protected] [email protected]
Logging: store window messages such as keystrokes, mouse movements and mouse clicks in persistent storage. Replay: replay logged window messages.
Additional features include (1) support of file rollback; (2) support of multiple threads; (3) console display recovery; (4) partial replay of window messages; (5) configurable replay speed.
2 2.1
User Interface Graphical User Interface
The Winckp GUI is shown in Fig. 1. The user clicks the “Run App” button to start an application, and clicks the “Snapshot” button at any time to checkpoint the application; the user clicks the “Recover” button to roll back the application to the last checkpoint. If the application process has died, the user can click the “Run App” button again to start the last application and click the “Recover” button to recover the last checkpointed state in the new process. An application process state consists of data, stack, heap, thread context, and some kernel objects. If the recovery of an application requires a rollback of files, the user must check the "Include files" box before running the application. Winckp can also be used to log and replay mouse and keyboard events of an application. The feature is turned on by clicking the “Start recording” button. The event log is truncated every time the “Snapshop” button is clicked. If the “Playback” button is clicked, Winckp replays all mouse and keyboard events starting from the last snapshot.
The Winckp GUI is suitable for Window-based applications that are constantly waiting for a mouse or keyboard event to change its state, such as games. Winckp can save the state between moves and rollback can be used as a generic undo operation.
embed checkpointing or logging in their applications by linking with either of these two DLLs.
3 3.1
Implementation Checkpoint and Rollback
Winckp is implemented from scratch to use only Win32 APIs. Winckp leverages a set of Win32 functions that allow a process to alter another process. Microsoft created many of these functions for use by debuggers [Richter97-18]. Winckp starts the application by calling CreateProcess(). To take a snapshot, Winckp suspends all threads and stores state information into files. During a rollback operation, all threads are also suspended. The state information is restored from files. Winckp resets all thread contexts then resumes all threads.
Figure 1 The Winckp GUI
2.2
Command Line Interface
Winckp is also useful for long-running scientific programs which usually require periodic checkpointing. We design a command line interface, ckprun, for this purpose. The usage of ckprun is shown in Table 1.
If the application process has died, Winckp starts a new process, suspends its main thread, restores all state information, and then resumes all threads. Winckp injects a checkpointing thread into the target application. This thread is in charge of saving system-call-related info in stable storage and replaying the system calls during rollback or recovery.
3.2 Usage: CKPRUN [/F ckpf] [/I t] [/X] app.exe [/F ckpf] specifies checkpoint file name [/I t] specifies the time between checkpoints. [/X ] No checkpoint and recovery of files. [/R ] Recover execution from checkpoint
Threads
Winckp and the target applications are running as two separate processes. NT allows a process to obtain and set the thread context for threads in a different process. Winckp obtain and restore the thread context using GetThreadContext() and SetThreadContext(). Winckp also recreates and terminates threads if the number of threads at the time of the last snapshot is different from the number at the recovery time.
Table 1 Ckprun command line interface
2.3
Library Interface
The Winckp GUI and ckprun are two separate executables. The basic functions for checkpointing and recover are implemented in a library, libwinckp.dll and the functions for logging and replaying window messages are in libwinrec.dll. This architecture allows people to
3.3
Memory
Winckp saves an application’s memory including data, heap and stack. The memory image is obtained and restored using ReadProcessMemory() and WriteProcessMemory(). Winckp determines the address and the amount of memory to save as follows. An NT process has
about 2GB of private address space, ranging from 0x00010000 to 0x7FFEFFFF [Richter973], but not every region in this space needs to be saved. The VitualQueryEx() system call allows us to examine the space region by region. It is necessary to save a memory region only if its write access is enabled and if its physical storage is committed [Richter97-5]. Winckp also stores the MEMORY_BASIC_INFORMATION structure along with each memory region. During a rollback operation, all threads are suspended. Winckp calls WriteProcessMemory() to restore the memory content.
3.4
Files
To checkpoint and recover files, Winckp uses a system call interception mechanism to intercept file-related system calls, such as CreateFile(), WriteFile(), etc. Winckp records each handle value and the parameters used to create the handle. In recovery, Winckp recreates all the handles by replaying the calls with the recorded parameters. An issue is that the values of the recovered handles may be different from their checkpointed values. Winckp creates a mapping of old handle values to the new handle values. The system call interception routines replace the old handle values with the new values before making the real call. To make sure file content is consistent, Winckp generates idempotent undo logs for all file updates. The undo logs are played at the recovery time.
3.5
Window objects
Winckp intercepts a subset of functions in USER32.dll and GDI32.dll, such as CreateWindow() and CreateDC(). The recovery of window handles and other GDI objects has been a very difficult task due to the dependency among these objects and the fact that they are not isolated from other processes or kernel. To recover a window handle, Winckp first starts a new process and let the new process create all related windows. Afterwards, Winckp creates a mapping between the old window handle value and the new window handle value. We replace the old value by new value in the checkpoint file before restoring it back to memory. It works for a small set of applications, such as winmine.exe and solitaire.exe. However, a more generic
mechanism is necessary and it depends on an indepth understanding of NT windows management.
3.6
Window messages
Winckp records events coming from the mouse and keyboard. The Win32 subsystem provides a hook that allows a user application to monitor system events such as keyboard strokes, window messages, debugging information, etc., and to react to these events through a user-defined callback procedure. User applications may specify those system events of interest and install the corresponding callback procedures via the Win32 API. Winckp captures these events by calling SetWindowsHookEx() with flag WH_JOURNALRECORD. Keyboard events and mouse events are copied from the Win32 system’s message queue to our callback procedure. These events are kept in a log file. To replay, we insert these events one by one in their timestamp order back to the Win32 system message queue by installing the WH_JOURNALPLAYBACK callback procedure. The Win32 system temporarily disables the inputs from keyboard and mouse when the WH_JOURNALPLAYBACK callback procedure is installed. It executes only the events fed from the callback procedure until our event log is over or the WH_JOURNALPLAYBACK callback procedure is uninstalled.
4
Limitations
Winckp currently has the following limitations: (1) System calls: If a snapshot takes place in the middle of a system call, Winckp does not store any kernel state. It may become a problem during recovery. (2) Thread ID and thread handle value: Current Winckp rolls back the number of threads to the number in the last snapshot state. A dead thread is recreated. The recreated thread usually has a different thread handle and thread ID. Applications depend on thread handle and thread ID may not work. (3) Synchronization objects: All kernel objects used for synchronization are not recovered if they no longer exist, such as events and mutexes. (4) Interprocess communication: The current implementation does not support recovery of interprocess communications, such as COM calls or socket calls.
(5) Playback nondeterminism: Since playback involves the whole display area, it involves all windows in other processes. There is no guarantee from Winckp that the state of all windows at the beginning of playback is the same as the beginning of recording. In addition, changing the playback speed may also affect the outcome. For instance, the Windows may interpret two single clicks as a double click or vice versa.
5 5.1
Related work Windows Checkpointing Software
A similar checkpointing facility was developed at Intel [Srouji98]. The main differences are (1) it requires a different build process to replace an application’s startup function with their checkpoint DLL startup code; (2) it utilizes the QueueUserAPC() mechanism on NT to replay all system calls at recovery by application threads, while our system calls are replayed by the injected thread; (3) it does not roll back any persistent storage; (4) it does not handle GUI programs; (5) it does not log or replay window messages. There is a checkpointing facility on the Brazos parallel programming environment [Abdel-Shafi 99]. To support thread migration, Brazos currently implements a subset of Winckp functions including saving the thread context, stack and memory.
5.2
Acknowledgement
The authors would like to thank Binh Vo for his help in the development of Winckp, Yi-Min Wang for his discussions on the UNIX libckp implementation, and Chandra Kintala for his support of the work.
8
Reference:
[Abdel-Shafi 99] H. Abdel-Shafi et al, “Efficient User-Level Thread Migration and Checkpointing on Windows NT Clusters”, to appear in the 3rd USENIX Windows NT Symposium, 1999. [Huang98] Y. Huang, P. Chung, C. Kintala, C. Wang and D. Liang, “Software Implemented Fault Tolerance on Windows NT”, 2nd USENIX Windows NT Symposium, pp. 47-55, Seattle, Washington, August, 1998. [Litzkow 97] M. Litzkow et al, “Checkpoint and Migration of UNIX Processes in the Condor Distributed Processing System,” Univ. of Wisconsin-Madison, CS TR #1346, April 1997. [Richter97-3] J. Richter, “Processes”, Chapter 3, in Advanced Windows, Ed. 3, pp.33-72, Microsoft Press, 1997. [Richter97-5] J. Richter, “Win32 Memory Architecture”, Chapter 5, in Advanced Windows. [Richter97-18] J. Richter, “Breaking Through Process Boundary Wall”, Chapter 18, in Advanced Windows.
UNIX Checkpointing Software
On UNIX, transparent checkpointing is now very well studied. The discussions can be found in [Litzkow 97][Plank 95][Wang 95].
6
7
Summary
Windows NT is very complex and it is open to all sorts of hardware/driver configurations. As a result, it has been perceived as less reliable than UNIX and there is a rapidly increasing need for application-level fault-tolerance on Windows NT. In this paper, we described our experience in implementing checkpointing on Windows NT. In general, NT supports more mechanisms needed to enable the implementation of checkpointing. However, the recovery of NT kernel objects or GUI objects is not yet well understood. There remain a lot of challenges.
[Plank 95] J. Plank, M. Beck and G. Kingsley, “Libckpt: Transparent Checkpointing Under UNIX”, 1995 Usenix Conference. [Srouju 98] J. Srouji et al., “A Transparent Checkpoint Facility on NT,” Proceedings of the USENIX NT Symposium, Seattle, 2nd Washington, pp. 77-85, 1998. [Wang95] Y. Wang and Y. Huang and K. Vo and P. Chung and C. Kintala, “Checkpoint and its applications”, Proceedings of the 25th IEEE Fault Tolerant Computing Symposium, Pasadena, California, pp. 22-31, 1995.
