termine the degree of authenticity. ... With the emulation approach, obsolete computer hardware is emulated by ... At the same time, programmers were to.
Implementing the Universal Virtual Computer Nico Krebs, Lothar Schmitz, and Uwe M. Borghoff Universit¨ at der Bundeswehr M¨ unchen, Institute for Software Technology 85577 Neubiberg, Germany {Nico.Krebs,Lothar.Schmitz,Uwe.Borghoff}@unibw.de
Abstract. In order to keep digital objects for an indefinite period of time, one needs a very contrived archiving system. One challenge is to sustain the accessibility of document formats that are becoming obsolete, another is to guarantee their authenticity. The Universal Virtual Computer (UVC) is a simple yet powerful approach to preserve digital objects on a very long-term scale. Its main attraction is that documents do not have to be processed and transformed during their whole archive lifetime. In contrast, when using the migration approach, all documents have to be processed in regular intervals. This is not only time-consuming; also, after a number of migration steps a document’s authenticity is seriously threatened. UVC does not share these problems. With UVC, the main effort occurs before ingest time: rendering software for interpreting documents of a given format on UVC must be developed and archived. The effort spent in the development of the rendering software will determine the degree of authenticity. In order to access archived objects, an implementation of UVC must be available. The focus of this paper is on implementing UVC. So far, only proof-of-concept implementations of UVC were available. We have gained practical experience by implementing UVC on different platforms based on a collection of vintage, but still fully working supercomputers. These efforts have led to an improved specification of the UVC, which simplifies implementation even more. Keywords: UVC, long-term archiving, virtual machine, datArena.
1
Introduction
Long-term archiving of digital documents – and other digital artifacts – still remains one of the big problems to be solved by our generation. An ever increasing number of digital documents is being generated. They have to be kept safe in a way which guarantees unimpeded access to authentic reproductions of the originals even after many years. The two main approaches discussed below for solving this problem are migration and emulation. With both approaches, digital documents are archived in data formats that were up-to-date and widespread at ingest time. Experience shows that access to digital documents becomes very costly as soon as their data formats are obsolescent, i.e., are not supported by current computer systems any more. R. Moreno-D´ıaz et al. (Eds.): EUROCAST 2011, Part I, LNCS 6927, pp. 153–160, 2012. c Springer-Verlag Berlin Heidelberg 2012
154
N. Krebs, L. Schmitz, and U.M. Borghoff Emulation Approach
past
Migration Approach
future
present
Application
Emulator Application
Emulator chain Application
Fig. 1. Migration vs. Emulation
The essence of the migration approach is to continuously convert digital documents into some current and widely accepted data format. The tasks of regularly checking all data formats for imminent obsolescence, finding suitable new formats, procuring reliable conversion programs, applying them to all documents of those formats, and checking the authenticity of the converted documents are very time-consuming and can only be handled using automated processes. This, in turn, inevitably compromises authenticity and entails the risk of losing information past recovery, as suggested by gradually changed detail in Fig. 1. With the emulation approach, obsolete computer hardware is emulated by software running on some current system. The original bit stream representing the digital document is left unchanged all the time. Application software that was originally used to render the digital document (including systems software) is also stored in the archive. In order to access a digital document, this software and the bit stream representing the document are loaded into the emulator. The authenticity of this approach cannot be surpassed [2]. Both approaches require very little effort at ingest time. However, keeping documents accessible requires time-consuming processes such as indicated above for the migration approach. With the emulation approach, a new emulator has to be developed as soon as obsolescence of a current hardware is imminent. This is rather costly, in particular since both hardware and systems software are typically produced in many different versions. Finally, together with a hardware system X all other hardware supported by emulators running on X may become obsolete. The use of chained emulators as sketched in Fig. 1 may reduce the effort, but will slow down emulation and possibly lead to subtle errors, like the small deviations introduced by repeated migration. Still, an emulator which ensures access with a high level of authenticity is very attractive. Lorie et al. [9] have introduced the Universal Virtual Computer (UVC) which provides an improved approach to the long-term preservation of digital documents: Instead of (re)creating emulators for all obsolete hardware, one single target machine, the UVC, is used for all rendering software. The rendering software (any software, indeed) needs to be implemented in UVC machine code only once to live on as long as UVC runtime environments are available. UVC-based applications can be executed in any UVC runtime environment the same way at any time. Neither the digital object itself nor its corresponding
Implementing the Universal Virtual Computer
155
software have to be changed. Since the UVC is a simple yet powerful machine, the effort for developing a UVC runtime environment is fairly low. Until recently, the UVC was only available as proof-of-concept. Meanwhile, we have implemented the UVC on different architectures, ranging from vintage supercomputers to current personal computers. Since our focus is on long-term archiving of digital documents, we have evaluated our UVC implementations primarily in this respect. We also have implemented other complex UVC application software, but this is beyond the scope of this paper. The reminder of this paper is organized as follows. We describe the UVC and related work in more detail in Section 2. Our evaluation experiments are outlined in Section 3. In Section 4, we present lessons learned. Section 5 briefly sums up and indicates our plans for further research.
2
UVC and Related Work
UVC
Input
UVC was developed to satisfy two main objectives: In order to ensure its portability it was designed to be simple enough to be implemented within very short time on almost any kind of computer. At the same time, programmers were to be endowed with a small, but comfortable and powerful instruction set. Segmented store
Program Processor
Segment
Memory
Register Register
Register
…
…
Output
Section
Fig. 2. The UVC architecture
Consequently, the UVC architecture (see Fig. 2) is rather simple. There are only 25 instructions and an arbitrary number of segments that each contain an unlimited amount of bitwise accessible memory, along with an unlimited number of registers assigned to these segments. Each register has a sign and an arbitrary number of bits. This design allows for both high flexibility and independence from hardware restrictions [9]. Of course, one of the challenges of implementing the UVC runtime is to map unbounded memory segments and registers onto current hardware. Figure 3 illustrates a well-tested UVC use case [10], where archived images are stored in compressed JPEG format for storage efficiency. However, compression might impose a barrier to future access. Therefore, a JPEG decoder was developed by IBM as a UVC application. This application decodes the given bitstream from the JPEG into a simple ”pixel-stream”, where each pixel is given by its coordinates and RGB color values. In order to access archived JPEG image in future, only a UVC runtime is needed to decode the JPEG into the simple format, which in turn is rendered using trivial viewer software.
156
N. Krebs, L. Schmitz, and U.M. Borghoff 1101011010111010001 1011011011001011001 0100110001101011010 0101110011010001001 1110011001010110100
decoder
1011000110101011101 1111100000101101000 1110101010100010010 1001010010010100101 0100111110010010111
UVC
... Next x = red Next ...
pixel 57, y = 80, = 30, green = 50, blue = 37 pixel
viewer
JPEG
Fig. 3. Using the UVC to decode images
The Designers of the UVC stated that in order to ensure the practicability of the approach [5] the effort of implementing the UVC runtime would have to be less than a person-year. In Section 3, we will show that actually far less time is needed. The designers offer a proof-of-concept UVC implementation for the PC architecture1 . Although the UVC approach is based on a stable specification, the developers are aware that further refinements may be needed [9] which must not hamper backward-compatibility: There must always be a viable migration path for existing UVC applications. The UVC was initially evaluated in a joint study between IBM and the Koninklijke Bibliotheek of the Netherlands. For that purpose, a UVC runtime was developed in Java [8]. Later IBM published a second, about four times slower implementation written in C++. All these implementations were designed for current computer architectures. This to some degree supports the claim of UVC’s universality, because Java Virtual Machines and C++ compilers exist for many different computer architectures, albeit only for current ones.
3
Evaluation Experiments
The primary aims of our evaluation were to investigate the universality, simplicity and self-containment of the UVC from a practical perspective. Here, universality means that the specification can be implemented on any hardware architecture at any time. Unfortunately, we cannot foresee future developments. However, we can look back on nearly 60 years of computer history that have produced a great variety of architectures and principles. Therefore, our working hypothesis is that if the UVC can be implemented on different systems from different computer ages without any problems, we may safely assume this to be true for future systems, as well. For our work, the datArena computer museum has provided us with a sampler of outdated, but still fully operational computer systems, including supercomputers, vector and parallel computers, as well as desktop computers. We have developed UVC runtime environments for three different supercomputers. The first was a Cyber 180/960-31 introduced 1988 by Control Data Corporation (CDC). We let the system always run in pure CYBER 170 State, which is backward compatible with the CDC 6600, launched in 1964, beginning the era of supercomputers [3]. The second machine was a CDC 4680, introduced 1
http://www.alphaworks.ibm.com/tech/uvc
Implementing the Universal Virtual Computer
157
in 1990 as a widespread successor to the Cyber. The newest supercomputer we used was a SUN Enterprise 10000, introduced in 1997. In order to close the remaining time gap we have also developed UVC runtimes for current Windows and Linux systems. Thus, our evaluation covers a history of almost 50 years. As a benchmark application we have used the UVC-based JPEG decoder described above to recover the bit stream of a 16x16 pixel color picture. Also, when implementing the UVC runtime environments we have made a conscious effort to employ contemporary programming models and software technology. 3.1
CDC Cyber 180/960-31
The first UVC runtime was implemented on the Cyber in FORTRAN IV. As FORTRAN IV offers no dynamic use of memory, we were forced to implement an own memory manager for the unbounded data structures involved. Considering the available algorithms around the year 1964, we made our memory manager implementation rely heavily on linked lists based on a static array and free lists. The next challange was that integer arithmetic is automatically mapped to the corresponding floating point operations. In order to obtain predictable results [4], integer multiplication and division must not use more than 48 bits of the 60-bit words. For efficiency reasons, bits had to be shifted around a good deal and the code contains a huge number of GOTOs. This not only effects the UVC commands MULT and DIV, it also bears on UVC register representation in memory. The specification of the UVC prescribes UVC-applications to be binary files, but with FORTRAN IV all inputs and outputs are record oriented. Thus, for the benchmark the JPEG decoder had to be reformatted. In total this implementation uses 4160 lines of code (LOC) and was completed by one skilled programmer within six weeks. Running on top of this UVC runtime, the IBM JPEG decoder took 28 minutes to produce output for a 16x16 picture. Since the Cyber is about three times faster than the CDC 6600, in 1964 the running time would have been about 90 minutes. 3.2
CDC 4680
The operating system EP/IX of the CDC 4680 provides a tool for migrating FORTRAN IV programs written for Cyber systems to FORTRAN 77. Using that tool and its manual, we ported the tool within two days. Only minor changes were necessary: switch to the provided INTEGER*8 type and adapt to the new names of intrinsic functions. The ported version uses 4291 LOC and produces the output of the 16x16 picture whithin 29 minutes. This was somewhat disappointing considering that the CDC 4680 should be at least three times faster than the Cyber. Therefore, we decided to reimplement the UVC runtime from the scratch, using the fast 32 bits provided by the hardware, and the extensions like STRUCTURE and POINTER provided by the FORTRAN 77 compiler. This was done by the same programmer, who was familar with the previous implementation and produced
158
N. Krebs, L. Schmitz, and U.M. Borghoff
the new UVC runtime using 4702 LOC within 4 weeks. As a result all GOTOs were gone and the benchmark was completed within 66 seconds. Because the FORTRAN 77 compiler provides pointers and a fast memory manager, we decided to employ the B-Trees invented in 1970 [1]. The UVC must be able to hold an arbitrary number of registers and memory blocks. Minor changes at these points in the code and 401 additional LOCs for the B-Tree implementation resulted in a much improved running time of only 11 seconds. 3.3
SUN Enterprise 10000
The SUN machine gains speed by parallel execution. Since the UVC is a sequential machine, parallelism could only be used to speed up arithmetic operations. This, however, would not seem to improve present UVC applications. For this machine, we have ported both the FORTRAN IV implementation and the FORTRAN 77 implementation. Each port was produced within a few hours. The ported UVC runtimes completed the benchmark in 14.3 and in 0.91 seconds, respectively. A re-implementation using the available 64 bit words to full advantage is currently under way. First results, however, indicate no surprising speed advantages. Using some extensions provided by the installed FORTRAN 95 compiler (later included in the FORTRAN 2003 standard), binary coded UVC applications can now be loaded in compliance to the UVC specification. 3.4
Implementation for Personal Computers
For current personal computers, IBM’s Java and C++ based UVC runtimes are available. In order to obtain comparable results we have reimplemented UVC for the same platforms in an imperative language which is closer to FORTRAN: Ada83 does not supply programmers with useful libraries providing lists, trees or multiprecision arithmetic. Therefore, the Ada code covers the same parts as the FORTRAN implementations. Comparing IBM’s runtimes with our own – all of them running on the same system with the same picture – our UVC runtime is more than two times faster than the Java based and eight times faster than the C++ based runtime: It produces the output for the 16x16 picture within 0.29 seconds.
4
Lessons Learned
While our experiments cannot prove the universality of the UVC, they have shown for the first time that the UVC runtime is implementable on different hardware architectures. The UVC execution speed depends both on the underlying hardware and the operating system, in particular on fast memory access and dynamic allocation. If these features are not supported by the implementation language, this will lead to another dependency and make it more difficult to achieve conformance with the UVC specification. Two examples are FORTRAN IV’s static memory model and its record-based input/output.
Implementing the Universal Virtual Computer Segmented Memory
159
Representation of Registers Multiprecision Arithmetic
Instruction Set, Processor Input/Output Memory Management
Initial Constants and Program Loading
Fig. 4. The distribution of development time on the parts of the UVC architecture
Our evaluation shows that the implementation time obviously does not depend on the hardware or on the implentation language used: In all cases a full working runtime was completed within six weeks. This information will help archivists to calculate the cost of the UVC archiving approach. In passing note that a ported runtime is a quick solution to keep access functionality. But in order to really benefit from faster hardware it is often worthwhile to do a complete reimplementation. Figure 4 shows the development time assigned to specific components of a runtime implementation. Both items on the right hand side offer potential to shorten development time significantly: The UVC specification allows for different representations of numbers within registers by allowing leading zeros. This made direct use of FORTRAN multiprecision arithmetic packages impossible. A wrapper implementation would require additional code to permanently copy and convert register contents. Therefore, we propose to tighten the UVC specification and allow only one standard representation of numbers. This would reduce both implementation effort and the probability of implementation errors significantly. As a second opportunity for improving the UVC specification we have identified the loading of initial constants. No one really needs this feature, whose effect could be achieved by using the command LOADC at the beginning of the code. Therefore, we propose to remove this feature from the UVC specification. Concerning UVC execution we observe a correlation between implementation effort and execution speed: By holding all registers of a segment in a tree instead of a linked list we have gained a speed-up of six. It is not clear whether the specification allows for self-modifing code. If not, then an additional speed-up may be achieved by parsing the code of an UVC application and align the command’s bitcodes in memory. Currently we are working on a new UVC specification proposal which combines the above suggestions concerning the UVC runtime with experiences developing UVC applications [6]. The execution speed of UVC applications could be improved by providing a few additional commands, whose implementation needs only a few lines of code each, e.g. commands that accept a constant value instead of a register.
5
Conclusion and Future Work
In order to show that the UVC specification is universal, we have implemented the UVC runtime environment on many different architectures, ranging from
160
N. Krebs, L. Schmitz, and U.M. Borghoff
1960s supercomputer architectures to today’s PCs. We have learned that the implementation is an easy task for a skilled programmer, taking six weeks at most. During our evaluation, we have identified some potential, wich will lead to an improved UVC specification. We have discussed several improvements for the three UVC aspects: implementing a UVC runtime in short time, improving its efficiency, and reducing the effort required for developing UVC applications. Our enhanced specification will allow for shorter development times and faster program execution, thus enabling the development of more complex applications. One important challenge has not been addresses as yet: interactivity. Archives should be prepared to preserve active documents. When offering interactivity on a larger scale, parallel processes become an important issue. Lorie has proposed to link different UVC instances to allow for parallel processing [7]. E.g., when creating many instances of the JPEG decoder, every instance could process a separate part of the image. However, parallel programming makes it hard to ensure deterministic behavior and the authentic preservation of an interactive object. We will focus on these issues, too. Finally, we consider the shift from the UVC interpreter towards a compiler, which again will increase execution speed. Preliminary results include a JPEG viewer wich is six times faster than its interpreted counterpart. Future work will also include the development of more user-friendly UVC development tools.
References 1. Bayer, R., McCreight, E.: Organization and maintenance of large ordered indices. In: ACM SIGFIDET, pp. 107–141. ACM, New York (1970) 2. Borghoff, U.M., R¨ odig, P., Scheffczyk, J., Schmitz, L.: Long-Term Preservation of Digital Documents: Principles and Practices. Springer, Heidelberg (2006) 3. Ceruzzi, P.E.: A History of Modern Computing. MIT Press, Cambridge (2003) 4. Control Data Corporation: FORTRAN Extended Version 4, Reference Manual, Rev. J, Sunnyvale, California (1983) 5. Gladney, H.M., Lorie, R.A.: Trustworthy 100-Year Digital Objects: Durable Encoding for When It’s Too Late to Ask. ACM Trans. Inf. Syst. 23(3), 299–324 (2005) 6. Krebs, N., R¨ onnau, S., Borghoff, U.M.: Fostering the Universal Virtual Computer as Long-Term Preservation Platform. In: ECBS 2011, pp. 105–110 (2011) 7. Lorie, R.A.: Long term preservation of digital information. In: JCDL 2001, pp. 346–352. ACM, New York (2001) 8. Lorie, R.A.: The UVC: A Method for Preserving Digital Documents: Proof of Concept. Technical report, IBM and Koninklijke Bibliotheek (2002) 9. Lorie, R.A., van Diessen, R.J.: UVC: A universal virtual computer for long-term preservation of digital information. Technical report, IBM Res. rep. RJ 10338 (2005) 10. Oltmans, E., van Wijngaarden, H.: Digital preservation in practice: the e-Depot at the Koninklijke Bibliotheek. VINE 32(6), 21–26 (2004)
