From Tuples to Files: a Fast Transactional System Store and File System

Abstract

Traditional file systems are designed to store a moderate number of large objects. However, an increasing number of applications need also to store a large number of interrelated smaller objects, to query and update these objects and their relationships, and to maintain consistency and recoverability. Current approaches require applications to interact with multiple interfaces for different data types, making it difficult for programmers to develop error-free, efficient, and portable applications. Researchers have tried to solve this problem by using additional layers of abstraction to unify these disparate interfaces but continue to use traditional storage formats and algorithms that are optimized only for specific workloads. We have built a transactional system store that can efficiently manage a continuum of interrelated objects from small to large. Our system is based on a data structure, the VT-tree, which is an extension of the log-structured merge-tree data structure (LSM). In this thesis we describe a transactional system store design and implementation that supports high levels of concurrency and larger-than-RAM snapshot-based transactions. We then describe the design of a new transactional file system, KVFS, which is based on our transactional VT-tree. In our system, applications can perform key-value storage and POSIX file operations in the same ACID system transaction, providing support for operations such as file indexing and tagging, meta-data search, and package installation---all in a generic and flexible manner. Our experiments indicate that KVFS's performance is comparable to that of existing native file systems and its elegant transactional interface adds a minimal overhead and supports highly concurrent transactions.