Reinventing Computing
Anyway, recently I attended a talk by Microsoft Technical Fellow and computing architecture and HPC guru, Burton Smith. The talk was very interesting though not specifically because of the subject matter (which is familiar), but because of the broad perspective. The premise is the same thing I’ve been hearing in every other talk for the past few years (which I’ve ranted about before) — namely, parallelism is going mainstream and we have to deal with it. Most people then follow with their specific sales pitch, but Burton followed with a broad overview of many different views of the issue and a wide range of different techniques and technologies.

One thing I really liked about his overview is that he shares my philosophy of “pragmatism over dogmatism.” He discussed potential alternatives and stated that there is a need to utilize a variety of different solutions, even those that are often seen as mutually opposing philosophies: for example, he said he believed that we need both message passing AND shared state concurrency, mutable state with transactions AND immutable functional objects, declarative programming AND imperative programming, etc. This is in contract to presentations where you hear that approach X is the “right” way forward. While it’s conceptually appealing to say that one approach is uniformly better, real world constraints often limit practical applicability — we probably need a tool chest, not just one really fancy hammer.

One thing he mentioned was the concept of viewing resource allocation in multi-core systems as a “2D bin packing” problem. He showed a view of traditional CPU scheduling as a one-dimensional problem of time-multiplexing single runnable kernel threads over each single processor (with a small number of processors total). He then showed the alternate view as a two dimensional problem of assigning chunks of processors over time to applications (i.e. time is the X-axis and processors form the Y-axis). This is reminiscent of gang scheduling or co-scheduling, except the internal scheduling of work within an application with a chunk of processors would be handled at the user level and kernel level time-slicing may not occur in a standard manner anymore. This reminded me of several pieces of current (and classical) related work.

One idea is the following: “why time slice at all on massively multi-core systems?” If you have 256 processors, just dynamically assign chunks of them like spatial multiplexing of memory.

Corey
Corey, a research OS for many-core systems, will be presented at OSDI this year (OSDI 08 program) and follows this principle — “Corey allocates physical cores to applications rather than presenting a time-shared virtual processor abstraction.” Another related concept is that Corey also allows the allocation of dedicated kernel cores for running the kernel, so kernel calls are handled “via fast shared-memory IPC rather than slow traps.” I remember Rik Farrow suggested this same idea in a 2006 Google Tech Talk titled “Security is Broken”. Corey is organized like an earlier OS called an Exokernel (and in fact M. Frans Kaashoek and his PDOS group are involved in both). Like the Exokernel, Corey delegates scheduling policy of an allocated set of cores to the “library operating systems.” This basically amounts to the same thing as user-level scheduling in a normal OS, since the library OS generally runs within the address space of an application (the authors note that the library OS doesn’t need to further isolate itself from the application because the exokernel doesn’t trust the library OS).

User-level Scheduling
One thing that the idea of user-level scheduling reminds me of is the infamous two-level scheduling of Scheduler Activations or Solaris M:N threading. The goals were similar: in theory, at the user (application) level, you can make better scheduling decisions via custom scheduling policies or just better information, and scheduling is also cheaper. In practice, the extra complexity didn’t pay off and the two levels of scheduling (at kernel and user) level often interacted in negative ways. Subtle interference between decisions made at different levels could cause significant and often unexpected performance issues, and to really take advantage of it, you needed to make sure that both levels of scheduling were not working at cross purposes — to do that really requires propagating the application level scheduling decision information to the kernel level scheduler too, which is messy and complicated.

So although, at first glance, the phrase “user-level scheduling” appearing on the slides brought the aforementioned black eye to my mind, I think the situation in the case of Corey and the kind of system Burton is proposing will be different because it’s really not the same kind of combination. In these scenarios, we have a lot more cores and the kernel level “scheduling” is at much longer time scales — instead of time slicing, it’s easier to think of it like allocating physical page frames of memory to processes’ address spaces. Of course, the allocated number of cores is less “transparent” than virtual memory, but model of holding on to a resource is more similar than time slicing over a small number of processors.

Concurrency Runtime
Another project that this brought to mind is Microsoft’s Concurrency Runtime (not to be confused with Microsoft’s similarly named Concurrency and Coordination Runtime). Not only is it related to the idea of user-mode domain-specific scheduling for multi-core applications, but it is also designed to allow different concurrency solutions to work together (thus supporting Burton’s view of utilizing many solutions with different strengths, potentially in the same program). The idea behind the Concurrency Runtime is providing a common resource management framework to allow various concurrency solutions to interoperate and “play nice” together. One problem with current solutions likes OpenMP or Intel TBB, etc. is that they all think they “own the machine.” If you want to use multiple solutions together, they interact poorly because they are all oblivious of each other. The concurrency runtime provides a user-mode common resource management framework underneath the various concurrency solutions which can arbitrate between these different requests. It also provides a bunch of richer primitives for building these solutions (i.e. higher level concepts of tasks, groups, events, thread pools, etc.), but I can’t find too much documentation on it so far (most of the references are in the form of interviews and presentations).

Snake Oil
At the beginning of this post, when I was talking about how Burton Smith’s talk and approach are different than what you usually hear, it reminded me of a recent post by Sun engineer Bryan Cantrill (an outspoken, frequently provocative* fellow — as an aside, his chapter in Beautiful Code was one of the most enjoyable to me, coming from a systems background). Anyway, as I mentioned, most talks follow the common “concurrency is here, we must deal with it” introduction with a sales pitch for a specific tool. One hot area currently is transactional memory, which can be implemented in hardware or software, and I’ve seen and read a lot of papers on this in the past few years. Anyway, Bryan recently posted a scathing post about transactional memory titled “Concurrency’s Shysters”. I think this kind of critique comes as an inevitable backlash against over-hyped/newly in vogue solutions and the dogmatic selling of some technique as a the fix for what ails you. Of course, for the sake of marketing, it’s very hard to present a nuanced view and still be convincing, so maybe it’s out of necessity, but it’s off-putting for people who feel like the presentation of a technology is unbalanced and unrealistic.

* Bryan Cantrill is infamous for the “Have you ever kissed a girl?” Usenet quip in historical Solaris v. Linux performance wars, and more recently for dumping on “Dreaming in Code” in a Google Tech Talk on DTrace. That’s random trivia, but I quite enjoy watching strongly opinionated and outspoken technical people duke it out.

BTW, if you like following all of the various concurrency solutions (which are popping up fast and furious), you might want to check out Alex Miller’s Concurrency feed.

I’ve spent most of my past seven or so months working on my thesis proposal and preparing a conference submission. On the topic of CS conferences in my area (Systems), I wanted to highlight a USENIX-sposored meta-workshop I found serendipitously — WOWCS ‘08: Workshop on Organizing Workshops, Conferences, and Symposia for Computer Systems. The WOWCS 08 PC and accepted authors is a list of seasoned veterans in systems research. Given that, and the improvement-based focus of the venue, many papers detail a lot of what some people see as “broken” in current systems academic venues (reviews, PC meetings, etc.).

One paper in particular that somewhat confirmed some disheartening truths about the nature of conference and workshop reviews is Overcoming Challenges of Maturity by Ken Birman — Ken is an ACM Fellow, well known for his work in systems and networking. Some of his gripes are from his experience chairing SOSP (in 2005), which is one of the most prestigious (and oldest) systems venues. Ken said,

Overwhelmed by the huge numbers of submissions, most PCs have turned to multi-round processes in which the first-round reviews are farmed out, often to students who may do an erratic reviewing job.
…
Most of us are learning to write papers in a manner calculated to appear to those beleaguered first-round reviewers. To get into SOSP or SIGCOMM a paper has to survive two thresholds: it must get past the two randomly selected students, and then must get past the six or so PC members who are most knowledgeable about the topic.
…
a PC chair today assigns some paper to PC member X, who then randomly hands it to students Y and Z, producing completely random reviews from people who have never been a part of the community and who are naturally inclined to be overly critical and to overly favor work in their own areas of interest: our mature researchers have long since shed these flaws of youth.

The above content encourages somewhat cynical views of publishing in such academic venues. Ken points out that the quality of the top conferences isn’t really diminished by these schizophrenic, semi-random first round eliminations because there are enough good papers remaining to fill the program. But it’s still somewhat unfair and very frustrating to authors. He also said,

Who hasn’t had papers that were rejected in the first round of reviews at a top conference, with just two reviews, one or both of which seemed almost completely clueless? Who hasn’t expressed anger at the system? Here are two little “factoids” to illustrate the depth of the issue: when I sent out the SOSP reviews, we discovered that in one case, a rejected paper had missed the initial cut on the basis of a review that was clearly written about some other paper.

Anyway, I’ve had pretty good experiences so far with my current primary research work, but some projects I’ve collaborated on as a secondary/advisory participant have experienced treatment like that (”completely clueless” reviews). Sometimes it could be chalked up to clarity issues in the paper, but other times it left me wondering if certain reviewers actually read the paper or just the abstract. Oh well… it helps to know that even highly-regarded, established researchers experience this sort of thing too.

Some higher-level languages — most notably Java — have defined language-level memory models: Java’s memory model defines how threads interact through memory and precisely what it means for a program to be data race free / well synchronized. It provides guarantees that a compiler/JIT won’t perform optimizations that break race free code. In a language like C without a defined memory model, these things can be tricky and surprising because it is unclear what a data race means when considering the mapping from language level statements to actual machine code. A programmer may have two independent threads concurrently assigning values to two different variables. If these variables are small (chars, for example) and stored adjacently in the same machine word, this may be a data race on some architectures. Since these kinds of decisions are often left unspecified and up to the compiler (or possibly the linker), there is no guaranteed way to write portable and robust code. Additionally, the compiler may introduce extra stores or perform other optimizations that, while fine for single-threaded code, introduce races into otherwise race-free code. People like me, who (try to) make use of lock-free algorithms and atomic operations, end up with code that may be quite fragile to even minor compiler optimization changes. More on this later.

One other nice thing about Java’s memory model is that it also constrains the language implementation on what can happen to incorrectly synchronized code; languages like C and C++ often say that the result of illegal code (e.g. modifying a variable twice without an intervening sequence point) is undefined — and undefined behavior can allow anything at all to happen. As one example, Java specifies that the implementation cannot introduce values into improperly synchronized code that appear “out of thin air.”

The videos are as follows:

• IA Memory Ordering — Richard Hudson explains Intel’s newly clarified memory ordering semantics for x86.
• Advanced Topics in Programming Languages: The Java Memory Model — Jeremy Manson describes the current Java memory model as revised by JSR-133 and Java thread semantics. The talk covers basics about model ordering guarantees, the meaning of locking/synchronization primitives and volatile, as well as common pitfalls.
• Getting C++ Threads Right — Hans Boehm, the well-known programming languages/compilers researcher, talks about the effort to provide better threads support in the upcoming C++ standard (C++0x). More importantly, he talks about the general problems plaguing implementation of correct multi-threaded programs in languages like C and C++.
• Towards a Memory Model for C++ — Not a Google Tech Talk, but another talk by Hans Boehm very similar to that above.

C++ Threads / Memory Model
Hans Boehm covered similar ground in his 2005 PLDI paper, “Threads Cannot be Implemented as a Library”. Some people felt the paper was trivial or hyping a non-problem by saying that the compiler and language have to provide a few extra guarantees and can’t be completely oblivious (I recall the LTU discussion — one commenter said, “Ayone who was paying attention already knew that”). But he does bring up a whole set of issues which is becoming increasingly important. Concurrent programming is already more difficult than regular sequential programming. On top of that, you have poorly specified semantics or broken implementations which actually cause problems. It’s hard enough already to get your part right without worrying about the compiler breaking things behind your back or the underlying language implementation not properly obeying the specification. His concerns aren’t just theoretical “cleanliness” issues, either.

Part of what prompted this post is the recent thread on LKML: “Is gcc thread-unsafe?” This is a perfect example of the problem that Boehm alludes to with regard to compiler optimization. The sample code in the gcc thread:

int trylock() {
  int res;

  res = pthread_mutex_trylock(&mutex);
  if (res == 0)
    ++acquires_count;

   return res;
}

That code attempts to acquire the mutex and increments acquires_count only if it succeeded in locking the mutex. With -O1, gcc 4.3 generated code that always reads and writes the acquires_count variable (load, conditional add, store) regardless of whether the mutex is obtained. Typical language lawyers pointed out that the C standard allows this optimization, even though it introduces a fairly nasty race condition. Some of the gcc developers tend to take a very defensive language lawyer stance, vigorously defending things that are technically permitted but practically useless. Standard C says nothing of threads and imposes very little on the compiler in this regard.

The Java Memory Model
The Java Memory Model is quite useful today, but it wasn’t always perfect. Java 5 incorporated JSR 133: Java Memory Model and Thread Specification Revision — a revision making some substantial changes to the original Java Memory Model, which was regarded as “broken.” I was still in high school and hadn’t even taken my first CS course when William Pugh (also known for the invention of the skip list, a wonderful data structure that works well in concurrent situations), published a paper titled The Java Memory Model is Fatally Flawed, a revised version of Fixing the Java Memory Model from a year earlier. I learned about the controversy a year or two later, and I was finishing my Masters by the time the new, fixed memory model was finalized and adopted. Although it seems somewhat strange that people tolerated and obviously wrote multi-threaded Java with a “fatally flawed” memory model for so long, the situation is roughly analogous to the current situation with C/C++ and threads: generally stuff works, and compilers/runtimes often do the “right thing” anyway, but we’d rather have stronger guarantees in this area.

Double-checked locking, a trick typically used to avoid synchronization on lazily-initialized fields, is one of the things broken by Java’s old memory model. Here is an example:

public Object getA() {
  if(a == null) {
    synchronized(this) {
      if(a == null)
        a = new Object();
    }
  }
  return a;
}

Despite the fact that the idiom was commonly-used within some Java libraries, it was technically incorrect and there was no satisfactory way to fix it (using thread-local storage made it possible, but that was a hack and often too expensive to be worth it). Under the current Java Memory Model, double-checked locking works if the field is made volatile. Under the old memory model, volatile was not very useful because volatile reads and writes could be reordered with respect to regular reads and writes, so you couldn’t use a write to a volatile value to, for instance, indicate to another thread that an object was initialized (because the write to signal the initialization might be reordered to before the actual initialization). The new memory model prevents this reordering. Since there are a lot of resources available on the web about the Java Memory Model, I won’t say too much more except that Java has been somewhat of a trailblazer in the area of really providing clear and useful semantics for multi-threaded programs.

William Pugh has a fairly comprehensive page with Java Memory Model resources and Doug Lea has a JSR-133 Cookbook for Compiler Writers and a tutorial titled Synchronization and the Java Memory Model (the latter is an excerpt from Doug’s nice Concurrent Programming in Java book).

IA Memory Ordering
x86 (IA32) has used, in the past, fairly strong memory ordering semantics called “processor ordering.” Intel has clarified/more robustly specified the memory ordering semantics in a document titled Intel® 64 Architecture Memory Ordering White Paper. The newly clarified memory ordering semantics apply to both 32-bit and 64-bit x86 and are described as “Total Lock Ordering + Causal Consistency.” Locked instructions are totally ordered across all processors, and memory obeys causal consistency, which is a fairly natural concept and provides publication safety. These memory ordering semantics don’t apply to all memory types (I/O would be different), but these semantics are what would be encountered in regular user code dealing with standard heap or stack memory. Precisely knowing the target architecture’s memory consistency guarantees is critical to correct and efficient implementation of higher-level language memory models and systems software (and lock-free code). AMD also has their own memory ordering reference which is very close or identical to Intel’s.
A long time ago, I read a paper titled Memory Ordering in Modern Microprocessors by Paul McKenney which explains the memory ordering guarantees provided by many modern microprocessor architectures (in the context of Linux’s memory barrier primitives). It is interesting just how weak Alpha’s (and to some extent PowerPC’s) ordering guarantees can be. Alpha seems to allow just about everything short of the processor simply making up bogus values for memory reads.

The Binomial distribution with p=0.5 over [0, N] looks approximately normal when N becomes very large, so I decided to generate binomially-distributed integers. Even when there isn’t a simple, closed-form transform from uniformly random integers to another distribution, it seemed like you could just generate the cumulative distribution function; then you generate a uniform random number u_1 and then binary search for the greatest index i where cdf(i) <= u_1 (basically inverting the cdf). So my plan was to generate the probability mass function straight from the definition and then sum it to get the cdf. With 128-bit floating point and care in implementing the binomial coefficient (i.e. don’t perform factorials directly and divide, evaluate it a non-canceled term at a time), this worked fine for my tests the range of a few thousand. However, I quickly ran into trouble since N=72,000 in the real test. The definition of the pmf is:

Obviously you can see where this is going. That definition works somewhat well for smaller integers, but once you have quantities like 0.5^(72000), you can easily get underflows and overflows. I tried some term rearrangement, but I still ended up with a mass function that was +Inf in the middle and 0 everywhere else, which is completely worthless. Not wanting to spend a lot of time implementing this, I figured GNU R — the excellent statistical computing language/environment — could probably do this. So I ended up using R to generate the cdf (with the pbinom function), and wrote the results out to a file which I read in when I needed the cdf.

Later I looked into how R actually implements pbinom, and it delegates the problem to the Beta distribution cdf implementation, pbeta. The work in pbeta is all done by a function called bratio, which is the interesting part. The bratio function is in toms708.c. A comment at the top explains the name:

/*      ALGORITHM 708, COLLECTED ALGORITHMS FROM ACM.
 *      This work published in  Transactions On Mathematical Software,
 *      vol. 18, no. 3, September 1992, pp. 360-373z.
 */

Sure enough, you can locate “Algorithm 708; significant digit computation of the incomplete beta function ratios” in ACM’s digital library. The the bratio function computes the Incomplete beta function, and toms708.c is almost 2300 lines of fairly scary looking numerics code (lots of gotos and labels, and a fair number of magic looking constants). The original code from ACM TOMS was in Fortran, and the top of the source file says “Based on C translation of ACM TOMS 708.” They don’t explicitly say whether it was manual or if machine translation was involved at any point, but I found several comments above some variable declarations that says “System generated locals”. Googling on that phrase brings up references to f2c, the Fortran to C translator. That probably also explains the abundance of labels with names corresponding to line numbers in the original Fortran source.

Anyway, I just thought this was interesting. Numerical algorithms are fascinating: they can take a lot of care to get right and often require complex tricks to avoid over/under-flows and maintain numerical stability. Just working with floating point is perilous on its own, because there are things like cancellation errors, non-associativity of operations, non-intuitive notions of equality, etc. This is one area that wasn’t well covered in my undergraduate classes (although it might have been covered in graphics classes, which I never took). In the introductory systems class, they discussed floating point representations and had us encode/decode some numbers from scientific notation into IEEE 754, but no class really went into the pragmatics and pitfalls of using floating point numbers in mathematical algorithms. I learned about that mostly a) from reading the classic report, “What Every Computer Scientist Should Know About Floating-Point Arithmetic” and later; b) from working at the Federal Reserve in Economic Research, helping economists make use of distributed computing with their simulations and such.

Through my experience at the Fed, I gained a lot of insight into just how much effort it takes to do correct numerical algorithms, let alone fast ones. Of course, most of the economists relied on highly-tuned primitive libraries like IMSL, but it still takes work to make sure your own calculations (built on top of those primitives) don’t introduce accumulating error. Fortran and Matlab were, by far, the most popular languages, with a few economists choosing C or Mathematica. Anyway, this is a deep and interesting area.

As an aside, it’s not too hard to generate Zipf distributed numbers using a technique called “Rejection-Inversion” (it doesn’t require generating the cdf and searching). Take a look at FastBit which has an implementation in twister.h

/// Discrete Zipf distribution: p(k) is proportional to (v+k)^(-a) where a
/// > 1, k >= 0.  It uses the rejection-inversion algorithm of W. Hormann
/// and G. Derflinger.  The values generated are in the range of [0, imax]
/// (inclusive, both ends are included).

The referenced paper is “Rejection-inversion to generate variates from monotone discrete distributions”; the code to generate Zipf-distributed integers is only about 20 lines of C++, and it’s actually understandable.

“The Causes of Bloat, The Limits of Health”

The first paper I mentioned is “The Causes of Bloat, The Limits of Health” (Nick Mitchell and Gary Sevitsky) from OOPSLA ‘07. I found this paper interesting for a few reasons, but it shows how application design choices in mapping a data model onto concrete data structures may lead to chronically wasteful memory usage where the amount of real data is overwhelmed by collection metadata/overhead (like pointers). The paper explores these issues in the context of Java and Java applications; the basic premise is applicable in any language, but things like Java’s object headers exacerbate the problem when compared to a comparable data structure in C. A key metric here, collection health, is a comparison of actual application data to overhead imposed by object headers, collection metadata, pointers, etc. One example they provide is putting Java Strings in collections:

Observe that a String must have at least 140 characters in order to achieve an S < 1.2 (i.e. no more than 80% actual data). When Strings are placed in a standard Java HashSet, they must have at least 270 characters to achieve this level of health. On the flip side, placing 10-character Strings into a HashSet will result in an S of no less than 3.7 (i.e. no more than 27% actual data), no matter how many Strings are placed into the HashSet.

They define “good health” for a data structure as having at least an 80% real data (actually, less than 20% overhead/data ratio). Even for a single string to qualify, it needs to at least 140 characters, because a String is actually fairly complicated under the hood: a String is an Object and has an object header and some fields (a cached hash code, a length and potentially an offset into the backing array), as well as a pointer to a char[], which also has a header (containing the length and other VM bookkeeping information). Ignoring the String fields, they put the overhead at 28 bytes per String, and they note that, in some cases, object headers could be up-to 20 bytes a piece.

Some of the applications they tested had less than 20% actual data due to inappropriate choice of Java collections or container objects. It’s not just about choosing the wrong data structure or having the default collection size be too big, it’s also about choosing the wrong object “containers” to hold values (decisions that may seem inconsequential). Java doesn’t have tuples (a huge gripe of mine), and one application needed to hash a pair of items, an int and some other Object. The developer chose java.util.Arrays$ArrayList, an inner-class of java.util.Arrays used in the asList method, because it was convenient to specify literal values by making a simple call like this: Arrays.asList(new Object[] {1, obj}). You couldn’t necessarily just put the Object array in the collection directly because array equality is object identity rather than content equality (i.e. it just compares the addresses of the arrays). Anyway, using Arrays$ArrayList causes a lot of overhead because the ArrayList is itself an Object with attributes and it also keeps a separate backing array (which has its own header overhead). Since the backing array is an Object array, storing an int actually requires boxing the primitive int into an Integer object wrapper, which is even more overhead, and every little bit adds up when you multiply it by the number of these Arrays$ArrayList objects you will create. If the author had instead created a Pair class with an int field and an Object field, it would reduce the memory usage of all of these (int, Object) pairs in this application to about a third of the original size (1.7MB versus 4.9MB).

This kind of study makes you think a little more carefully about choices you might take for granted or overheads you might dismiss as negligible. I know in the past I’ve done things like the above scenario with ArrayList (although not using that exact class) — but simply taking an existing Java class already present in the API that is close to what I want rather than making my own trivial two element wrapper class to work around the lack of tuples. If it’s a one-off it won’t be a big deal, but if it goes in a collection with many like objects, eventually the overhead may become unreasonable. This paper also reminded me of the sometimes significant size overhead of Java objects compared to plain data, and that’s good to keep in mind. Now don’t get me wrong, I’m not one of those systems programmers who likes to beat his chest and loudly proclaim that Java is slow and for n00bs and that we should all be coding everything in hand-tuned assembly language. I spend a lot of my time coding in C, and when I end up doing stuff in Java, it’s often a breath of fresh air (except when I need unsigned integers). Even in a high-level language, though, I personally like knowing what’s going on underneath.

“Quantifying the Performance of Garbage Collection vs. Explicit Memory Management”

The second paper is “Quantifying the Performance of Garbage Collection vs. Explicit Memory Management” (Matthew Hertz and Emery D. Berger) from OOPSLA ‘05. A post on Lambda The Ultimate a few months ago referenced this paper. I had heard the conclusion of the paper cited before, but I’d never actually read the paper so I put it in my reading queue. The authors note that the garbage collection process visits more pages of memory than an application would using explicit allocation. This is bad for locality and increases pressure at many levels of the memory hierarchy. In the presence of physical memory pressure and demand-paged virtual memory, it causes significantly more paging overhead which is very expensive (it can cause an order of magnitude performance degradation). In addition, the authors note that you need a larger heap to achieve performance parity with manual allocation because smaller heaps increase (internal) memory pressure, which leads to more frequent collections, which has baseline overhead and of course leads to visiting a lot of pages. With 5x as much memory, the garbage collector will generally equal or surpass manually allocation, and 3x memory gives an average 17% performance penalty over manual allocation, while lower factors degrade significantly.

Again, I’m not one of those “macho programmers” who likes to complain about garbage collection being too expensive and never appropriate. I think managed languages and garbage collection are really a good thing and even though it’s often more expensive, I don’t think it’s too expensive for most applications. And with the imminent rise in concurrency and parallel programming, the usefulness of garbage collection is even greater; manual storage management in the face of concurrency is often even more painful — a lot of novel concurrent data structures just assume the presence of garbage collection. There’s always programming features that we eventually take for granted which people gripe about being too expensive at one time: people complained about the overhead of operating systems (versus running applications on base metal), and later it was compilers/high-level languages that were “too expensive” (versus assembly language). The situation with garbage collection may not be exactly analogous, but current trends seem to indicate that it’ll be a given in due time.

“Reconsidering Custom Memory Allocation”

The last paper is “Reconsidering Custom Memory Allocation” (Emery D. Berger, Benjamin G. Zorn and Kathryn S. McKinley). The authors tested custom memory allocators versus the general purpose Lea allocator, which is just a really good general purpose allocator created by the amazing Doug Lea (who is also one of the people responsible for the fantastic java.util.concurrent package added via JSR 166). The Lea allocator is also used in glibc in a modified form (ptmalloc/ptmalloc2/ptmalloc3, which is basically a Lea allocator enhanced for multi-threaded allocation). The authors test various applications that use custom allocators and found that custom allocators were rarely worth it. In their conclusion, they state:

Despite the widespread belief that custom allocators should be used in order to improve performance, we come to a different conclusion. In this paper, we examine eight benchmarks using custom memory allocators, including the Apache web server and several applications from the SPECint2000 benchmark suite. We find that the Lea allocator is as fast as or even faster than most custom allocators. The exceptions are region-based allocators, which often outperform general-purpose allocation.

The fact that the Lea allocator outperforms most custom allocators isn’t a coincidence, it was an explicit design goal. Doug Lea says on his malloc webpage:

I soon realized that building a special allocator for each new class that tended to be dynamically allocated and heavily used was not a good strategy when building kinds of general-purpose programming support classes I was writing at the time. (From 1986 to 1991, I was the the primary author of libg++ , the GNU C++ library.) A broader solution was needed — to write an allocator that was good enough under normal C++ and C loads so that programmers would not be tempted to write special-purpose allocators except under very special conditions.

I’d say this paper shows that he largely succeeded. But I think the paper’s conclusion is just a classic lesson in optimization. Knuth famously stated, “premature optimization is the root of all evil.” This isn’t to necessarily say that the developers of applications like Apache and gcc were oblivious and guilty of premature optimization. The authors note that this may be a factor of general purpose allocators getting better as well as program evolution. At one time, gcc’s runtime was dominated by parsing, which benefited from the custom allocator; now optimization is where more cycles are spent and so the game has changed. In any event, the take-home message is about future practice; don’t rush in to custom memory allocators just because you think they’re faster. Most applications won’t benefit, and you’ll save yourself a lot of trouble.

When I was more of a novice, I was tempted to perform premature optimization and it rarely paid off in terms of the time investment. Now I’m much more concerned with getting the structure of the system so that it doesn’t impose high overhead in general — a holistic view of the entire system rather than micro-optimizing specific operations. After a while you just get a feel for how you can structure a system so that cumulative overhead is avoided. Obviously part of that is in using the right data structures and algorithms (from an asymptotic complexity standpoint, but also considering constant factors when they matter), but also thinking about things like how data flows through the system, so you can avoid unnecessary copies. After the general structure is there, you profile and you can always micro-optimize hot paths. If you micro-optimize while the structure is still in flux, Murphy’s Law dictates that whatever part you optimize will change to foil you. Anyway, this post is getting quite long, so I just wanted to mention some high-performance, drop-in replacement (general-purpose) malloc implementations:

• ptmalloc3 — newer and faster than ptmalloc2, which is in glibc.
• Hoard — fast allocator developed as a research project at UMass Amherst under Prof. Emery Berger (a co-author of two of the papers I mentioned)
• tcmalloc — Google’s thread caching malloc. At one time it was the fastest out there, but now various others may be equally competitive or faster.
• nedmalloc — supposedly faster than Hoard, ptmalloc2 and tcmalloc.
• libumem — Solaris’s umem allocator, made portable.

For the first few installfests where we used the PXE server, I printed out several copies of a sheet with the various network mirrors for different distros. It would be something like this:

• Red Hat Enterprise Linux 5.1 — http://10.0.0.2/rhel5.1/i386/
• Ubuntu 7.10 (gutsy) — http://10.0.0.2/pub/ubuntu/
• Fedora 8 — http://10.0.0.2/fc8/i386/os/
• Debian testing — ­ http://www.gtlib.gatech.edu/pub/debian/
• openSUSE 10.3 –­ http://128.61.111.11/pub/opensuse/distribution/10.3/repo/oss/
• Mandriva 2008.0 –­ http://www.gtlib.gatech.edu/pub/mandrake/official/2008.0/i586/
• Gentoo 2007.0 — http://10.0.0.2/gentoo (stage3 tarballs)

Now, obviously it was a minor pain to fill in that information repeatedly for each install, so I decided to automate it. For RedHat-based distros, I used a kickstart configuration file to specify the mirror, and for Debian-derived distros I used the installer “preseed” mechanism. Mandriva and openSUSE also provide features to do the same. Here’s how I set up each distro to automatically find the mirrors:

Red Hat Enterprise Linux / Fedora
For each distro version and architecture combination, create a kickstart configuration file available via HTTP. Here’s an example for Fedora 8 i386:

interactive
network --bootproto dhcp --noipv6
url --url http://10.0.0.2/fc8/i386/os/
firstboot --enable

The “interactive” and “firstboot –enable” parts are important because the installer assumes that you are doing a semi- or entirely automated installation if you use a kickstart configuration file. Without the “firstboot” line, it won’t give you the first boot system configuration dialogs where you create a non-root user and configure sound, video, etc. I wanted these installs to be basically identical to a manual install except with the network mirror pre-selected.

Now, all you have to do to use the kickstart config is edit your pxelinux configuration file. Append “ks=url_to_file” lines to the kernel boot parameters of each entry. E.g.:

LABEL fedora8_x86_64
kernel fedora/8/x86_64/vmlinuz
append initrd=fedora/8/x86_64/initrd.img ks=http://10.0.0.2/fc8/x86_64/ks.cfg

Debian / Ubuntu
The Debian installer (which is also used for Ubuntu network and alternate installs) allows you to “preseed” answers to all installer prompts. For each distro, you only need a single file for all version and architecture combinations assuming they all use the same mirror site (versions and architecture are not explicit in the mirror URL). Create a preseed configuration file like this:

d-i mirror/protocol string http
d-i mirror/country string enter information manually
d-i mirror/http/hostname string 10.0.0.2
d-i mirror/http/directory string /pub/ubuntu/
d-i mirror/http/proxy string
d-i apt-setup/security_host string

After creating the preseed file, simply append “preseed/url=url_to_file” to the kernel boot parameters of each entry. E.g.:

LABEL ubuntu_gutsy_i386
kernel ubuntu/gutsy/i386/ubuntu-installer/i386/linux
append vga=normal initrd=ubuntu/gutsy/i386/ubuntu-installer/i386
/initrd.gz preseed/url=http://10.0.0.2/ubuntu/lug.cfg --

One problem with using a local mirror for Ubuntu (10.0.0.2/pub/ubuntu) is that we have to go back and change the /etc/apt/source.list file to point to a public mirror after install — otherwise, after leaving the installfest, a user would be trying to use a non-existent Ubuntu mirror on RFC1918 IP space. I also used the pre-seed configuration file to automatically replace the source.list file after installation. The “preseed/late_command” option allows you to run stuff just before the install finishes (the root of the new system is in /target at this point). Here is the slightly hackish entry I use to fix the sources.list:

d-i preseed/late_command string cd /target/tmp ; wget http://10.0.0.2/ubuntu/fix_sources.sh ; cd /target ; sh tmp/fix_sources.sh

The fix_sources.sh simply replaces the entries in sources.list to point to a public mirror.

One other trick I did with the Ubuntu installer is disable the supremely annoying “Automatic Keyboard layout detection” mechanism. It prompts you “Yes/No,” but the default is Yes, so many people select it unwittingly. The result is a long and irritating process of pressing various keys on the keyboard which could have been solved in 1 second by simply selecting “American English” (99.9% of the time) from the keyboard layout menu. If you append “console-setup/ask_detect=false” to the kernel parameters to the installer image, it will go directly to the keyboard layout menu as if you selected “No” to keyboard autodetection.

openSUSE
The openSUSE installer supports setting the installation mirror source by passing “install=url_to_repository” as a kernel parameter to the install. The mirror path does not change with different architectures (like Debian), but it does change between versions. For example:

LABEL opensuse10.3_i386
kernel opensuse/10.3/i386/linux
append initrd=opensuse/10.3/i386/initrd splash=silent showopts install=http://128.61.111.11/pub/opensuse/distribution/10.3/repo/oss

The url provided uses a numeric IP address rather than a hostname because the openSUSE installer doesn’t handle DNS resolution (at least last time I checked; it may have been fixed in the meantime).

Mandriva
Mandriva supports mirror selection through the “automatic=config_list” kernel parameter, where config_list is a list of comma-separated key/value pairs in the form of “key:value.” To set the mirror, one could specify the string as follows: “automatic=method:http,network:dhcp,server:mirror_hostname,directory:mirror_path.” For example, here is an entry:

LABEL mandriva2008.0_i586
kernel mandriva/2008.0/i586/vmlinuz
append initrd=mandriva/2008.0/i586/all.rdz vga=788 splash=silent automatic=method:http,network:dhcp,server:www.gtlib.gatech.edu,directory:/pub/mandrake/official/2008.0/i586

The full directory above is “/pub/mandrake/official/2008.0/i586,” but fixed-width “code” entries don’t word-wrap without putting extra spaces. Note that Mandriva’s mirror URLs also include both architecture and distro version.

Changing traditional storage layering for distribution
In very large distributed filesystems (getting into the multi petabyte range), traditional RAID is often too weak and constrained for redundancy. By traditional RAID, I mean RAID that is implemented in hardware or software and is effectively invisible to the file system (sits below the block level). Say you had thousands of multi-disk RAID-5 arrays. In this situation, the probability that you’ll lose an entire array at some point is probably going to be make people nervous, particularly the kind of people who would have such massive storage systems. Depending on the scale, you could try to manage with hot spares and round-the-clock IT staff or increase the redundancy by going to RAID-6 or RAID-1, but you still have dangerous and constraining locality in your redundancy. This is more important when distribution is involved: what if you lose a controller or network connection or some other local aspect that takes an entire array effectively offline (or an entire chassis/rack or entire SAN or even an entire datacenter)?

A presentation titled “Storage Challenges for Petascale Systems” given by Dilip D. Kandlur, Director of IBM’s Storage Systems Research talks about these challenges in the context of petaflop systems with tens or even hundreds of petabytes of storage. These systems might have 100k-150k disk drives! The presentation notes:

RAID-5 is dead at petascale; even RAID-6 may not be sufficient to prevent data loss
Simulations of file system size, drive MTBF, failure probability distribution show 4%-28% chance of data loss over five-year lifetime for 8+2P code.

The probability of failure is unacceptably high even with the double parity of RAID-6, but triple parity gives you several orders of magnitude lower mean time to data loss. With these challenges in mind, GPFS is adding software RAID to support such stronger RAID codes not typically supported by RAID controller hardware (triple parity). In addition, it will support what is called “declustered RAID” (see Parity Declustering for Continuous Operation in Redundant Disk Arrays) which significantly improves load balancing during rebuild (see slides 11-13 for a great visual depiction of the way declustered RAID works). See also “The Challenges of Storage System Growth” a presentation by Denis Serenyi of Symantec, which covered some related issues.

The stronger software RAID helps at one level and allows you to using striping for throughput, but it doesn’t really deal with the location-based redundancy. The context of the previous presentation is primarily extremely large but single-site HPC systems, so it doesn’t discuss this issue much, but when you have a distributed system spanning many locations you need to consider it. In principle you could span multiple datacenters with your RAID layout, but that wouldn’t work very well from a performance perspective. RAID works best with symmetric (and predictable) latencies between devices; moreover, it’s unnecessary if you just want to deal with failure, because you can handle it much more robustly at a higher layer. Most large distributed systems provide for redundancy at the level of larger storage granules: files, objects (if you’re using an object-store based system), or perhaps larger blocks or “chunks.” For example, the Google File System (GFS, not to be confused with Red Hat’s Global File System, another distributed filesystem also named GFS) stores files as a series of 64MB chunks and each chunk is replicated. The paper notes the importance of replicating chunks on different racks:

We must also spread chunk replicas across racks. This ensures that some replicas of a chunk will survive and remain available even if an entire rack is damaged or offline (for example, due to failure of a shared resource like a network switch or power circuit). It also means that traffic, especially reads, for a chunk can exploit the aggregate bandwidth of multiple racks. On the other hand, write traffic has to flow through multiple racks, a tradeoff we make willingly.

Ceph, a recent distributed filesystem (see Ceph: A Scalable, High-Performance Distributed File System in OSDI ‘06), uses an underlying object store model and replicates at the level of objects:

In contrast to systems like Lustre [4], which assume one can construct sufficiently reliable OSDs using mechanisms like RAID or fail-over on a SAN, we assume that in a petabyte or exabyte system failure will be the norm rather than the exception, and at any point in time several OSDs are likely to be inoperable. To maintain system availability and ensure data safety in a scalable fashion, RADOS manages its own replication of data using a variant of primary-copy replication [2], while taking steps to minimize the impact on performance. Data is replicated in terms of placement groups, each of which is mapped to an ordered list of n OSDs (for n-way replication).

In Ceph, the data for a traditional file at the filesystem-level may consist of many underlying objects in the object store (a file is striped across objects named by combining an inode and a stripe number), so this is similar to replicating at a “chunk” or large block level. GPFS, in addition to the planned lower level declustered/striped strong parity strategy, already has file data and metadata replication (which can be controlled on a per-file basis). These features are actually a part of a rich set of ILM (Information Lifecycle Management) features that allow you to define different policies for various data on the same filesystem in a SQL-like declarative language. For example, you can create a policy that a certain directory subtree should be stored on a pool of faster disks and have a specific, higher replication factor than other files. Or you could make a policy to have the system gradually decrease the replication factor of files that haven’t been accessed in a long time, finally migrating it to offline, external storage after a certain threshold.

The various DHT-based filesystems/storage systems (CFS, Ivy, PAST, Pastiche, OceanStore, etc.) mentioned in my previous post also replicate pieces of files or entire files on multiple nodes. These systems are designed for more widely distributed and dynamic environments so they have to deal with things like significant node churn (nodes not being powered on/connected all the time or leaving the system permanently); it is critical to adopt a replication strategy that is easy to maintain in such circumstances. In such systems is it also accepted that some files may be temporarily unavailable or lost permanently due to a loss of all replicas — most files will be fine, but you don’t necessarily set replication parameters for losing a given file to the same low probability of failure as RAID type strategies. Note the difference in the nature of redundancy and the reasons for doing so: losing a piece of a file might be bad for the user of a file, but the rest of the filesystem is fine. In the case of RAID, where a coherent filesystem’s data and metadata are striped indiscriminately across several disks, losing all replicas of a block could mean that the filesystem’s metadata is damaged, which could cause serious problems.

Anyway, I just think it’s interesting to note how the traditional storage layering evolves in the face of distribution and large datasets. With local disks and filesystems, people tend to put replication below everything and provide a replicated block device. When distribution is involved, it becomes more flexible to think of replication in the context of filesystem entities like files or chunks.

A violation of layering by DHash
My last post got quite long so I didn’t remember to include every interesting footnote and piece of trivia. One interesting “layering violation” for efficiency in the related work I listed is in DHash, the distributed block storage layer built on top of Chord. The authors of the CFS SOSP paper note:

DHash has its own implementation of the Chord lookup algorithm, but relies on the Chord layer to maintain the routing tables. Integrating block lookup into DHash increases its efficiency. If DHash instead called the Chord find successor routine, it would be awkward for DHash to check each server along the lookup path for cached copies of the desired block. It would also cost an unneeded round trip time, since both Chord and DHash would end up separately contacting the block’s successor server.

That’s obviously a case in which duplicating code and violating layering is a good tradeoff, since it eliminates a costly network round trip. Also, since DHash and Chord are maintained by the same entity, it is unlikely to be particularly painful. However, it does make me wonder if Chord’s dead simple interface is just too austere. The only function it provides is the ability to find the successor for a node, which is too basic for DHash (at least without compromising performance). Most competing DHT solutions (e.g. Pastry, Tapestry, CAN) didn’t separate the hash lookup primitive into a separate externally-distinguished artifact. I really like the idea of separating the hashing/routing from storage policy, and the single successor primitive is appealing for its simplicity, but perhaps the interface at the split should have been richer.

Personally, I don’t understand the reason for the large hype over ZFS, particularly in the context of OS X (more on that later). Now, it seems to be a fairly impressive engineering effort, but it is concentrating on an artifact that is somewhat pedestrian by now: a purely local filesystem. ZFS seems to be a good but still incremental improvement in local filesystem capabilities (with some unorthodox choices, but more on that later) — they took functionality that was previously available in different storage layers and increased coupling to improve performance and flexibility. Don’t get me wrong; there are certainly great things about ZFS, but in my book “the last word in file systems” would at least have to be distributed. From my perspective, the bulk of “cutting-edge” research in filesystems over the last decade has been on distributed or cluster filesystems. Of course, my research is generally in distributed systems and I worked on a distributed/parallel filesystem (IBM’s GPFS) this past summer, so I’m not an impartial bystander, but I think it’s non-controversial to say that storage transparently interfacing with the network is important now and will only become more important in the future. Now, Sun has indicated that they are going to use ZFS as a local storage backend for Lustre (since it currently uses ext3, this would be a big improvement), but that’s slightly different. Alternately, maybe if ZFS was like the eternal vaporware relational filesystem WinFS, I could see the hype being justified, particularly from end-users.

One of the reasons I’m baffled about the hype of ZFS on FreeBSD/OpenBSD/Mac OS X is because the port is not stable yet and there are some general ZFS issues that would limit its wide use. First of all, it needs a lot of memory and can panic or deadlock if it runs out of memory. It can really only run reliably on 64-bit machines (because it tends to exhaust kernel resources on 32-bit machines), but Sun is upfront about this. It also seems to be somewhat finicky and require manual tuning to get good performance (and sometimes just not crashing). A post to the FreeBSD mailing list titled “ZFS Honesty” summarizes the issues nicely:

But let’s also be honest about ZFS in the 64-bit world. There is ample evidence that ZFS basically wants to grow unbounded in proportion to the workload that you give it. Indeed, even Sun recommends basically throwing more RAM at most problems. Again, tuning is often needed, and I think it’s fair to say that it can’t be expected to work on arbitrary workloads out of the box.

A followup added:

I guess what makes me mad about ZFS is that it’s all-or-nothing; either it works, or it crashes. It doesn’t automatically recognize limits and make adjustments or sacrifices when it reaches those limits, it just crashes. Wanting multiple gigabytes of RAM for caching in order to optimize performance is great, but crashing when it doesn’t get those multiple gigabytes of RAM is not so great, and it leaves a bad taste in my mouth about ZFS in general.

Anyway, I’m not trying to dump on ZFS for these problems, because some are related to the BSD port and manual tuning is not unreasonable in high-end storage applications. The thing that gets me is that the hype is generally among user groups where such constraints would not be appropriate. For example, all of the buzz about OS X getting ZFS (and possibly being the default filesystem): based on the information above, it does not fit into the “Mac ethos” of “it just works.” Sure, it may get there one day, but why did a lot of people get all worked up about the availability of OS X binaries when they probably won’t be ready for general use for a long time? I guess it’s pre-excitement.

Now, as for ZFS’s unorthodox design choices: the designers essentially decided to collapse (or induce a tighter coupling between) many storage layers, including volume management and striping/RAID and make them a more integrated part of the filesystem. In fact, Linux kernel developer Andrew Morton famously called ZFS a “rampant layering violation” (ZFS developer Jeff Bonwick replies to Morton’s comment here). Normally you have a separate filesystem agnostic volume manager (like FreeBSD’s Vinum and Linux’s LVM/LVM2), and potentially RAID below that. The presence of those layers, however, effectively virtualize some aspect of the underlying disks and may hide information crucial for making layout decisions impacting performance. In the proceedings of HotOS X (2005), this point is well articulated in Lex Stein’s “Stupid File Systems Are Better.” In that paper, the author argues that a simple (”stupid”) filesystem that does random block layout has more uniformly good performance than filesystems with sophisticated layout policies (because the policies make assumptions about the underlying layout which may be completely invalidated by striping or other issues). Sun took the opposite approach: instead of making their filesystem stupid, Sun removed the layers between the disks and the filesystem, thus giving the filesystem more information to make layout decisions. Portability is another good reason not to rely on layering; if you need to ensure that certain volume management features are available on all platforms, you may have to “bring your own.” I don’t think that was a major factor in Sun’s ZFS, however, because I believe it was meant to be a compelling reason to use Solaris.

I’m somewhat ambivalent about the decision to couple layers, because there are good arguments both for and against. Sun has invested significant effort in engineering complex artifacts where the added complexity ultimately didn’t pay off, like Solaris M:N threading, which was an impressive effort but was ultimately ditched for simpler 1:1 user/kernel thread designs (Sun’s whitepaper explaining the Solaris 9 switch from M:N to 1:1 threading)*. An influential systems paper “Scheduler Activations: Effective Kernel Support for the User-Level Management of Parallelism” (appearing in SOSP and later in TOCS) argued for the two-level, M:N approach, but also noted it was critical to share information between the user-level scheduler and the kernel level thread scheduler. In this case, strict layering with both schedulers oblivious to each other would lead to poor performance decisions (and possibly deadlock).

But, in the realm of filesystems, there are also good arguments for clean layering (particularly once distribution is involved). Chandu Thekkath of Microsoft Research is a very strong advocate of structuring filesystems/storage abstractions in clean simple layers. For example, Frangipani, a distributed filesystem, is built on top of a virtual distributed disk, Petal. In the Frangipani paper, the authors praise the layered approach for making the filesystem very simple and quick to develop. In addition, the layering itself provides parallelism and the simplicity of the implementation allows it to be quite fast, despite the fact that information sharing in non-layered implementations may open up potential optimization opportunities. A more recent paper of his describes Boxwood, which is another distributed storage mechanism. Instead of a filesystem-like interface, however, it provides either distributed, replicated block-like storage or persistent data structures (also distributed and replicated). It again argues convincingly about the benefits of designing a system with clean and simple layers rather than complex tight coupling.

More widely distributed/peer-to-peer storage systems like CFS, described in “Wide-area cooperative storage with CFS”, are commonly built in several layers. CFS is a read-only filesystem built on top of a distributed block storage system DHash, which is itself built upon Chord, a peer-to-peer overlay network middleware (basically like half of a DHT — hashing/routing without storage; DHash provides storage). DHash and Chord were later used to implement Ivy, a peer-to-peer read-write filesystem. Similarly, both PAST, a peer-to-peer data publishing/archival system (immutable data; not a read-write filesystem) and Pastiche, a cooperative-storage based backup system, are layered on Pastry, a feature rich peer-to-peer routing/DHT like system. OceanStore, another wide-area distributed storage system, was itself built upon Tapestry, another peer-to-peer DHT/overlay network.

Anyway, now I’m just rambling through tangentially related work in distributed filesystems, but I guess the question is whether the added complexity of ZFS will pay off versus something like btrfs or HAMMER on top of a good volume manager and RAID. I guess time will tell, but I’m sympathetic to arguments for both alternatives: on one hand, increasing coupling between layers allows you to optimize, but decreasing coupling may make each layer simpler. Given finite development time/effort, it’s easier to perfect and optimize simple artifacts than complex ones. As for my previous examples, one might say they aren’t directly relevant in that distribution nearly always suggests a clean layered design because the complexity is just too high otherwise, whereas a local filesystem with locally attached disks may benefit from cooperation between layers because they do similar things and you gain performance and flexibility within the confines of a specific filesystem. On the other hand, separate layers give you more horizontal flexibility (i.e. if you definitely need to use something other than ZFS): for example, Linux’s LVM2 support snapshots on many filesystems at the volume manager level, but they’re not as flexible or fast as ZFS.

Anyway, I guess the whole impetus behind this post was that I’m bothered by the level of hype I’m seeing in certain circles over ZFS (and the marketing label “the last word in file systems” doesn’t help). Sure, it seems like impressive engineering, but nothing particularly groundbreaking or revolutionary. I’ll be interested to see how it ultimately turns out in competing against various other filesystems in development.

* Incidentially, FreeBSD, one of the last major holdouts with M:N threading (along with NetBSD), is also purportedly switching to 1:1 threading by making the 1:1 libthr the default threading library in very-soon-to-be-released FreeBSD 7.

Gentoo PXE Install

• Grab a minimal install ISO. For this example, I’ll use install-x86-minimal-2007.0-r1.iso.
• Mount the iso and copy the following files: image.squashfs, isolinux/gentoo, isolinux/gentoo.igz (and isolinux/isolinux.cfg for reference)
• Make a temporary directory and unpack gentoo.igz: mkdir tmp; cd tmp; zcat ../gentoo.igz | cpio -idv
• Make a mnt/cdrom subdirectory (mkdir -p mnt/cdrom) and copy the image.squashfs into it
• Patch the init script so that it looks for the squashfs image in the right place. Here’s my gentoo_init.patch.
• Repack the gentoo.igz file: find * | cpio --quiet --dereference -o -H newc | gzip -9 > ../gentoo.igz

Now, add Gentoo entries to your pxelinux config following the isolinux.cfg for reference, but add real_root=/ to the kernel parameters (the append=) line (assuming you are using my patch).

GParted Live PXE
So the process for making the GParted LiveCD PXE bootable is similar. Once I have performed the above modifications on the .igz file (to pack the squashfs right into the initrd), I could take the iso/syslinux entries for the GParted boot options and add them directly to my pxelinux boot menu. Instead, I decided to make a two-level menu with a self-contained GParted disk image. In other words, you select GParted from the PXE boot options and it loads a disk image which itself boots into syslinux, prompting for GParted-specific options. This is slightly more complicated, but it separates the maintenance of GParted-specific options from the pxelinux.cfg file. One could accomplish this kind of two-level separation in other ways, too, but I’ll describe what I did.

• Using syslinux’s mkdiskimage tool, make a disk image to contain the files: mkdiskimage disk.img 7 255 63 (make sure it is big enough to hold everything)
• Mount the FAT filesystem part of the disk image: mount -o loop,offset=32256 disk.img files
• Copy the appropriate files from the LiveUSB stuff to the image (gparted kernel, gparted.igz, properly edited syslinux.cfg, options.msg, boot.msg, splash.lss, etc.)
• Unmount the loopback image: umount disk.img
• Run syslinux on the filesystem: syslinux -o 32256 disk.img

Now I can copy this disk image somewhere and add an entry to my pxelinux.cfg similar to this:

label gparted
  kernel memdisk
  append initrd=gparted.img c=7 h=255 s=63 noedd

If it doesn’t work, make sure after copying the files to the loopback image that you don’t have errors due to going “out of bounds” on the image. I’ve found that when the size of the files are close to the size of the filesystem, and I copy and recopy certain files, I’ll get error messages like the following:

kernel: loop1: rw=1, want=112452, limit=112392
kernel: lost page write due to I/O error on loop1

I find starting over with a fresh filesystem and copying the files once only fixes this.

Bootable DOS CD
Finally, I also wanted to make a bootable DOS CD PXE bootable, so I tried a couple of methods before finding one that worked well. I won’t name the CD because it is probably against the terms of the license to do something like this, but it’s just convenient to not have to have media handy for personal uses. I used some utilities from the mtools suite as well as the geteltorito tool which is included in Debian’s genisoimage package.

• Extract the El Torito bootable image from the CD: geteltorito -o eltorito.img cd.iso
• Grab the DOS boot block from the El Torito image: dd if=eltorito.img of=bootblock.img count=512 bs=1
• Set up mtools to create a loopback disk image. For this, I edited my ~/.mtoolsrc to contain drive x: file="/tmp/floppyimage"
• Use mformat to format the disk image that can hold the relevant files and give it the boot block: mformat -C -t 160 -s 36 -h 2 -B bootblock.img x:
• After that, I had to edit the autoexec.bat and other pieces of the CD startup files so it referred to files in the right places. This is the non-automatic part of the process. Previously, the system would boot with A: being the El Torito image, load mscdex.exe and then refer to Y: for files on the CD. When PXE booting, all of the relevant files will be on A: and mscdex won’t need to be loaded.
• After editing the proper files, copy them to the disk image: mcopy -s * x:

In the last case, make sure to get the relevant DOS system and startup files. For the CD I was modifying, it was PC-DOS, so I needed ibmio.com, ibmdos.com, command.com as well as startup files like autoexec.bat and config.sys. After the disk image is made, I could simply add an entry to pxelinux.cfg like the following:

label dosboot
  kernel memdisk
  append initrd=dosimage.img.gz c=160 h=2 s=36 floppy noedd

In this case, I gzipped the image. For gparted, I didn’t gzip the outer image because the contents were already mostly compressed (e.g. the igz and kernel image).

Look at stuff that gets published in SIGGRAPH, ICCV, ACM Multimedia and similar venues. Compare with venues like SOSP, OSDI, Usenix, etc. In terms of the ability to appeal to non-technical people, I think systems work is soundly beaten. Now, of course popular appeal is not the point of these venues, and I don’t think it’s something to “fix,” I’m just using them as representative samples of their respective subdisciplines. I guess what makes the output of some disciplines more accessible to outsiders is their connection to the real world (the parts that people interact with, at least). Graphics and information visualization deal with visual output, and computer vision deals with visual input. Robots navigate in and manipulate the real world. Systems work is building software that either interfaces with computer hardware or other layers of software. I guess in that respect, middleware is sort of the ultimate “boring” and unappreciated artifact.

On the topic of cool graphics demos, here are a few that immediately come to mind:

• Dual Photography from SIGGRAPH 2005 — check out the demo where they reconstruct the face of a playing card with its back to the camera
• Photo Tourism from SIGGRAPH 2006 — Microsoft Live Labs has turned this into Photosynth, and they have been doing demos of it quite a bit for the past year or so. Blaise Aguera y Arcas gave a MS Photosynth and Seadragon demo as a TED talk.
• Scene Completion Using Millions of Photographs from SIGGRAPH 2007 — Applying “Google-sized” data sets to hard vision problems. Alexei Efros, the faculty advisor of this project, gave a Google Tech talk recently titled “Using Data to ‘Brute Force’ Hard Problems in Vision” talking about the results of this paper and related data-heavy efforts. Again, the talk has lots of cool, compelling visual examples of the research.