In recent months, my student Peter Bailis and his teammate Shivaram Venkataraman took a different tack on the whole EC analysis problem which they call PBS: Probabilistically Bounded Staleness. The results are interesting, and extremely relevant to current practice.  (See, for example, the very nice blog post by folks at DataStax).

Many people today deal with EC in the specific context of replica consistency, particularly in distributed NoSQL-style Key-Value Stores (KVSs). It is typical to configure these stores with so-called “partial” quorum replication, to get a comfortable mix of low latency with reasonable availability. The term “partial” signifies that you are not guaranteed consistency of writes by these configurations — at best they guarantee a form of eventual consistency of final writes, but readers may well read stale data along the way. Lots of people are deploying these configurations in the field, but there’s little information on how often the approach messes up, and how badly.

Jumping off from earlier theoretical work on probabilistic quorum systems, Peter and Shivaram answered two natural questions about how these systems should perform in current practice:

1. How many versions ago?  On expectation, if you do a read in a partial-quorum KVS, how many versions behind are you? Peter and Shivaram answer this one definitively, via a closed-form mathematical analysis.
2. How stale on the (wall-)clock?  On expectation, if you do a read in a partial-quorum KVS, how out-of-date will your version be in terms of wall-clock time? Answering this one requires modeling a read/write workload in wall-clock time, as well as system parameters like replica propagation (“anti-entropy”). Peter and Shivaram address this with a Monte Carlo model, and run the model with parameters grounded in real-world performance numbers generously provided by two of our most excellent colleagues: Alex Feinberg at LinkedIn and Coda Hale at Yammer (both of whom also guest-lectured in my Programming the Cloud course last fall.)  Peter and Shivaram validated their models in practice using Cassandra, a widely-used KVS.

On the whole, PBS shows that being sloppy about consistency doesn’t bite you often or badly — especially if you’re in a single datacenter and you use SSDs. But things get more complex with magnetic disks, garbage collection delays (grr), and wide-area replication.

Interested in more detail?  You can check out two things:

• The paper (currently under submission to a conference) is available as a Berkeley Tech Report.
• Peter put up a web-based version of the Monte Carlo simulation that allows you to specify quorum parameters and workload parameters, and observe the tradeoff between those parameters and the probability of various levels of staleness.

I’d be very grateful if you would add your favorites below to the comments, preferably with a link to a source that either introduces or references the maxim.  It’s OK to agree or disagree with the maxim.

I’d enjoy seeing people’s support/critiques for these below as well — may merit more focused posts another day.

Examples:

• The End-to-End Argument, Saltzer/Reed/Clark, ICDCS ’81.
• Many examples in Lampson’s “Hints for Computer System Design“, SOSP ’83.
• “All problems in computer science can be solved by another level of indirection”. — also attributed to Butler Lampson

What else?

Dan pointed out that he and Brian and others took steps in this direction years ago at Berkeley, by introducing MapReduce and Hadoop in our initial 61A course.  I have argued elsewhere that this is a Good Thing, because it gets people used to the kind of disorderly thinking needed for scaling distributed and data-centric systems.

But as a matter of both pedagogy and system design, I have begun to think that Google’s MapReduce model is not healthy for beginning students.  The basic issue is that Google’s narrow MapReduce API conflates logical semantics (define a function over all items in a collection) with an expensive physical implementation (utilize a parallel barrier). As it happens, many common cluster-wide operations over a collection of items do not require a barrier even though they may require all-to-all communication.  But there’s no way to tell the API whether a particular Reduce method has that property, so the runtime always does the most expensive thing imaginable in distributed coordination: global synchronization.

From an architectural point of view, a good language for parallelism should expose pipelining, and MapReduce hides it. Brian suggested I expand on this point somewhere so people could talk about it.  So here we go.

The canonical example here is the relational join, which pairs up matching objects from two different input streams.  One of the early observations in shared-nothing parallel databases was that parallel joins can work in a fully streaming fashion even though they require all-to-all communication (Wilschut and Apers’ Pipelining Hash Join).  This is impossible to implement in stock Hadoop.  Which means, for example, that the NSA—apparently a big Hadoop user—can’t use MapReduce to match live feeds of suspicious activity reports with a repository of intelligence on suspicious people.  (Whether you think it’s good or bad that the NSA can’t use Hadoop for this task may depend on your politics regarding national security, open source, or both.)

That example is easy to understand, but maybe too easy to dismiss as the special case of “stream processing”.  Sure, Hadoop wasn’t designed for that, so maybe you should dismiss my point as saying that somebody should just build a streaming edition of Hadoop for special uses.  (Oh yeah, somebody did that.)

A more telling pedagogical example is the standard discussion of implementing PageRank in MapReduce.  What an awesome lecture: the two most famous pieces of Google technology in action at once!  The thing is, at a logical level PageRank is a simple self-join query with transitive closure: it pairs up items in the Nodes table with their neighbors in that same table, and outputs new weights until a stopping condition is reached. The matching of nodes and neighbors needs to be hash-partitioned across the network, but there is no reason to require the entire cluster to do a barrier synchronization at each iteration boundary.  The subtext of this lecture’s awesomeness is to deeply implant a misleading (and potentially very inefficient) idea about the need for synchronized execution in parallel linear algebra and query processing tasks.  That sounds like a bad way to teach.

And it goes deeper.  At a very basic level, asynchronous parallel computation is fundamentally a matter of streaming rendezvous (join) between event channels and buffers.  We converted this observation into a first-class feature of the Bloom language for distributed systems. When you write event-handling code in Bloom, you express it as the parallel join of a partitioned event stream with partitioned program state. Whether or not this metaphor makes sense to you (yet!), I assure you that it is fundamentally what goes on under the covers of any asynchronous distributed system: the matching of streams of requests or responses with stored state.  By teaching students the Google MapReduce model (as represented in Hadoop), we teach them that distributed joins need to block, and therefore cannot be the basis of the kind of streaming computation that servers must do by their very nature.  That is not only bad, it’s paradoxical: underneath the Hadoop implementation is a message handling dataflow that effectively does streaming joins.

So there—at Brian’s request I wrote down some cautionary words about using Hadoop in school.  Lest somebody read this as my declaration of allegiance to the “get off my lawn” side of the AntiNoSQLdisestablishmentarianism argument, let me clarify my view on a few key things:

• Google MapReduce and the Hadoop open-source implementation opened the floodgates for discussing big data and parallelism in the core of the computing curriculum.  This has been a critical step in moving computing education into the era of abundant computing and data resources.  Bully for that.
• I do understand that Google put barriers into MapReduce for fault tolerance purposes.  I also get that fault-tolerance is important in scale-out of parallel dataflows (we did some work on this) and that Google uses a lot of machines at once.  But as a practical matter, many Hadoop users run on a small handful of nodes and don’t need Google-scale fault tolerance features.  More to the point, when we teach computing, we shouldn’t focus on only one design point; we should focus on the fundamentals.  This is a case where the fundamentals get taught in a tangled way that only makes sense at extreme scale.
• For the record, as languages go I like MapReduce about as much as I like SQL. That is to say I would invite them both to my birthday party. But I wouldn’t invite either to the prom.  My sweetheart here is Bloom.

If you’ve been following our Bloom work, you know where this discussion is coming from: the CALM Theorem says that the real reason to use coordination is to manage non-monotonic reasoning.  Many (most?) Reduce functions in the wild are in fact monotonic w.r.t. subsets of their inputs, but the standard Reduce API defies monotonicity analysis.  And introducing barriers or monotonic computation is a waste of “time”, in the temporal logical sense of the word.

Forget your previous associations with low-tech SQL analytics, including so-called “business intelligence”, “olap”, “data cubes” and the like. This is the real deal: statistical and machine learning methods running at scale within the database, massively parallel, close to the data. Much of the code is written in SQL (a language that doesn’t get enough credit as a basis for parallel statistics), with key extensions in C/C++ for performance, and the occasional Python glue code. The suite of methods in the beta includes:

• standard statistical methods like multi-variate linear and logistic regressions,
• supervised learning methods including support-vector machines, naive Bayes, and decision trees
• unsupervised methods including k-means clustering, association rules and Latent Dirichlet Allocation
• descriptive statistics and data profiling, including one-pass Flajolet-Martin and CountMin sketch methods (my personal contributions to the library) to compute distinct counts, range-counts, quantiles, various types of histograms, and frequent-value identification
• statistical support routines including an efficient sparse vector library and array operations, and conjugate gradiant optimization.

More methods are planned for future releases.  Myself, I’m working with Daisy Wang on merging her SQL-based Conditional Random Fields and Bayesian inference implementations into the library for an upcoming release, to support sophisticated text processing.

MADlib 0.20beta was designed as a drop-in addition to the popular open-source PostgreSQL database.  The routines also all run massively parallel in the Greenplum database, which inherits the extensibility interfaces of PostgreSQL. (Non-commercial users can get the parallelism for free—no crippleware!—via Greenplum’s Community Edition.)

Obviously the first beta release is still experimental code.  But we’ve come a long way since our alpha announcement at the Strata conference last February in both breadth and usability. For me, the most exciting aspect of the beta release is that we’re about ready to grow our developer community beyond the initial committers at Berkeley and EMC-Greenplum.  If you’re interested in adding methods to MADlib—or in developing ports for other DBMS platforms (something I’d love to see happen!)—please get in touch.

And hats off to EMC-Greenplum for putting significant development resources behind this open-source effort.  I started this project at Greenplum before they were acquired, and have been happy to see EMC embrace it and push it further.  As I discussed at Strata last year, I think this is a very healthy direction for open-source software development.

Meanwhile, some of the experience is even more relevant to current technology trends than it was 4 years ago, so hopefully folks interested in social computing, software engineering, image processing, crisis response, and other related areas will find something of use in there.

For those of you whose work is represented (or underrepresented) by the article, my apologies for its shortcomings.  I still don’t have the full picture of what happened—nobody does, really.  As a result I decided to avoid using personal names of volunteers in general to avoid attributing credit unevently. I know the result seems oddly impersonal.  Setting the tone of the article was as hard as capturing the content.

Meanwhile, I encourage you to add corrections and perspective to the article in the comment box at the end of the CACM link above. Comments are welcome here too, but they might not get as well-viewed or -archived.

I’ve written a lot about Bloom in research papers and on the new Bloom website, and I have lots to say about distributed programming that I won’t recap. Instead, I want to focus here on the tangible: working code. If you’re looking for something serious, check out the walkthrough of the bfs distributed filesystem, a GFS clone. But to get the flavor, consider the following two lines of code, which implement what you might consider to be “hello, world” for distributed systems: a chat server.

nodelist <= connect.payloads
mcast <~ (mcast * nodelist).pairs { |m,n| [n.key, m.val] }

That’s it.

The first line says “if you get a message on a channel called ‘connect’, remember the payload in a table called ‘nodelist’”. The second says “if you get a message on the ‘mcast’ channel, then forward its contents to each address stored in ‘nodelist’”. That’s all that’s needed for a bare-bones chat server.  Nice, right?

The chat client code bootstraps by connecting to the server:

connect <~ [[@server, [ip_port, @nick]]]

and then runs the following logic:

mcast <~ stdio do |s|
  [@server,
   [ip_port, @nick, Time.new.strftime("%I:%M.%S"), s.line]]
end
stdio <~ mcast { |m| [pretty_print(m.val)] }

The first statement of this batch takes input from the terminal (‘stdio’), formats it, and sends it to the server on the ‘mcast’ channel.  The second statement takes messages from the ‘mcast’ channel (forwarded by the server) and prints them on the terminal.

I often hear that many of the leading data analysts in the field have PhDs in physics or biology or the like, rather than computer science.  Computer scientists are typically interested in methods; physical scientists are interested in data.

Another thing I often hear is that a large fraction of the time spent by analysts — some say the majority of time — involves data preparation and cleaning: transforming formats, rearranging nesting structures, removing outliers, and so on.  (If you think this is easy, you’ve never had a stack of ad hoc Excel spreadsheets to load into a stat package or database!)

Putting these together, something is very wrong:  high-powered people are wasting most of their time doing low-function work.  And the challenge of improving this state of affairs has fallen in the cracks between the analysts and computer scientists.

DataWrangler is a new tool we’re developing to address this problem, which I demo’d today at the O’Reilly Strata Conference.  DataWrangler is an intelligent visual data transformation tool that lets users reshape, transform and clean data in an intuitive way that surprises most people who’ve worked with data.  As you manipulate data in a grid layout, the tool automatically infers information both about the data, and about your intentions for transforming the data.  It’s hard to describe, but the lead researcher on the project — Stanford PhD student Sean Kandel — has a quick video up on the DataWrangler homepage that shows how it works.  Sean has put DataWrangler live on the site as well.

Tackling these problems fundamentally requires a hybrid technical strategy.  Under the covers, DataWrangler is a heady mix of second-order logic, machine learning methods, and human-computer interaction design methodology.   We wrote a research paper about it that will appear in this year’s SIGCHI.

If you’re interested in this space, also have a look at Shankar Raman’s very prescient Potter’s Wheel work from a decade ago, the PADS project at AT&T and Princeton, recent research from Sumit Gulwani at Microsoft Research, and David Huynh’s most excellent Google Refine.  All good stuff!

Shatner? Or Not(Shatner)? Having now changed history, he could not have been born, meaning he could not have traveled back in time and changed history, meaning he was indeed born, meaning…?

You see where this goes.  It’s the old grandfather paradox, a hoary chestnut of SciFi and AI.  Personally I side with Captain Kirk: I don’t like mysteries. They give me a bellyache. But whether or not you think a discussion of “p if Not(p)” is news that’s fit to print, it is something to avoid in your software.  This is particularly tricky in distributed programming, where multiple machines have different clock settings, and those clocks may even turn backward on occasion. The theory of Distributed Systems is built on the notion of Causality, which enables programmers and programs to avoid doing unusual things like executing instructions in orders that could not have been specified by the program that generated them. Causality is established by distributed clock protocols. These protocols are often used to enforce causal orderings–i.e. to make machines wait for messages. And waiting for messages, as we know, is bad.

So I’m here to tell you today that Causality is overrated, and we can often skip the wait. To hell with distributed clocks: time travel can be fine.  In many cases it’s even fine to change history. Here’s the thing: Casuality is Required Only to control Non-monotonicity. I call this the CRON principle.

Let’s agree for purposes of discussion that p and Not(p) is not meaningful, and our Twilight Zone episode the usual logical nonsense dressed up with campy schmacting. Suppose we change the plot so that Shatner goes back in time and does something positive, in the logical sense of monotonicity: generating additional information without negating any information. For example, suppose he falls in love with a young woman in the past, they marry, and have a child. Then one day, to his horror, he realizes that his wife is his own grandmother! Distasteful, I agree, but logically monotonic! If you accidentally mate with your ancestors, it doesn’t matter (from a logical point of view) whether you use a time machine to do it. This is one direction of CRON: Purely monotonic programs will have consistent outcomes even in the face of “non-causal” orderings.

That is a big deal. It means that there is a large class of distributed programs where causal orderings—a foundational topic in distributed systems theory—are not required. Note that monotonic fixpoint logic (if you allow an ordered domain like the integers) can express all polynomial-time programs. And after all, who would run an exponential program in a distributed system?  So you might justifiably argue that distributed systems have been “wasting time” all these years for no good reason.

But let’s be real: not all polynomial-time programs are equally good, and it’s handy for programmers to be able to express non-monotonic things like … say … updating a variable (or shooting somebody evil). So when can we allow that to happen without enforcing the forward march of time?

Answer: when the consequences of the non-monotonic expression do not entail its antecedents. I.e. when the ability to shoot does not depend on the fate of the victim. Now in the physical world, we might worry a la chaos theory (or Edith Keeler) that everything is sensitive to the fate of everything else. In computer programs, though, we can plausibly guarantee the property we want here in many cases. We might be able to prove for a given program that, effectively, victims will never be ancestors of any shooter. When we can guarantee that–statically or dynamically–we can admit the non-monotonicity into the program without requiring an implementation of distributed clocks and waiting.

So when do we need distributed clocks?  Only when–exactly when–the grandfather paradox would arise in their absence. Causality is Required Only for controlling cycles in Non-monotonicity.

(This discussion comes out of my PODS talk on the Declarative Imperative from last June.  Bill Marczak, Peter Alvaro and I are in the midst of writing up a formal proof. Input welcome!)

12/16/2010: final version of CALM/Bloom paper for CIDR now posted

Conventional Wisdom:
In large distributed systems, perfect data consistency is too expensive to guarantee in general. “Eventually consistent” approaches are often a better choice, since temporary inconsistencies work out in most cases. Consistency mechanisms (transactions, quorums, etc.) should be reserved for infrequent, small-scale, mission-critical tasks.

Most computer systems designers agree on this at some level (once you get past the NoSQL vs. ACID sloganeering). But like lots of well-intentioned design maxims, it’s not so easy to translate into practice — all kinds of unavoidable tactical questions pop up:

Questions:

• Exactly where in my multifaceted system is eventual consistency “good enough”?
• How do I know that my “mission-critical” software isn’t tainted by my “best effort” components?
• How do I maintain my design maxim as software evolves? For example, how can the junior programmer in year n of a project reason about whether their piece of the code maintains the system’s overall consistency requirements?

If you think you have answers to those questions, I’d love to hear them. And then I’ll raise the stakes, because I have a better challenge for you: can you write down your answers in an algorithm?

Challenge:
Write a program checker that will either “bless” your code’s inconsistency as provably acceptable, or identify the locations of unacceptable consistency bugs.

The CALM Conjecture is my initial answer to that challenge.

CALM stands for “Consistency As Logical Monotonicity”. The idea is that the family of eventually consistent programs are exactly those that can be expressed in monotonic logic. By contrast, distributed non-monotonicity (e.g. destructive state modification, aggregation or “reduce”) can and must be resolved via distributed coordination logic (e.g. two-phase-commit, Paxos). The idea that “temporary inconsistencies work out” amounts to ensuring that the data in question is contained within a properly-protected monotonic component of the system.

Presuming that CALM can be proven with sufficient generality (restricted formal versions of it are obvious, but I’m quite sure there’s more that can be done), the next step is to translate this idea into a useful program checker. Like many other verification tasks, this is only tractable in a sufficiently high-level language — in this case, one where the constructs of the language can be translated into an underlying logic. Bloom is our language along these lines, and we have the initial program checks in place. Given a Bloom program, we can do the following:

1. bless programs as monotonic, and hence safe to run coordination-free
2. identify non-monotonic “Points of Order” in Bloom programs. These can be resolved by either of the following (which we intend to automate):
   • add coordination logic (e.g. quorum consensus) to enforce the ordering, or
   • augment the program to tag downstream data as “tainted” with potential inconsistency
3. visualize the Points of Order in a dependency graph, to help programmers reason about restructuring their code for more efficient consistency enforcement.

We wrote up our initial ideas on this topic in a short submission to CIDR 2010 CIDR 2010 paper, including an intro to the current state of the Bloom language, and an example of analyzing a replicated shopping-cart application.  This follows from the discussion in my companion paper to my PODS keynote talk [slides].

I’d love feedback on these ideas, which are still a work in progress.

That’s necessary but not sufficient: you also need correct failure-handling protocols and faithful implementations.

So, do today’s popular distributed systems handle component failure well? Or, taking the longer view: what kinds of tools will help the engineers who build those systems ensure that they handle component failure well?

My postdoc Haryadi Gunawi and his team have taken some big steps to answer these questions, and written them up in their report on FATE and DESTINI: A Framework for Cloud Recovery Testing. They take a systematic combinatorial approach to generating faults (FATE) and a formal approach to specifying correctness (DESTINI) that grows out of our work on declarative languages. Upshot: research that produces real tools, which help developers find (and then fix) real failure-handling bugs, including 16 new bug reports to HDFS (7 design bugs and 9 implementation bugs). Pretty nice, given the intricacies of failure-recovery protocols.

Haryadi conceived of and drove this project, and like his earlier work fixing the sorry state of file system checkers, it takes good clean research designs and uses them to improve substantially on real-world practice.

Haryadi started the project like a social scientist, combing through the Jira issue-tracker reports for HDFS and classifying the recovery bugs.  Then he set about generalizing issues, designing techniques, and building tools.  After that he used the Jira’s to make sure his tools were getting good coverage (they found the already-reported bugs automatically) and generating tangible benefits (surfacing new bugs). He then transitioned from his first experimental setting (HDFS) to two new ones: Cassandra and Zookeeper. Results there are preliminary but look promising.

In the end, the whole package feels simple, sensible, and useful.  And it’s exactly because of the way he combined the grounding in practice with the elegance of his research ideas,  Now that the initial results are written up, Haraydi is beginning the discussions with Yahoo, Facebook, Cloudera and others to get this stuff out in the field.