Before I talk more about the background let me say this: We Are Hiring. We’re looking for people with passion and talent in Interaction Design, Data Visualization, Databases, Distributed Systems, Languages, and Machine Learning. We’re looking for folks who want to reach across specialties, and work together to build integrated, rich, and deeply satisfying software. We’ve got top-shelf funding and a sun-soaked office in the heart of SOMA in San Francisco, and we’re building a company with clear, tangible value. It’s early days and the fun is ahead. If you ever considered joining a data startup, this is the one. Get in touch.

The genesis of the company goes back many years. I’ve known Jeff since his days a grad student at Berkeley. Since then of course he’s made a huge splash both in the HCI research community and in the open source world via tools like D3.js and Protovis. After serving on Jeff’s thesis committee and watching him move to a faculty position at Stanford, I was determined to continue working with him.

As part of the d^p project, Jeff and I began co-advising Sean Kandel. Sean has proven to be a research monster, with an unbroken string of accepted papers during 3 short years at Stanford. He also brought his experience as a financial analyst to the table, and a thirst for entrepreneurship.

Trifacta is the next phase of this collaboration. The name refers to the 3 issues that need to be addressed simultaneously to achieve this vision: People, Data, and Computation.  We’re entering a period in computing history in which Data has become ubiquitous and Computation has turned into an inexpensive commodity. At this point, People are the last bottleneck to useful data analysis — both because human skills don’t improve with Moore’s Law, and because the growth in the other resources has increased the demand for people to deal with data and do analysis.

Another day I will write more about Trifacta’s initial products, but suffice it to say for now that our mission is to build software that ameliorates the human bottlenecks in analysis. Our goal is both to remove data drudgery from the workflows of experienced data scientists, and to enable self-service solutions for end-users who otherwise would not be able to take advantage of new sources of data. It’s an inherently inter-disciplinary problem, with interaction and visualization at its core, but with unique demands on systems and inference algorithms to drive an intelligent user experience over large volumes of data. And of course there’s a role for the kind of high level languages for scalable systems that I’ve been focused on for years with Bloom and the like as well.

If you made it this far, please wander over to the Trifacta website for more info. We’d love to follow up.

Bill Marczak, right, in NY Times

Bill Marczak, a PhD student in my group, does interesting research on algebraic programming languages, which I hope to describe in more detail here soon.

But Bill has recently received significant attention for work he did in his spare time—a dramatically successful cyber-espionage effort to expose government misuse of commercial surveillance software in Bahrain, the nation where Bill attended high school. The story picked up major press coverage in venues including Bloomberg and the New York Times, which also ran a more detailed article in their Bits Blog.

I’m always happy to see the press pick up on my students’ work, but this one is special.

Matt Welsh of Google—formerly of Harvard, Berkeley and Cornell—is a deservedly well-read blogger in the computing community.  He’s also somebody I’ve admired since his early days in grad school as a smart, authentic person.

Matt’s been working through his transition from Harvard Professor to Googler in public over the last year or so, and it’s been interesting to watch what he says, and the discussion it provokes.  His latest post was a little more acid than usual though, with respect to the value of academic computer science.  My response got pretty long, and in the end I figured it’d be better to toss it up in my own space.

Matt:

Rather than run down work you don’t like—including maybe your own prior work, as assessed on one of your dark days—think about the academic work over the last 50 years that you admire the hell out of. I know you could name a few heroes. I bet a bunch of your blog’s readers could get together and name a whole lot more. Now imagine the university system hadn’t been around and reasonably well-funded at the time, because it was considered “inefficient when it comes to producing real products that shape the world”.   It’s sad to consider.

Here’s another thing you and your readers should consider: Forget efficiency. At least, forget it on the timescale you measure in your current job. Instead, aspire to do work that is as groundbreaking and important as the best work in the history of the field. And at the same time, inspire generations of brilliant students to do work that is even better—better than your very best. That’s what great universities are for, Matt. Remember? Sure you do. And yes—it’s goddamn audacious. As well it should be.

Now, can you get up and be actively audacious every day? Nobody can, not even the Greats. So we take a portfolio approach at all levels: lots of universities, lots of research grants, lots of individual faculty and students, and—for some individuals—lots of different career phases to see where your changing interests and skills fit best. It’s good to move around. But all the while, you have to respect the portfolio, Matt. It’s been working fabulously over the long term. And it’s done so on a tiny fraction of the budget of corporate America, a tiny fraction of the tax base.

The funny thing about your timing here is that the short-term view is actually really rosy right now—times are great for academic/industrial fluidity in computing. Companies and VCs and pundits are flocking to academic leaders for ideas and guidance. Academics are more aware than ever about what’s going on in industry, and some of the really good ones have taken the plunge into companies to do the tech transfer. Times are really good, even on the small timescale.

Can things get better? Sure they can. Is the university research model perfect? Of course not. We should have a portfolio of experiments there too, including new models for tech transfer and collaboration between academia, industry, and venture funding. (I blogged about one of my experiments recently). So by all means go back to your post and distill down some of the constructive parts.  Spend time on some of them.  You’re not alone in wanting to experiment with these models and improve the pipeline, and there’s a good constructive conversation to have.

Meanwhile, stay positive! Life is good. Research is good! You have a big platform there (and I don’t mean all those computers you can log into). Use it for good. Remember your heroes. Emulate and inspire.

I kicked off MADlib as a part-time consulting project for Greenplum during my sabbatical in 2010-2011.  As I built out the first two methods (FM and CountMin sketches) and an installer, Greenplum started assembling a team of their own engineers and data scientists to overlap with and eventually replace me when I returned to campus.  They also developed a roadmap of additional methods that their customers wanted in the field.  Eighteen months later, Greenplum now contributes the bulk of the labor, management and expertise for the project, and has built bridges to leading academics as well.

While I’ve encouraged Greenplum’s investment all along, it has been equally important to me that MADlib avoid becoming Yet Another Proprietary Library for Analytics.  There are a bunch of those libraries provided by DBMS vendors and third parties; they are all-but-invisible to researchers, and are immutable black boxes to data scientists in the field. So while some of them may be quite good (who knows?!) they don’t create or benefit from a network effect in the technical community.  Given the current excitement about Big Data and scarcity of data scientists, that kind of isolation could be deadly to both adoption and evolution.

As a result of those considerations, MADlib is open source, and I’ve insisted all along that it be maintained on at least one open-source platform: PostgreSQL.  I’m also genuinely eager to see MADlib ports to other massively-parallel commercial DBMSs.  This will require some work, but it’s tractable and well-scoped engineering work, and I hope that incentives will arise  to get it done and expand the scope of the community.

Finally, I’m happy to see academics like Chris Ré, Daisy Wang and their students starting to contribute to the project.  I hope their research will help out data scientists in the field, and generate the kind of user feedback that is so hard to get in academia. I also hope that other researchers will join them and get involved.  I think it’s great experience to write meaty research code that actually ships, and I hope MADlib can be a mechanism to encourage more of that experience in the community.

In recent months, my student Peter Bailis and his teammate Shivaram Venkataraman took a different tack on the whole EC analysis problem which they call PBS: Probabilistically Bounded Staleness. The results are interesting, and extremely relevant to current practice.  (See, for example, the very nice blog post by folks at DataStax).

Many people today deal with EC in the specific context of replica consistency, particularly in distributed NoSQL-style Key-Value Stores (KVSs). It is typical to configure these stores with so-called “partial” quorum replication, to get a comfortable mix of low latency with reasonable availability. The term “partial” signifies that you are not guaranteed consistency of writes by these configurations — at best they guarantee a form of eventual consistency of final writes, but readers may well read stale data along the way. Lots of people are deploying these configurations in the field, but there’s little information on how often the approach messes up, and how badly.

Jumping off from earlier theoretical work on probabilistic quorum systems, Peter and Shivaram answered two natural questions about how these systems should perform in current practice:

1. How many versions ago?  On expectation, if you do a read in a partial-quorum KVS, how many versions behind are you? Peter and Shivaram answer this one definitively, via a closed-form mathematical analysis.
2. How stale on the (wall-)clock?  On expectation, if you do a read in a partial-quorum KVS, how out-of-date will your version be in terms of wall-clock time? Answering this one requires modeling a read/write workload in wall-clock time, as well as system parameters like replica propagation (“anti-entropy”). Peter and Shivaram address this with a Monte Carlo model, and run the model with parameters grounded in real-world performance numbers generously provided by two of our most excellent colleagues: Alex Feinberg at LinkedIn and Coda Hale at Yammer (both of whom also guest-lectured in my Programming the Cloud course last fall.)  Peter and Shivaram validated their models in practice using Cassandra, a widely-used KVS.

On the whole, PBS shows that being sloppy about consistency doesn’t bite you often or badly — especially if you’re in a single datacenter and you use SSDs. But things get more complex with magnetic disks, garbage collection delays (grr), and wide-area replication.

Interested in more detail?  You can check out two things:

• The paper (currently under submission to a conference) is available as a Berkeley Tech Report.
• Peter put up a web-based version of the Monte Carlo simulation that allows you to specify quorum parameters and workload parameters, and observe the tradeoff between those parameters and the probability of various levels of staleness.

I’d be very grateful if you would add your favorites below to the comments, preferably with a link to a source that either introduces or references the maxim.  It’s OK to agree or disagree with the maxim.

I’d enjoy seeing people’s support/critiques for these below as well — may merit more focused posts another day.

Examples:

• The End-to-End Argument, Saltzer/Reed/Clark, ICDCS ’81.
• Many examples in Lampson’s “Hints for Computer System Design“, SOSP ’83.
• “All problems in computer science can be solved by another level of indirection”. — also attributed to Butler Lampson

What else?

Dan pointed out that he and Brian and others took steps in this direction years ago at Berkeley, by introducing MapReduce and Hadoop in our initial 61A course.  I have argued elsewhere that this is a Good Thing, because it gets people used to the kind of disorderly thinking needed for scaling distributed and data-centric systems.

But as a matter of both pedagogy and system design, I have begun to think that Google’s MapReduce model is not healthy for beginning students.  The basic issue is that Google’s narrow MapReduce API conflates logical semantics (define a function over all items in a collection) with an expensive physical implementation (utilize a parallel barrier). As it happens, many common cluster-wide operations over a collection of items do not require a barrier even though they may require all-to-all communication.  But there’s no way to tell the API whether a particular Reduce method has that property, so the runtime always does the most expensive thing imaginable in distributed coordination: global synchronization.

From an architectural point of view, a good language for parallelism should expose pipelining, and MapReduce hides it. Brian suggested I expand on this point somewhere so people could talk about it.  So here we go.

The canonical example here is the relational join, which pairs up matching objects from two different input streams.  One of the early observations in shared-nothing parallel databases was that parallel joins can work in a fully streaming fashion even though they require all-to-all communication (Wilschut and Apers’ Pipelining Hash Join).  This is impossible to implement in stock Hadoop.  Which means, for example, that the NSA—apparently a big Hadoop user—can’t use MapReduce to match live feeds of suspicious activity reports with a repository of intelligence on suspicious people.  (Whether you think it’s good or bad that the NSA can’t use Hadoop for this task may depend on your politics regarding national security, open source, or both.)

That example is easy to understand, but maybe too easy to dismiss as the special case of “stream processing”.  Sure, Hadoop wasn’t designed for that, so maybe you should dismiss my point as saying that somebody should just build a streaming edition of Hadoop for special uses.  (Oh yeah, somebody did that.)

A more telling pedagogical example is the standard discussion of implementing PageRank in MapReduce.  What an awesome lecture: the two most famous pieces of Google technology in action at once!  The thing is, at a logical level PageRank is a simple self-join query with transitive closure: it pairs up items in the Nodes table with their neighbors in that same table, and outputs new weights until a stopping condition is reached. The matching of nodes and neighbors needs to be hash-partitioned across the network, but there is no reason to require the entire cluster to do a barrier synchronization at each iteration boundary.  The subtext of this lecture’s awesomeness is to deeply implant a misleading (and potentially very inefficient) idea about the need for synchronized execution in parallel linear algebra and query processing tasks.  That sounds like a bad way to teach.

And it goes deeper.  At a very basic level, asynchronous parallel computation is fundamentally a matter of streaming rendezvous (join) between event channels and buffers.  We converted this observation into a first-class feature of the Bloom language for distributed systems. When you write event-handling code in Bloom, you express it as the parallel join of a partitioned event stream with partitioned program state. Whether or not this metaphor makes sense to you (yet!), I assure you that it is fundamentally what goes on under the covers of any asynchronous distributed system: the matching of streams of requests or responses with stored state.  By teaching students the Google MapReduce model (as represented in Hadoop), we teach them that distributed joins need to block, and therefore cannot be the basis of the kind of streaming computation that servers must do by their very nature.  That is not only bad, it’s paradoxical: underneath the Hadoop implementation is a message handling dataflow that effectively does streaming joins.

So there—at Brian’s request I wrote down some cautionary words about using Hadoop in school.  Lest somebody read this as my declaration of allegiance to the “get off my lawn” side of the AntiNoSQLdisestablishmentarianism argument, let me clarify my view on a few key things:

• Google MapReduce and the Hadoop open-source implementation opened the floodgates for discussing big data and parallelism in the core of the computing curriculum.  This has been a critical step in moving computing education into the era of abundant computing and data resources.  Bully for that.
• I do understand that Google put barriers into MapReduce for fault tolerance purposes.  I also get that fault-tolerance is important in scale-out of parallel dataflows (we did some work on this) and that Google uses a lot of machines at once.  But as a practical matter, many Hadoop users run on a small handful of nodes and don’t need Google-scale fault tolerance features.  More to the point, when we teach computing, we shouldn’t focus on only one design point; we should focus on the fundamentals.  This is a case where the fundamentals get taught in a tangled way that only makes sense at extreme scale.
• For the record, as languages go I like MapReduce about as much as I like SQL. That is to say I would invite them both to my birthday party. But I wouldn’t invite either to the prom.  My sweetheart here is Bloom.

If you’ve been following our Bloom work, you know where this discussion is coming from: the CALM Theorem says that the real reason to use coordination is to manage non-monotonic reasoning.  Many (most?) Reduce functions in the wild are in fact monotonic w.r.t. subsets of their inputs, but the standard Reduce API defies monotonicity analysis.  And introducing barriers or monotonic computation is a waste of “time”, in the temporal logical sense of the word.

Forget your previous associations with low-tech SQL analytics, including so-called “business intelligence”, “olap”, “data cubes” and the like. This is the real deal: statistical and machine learning methods running at scale within the database, massively parallel, close to the data. Much of the code is written in SQL (a language that doesn’t get enough credit as a basis for parallel statistics), with key extensions in C/C++ for performance, and the occasional Python glue code. The suite of methods in the beta includes:

• standard statistical methods like multi-variate linear and logistic regressions,
• supervised learning methods including support-vector machines, naive Bayes, and decision trees
• unsupervised methods including k-means clustering, association rules and Latent Dirichlet Allocation
• descriptive statistics and data profiling, including one-pass Flajolet-Martin and CountMin sketch methods (my personal contributions to the library) to compute distinct counts, range-counts, quantiles, various types of histograms, and frequent-value identification
• statistical support routines including an efficient sparse vector library and array operations, and conjugate gradiant optimization.

More methods are planned for future releases.  Myself, I’m working with Daisy Wang on merging her SQL-based Conditional Random Fields and Bayesian inference implementations into the library for an upcoming release, to support sophisticated text processing.

MADlib 0.20beta was designed as a drop-in addition to the popular open-source PostgreSQL database.  The routines also all run massively parallel in the Greenplum database, which inherits the extensibility interfaces of PostgreSQL. (Non-commercial users can get the parallelism for free—no crippleware!—via Greenplum’s Community Edition.)

Obviously the first beta release is still experimental code.  But we’ve come a long way since our alpha announcement at the Strata conference last February in both breadth and usability. For me, the most exciting aspect of the beta release is that we’re about ready to grow our developer community beyond the initial committers at Berkeley and EMC-Greenplum.  If you’re interested in adding methods to MADlib—or in developing ports for other DBMS platforms (something I’d love to see happen!)—please get in touch.

And hats off to EMC-Greenplum for putting significant development resources behind this open-source effort.  I started this project at Greenplum before they were acquired, and have been happy to see EMC embrace it and push it further.  As I discussed at Strata last year, I think this is a very healthy direction for open-source software development.

Meanwhile, some of the experience is even more relevant to current technology trends than it was 4 years ago, so hopefully folks interested in social computing, software engineering, image processing, crisis response, and other related areas will find something of use in there.

For those of you whose work is represented (or underrepresented) by the article, my apologies for its shortcomings.  I still don’t have the full picture of what happened—nobody does, really.  As a result I decided to avoid using personal names of volunteers in general to avoid attributing credit unevently. I know the result seems oddly impersonal.  Setting the tone of the article was as hard as capturing the content.

Meanwhile, I encourage you to add corrections and perspective to the article in the comment box at the end of the CACM link above. Comments are welcome here too, but they might not get as well-viewed or -archived.

I’ve written a lot about Bloom in research papers and on the new Bloom website, and I have lots to say about distributed programming that I won’t recap. Instead, I want to focus here on the tangible: working code. If you’re looking for something serious, check out the walkthrough of the bfs distributed filesystem, a GFS clone. But to get the flavor, consider the following two lines of code, which implement what you might consider to be “hello, world” for distributed systems: a chat server.

nodelist <= connect.payloads
mcast <~ (mcast * nodelist).pairs { |m,n| [n.key, m.val] }

That’s it.

The first line says “if you get a message on a channel called ‘connect’, remember the payload in a table called ‘nodelist’”. The second says “if you get a message on the ‘mcast’ channel, then forward its contents to each address stored in ‘nodelist’”. That’s all that’s needed for a bare-bones chat server.  Nice, right?

The chat client code bootstraps by connecting to the server:

connect <~ [[@server, [ip_port, @nick]]]

and then runs the following logic:

mcast <~ stdio do |s|
  [@server,
   [ip_port, @nick, Time.new.strftime("%I:%M.%S"), s.line]]
end
stdio <~ mcast { |m| [pretty_print(m.val)] }

The first statement of this batch takes input from the terminal (‘stdio’), formats it, and sends it to the server on the ‘mcast’ channel.  The second statement takes messages from the ‘mcast’ channel (forwarded by the server) and prints them on the terminal.