The reason you see a lot of people talk about using Hadoop on plain text data is that many data sets come into a system that way and are not queried often enough to justify converting to anything else. A perfect example is web search. Every night, you might crawl a number of web pages and end up with raw text data. You then scan these pages to update an index, and never do anything else with them. It wouldn’t make sense to convert the data to some optimized binary format first if you can just run your analysis on the raw text files. Other web backend applications, such as spam detection, have the same “run-once” property (you never check a new email twice to see whether it’s spam). For these “run-once” applications, Hadoop provides a convenient means to run over raw data and get your answer quickly without wasting time on building indexes, packing the data into your database’s favorite storage format, etc. From my understanding, Greenplum and others are starting to support similar “run over raw data” functionality.

Wrt pipelining vs materialization, the premise of the original blog post, and Joydeep’s comment, a Yale undergraduate who took my database architecture and implementation class I teach every spring played around with storing Map output in an in-memory fs. He found this had limited affect on performance; however, this is partly because it is hard to create queries where Hadoop’s pessimistic checkpointing (to borrow Joe’s language) is the performance bottleneck. Some might argue that “sort” is a little bit unusual in that the Map task does zero filtering (the output size is the same as the input size).

The generalization of the problem you speak of would involve pipelined DAGs (or maybe even a simple pipeline). The scheduling choices there would involve materializing intermediate outputs vs restarting the pipeline. The twist here is that some operations (such as sort) break the pipeline anyway.

Here’s an example of a work in this area:
Matching and Scheduling Algorithms for Minimizing Execution Time and
Failure Probability of Applications in Heterogeneous Computing
Atakan Dogan, Student Member, IEEE, and FuÈ sun OÈ zguÈ ner, Member, IEEE

Link: http://hpcnl.ece.ohio-state.edu/pdfs/match_sched_alg.pdf