Comments on: First GigaOm post on data-parallelism is up http://databeta.wordpress.com/2008/11/08/first-gigaom-post-on-data-parallelism-is-up/ on computing and data .. in permanent beta Wed, 21 Oct 2009 05:02:21 +0000 http://wordpress.com/ hourly 1 By: Siva http://databeta.wordpress.com/2008/11/08/first-gigaom-post-on-data-parallelism-is-up/#comment-16 Siva Wed, 25 Feb 2009 02:58:54 +0000 http://databeta.wordpress.com/?p=17#comment-16 There's also the Active Data Repository project (http://www.cs.umd.edu/projects/hpsl/chaos/ResearchAreas/adr/) that proposed using a MapReduce-like paradigm to represent certain computations on multi-dimensional scientific data. There’s also the Active Data Repository project (http://www.cs.umd.edu/projects/hpsl/chaos/ResearchAreas/adr/) that proposed using a MapReduce-like paradigm to represent certain computations on multi-dimensional scientific data.

]]> By: jmh http://databeta.wordpress.com/2008/11/08/first-gigaom-post-on-data-parallelism-is-up/#comment-15 jmh Tue, 24 Feb 2009 21:45:33 +0000 http://databeta.wordpress.com/?p=17#comment-15 I'm not sure how to define equivalent, but any discussion of parallel dataflow software should start with <a href="http://portal.acm.org/citation.cfm?doid=129888.129894" rel="nofollow">Gray and DeWitt's CACM survey</a> on parallel databases. You can work backwards from there through the Gamma and Bubba projects. More recently there's <a href="http://portal.acm.org/citation.cfm?id=1272996.1273005" rel="nofollow">Dryad</a> from Microsoft Research. I keep meaning to read the <a href="http://portal.acm.org/citation.cfm?id=1453856.1453865" rel="nofollow">Clustera</a> work from Wisconsin. And there's been a variety of academic work around Hadoop itself in the last year or two, including both internals issues and languages (Pig/JAQL/etc). The Hadoop-centric stuff should be relatively easy to find with a web search. MapReduce is not so far from data streams. Our work on FLuX for <a href="http://portal.acm.org/citation.cfm?id=1007568.1007662" rel="nofollow">Fault-tolerant</a> and <a href="http://portal.acm.org/citation.cfm?id=375663.375773" rel="nofollow">Load-balanced</a> parallel data streams came out about the same time as Google's MapReduce, and it's interesting to compare them. This grew out of earlier work by the <a href="http://pages.cs.wisc.edu/~remzi/" rel="nofollow">Arpaci</a>-<a href="http://pages.cs.wisc.edu/~dusseau/" rel="nofollow">Dusseaus</a> on <a href="http://portal.acm.org/citation.cfm?id=592637.592639" rel="nofollow">River</a> and <a href="http://now.cs.berkeley.edu/NowSort/" rel="nofollow">NOW-Sort</a>. I’m not sure how to define equivalent, but any discussion of parallel dataflow software should start with Gray and DeWitt’s CACM survey on parallel databases. You can work backwards from there through the Gamma and Bubba projects. More recently there’s Dryad from Microsoft Research. I keep meaning to read the Clustera work from Wisconsin. And there’s been a variety of academic work around Hadoop itself in the last year or two, including both internals issues and languages (Pig/JAQL/etc). The Hadoop-centric stuff should be relatively easy to find with a web search.

MapReduce is not so far from data streams. Our work on FLuX for Fault-tolerant and Load-balanced parallel data streams came out about the same time as Google’s MapReduce, and it’s interesting to compare them. This grew out of earlier work by the Arpaci-Dusseaus on River and NOW-Sort.

]]> By: Siva http://databeta.wordpress.com/2008/11/08/first-gigaom-post-on-data-parallelism-is-up/#comment-14 Siva Tue, 24 Feb 2009 19:09:41 +0000 http://databeta.wordpress.com/?p=17#comment-14 It would be great if you could document equivalent concepts to MapReduce that have been explored in academia. It would be great if you could document equivalent concepts to MapReduce that have been explored in academia.

]]>