A major source of frustration in distributed programming is that contemporary software tools—think compilers and debuggers—have little to say about the really tricky bugs that distributed systems developers face. Sure, compilers can find type and memory errors, and debuggers can single-step you through sequential code snippets. But how do they help with distributed systems issues? In some sense, they don’t help at all with the stuff that matters—things like:
- Concurrency: Does your code have bugs due to race conditions? Don’t forget that a distributed system is a parallel system!
- Consistency: Are there potential consistency errors in your program due to replicated state? Can you get undesirable non-deterministic outcomes based on network delays? What about the potential for the awful “split-brain” scenario where the state of multiple machines gets irrevocably out of sync?
- Coordination performance: Do you have performance issues due to overly-aggressive coordination or locking? Can you avoid expensive coordination without incurring bugs like the ones above?
These questions are especially tricky if you use services or libraries, where you don’t necessarily know how state and communication are managed. What code can you trust, and what about that code do you need to know to trust it?
Peter Alvaro has been doing groundbreaking work in the space, and recently started taking the veil off his results. This is a big deal. Read More »
Bright and early next Monday morning I’m giving the keynote talk at PODS, the annual database theory conference. The topic: (a) to summarize seven years of experience using logic to build distributed systems and network protocols (including P2, DSN, and recent BOOM work), and (b) to set out some ideas about the foundations of distributed and parallel programming that fell out from that experience.
I posted the paper underlying the talk, called The Declarative Imperative: Experiences and Conjectures in Distributed Logic. It’s written for database theoreticians, and in a spirit of academic fun it’s maybe a little over the top. But I’m hopeful that the main ideas can clarify how we think about the practice of building distributed systems, and the languages we design for that purpose. The talk will be streamed live and archived (along with keynotes from the SIGMOD and SOCC conferences later in the week.)
Below the break is a preview of the big ideas. I’ll post about them at more length over the next few weeks, hopefully in more practical/approachable terms than I’m using for PODS.
Read More »
We were happy to find out this week that our BOOM project and and Bloom langauge have been selected by Technology Review magazine as one of the TR10, their “annual list of the emerging technologies that will have the biggest impact on our world.” This was news to us — we knew they were going to run an article, but weren’t aware of the TR10 distinction. Pretty neat.
I’ve been getting a lot of questions since the article launched about the project and language. So while folks are paying attention, here’s a quick FAQ to answer what the project is all about and its status.
Read More »
It’s been about 6 years now that we’ve been working on declarative programming for distributed systems — starting with routing protocols, then network overlays, query optimizers, sensor network stacks, and more recently scalable analytics and consensus protocols.
Through that time, we’ve struggled to find a useful middle ground between the pure logic roots of classical declarative languages like Datalog, and the practical needs of real systems managing state across networks. Our compromises over the years allowed us to move forward, build real things, and learn many lessons. But they also led to some semantic confusion — as noted in papers by colleagues at Max Planck and AT&T.
Well, no more. We recently released a tech report on Dedalus, a formal logic language that can serve as a clean foundation for declarative programming going forward. The Dedalus work is fairly theoretical, but having tackled it we’re in a strong position to define an approachable and appealing language that will let programmers get their work done in distributed environments. That’s the goal of our Bloom language.
The key insight in Dedalus is roughly this:
Time is essential; space is a detail.
Read More »
It’s official: the name of the programming language for the BOOM project is: Lincoln Bloom.
I didn’t intend to post about Bloom until it was cooked, but two things happened this week that changed my plans. The first was the completion of a tech report on Dedalus, our new logic language that forms the foundation of Bloom. The second was more of a surprise: Technology Review decided to run an article on our work, and Bloom was the natural way to talk about it.
More soon on our initial Dedalus results.