This week I joined Decodable. I’m grateful to my former colleagues at Treeverse for allowing me to join them on the journey with lakeFS - but something about the streaming world was too strong to resist 😁.
I spent several years previously at Confluent bringing my data engineer’s view of the world to help advocate for what was being built within Confluent as ksqlDB—and the broader Apache Kafka community in Kafka Connect—as a coherent platform on which analytics and data integration solutions could be built.
Turns out, stream processing was only really just getting started. In the last 12-18 months a veritable plethora of stream processing projects and companies have shown up, demonstrating that there is not only demand for it but also different approaches to take too.
The advent of real-time data stores such as Apache Druid, Apache Pinot, and Clickhouse further complicates the picture (or makes it more interesting, depending on your view). How much do you serve from your operational application, your streams, or your dedicated realtime store? What’s driving the need for the data, and what kind of access patterns do you have?
In short, I have unfinished business with streaming.
Unsuprising really, since event streams are unbounded 🤓 <groan/>.
A.B.L. (Always Be Learning)
One of the many things that excites me about joining Decodable is the opportunity to dive in and learn Apache Flink. I’m planning to blog my journey with it (similar to how I did as a complete n00b going into Go), so if you have any burning questions about it already then make sure to drop me a line and I’ll be happy to use it as an excuse to go and find out the answer. And find out the answer I shall, because one of the other many reasons that I’m so excited about Decodable is my colleagues here:
Never Be The Smartest in the Room
Well, there’s never been any danger of that at any of my previous gigs—I’ve been fortunate to work with some excellent people—and the same holds true for Decodable.
One of the things that really sealed the deal for me during the interview process was just how damn smart the folk were. Not in a “I ask you stupidly pointless question to highlight how smart I am, you wriggle for an answer that I’ll not really listen to, rinse & repeat” kind of way. But in a focussed and probing and inquisitive way, with true dialogue and listening. That’s the kind of stuff you can’t LeetCode for, and is a hugely positive indicator.
As well as being lovely people to chat to and sharp as a sharp thing that’s just been sharpened, there’s some extensive and deep knowledge about the stream processing world in general and the projects which Decodable uses. These include folk such as Eric Sammer, Sharon Xie, Flink PMC Chair Robert Metzger, and former lead on the Debezium project Gunnar Morling.
What Will I Be Doing at Decodable?
Who knows?! That’s the fun part of joining a 30-person startup 😉.
Srsly tho: my starting point for this answer is my blog from earlier this year What Does This DevEx Engineer Do?. But very shortly after that my answer is: getting stuck in and helping build a really awesome streaming platform. Friction logs, blogs, shitposting, community, docs, making videos…whatever needs doing to make the experience for developers the very best 🏆.
Stream Processing - We’ve Only Just Got Started
So, time for me to get back into stream processing land. I’ll save you the fluff and gumpf about the business value of stream processing, the use cases, all the evils and perils of batch, yada yada. If you know, you know. And if you don’t, well, join me for the ride 🙃.
