Thank you for a quality rebuttal, and for taking the time to read and analyse my original post. I'm not going to comment on the individual points, as I agree with most of them — taken in isolation. I particularly appreciate the point on securing database views; I may incorporate it into the post if you don't mind. And top points for sticking to the original Top Gear theme and incorporating The Stig.

My comment is going to relate to the overall piece.

I think the crux of the original post was misunderstood, or quite possibly, I may have failed to express the point clearly. The post doesn't conclude that a Kafka-like messaging system couldn't be implemented on top of a database. It most certainly can. In fact, the complete title reads "You can replace Kafka with a database, but for goodness' sake don't". (Unfortunately, Medium sometimes splits the title and subtitle when it features the posts.) My argument was that, while conceptual simplicity is alluring, building even simple DB-backed messaging systems involves a lot more incidental complexity than meets the eye; the complete 'austere' solution can easily end up more 'lavish' than just using an off-the-shelf broker. It also involves a lot more engineering skill that isn't easily commoditised. I can clearly see that your knowledge of certain database products is higher than an average engineer, which is a case in point. Your knowledge of Brewer's CAP theorem already puts you into the 95th percentile.

As a purely anecdotal example, I recall a particular client who "started small" and built a queue on top of MySQL. That was sufficient for them in 2005, being a startup at the time. They are a listed company now. In an ideal world, that aspect of the system should have been decommissioned long ago and replaced with a message broker, as the business grew, the number of systems using the shared database ballooned, and the performance demands skyrocketed. But it wasn't. The developers kept trying to optimise their home-grown queue, trying to extract every last iota of performance — flogging a dead horse. I suggest that "starting small" should still involve selecting the right tool for the job, unless the alternative is genuinely significantly simpler/cheaper/quicker and offers a faster time to market or some other material competitive advantage. I don't believe that implementing and maintaining a messaging platform inside a DBMS is easier or more cost-effective than spinning up a dedicated message broker.

Software architect, an engineer, and a dad. Also an avid storyteller, maintainer of Kafdrop, and author of Effective Kafka.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store