CQRS - The Cult of Shiny Things

CQRS has become another casualty of the buzzword culture and the cult of 'shiny things'

What started as a collection of some reasonably good principles has now turned into an almost religious mantra for some, with more and more outrageous claims, and almost no basis in fact or experience.

Scalability

Claims like "almost infinitely scalable systems" associated to CQRS are ludicrous, especially as none of the currently "near infinitely scalable systems" use CQRS or have probably even heard of it. The largest systems on the planet have no need of CQRS, they use many tricks and techniques that have been associated with CQRS - but they don't use it, nor are they ever likely to use it, because when you have really large systems, things like CQRS become totally irrelevant.

The scaling benefits of CQRS come from understanding eventual consistency and messaging - concepts that were around long before CQRS, and that will be around long after it too.

Task Based UIs

Well, yes CQRS may require a task based UI, but task based UIs were again around long before CQRS, and will again outlast it. This is just a different way of thinking about how people interact with computers - this is a UX issue, not a technical one.

In fact, if anything, CQRS can be a major cause of issues here - when you really have a CRUD style system, CQRS is going to cause you all sorts of issues you will need to resolve. And much as I dislike CRUD based UIs, the reality is we have abused users to work this way in many cases - so sometimes that is just how we have to make our next UI work.

Side Effect Free

Another aspect of CQS (and by extrapolation CQRS) is that functions employing CQS are side effect free - you don't accidentally introduce subtle bugs by writing and reading in one operation.

And while this is nice in theory, the scale of system where this preventative measure would have benefit is the one where this problem is just as easily solved by many other simpler means, like just writing smart code. Larger systems have so many points in them where read and write are already being mixed, CQRS here just lends an illusion of safety.

Denormalised Read Models

A valuable benefit of CQRS is the concept that the views have specific read models that are denormalised from the (presumably) complex domain model.

This has a number of benefits, though these are largely boiled down to improved efficiency of querying (and action that tends to happen frequently). These models can also be extrapolated out to provide reporting and MI models too.

But again, we have been doing this for many years already, and usually to account for overloaded database servers that we don't wish to query against, particularly with MI.

And querying these days, even with horribly complex joins on highly complex models is becoming faster and faster. Push querying to a slave server, and you gain all the benefits of a rich model, and fast queries. And for even more speed you can look to NoSQL solutions - which are effectively based on the principle that you denormalised your data anyway.

Performance Considerations

Part of the theoretical benefit of CQRS is that your queries will be faster as your denormalised datastores are updated at write time, so you don't have to do horrible joins on queries (which in theory on most systems happen far more frequently than writes)

These days, the performance of relational databases and especially NoSQL databases is advancing rapidly. The new breed of databases takes things like sharding and replication very seriously, and as first class features, not afterthoughts like the databases of old did.

Replication in MySQL takes roughly 0.0002 of a second (see http://nr-content.s3.amazonaws.com/railslab/videos/17-ScalingRails-Scaling-Your-Database-Part-1.mp4 at 6 minutes in) - compare that with how long a message based approach like CQRS will take to push that to your secondary datastores and you are talking orders of magnitude faster for the dumb replication approach. Apparently it operates at this speed for up to 10 slave databases before degrading.

Now, think about this bit, the 37 Signal systems (a huge system) operates on a single MySQL instance running with 105Gb of memory caching. And 37 Signal could be considered the archetype of an Active Record approach, after all it is the reason Ruby on Rails exists. Are you writing a system that big?

Maybe you are so let's examine a larger system - Wordnik stores over 12 billion documents, 3Tb of data per node, 500k requests per hour and 4 times that at peak load, and yet their fetch time is around 60ms. (http://blog.wordnik.com/12-months-with-mongodb)

Let's not even begin to discuss Twitter, Ebay, Facebook or other similar "mega systems" - they still respond in milliseconds with tens or hundreds of millions of users - think that's down to CQRS?

Dealing with Complexity

Yep - CQRS deals with complexity to one degree - it forces you to explicitly think about what each call is doing - "Is this writing data or reading data?" - and that thought process is a good thing.

But it introduces another layer of complexity that is more subtle and harder to deal with - the replication to secondary datastores is complex in itself - it effectively needs to do that horrible join you are trying to avoid in your queries so that it can update your denormalised tables. That in itself isn't simple.

But it also needs to be able to maintain those secondary datastores if, for example they are corrupted and need recreating or if they miss some messages. In normal replication this is trivial, drop the slave, add a new slave - wait a short while - someone already wrote all that for you on a database system - you are going to need to write it all yourself with CQRS.

So Why Have I Implemented CQRS Before?

Well, at the time I first implemented CQRS I had two reasons for doing so - and oddly neither had to do with any of the aforementioned technical aspects. In fact, one was a technical problem and one was a people problem.

Firstly this was nearly two years back, and the system that we were writing a layer over was a very old and crumbling legacy database system. It had no real normalised schema, and was open to abuse from just about every development team in the organisation without warning to others - it was also horribly prone to outages.

One project had just "failed" (though I'm sure some would say it was just a lesser success), primarily due to trying to put a domain model on top of the legacy database.

So, the problem we had was to create a disconnected system that could operate independently of the legacy system, but also use the legacy system as it's true authority for the information it used. Because of the horrible complexity in the legacy system, using CQRS allowed us to write what were frankly quite horrible SQL statements for reading data from legacy, while using a pseudo-document database for the daily operations. We then used horrible SQL statements to replicate this data back into legacy.

Of course, what we were really writing was a standard message based architecture, but the term CQRS encompassed that quite nicely.

We could have approached this without the CQRS badge - but it served as a useful political tool - a number of the existing team had heard of it and were quite enthusiastic, and management would be confounded enough by it to not get in the way too much. It also broke the mental model the development teams had with "put an ORM over the database".

So the secondary benefit of CQRS here was a people problem - it was a new buzzword - it got new enthusiasm from a team that was demotivated from the previous project - and more than anything - that motivation was going to be the key to the success or failure of the new project.

So Why Have I Presented On It Before?

Well two reasons really, the first being that I have always tried to dispel the mythology around CQRS and to explain the real benefits of a flexible and scalable architecture - it just so happens that people want the buzzword explaining - you have to admit it's a pretty vague and fuzzy term with a myriad of definitions and explanations.

The secondary reason being, as my last presentation at DDD Sydney explained - my objective was to make you think. There's a lot of good things hiding under the CQRS banner, and I really would like people to think about better ways of writing software. 

Conclusion

In the few years since CQRS was "invented" it has got a bit of a cult buzzword status. It is talked about as though it fixes a multitude of problems.

But in that time, NoSQL has really taken off. RBDMS' have got far more powerful. And memory and processing power has increased significantly. Database caching is exceptionally impressive. REST gives us proxy caching for near enough free. Memcached and Velocity make light of massive data loads. We even have the nefarious "cloud" now, where processing power and memory is cheaper than the effort involved in maintain complex architectures.

Scaling systems is now a pretty commonplace business problem for most of the startups taking off  -and they don't need CQRS - there is a very good chance you don't either. We are past that point in time where we had to prematurely optimise things like database queries.

If it's about thinking differently, then I can tell you all you need to know about CQRS in a simple paragraph:

Learn to think of systems as being inconsistent at all times, no matter what illusion they present of 'real time'. Learn to embrace messaging, REST, caching, replication, sharding, and a hundred other techniques to solve these problems.  Embrace user centric task based UIs, embrace strong UX, embrace strong domain driven design, embrace business led functionality and delivering systems that match real requirements. Embrace thinking differently.

CQRS is no silver bullet, in fact it has almost now become a solution to a problem that doesn't exist any more. If you can give me a problem that CQRS claims to solve, I'm fairly sure I can solve it more easily than I could have done 2 years ago when I started using CQRS. I'm sure I'll still use a lot of the things that CQRS encapsulates, because they are after all very good principles - I'm just pretty much over playing buzzword bingo!

 

 

 


Posted 10-29-2010 1:12 AM by Jak Charlton
Filed under: , ,

[Advertisement]

Comments

mynkow wrote re: CQRS - The Cult of Shiny Things
on 10-29-2010 3:30 AM

"...than the effort involved in maintain complex architectures."

Man, cqrs is even simpler than regular CRUD systems. You have to write more simpler code, that is it.

PS: Update your post and talk about the audit. How this is achieved in CRUD systems and then in CQRS systems. Btw, CQRS is good only for 4% of the projects.

Elliot wrote re: CQRS - The Cult of Shiny Things
on 10-29-2010 6:05 AM

I assumed Greg named it CQRS sometime last year to distinguish it from CQS. Was it ever touted as something new other than being a new anacronym to consolidate a bunch of existing approaches to building distributed systems?

You've touched on the technical aspects that you tend to get for free when taking a CQRS approach but not the main benefit of being able to more effectively model the business problem that in turn results in simpler code.

Jonathan Oliver wrote re: CQRS - The Cult of Shiny Things
on 10-29-2010 7:35 AM

@Jak,

This "additional layer of complexity" (which is actually pretty dang simple) is only for those that *choose* to do introduce and/or leverage eventual consistency between the read and write models.  We can still keep things fully consistent without any kind of message-based replication.  In fact, Greg strongly advocates this approach for adopters of CQRS and then recommends introducing eventual consistency if the load on the system demands it.

In fact, a good portion of your post is applying the name CQRS improperly.  CQRS has nothing to do with eventual consistency or messaging.  Those are *additional* patterns that can be leveraged on top of CQRS.   CQRS is merely the separation of reads from writes.  Greg has posted about this before on his blog.

DaRage wrote re: CQRS - The Cult of Shiny Things
on 10-29-2010 9:56 AM

Ohh... the cult of shiny things.. i agree with you completely on this. NoSql, DDD, CQRshit, etc. The problem is that they don't stick around long enough to validate these principles and build a useful thing out of them because before you know it they've jumped on a new buzzword.

I learned just to ignore noise, despite its loudness.

Julien Letrouit wrote re: CQRS - The Cult of Shiny Things
on 11-01-2010 1:14 PM

Mmmh, I think CQRS advantages are not really technical. Most of the benefit are business benefit (ease of integration, team segregation, etcc). See Greg Young presentation: skillsmatter.com/.../zx-546.

Then, CQRS is not something to apply everywhere. I agree with you that people are starting to advocate CQRS as a silver bullet, and that is not a good thing. You need a rich domain to start with. So where CQRS has a lot of value is when coupled with DDD. Most examples you are citing are irrelevant to DDD, so of course, CQRS is not a good fit for those. Which does not mean that in the right context, there could not be tremendous value. Again, it is a matter of using the right tool at the right place.

The only technical nice thing about CQRS is, yes, nothing is new in there, but the set of concepts it encapsulate make the odds of having good frameworks for that in the wild more likely. And make implementing DDD a little easier, a little more standard.

About The CodeBetter.Com Blog Network
CodeBetter.Com FAQ

Our Mission

Advertisers should contact Brendan

Subscribe
Google Reader or Homepage

del.icio.us CodeBetter.com Latest Items
Add to My Yahoo!
Subscribe with Bloglines
Subscribe in NewsGator Online
Subscribe with myFeedster
Add to My AOL
Furl CodeBetter.com Latest Items
Subscribe in Rojo

Member Projects
DimeCasts.Net - Derik Whittaker

Friends of Devlicio.us
Red-Gate Tools For SQL and .NET

NDepend

SlickEdit
 
SmartInspect .NET Logging
NGEDIT: ViEmu and Codekana
LiteAccounting.Com
DevExpress
Fixx
NHibernate Profiler
Unfuddle
Balsamiq Mockups
Scrumy
JetBrains - ReSharper
Umbraco
NServiceBus
RavenDb
Web Sequence Diagrams
Ducksboard<-- NEW Friend!

 



Site Copyright © 2007 CodeBetter.Com
Content Copyright Individual Bloggers

 

Community Server (Commercial Edition)