Flutterby™! : Once wrote the heavily threaded kernel 2014-04-23 22:15:06.962004+00

Once wrote the heavily threaded kernel

2014-04-23 22:15:06.962004+00 by Dan Lyke 5 comments

Once wrote the heavily threaded kernel to a distributed database, and had fewer concurrency issues than this business logic code. #sigh

You know, the interesting thing about REST is the fact you can implement REST-style systems with any cross-thread-call or IPC mechanism; not just with HTTP servers. The trick (and the hard part) is re- thinking the logic so that each transaction can be entirely atomic.

This means you can take a highly concurrent system with hundreds of threads and rethink it into a few transaction process servers -- each front-ended by some kind of message queue -- and a ton of transaction clients that simply submit a transaction into the proper queue and then block until they get a response. Done. (extra points for doing timeouts)

Of course this may not describe what you are working with to even a minimal extent. But I'll bet yours can be reworked that way; given time and resources.

Jack: Message Queing was something I saw from afar (IBM MQ style) a long time ago, that we know use as part of our core processes for almost everything. It all goes into queues and gets processed as soon as possible, without tying things up. It's allowed some incredible scaling of systems without a perceived difference in how things are working.

#Comment Re: made: 2014-04-25 01:16:31.279814+00 by: Jack William Bell

Right! The message queueing gives you the scalability. The REST architecture gives you the framework for organizing the transaction servers.

None of this is really new. The thing is, people tend to think of REST as being something you only use for websites. But it is an architectural pattern; you can use it anywhere that you do call and response between client and server. No HTTP required.

As you've noted, though, one has to be really careful with the solution design. A consultant group once built us a queue-based system, mostly because that was what they were used to, but it didn't scale well at all because the different pieces of data still had (and needed to have) dependencies - so stuff got backed up or disappeared into a retry hell.

One of the things I love about real SQL installations is that the model is in the server.

I've been trying to figure out a good API framework for reducing other operations to CRUD that allow me to put a distributed layer in front of the database that would still enforce the sorts of business rules you put in the model.

Right now I'm basically passing around hashes, and using a setup that lets me easily put pre and post hooks on the various ops, but there's some atomicity and side effects that I haven't all really got a good framework for yet.

But: Yeah, everything is CRUD (Create/Retrieve/Update/Delete), tell it which objects you want to operate on, enforce integrity in the model, write tests above the model layer.

On the MQ everywhere: I've been playing with Node.js recently, mostly with Derby, and when I get the right MQTT layer in there... What I need is that CRUD model over MQTT...