How to implement searchable persistent queues?

We will implement a system , which will receive quite a lot of requests through web service interface. We will have to queue the requests and process them asynchronously. Consequently we will have to have also some output queue with responses (QR).
The web service client will have the possibility to query his (and only his) processed requests in QR using another web service method.
Also, client will have the possibility to find the exact response from the QR by providing id of the request (returned after posting the request).
Also, client will be abel to delete the request from QR.
There will be cca 60 requests per second.

The question is how to implement QR. We need reliability and durability (nothing is allowed to be lost).

Now the question how this can be implemented … . There are several options, which come to my mind:

  1. Using RDBMS – but this is far from ideal as the performance would not be ideal
  2. employ something like Redis or memcached – having one incoming queue in redis and for each client list with processed items (responses), but i am sceptical about reliability and durability. We can not afford to lose any of the requests or responses.
  3. Some other noSQL solution ?


I think you’re overthinking things. You can use a traditional SQL-based RDBMS. It may or may not be fast enough (although I suspect you’re worrying prematurely about optimization), but the only way to tell would be to try it.

Just make sure that you write your code that interacts with the storage system in an abstract-enough way that it is simple to change to another storage method later if your RDBMS turns out not to be good enough.

As an aside, I see nothing in your problem description that is a good fit for a NoSQL database of any kind. You have no non-persistent or easily-reproduced data (so it is not appropriate for a memory-based database). You need strong consistency guarantees (so it is not appropriate for the various systems that let ACID compliance slide in order to get a performance boost). Your data sounds like it is relatively simply structured (so you won’t have Object/Relation “impedance mismatch” issues that could push you towards a document store or an object database). It doesn’t sound like the relationship between items is more important than the items themselves (so a graph database is unlikely to be the best choice). It sounds like you’ve picked on NoSQL because NoSQL has a reputation of being more scalable than RDBMSs — this is sort-of true, but only for applications where it is a natural good fit. Evaluate whether your application uses any of the specific fceatures of a NoSQL solution before just assuming it will create a performance boost.

Trả lời

Email của bạn sẽ không được hiển thị công khai. Các trường bắt buộc được đánh dấu *