Advice on App/Service Architecture

I’m starting a project that will have a web front end for the users coupled with a database. There will then be a stand-alone service running that will, on a specified interval, poll an API and update the database with the changes it found.

What I need advice on is how the service should be setup. When a user creates an account on the site, I”ll need to create a “hook” in the service that will then go poll the API for that specific user.

I looked at building the service using NodeJS because I could use the non-blocking structure of it along with callbacks then the API returns data that needs to be reflected in the database. I don’t know how I will be able to create more monitors when a user creates an account since it’s already running. Can NodeJS run in a continuous loop as as service?

I am familiar with .Net and could use a Windows Service, but how would that scale with polling when the user numbers grow?

I’m still a fuzzy on exactly what you are asking, but based on your comment, I think I can offer some advice. As I understand it, you’re mainly asking what you can be called by a “hook” and then ran in the background (not in real-time) to perform some sort of db-related update based on calls made to an in-house API. I imagine a simple diagram like the following:

 +----------------+                    +------------------+
 |                |                    |                  |
 |   web app      |    after-signup    |    background    |
 |                |+------------------>|    service       |
 |                |   async-call or    |                  |
 +----------------+      "hook"        +------------------+
                                      +  ^       +
                                      |  |       |
                                      |  |       |
                                      |  |       | update db
                             call API |  |       |
                                      |  |       |
                                      |  |       |
                                      v  +       v
                               +----------+     +------------+
                               |          |     |            |
                               |  API     |     |   Database |
                               +----------+     +------------+

So, in regards to how this “async-call” is made, there are a couple of common options that come to mind.

1. Using a Queue

Perhaps something like Redis or RabbitMQ. In this scenario, after the user has signed up, you’re web-application will push some values into the queue (all the data the background service would need). Your background service would then be a program that simply watches the queue in a continuous loop and processes an item when something is found on the queue.

While this is a very simple approach, it does require that you install some sort of queue server inside your infrastructure. In that case, the diagram would change sightly:

 +----------------+                 +---------+             +------------------+
 |                |                 |         |    pop      |                  |
 |   web server   |    push         |         |<-----------+|    background    |
 |                |+--------------->| queue   |+----------->|    service       |
 |                |   async-call    |         |   process   |                  |
 +----------------+                 |         |             +------------------+
                                    +---------+             + ^       +
                                                            | |       |
                                                            | |       |
                                                            | |       | update db
                                                   call API | |       |
                                                            | |       |
                                                            | |       |
                                                            v +       v
                                                     +---------+     +------------+
                                                     |         |     |            |
                                                     |  API    |     |   Database |
                                                     +---------+     +------------+

Fire and Forget

The other approach is to setup a simple web-server as your processing service. All your web-application will need to do is send a fire and forget HTTP request to your service. By fire and forget I mean that you do not wait for a response when you make the HTTP call. You’ll have to check what libraries you’ll need to use for your language of choice to make this happen.

Then your background service, another web service in this case, will accept the request and start it’s processing. Since the calling service is not waiting for a response, there is no rush here.

Node

In either of these scenarios, you can use NodeJS if you’d like. It will work fine for both approaches. However, Ruby’s EventMachine and Python’s Twisted frameworks will do equally as good if evented IO is your goal, in which case I would have to assume that you are planning on handling a high-volume of requests/queue-items from your we application.

Let me know if this answers your question. Like I had mentioned previously, the question seems kind of broad but I answered what I could.

Filed under: softwareengineering - @ 19:57

Thẻ: architecture, web, web-services

Thiết kế website giá rẻ

Danh mục

Advice on App/Service Architecture

1. Using a Queue

Fire and Forget

Node