I have a system running in Azure that accepts messages via a REST endpoint. The customer has a certain number of devices that send some data for example temperature and we present this data in a web interface. The part of the system that accepts messages is separate from the website (microservices!) but they access the same database (not so microservices?). I am building a new functionality that allows the customer to setup notifications. For example the customer defines that if any of his devices sends a temperature above 50°C then an e-mail should be sent. The notifications can be instant (or of relatively short period) or aggregated (daily, monthly, etc.).
The way I had built these things in the past on premise was to setup a Windows Service that polls the database periodically gets the notification definitions, gets the corresponding messages and sends the proper e-mails be it instant or aggregated. However something tells me that this way is not cloudy at all. Am I supposed to use a message queue, a service bus or something else? If yes which one? What about aggregated notifications? If not what is the best equivalent of a Windows Service in Azure today for my use case? Am I still supposed to use a Worker Role?
Also note that in the future the messages may be taken out of the relational database and put in a NoSQL solution and in practice they are much more complex than one number (the REST endpoint does some processing before storing them in the DB). The REST endpoint does have access to the notification definitions and could potentially process notifications itself but I’d like to avoid that because I like the clean separation and simplicity of the separate services I currently have. Also processing the message when received does not solve the aggregation problem. In case it matters we expect relatively small number of users but great number of messages and hopefully great number of devices sending messages. In case it is not clear there are multiple users each of them getting notification for his own data.
7
We ended up implementing the “Windows Service” scenario knowing that it would not scale but intending to build a scalable solution when we have to. With the current implementation a Web Job runs a console application that queries the DB for all notification definitions, runs appropriate queries and sends e-mails. It is easy to test (just run the console app) and easy to deploy (web jobs deploy with the web site). The best part is that if we go for the scalable solution we can just remove this part without affecting any other code because notification definition UI and database schema do not depend on the actual sending method.
We ended up deciding against Stream Analytics because Stream Analytics are not very useful for ever changing queries. Changing the query requires a restarting the Stream Analytics job. Putting the notification definitions in the job is hard. There is something called training data that can be used but it is not very suitable for this case.
After we implemented the solution Microsoft launched Azure Functions which at first sight seem like a good fit. I did not do a detailed research because we already had a working solution at that point. I tend to dislike how separate Functions tend to be from the rest of the codebase and I worry about deployment. I guess there are solutions to these problems but they were not in the demos.
The scalable solution that we had in mind and will evaluate again if we need to scale would work as follows – The Message processing part sends each and every message into azure queue. A consumer gets the messages one by one and checks if the message is eligible for any notification. If it is it is put in DocumentDB or Azure Tables bucket list which is accessible based on the notification ID(s). Another consumer periodically checks the notification buckets and sends all the messages and deletes the bucket list. The solution seems scalable because we can add whatever number of consumers we need and azure queues are built to be scalable. There seems to be no bottleneck part. Now of course with Azure Functions we can replace parts of this systems with if they turn out to be better suited for the task.
Of course I am still open to criticism and other ideas.