Introduction
Hi everyone,
in my company we are using microservice approach and of course are trying to do it as correct as possible. There is a new requirement coming up where I have laid out a proposal for an architectural design. I have received feedback to change the implementation, however this would require direct HTTP connection over 2 microservices. I wanted to ask if there are cases where chatty microservices are okay and maybe in my example we are not even talking about chatty microservices
Context
So, we want to GET and POST data to an external API that is outside our company. This API updates their data once a day so the idea was to use a TimerTrigger (Azure specific, like a cronjob) that synchronizes the data from the GET endpoints once a day.
As I mentioned we would also do some POST requests where we send data the user has created to their API. For that part I thought we put the user requests into a queue that a data synchronization service can read from and process whenever the time is there.
I will go into more detail now to first talk about my approach and then talk about the second approach that was suggested by our architects
Proposal one (My initial plan)
My plan is to have these things
- Microservice 1 (ApiService from now on) for our internal end-users without any connection to the external API
- Microservice 2 (DataSynchronizationService) for synchronizing data from the external API to our system to be consumed
- CosmosDb to store the synchronized data
- ServiceBus to use POST endpoints in an asynchronous way
So basically, we will have one ApiService which is responsible for allowing OUR users to retrieve data from. This ApiService WILL NOT have any direct connection to the external API but use the data that is synced by the DataSynchronizationService.
So basically, an end-user makes a request RETRIEVE DATA to the ApiService which takes all the data from the database without relying on the external api. The DataSynchronizationService will run once a day to update the data in the database.
When an end-user makes a request to POST DATA to the ApiService, the ApiService will add an item to the Queue which the DataSynchronizationService will pick up and send to the external API
However, I already have some points I’m not so sure about with this approach
- Is it okay that 2 microservices are sharing the db? I could create an EventHub in between these two but for me this feels like way too overengineered and duplicated as the one MicroService is just for DataSynchronization
- Should maybe all of it happen in ONE microservice, because everything is coupled together? But then I would send a queue to the same service just to run it in the background?
Proposal two (Suggested plan)
For the suggested plan we will have
- Microservice 1 (ApiService from now on) for our internal end-users which handles anything like synchronizing the data, but also providing it for the user
- Microservice 2 (DataSynchronizationService) which will only contain the cronjob and service bus
- CosmosDb to store the synchronized data
- ServiceBus to use POST endpoints in an asynchronous way
Basically their idea is the following
The ApiService will handle everything. The user will query the ApiService to retrieve the data, but also to create the data. The ApiService will also have an endpoint to synchronize the data. This endpoint is called by the DataSynchronizationService. So the DataSynchronizationService has just a TimerTrigger (Azure function like cronjob) that does nothing else but make an HTTP request to our API. For me, this already feels like an anti-pattern because the DataSynchhronizationService makes a http request to the ApiService which makes again multiple http requests to external-service.
Then, when ApiService will create a Queue Item to process the request of the user, the DataSynchronizationService will read the Queue and again make an HTTP Request to the ApiService which will in turn again make a request to the external-service.
For me, this brings multiple problems
- We are coupling the two services with multiple HTTP requests
- We build a queue to write in, just to read it from another service, just for triggering the initial service who created the message again
- We are creating a second microservice just for handling non-http specific things like ServiceBus and TimerTrigger
Maybe both solutions are wrong? Because I could also imagine that everything belongs together, so all of it should be one microservice? However in my proposal I thought about splitting it by
- One microservice for end-user communication
- One microservice for letting our system use data of external data
I would love to get your insights
11
As you identified yourself, the two services are very coupled together and that holds for both designs. Either they cannot be freely developed independently due to the shared database, or there is a strong temporal coupling that causes the synchronization to go wrong if the two services can’t communicate with each other at the wrong moment.
As you identified yourself, it should be one microservice that is responsible for maintaining a synchronized local view on that data from the external party.
One microservice means that it is maintained, versioned and presented to the outside world as one unit, but it is nowhere written that one microservice must also be deployed as a single executable. If your infrastructure makes it easier to deploy a microservice in two parts (one to listen to HTTP requests and another one to be executed by a scheduler), then that is an entirely valid deployment strategy. And if you want to use a Queue to communicate between those parts of your microservice, that is an implementation detail that nobody but the team working on the service should worry about.
So, I would go with your proposal, but call it one microservice that happens to be deployed in two parts.
5
Is it okay that 2 microservices are sharing the db?
It can be perfectly okay – although whether you can then still call them legit microservices is up for debate. It’s a tradeoff that depends on your context: what is the planned db load in the future, how much horizontal scalability do you need (a single db could hamper it), could the database schema be a contention point between 2 dev teams, etc.
Should maybe all of it happen in ONE microservice, because everything is coupled together? But then I would send a queue to the same service just to run it in the background?
I wouldn’t recommend it, as the features from microservice 2 will be highly dependent on the external system and you probably don’t want to redeploy everything at every small change in their API. The reasons for your main API to change will mostly be different from the integration microservice’s reasons to change. A separate service also brings robustness and fault tolerance, as one can crash or malfunction without the other being affected if appropriate communication buffers exist between them.
We are coupling the two services with multiple HTTP requests
Not necessarily a big deal if it stays at that, but the double coupling you raise in your second bullet point indeed looks like overkill. You basically get the complexity of a message queue without the benefits.
However in my proposal I thought about splitting it by
One microservice for end-user communication
One microservice for letting our system use data of external data
I’m not sure about the term “microservices” to describe that ecosystem, and I’m also not sure how this differs from Proposal one. But I’ve implemented similar solutions to similar integration problems in the past and it works well.
6