I’m working on designing a stream-processing analytics application that will handle incoming events at a rate of 10,000 per second. The architecture involves an application server receiving events from clients and putting them into a Distributed Message Queue (DMQ). Subsequently, a service will consume these events from the DMQ, perform in-memory analysis (as it is done every 5 minutes, so need low latency), and persist the results in a database. Data loss is not acceptable, as analytics results must be stored every 5 minutes.
Please suggest regarding RabbitMQ vs Kafka as the better DMQ option. I read somewhere that RMQ is mostly RAM-based and it also write to disk but that is not done for every message. Since, we can not afford data loss in our app that may not suit us well.