I’m designing an analytics system that feeds all events to Elasticsearch. The event lifecycle is as follows:
- Visitor does something.
- Custom analytics server gathers data, makes an event out of it and puts it into Elasticsearch.
- One a day or so, custom batch processor aggregates a large set of events from Elasticsearch, transforms them to smaller chunks of data and puts back to Elasticsearch.
I know Logstash is made to gather and process events. Why should I consider it over my own custom solution?
I have no log files in the process. Events travel from one component to another via HTTP. I don’t care transforming the data to events on my own either.
I’m especially concerned about not having the flexibility of dealing with the events and Elasticsearch directly from my own choice of programming language.
Of course you can use the ElasticSearch Index API directly: https://www.elastic.co/guide/en/elasticsearch/reference/1.4/docs-index_.html
Logstash’s main responsibility in the ELK stack is the efficient collection and conversion of a stream of logs into individual, indexable documents. If you already have individual, indexable documents, Logstash may be an unnecessary complication.
When you have an existing stream of data that you want to index, Logstash provides a single endpoint for aggregating, parsing and enriching that data. When you want to change how you process or present a log, you make that change in Logstash — not your application. This decoupling can allow an analytics team to efficiently manage an application that is slow to update, or that they don’t control themselves. Logstash also acts as buffer, protecting ES’ endpoints from thundering herd issues, though you can scale ES to diminish this problem.
Note that the enrichment is a very nice feature — it can allow you to add additional context to your documents logs without rebuilding the services that produce them, or to remap fields to mate a canonical logging format.