Relative Content

Tag Archive for high-performance

Cpu-heavy web server

When reading about web servers, frameworks, etc most of the time I notice that the goal is to have a technology that has the next features:

clojure/erlang/go for high volume server

I have a project that will need to handle 1000s of requests a second with a decent amount of processing for each. For the most part, the processing will be done on a list of items, basically filtering it and returning a smaller list. This process can be done in parallel fairly easily and I should also say, speed is important. I’ve narrowed my language of choice for the project down to clojure, erlang, and go, and have researched them all to a moderate degree.

Deduplication, Grouping for events table at scale

I’m working with an events table where different source tables trigger writes into this table with columns: entity_id and payload. These events are then published to a Kafka topic using a message relay service. The table is partitioned hourly based on event_time, handling a high scale of ~5M+ rows per hour. After a row is processed and published, we mark it as processed=true and drop partitions after 24 hours to avoid performance issues from deleting individual rows.

Socket Connection Data Insert

I have been working through a high-performance application where I have identified a bottleneck. The bottleneck is actually when the application must insert messages from a socket, it will record them in a database. The trouble is the speed at which the messages come through are so fast that the application cannot catch up due to poor architectural planning. Clearly, insertion needs to be improved.