Do you feel there is a trade-off between writing “nice” object
orientated code and writing very [sic] low latency code?

Yes.

That’s why the phrase “premature optimization” exists. It exists to force developers to measure their performance, and only optimize that code that will make a difference in performance, while sensibly designing their application architecture from the start so that it doesn’t fall down under heavy load.

That way, to the maximum extent possible, you get to keep your pretty, well-architected, object-oriented code, and only optimize with ugly code those small portions that matter.

Yes, the example I give is not C++ vs. Java but is Assembly vs. COBOL as it is what I know.

Both languages are very fast, but, even COBOL when compiled has many more instructions that are placed into the instruction set that do not necessarily need to be there vs writing those instructions yourself in Assembly.

The same idea can be applied directly to your question of writing “ugly looking code” vs. using inheritance/polymorphism in C++. I believe it is necessary to write ugly looking code, if the end-user needs sub-second transaction timeframes then it’s our job as programmers to give them that no matter how it happens.

That being said, liberal use of comments increases programmer functionality & maintainability greatly, no matter how ugly the code is.

Yes, a trade-off exist. By this, I mean that code that is faster and
uglier is not necessary better – the quantitative benefits from “fast
code” needs to be weighted against the maintenance complexity of the
code changes needed to achieve that speed.

The trade-off comes from business cost. Code that is more complex
requires more skilled programmers (and programmers with a more focused
skill set, such as ones with CPU architecture and design knowledge),
takes more time to read and understand the code and to fix bugs.
The business cost of developing and maintaining such code could be
in the range of 10x – 100x over normally-written code.

This maintenance cost is justifiable in some industries, in which
customers are willing to pay a very high premium for very fast software.

Some speed optimizations make better return-on-investment (ROI) than
others. Namely, some optimizations techniques can be applied with
lesser impact on code maintainability (preserving higher-level structure
and lower-level readability) compared to normally-written code.

Thus, a business owner should:

• Look at the costs and benefits,
• Make measurements and calculations
  • Have the programmer measure the program speed
  • Have the programmer estimate the development time needed for optimization
  • Make own estimate about the increased revenue from faster software
  • Have software architects or QA managers gauge qualitatively the drawbacks from reduced intuitiveness and readability of source code
• And prioritize the low-hanging fruits of software optimization.

These trade-offs are highly specific to circumstances.

These cannot be optimally decided without the participation of managers and
product owners.

These are highly specific to platforms. For example, desktop and mobile
CPUs have different considerations. Server and client applications
also have different considerations.

Yes, it is generally true that faster code looks different from
normally-written code. Any code that is different will take more time
to read. Whether that implies ugliness is in the eyes of the beholder.

The techniques that I have some exposure with are: (without trying to
claim any level of expertise) short-vector optimization (SIMD),
fine-grained task parallelism, memory pre-allocation and object reuse.

SIMD typically has severe impacts on low-level readability, even though
it typically doesn’t require higher-level structural changes (provided
that the API is designed with bottleneck-prevention in mind).

Some algorithms can be transformed into SIMD easily (the embarassingly-
vectorizable). Some algorithms require more computation rearrangements
in order to use SIMD. In extreme cases such as wavefront SIMD parallelism,
entirely new algorithms (and patentable implementations) have to be
written to to take advantage.

Fine-grained task parallelization requires rearranging algorithms into
data flow graphs, and repeatedly apply functional (computational)
decomposition to the algorithm until no further margin benefit can be
gained. Decomposed stages are typically chained with continuation-style,
a concept borrowed from functional programming.

By functional (computational) decomposition, algorithms which could
have been normally-written in a linear and conceptually clear sequence
(lines of code that are executable in the same order they are written)
have to be broken down into fragments, and distributed into multiple
functions or classes. (See algorithm objectification, below.) This
change will greatly impede fellow programmers who are not familiar
with the decomposition design process which gave rise to such code.

To make such code maintainable, the authors of such code must write
elaborate documentations of the algorithm – far beyond the kind of
code commenting or UML diagrams done for normally-written code.
This is similar to the way researchers write their academic papers.

No, fast code need not be in contradiction with object-orientedness.

Put in another way, it is possible to implement very fast software that
is still object-oriented. However, toward the lower-end of that
implementation (at the nuts-and-bolts level where the majority of
computation occurs), the object design may deviate significantly from
designs obtained from object-oriented design (OOD). The lower-level design
is geared toward algorithm-objectification.

A few benefits of object-oriented programming (OOP), such as
encapsulation, polymorphism, and composition, can still be reaped from
low-level algorithm-objectification. This is the main justification for
using OOP at this level.

Most benefits of object-oriented design (OOD) are lost. Most
importantly, there is no intuitiveness in the low-level design.
A fellow programmer cannot learn how to work with the lower-level
code without first fully understanding how the algorithm had been
transformed and decomposed in the first place, and this understanding
is not obtainable from the resulting code.

Yes sometimes code has to be “ugly” to make it work in the required time, all the code doesn’t have to be ugly though. Performance should be tested and profiled before to find the bits of code that need to be “ugly” and those sections should be noted with a comment so future devs know what is purposefully ugly and what is just laziness. If someone is writing lots of poorly designed code claiming performance reasons, make them prove it.

Speed is just as important as any other requirement of a program, giving wrong corrections to a guided missile is equivalent to providing the right corrections after impact. Maintainability is always a secondary concern to working code.

Some of the studies I have seen extracts of indicate that clean easy to read code is often faster than more complex hard to read code. In part, this is due to the way optimizers are designed. They tend to be much better at optimizing a variable into a register, than doing the same with an intermediate result of a calculation. Long sequences of assignments using a single operator leading to the final result may be optimized better than a long complicated equation. Newer optimizers may have reduced the difference between clean and complicated code, but I doubt they have eliminated it.

Other optimizations like loop unrolling can be added in a clean fashion when required.

Any optimization added to improve performance should be accompanied by an appropriate comment. This should include a statement that it was added as an optimization, preferably with measures of performance before and after.

I have found the 80/20 rule applies to the code I have optimized. As a rule of thumb I don’t optimize anything that isn’t taking at least 80% of the time. I then aim for (and usually achieve) a 10 fold performance increase. This improves performance about 4 fold. Most optimizations I have implemented haven’t made the code significantly less “beautiful”. Your mileage may vary.

If by ugly, you mean difficult to read/understand at the level where other developers will be re-using it or be needing to understand it, then I would say, elegant, easy-to-read code will almost always ultimately net you a performance gain in the long run in an app that you have to maintain.

Otherwise, sometimes there’s enough of a performance win to make it worth putting ugly in a beautiful box with a killer interface on it but in my experience, this is a pretty rare dilemma.

Think about basic work avoidance as you go. Save the arcane tricks for when a performance problem actually presents itself. And if you do have to write something that somebody could only understand through familiarity with the specific optimization, do what you can to at least make the ugly easy to understand from a re-use of your code point of view. Code that performs miserably rarely ever does so because the developers were thinking overly hard about what the next guy was going to inherit, but if frequent changes are the only constant of an app (most web apps in my experience), rigid/inflexible code that’s difficult to modify is practically begging for panicked messes to start popping up all over your code base. Clean and lean is better for performance in the long run.

Do you feel there is a trade-off between writing “nice” object
orientated code and writing very fast low latency code? For instance,
avoiding virtual functions in C++/the overhead of polymorphism etc-
re-writing code which looks nasty, but is very fast etc?

I work in a field which is a bit more focused on throughput than latency, but it’s very performance-critical, and I’d say “sorta”.

Yet a problem is that so many people get their notions of performance completely wrong. Novices often get just about everything wrong, and their entire conceptual model of “computational cost” needs reworking, with only algorithmic complexity being about the only thing they can get right. Intermediates get a lot of things wrong. Experts get some things wrong.

Measuring with accurate tools that can provide metrics like cache misses and branch mispredictions is what keeps all people of any level of expertise in the field in check.

Measuring is also what points out what not to optimize. Experts often spend less time optimizing than novices, since they’re optimizing true measured hotspots and not trying to optimize wild stabs in the dark based on hunches about what could be slow (which, in extreme form, could tempt one to micro-optimize just about every other line in the codebase).

Designing for Performance

With that aside, the key to designing for performance comes from the design part, as in interface design. One of the problems with inexperience is that there tends to be an early shift on absolute implementation metrics, like the cost of an indirect function call in some generalized context, as though the cost (which is better understood in an immediate sense from an optimizer’s point of view rather than a branching point of view) is a reason to avoid it throughout the entire codebase.

Costs are relative. While there is a cost to an indirect function call, e.g., all costs are relative. If you’re paying that cost one time to call a function which loops through millions of elements, worrying about this cost is like spending hours haggling over pennies for the purchase of a billion dollar product, only to conclude not to buy that product because it was one penny too expensive.

Coarser Interface Design

The interface design aspect of performance often seeks earlier on to push these costs to a coarser level. Instead of paying runtime abstraction costs for a single particle, for example, we might push that cost to the level of the particle system/emitter, effectively rendering a particle into an implementation detail and/or simply raw data of this particle collection.

So object-oriented design doesn’t have to be incompatible with designing for performance (whether latency or throughput), but there can be temptations in a language that focuses on it to model increasingly teeny granular objects, and there the latest optimizer can’t help. It can’t do things like coalesce a class representing a single point in a way that yields an efficient SoA representation for the memory access patterns of the software. A collection of points with the interface design modeled at the level of coarseness offers that opportunity, and allows iterating towards more and more optimal solutions as needed. Such a design is designed for bulk memory *.

_{* Note the focus on memory here and not data, as working in performance-critical areas for a long time will tend to change your view of data types and data structures and seeing how they connect to memory. A binary search tree no longer becomes solely about logarithmic complexity in such cases as possibly-disparate and cache-unfriendly memory chunks for tree nodes unless aided by a fixed allocator. The view does not dismiss algorithmic complexity, but it sees it no longer independently of memory layouts. One also starts to see iterations of work as being more about iterations of memory access.*}

A lot of performance-critical designs can actually be very compatible with the notion of high-level interface designs that are easy for humans to understand and use. The difference is that “high-level” in this context would be about bulk aggregation of memory, an interface modeled for potentially large collections of data, and with an implementation under the hood that may be quite low-level. A visual analogy might be a car that’s really comfortable and easy to drive and handle and very safe while going at the speed of sound, but if you pop the hood, there’s little fire-breathing demons inside.

With a coarser design also tends to come an easier way to provide more efficient locking patterns and exploit parallelism in the code (multithreading is an exhaustive subject that I’ll kind of skip here).

Memory Pool

A critical aspect of low-latency programming is probably going to be a very explicit control over memory to improve locality of reference as well as just the general speed of allocating and deallocating memory. A custom allocator pooling memory actually echoes the same kind of design mindset we described. It’s designed for bulk; it’s designed at a coarse level. It preallocates memory in large blocks and pools the memory already-allocated in small chunks.

The idea is exactly the same of pushing costly things (allocating a memory chunk against a general-purpose allocator, e.g.) to a coarser and coarser level. A memory pool is designed for dealing with memory in bulk.

Type Systems Segregate Memory

One of the difficulties with granular object-oriented design in any language is that it often wants to introduce a lot of teeny user-defined types and data structures. Those types can then want to be allocated in little teeny chunks if they’re dynamically allocated.

A common example in C++ would be for cases where polymorphism is required, where the natural temptation is to allocate each instance of a subclass against a general-purpose memory allocator.

This ends up breaking apart possibly-contiguous memory layouts into little itsy-bitsy bits and pieces scattered across the addressing range which translates to more page faults and cache misses.

Fields that demand the lowest-latency, stutter-free, deterministic response are probably the one place where hotspots don’t always boil down to a single bottleneck, where tiny inefficiencies can actually genuinely kind of “accumulate” (something a lot of people imagine happening incorrectly with a profiler to keep them in check, but in latency-driven fields, there can actually be some rare cases where tiny inefficiencies accumulate). And a lot of the most common reasons for such an accumulation can be this: the excessive allocation of teeny chunks of memory all over the place.

In languages like Java, it can be helpful to use more arrays of plain old data types when possible for bottlenecky areas (areas processed in tight loops) such as an array of int (but still behind a bulky high-level interface) instead of, say, an ArrayList of user-defined Integer objects. This avoids the memory segregation that would typically accompany the latter. In C++, we don’t have to degrade the structure quite as much if our memory allocation patterns are efficient, as user-defined types can be allocated contiguously there and even in the context of a generic container.

Fusing Memory Back Together

A solution here is to reach for a custom allocator for homogeneous data types, and possibly even across homogeneous data types. When tiny data types and data structures are flattened to bits and bytes in memory, they take on a homogeneous nature (albeit with some varying alignment requirements). When we don’t look at them from a memory-centric mindset, the type system of programming languages “want” to split/segregate potentially-contiguous memory regions apart into little teeny scattered chunks.

The stack utilizes this memory-centric focus to avoid this and potentially store any possible mixed combination of user-defined type instances inside of it. Utilizing the stack more is a great idea when possible as the top of it is almost always sitting in a cache line, but we can also design memory allocators that mimic some of these characteristics without a LIFO pattern, fusing memory across disparate data types into contiguous chunks even for more complex memory allocation and deallocation patterns.

Modern hardware is designed to be at its peak when processing contiguous blocks of memory (repeatedly accessing the same cache line, the same page, e.g.). The keyword there is contiguity, as this is only beneficial if there’s surrounding data of interest. So a lot of the key (yet also difficulty) to performance is to fuse segregated chunks of memory back together again into contiguous blocks that are accessed in their entirety (all surrounding data being relevant) prior to eviction. The rich type system of especially user-defined types in programming languages can be the biggest obstacle here, but we can always reach around and solve the problem through a custom allocator and/or bulkier designs when appropriate.

Ugly

“Ugly” is hard to say. It’s a subjective metric, and someone who works in a very performance-critical field will start to change their idea of “beauty” to one that’s a lot more data-oriented and focuses on interfaces that process things in bulk.

Dangerous

“Dangerous” might be easier. In general, performance tends to want to reach towards lower-level code. Implementing a memory allocator, for example, is impossible without reaching beneath data types and working at the dangerous level of raw bits and bytes. As a result, it can help to increase the focus on careful testing procedure in these performance-critical subsystems, scaling the thoroughness of testing with the level of optimizations applied.

Beauty

Yet all of this would be at the implementation detail level. In both a veteran large-scale and performance-critical mindset, “beauty” tends to shift towards interface designs rather than implementation details. It becomes an exponentially higher priority to seek “beautiful”, usable, safe, efficient interfaces rather than implementations due to coupling and cascading breakages that can occur in the face of an interface design change. Implementations can be swapped out any time. We typically iterate towards performance as needed, and as pointed out by measurements. The key with the interface design is to model at a coarse enough level to leave room for such iterations without breaking the entire system.

In fact, I would suggest that a veteran’s focus on performance-critical development will often tend to place a predominant focus on safety, testing, maintainability, just the disciple of SE in general, since a large-scale codebase which has a number of performance-critical subsystems (particle systems, image processing algorithms, video processing, audio feedback, raytracers, mesh engines, etc) will need to pay close attention to software engineering to avoid drowning in a maintenance nightmare. It’s by no mere coincidence that often the most astonishingly-efficient products out there can also have the fewest number of bugs.

TL;DR

Anyway, that’s my take on the subject, ranging from priorities in genuinely performance-critical fields, what can reduce latency and cause tiny inefficiencies to accumulate, and what actually constitutes “beauty” (when looking at things most productively).

Complex and ugly aren’t the same thing. Code that has many special cases, that’s optimized to eek out every last drop of performance, and that looks at first like a tangle of connections and dependencies may in fact be very carefully engineered and quite beautiful once you understand it. Indeed, if performance (whether measured in terms of latency or something else) is important enough to justify very complex code, then the code must be well designed. If it’s not, then you can’t be sure that all that complexity is really better than a simpler solution.

Ugly code, to me, is code that’s sloppy, poorly considered, and/or unnecessarily complicated. I don’t think you’d want any of those features in code that has to perform.

Not to be different, but here’s what I do:

1. Write it clean and maintainable.

2. Do performance diagnosis, and fix the problems it tells you, not the ones you guess. Guaranteed, they will be different from what you expect.

You can do these fixes in a way that is still clear and maintainable, but, you will have to add commentary so people who look at the code will know why you did it that way. If you don’t, they will undo it.

So is there a tradeoff? I don’t really think so.

You can write ugly code that is very fast and you can also write beautiful code that is as fast as your ugly code. The bottleneck will not be in the beauty/organization/structure of your code but in the techniques you chose. For example, are you using non-blocking sockets? Are you using single-threaded design? Are you using a lock-free queue for inter-thread communication? Are you producing garbage for the GC? Are you performing any blocking I/O operation in the critical thread? As you can see this has nothing to do with beauty.

What matters the end-user?

• Performance
• Features / Functionality
• Design

Case 1: Optimized bad code

• Hard maintenance
• Hardly readable if as an open-source project

Case 2: Non-optimized good code

• Easy maintenance
• Bad user experience

Solution?

Easy, optimize performance critical pieces of code

e.g.:

A program that consists of 5 Methods, 3 of them are for data management, 1 for disk reading, the other for disk writing

These 3 data management methods use the two I/O methods and depend on them

We would optimize the I/O methods.

Reason:
I/O methods are less likely to be changed, nor they affect the design of the app, and all in all, everything in that program depends on them, and thus they seem performance critical, we would use whatever code to optimize them.

This means we get good code and manageable design of the program while keeping it fast by optimizing certain parts of code

I am thinking..

I think bad code makes it hard for humans to polish-optimize and small mistakes might make it even worse, so a good code for a novice/beginner would be better if just well written that ugly code.