It sounds like the fundamental problem is not just code repository maintenance, but a lack of suitable architecture.

1. What is the core/essence of the system, that will always be shared by all systems?
2. What enhancements/deviations are required by each customer?

A framework or standard library encompasses the former, while the latter would be implemented as add-ons (plugins, subclasses, DI, whatever makes sense for the code structure).

A source control system that manages branches and distributed development would probably help also; I’m a fan of Mercurial, others prefer Git. The framework would be the main branch, each customized system would be sub-branches, for example.

The specific technologies used to implement the system (.NET, WPF, whatever) are largely unimportant.

Getting this right is not easy, but it is critical for long-term viability. And of course the longer you wait, the greater the technical debt you’ll have to deal with.

You may find the book Software Architecture: Organizational Principles and Patterns useful.

Good luck!

One company I’ve worked for had the same problem, and the approach to tackle the problem was this: A common framework for all new projects was created; this includes all stuff that has to be the same in every project. E.g. form generating tools, export to Excel, logging. Effort was taken to make sure that this common framework is only improved (when a new project needs new features), but never forked.

Based on that framework, customer-specific code was maintained in seperate repositories. When useful or necessary, bug fixes and improvements are copy-pasted between projects (with all the caveats described in the question). Globally useful improvements go into the common framework, though.

Having everything in a common codebase for all customers has some advantages, but on the other hand, reading the code becomes difficult when there are countless ifs to make the program behave differently for each customer.

EDIT: One anecdote to make this more understandable:

The domain of that company is warehouse management, and one task of a
warehouse management system is to find a free storage location for
incoming goods. Sounds easy, but in practice, a lot of constraints and
strategies have to be observed.

At one point in time, management asked a programmer to make a
flexible, parameterisable module to find storage locations, which
implemented several different strategies and should have been used in
all subsequent projects. The noble effort resulted in a complex module, which
was very difficult to understand and maintain. In the next project,
the project lead couldn’t figure out how to make it work in that
warehouse, and the developer of said module was gone, so he eventually
ignored it and wrote a custom algorithm for that task.

A few years later, the layout of the warehouse where
this module was originally used changed, and the module with all its
flexibility didn’t match the new requirements; so I replaced it with a
custom algorithm there, too.

I know LOC is not a good measurement, but
anyway: the size of “flexible” module was ~3000 LOC (PL/SQL), while a custom
module for the same task takes ~100..250 LOC. Therefore, trying to be flexible
extremely increased the size of the code base, without gaining the reusability
we had hoped for.

One of the projects I’ve worked on supported multiple platforms (more than 5) across a large number of product releases. A lot of the challenges you are describing were things we faced, albeit in a slightly different way. We had a proprietary DB, so we didn’t have the same types of problems in that arena.

Our structure was similar to yours, but we had a single repository for our code. Platform specific code went into their own project folders within the code tree. Common code lived within the tree based upon what layer it belonged to.

We had conditional compilation, based upon the platform being built. Maintaining that was kind of a pain, but it only had to be done when new modules were added at the platform specific layer.

Having all of the code in a single repository made it easy for us to make bug fixes across multiple platforms and releases at the same time. We had an automated build environment for all the platforms to serve as a backstop in case new code broke a presumed unrelated platform.

We tried to discourage it, but there would be cases where a platform needed a fix based upon a platform-specific bug that was in otherwise common code. If we could conditionally override the compile without making the module look fugly, we’d do that first. If not, we would move the module out of common territory and push it into platform specific.

For the database, we had a few tables that had platform specific columns / modifications. We would make sure that every platform version of the table met a baseline level of functionality so common code could reference it without worrying about platform dependencies. Platform specific queries / manipulations were pushed into the platform project layers.

So, to answer your questions:

1. Lots, and that was one of the best teams I’ve worked with. Codebase at that time was around 1M loc. I didn’t get to choose the approach, but it worked out pretty dang well. Even in hindsight, I haven’t seen a better way of handling things.
2. I recommend the second approach you suggested with the nuances I mention in my answer.
3. No books that I can think of, but I would research multi-platform development as a starter.
4. Institute some strong governance. It’s the key to making sure your coding standards are followed. Following those standards is the only way to keep things manageable and maintainable. We had our share of impassioned pleas to break the model we were following, but none of those appeals ever swayed the entire senior development team.

I worked for many years on a Pension Administration application which had similar issues. Pension plans are vastly different between companies, and require highly specialized knowledge for implementing calculation logic and reports and also very different data design. I can only give a brief description of part of the architecture, but maybe it will give enough of the idea.

We had 2 separate teams: a core development team, which was responsible for the core system code (which would be your 80% shared code above), and an implementation team, which had domain expertise in pension systems, and was responsible for learning client requirements and coding scripts and reports for the client.

We had all of our tables defined by Xml (this before the time when entity frameworks were time-tested and common). The implementation team would design all the tables in Xml, and the core application could be prompted to generate all the tables in Xml. There were also associated VB script files, Crystal Reports, Word docs etc. for each client. (There was also an inheritance model built into the Xml to enable reusing other implementations).

The core application (one application for all clients), would cache all the client specific stuff when a request for that client came, and it generated a common data object (kind of like a remote ADO record set), which could be serialized and passed around.

This data model is less slick that entity/domain objects, but it is highly flexible, universal, and can be processed by one set of core code. Perhaps in your case, you could define your base entity objects with only the common fields, and have an additional Dictionary for custom fields (add some kind of set of data descriptors to your entity object so that it has meta data for the custom fields.)

We had separate source repositories for the core system code and for the implementation code.

Our core system actually had very little business logic, other than some very standard common calculation modules. The core system functioned as: screen generator, script runner, report generator, data access and transport layer.

Segmenting core logic and customized logic is a tough challenge. However, we always felt it was better to have one core system running multiple clients, rather than multiple copies of the system running for each client.

I’ve worked on a smaller system (20 kloc), and found that DI and configuration are both great ways to manage differences between clients, but not enough to avoid forking the system. The database is split between an application specific part, which has a fixed schema, and the client dependent part, which is defined through a custom XML configuration document.

We’ve kept a single branch in mercurial that’s configured as if it was deliverable, but branded and configured for a fictional client. Bug fixes are mainlined into that project, and new development of core functionality only happens there. Releases to actual clients are branches off of that, stored in their own repositories. We keep track of large changes to the code through manually written version numbers and track bug fixes using commit numbers.

I am afraid that I do not have direct experience of the problem that you describe, but I do have some comments.

The second option, of bringing the code together into a central repository (as much as practicable), and architecting for customization (again, as much as practicable) is almost certainly the way to go in the long term.

The problem is how you plan to get there, and how long it is going to take.

In this situation, it is probably OK to (temporarily) have more than one copy of the application in the repository at a time.

This will enable you to gradually move to an architecture that directly supports customization without having to do it in one fell swoop.

The second approach seems to be more elegant, but we have many
unsolved problems in this approach.

I am sure any of those problems can be solved, one after a another. If you get stuck, ask here on or on SO about the specific problem.

As others have pointed out, having one central codebase / one repository is the option you should prefer. I try to answer your example question.

For example: how do handle changes/additions in your model/database. We are using .NET with Entity Framework to have strongly typed entities. I don’t see how we
can handle properties which are required for one customer but useless
for another customer without cluttering our data model.

There are some possibilities, all of them I have seen in real-world systems. Which one to choose depends on your situation:

• live with the cluttering to a certain degree

• introduce a tables “CustomAttributes” (describing names and types) and “CustomAttributeValues” (for the values, for example stored as a string representation, even if they are numbers). That will allow to add such attributes at install time or run time, having individual values for each customer. Don’t insist on having each custom attribute modeled “visibly” in your data model.

• now it should be clear how to use that in code: have just general code for accessing those tables, and individual code (perhaps in a separate plug-in DLL, which is up to you) for interpreting that attributes correctly

• another alternative is to give each entity table a big string field where you can add an individual XML-string.
• try to generalize some concepts, so they could be more easily reused across different customers. I recommend Martin Fowler’s book “Analysis patterns”. Though this book is not about customizing software pre se, it may be helpful for you either.

And for specific code: you can also try to introduce a scripting language into your product, especially for adding customer-specific scripts. That way you do not only create a clear line between your code and customer-specific code, you can also allow your customers to customize the system to some degree by themselves.

When I’m asked to start the development of B which is sharing 80% functionality with A, I will either:

1. Clone A and modify it.
2. Extract the functionality that both A and B share into C which they will use.
3. Make A configurable enough to fulfill the needs of both B and itself (therefore B is embedded in A).

You chose 1, and it doesn’t seem to fit your situation well. Your mission is to predict which of 2 and 3 is a better fit.

I have only built one such application. I’d say that 90% of the units sold were sold as is, no modifications. Each customer had their own customized skin and we served up the system within that skin. When a mod did come in that affected the core sections we tried using IF branching. When mod #2 came in for the same section we switched to CASE logic whiched allowed for future expansion. This seemed to handle most of the minor requests.

Any further minor custom requests were handled by implementing Case logic.

If the mods were two radical, we built a clone (separate include) and wrapped a CASE around it to include the different module.

Bug fixes and modifications on the core effected all users. We tested thoroughly in development before going to production. We always sent out email notifications that accompanied any change and NEVER, NEVER, NEVER posted production changes on Fridays… NEVER.

Our environment was Classic ASP and SQL Server. We were NOT a spaghetti code shop… Everything was modular using Includes, Subroutines and Functions.