Foreword

This is a daunting task indeed, and there’s a lot of ground to
cover. So I’m humbly suggesting this as somewhat comprehensive guide
for your team, with pointers to appropriate tools and educational
material.

Remember: These are guidelines, and that as such are meant to
adopted, adapted, or dropped based on circumstances.

Beware: Dumping all this on a team at once would most likely
fail. You should try to cherry-pick elements that would give you the
best bang-for-sweat, and introduce them slowly, one at a time.

Note: not all of this applies directly to Visual Programming
Systems like G2. For more specific details on how to deal with these,
see the Addendum section at the end.

Executive Summary for the Impatient

• Define a rigid project structure, with:
  • project templates,
  • coding conventions,
  • familiar build systems,
  • and sets of usage guidelines for your infrastructure and tools.
• Install a good SCM and make sure they know how to use it.
• Point them to good IDEs for their technology, and make sure they know how to use them.
• Implement code quality checkers and automatic reporting in the build system.
• Couple the build system to continuous integration and continuous inspection systems.
• With the help of the above, identify code quality “hotspots” and refactor.

Now for the long version… Caution, brace yourselves!

Rigidity is (Often) Good

This is a controversial opinion, as rigidity is often seen as a force
working against you. It’s true for some phases of some projects. But
once you see it as a structural support, a framework that takes away
the guesswork, it greatly reduces the amount of wasted time and
effort. Make it work for you, not against you.

Rigidity = Process / Procedure.

Software development needs good process and procedures for exactly the
same reasons that chemical plants or factories have manuals,
procedures, drills and emergency guidelines: preventing bad outcomes,
increasing predictability, maximizing productivity…

Rigidity comes in moderation, though!

Rigidity of the Project Structure

If each project comes with its own structure, you (and newcomers) are
lost and need to pick up from scratch every time you open them. You
don’t want this in a professional software shop, and you don’t want
this in a lab either.

Rigidity of the Build Systems

If each project looks different, there’s a good chance they also
build differently. A build shouldn’t require too much research or
too much guesswork. You want to be able to do the canonical thing and
not need to worry about specifics: configure; make install, ant,
mvn install, etc…

Re-using the same build system and making it evolve over the time also
ensures a consistent level of quality.

You do need a quick README to point the project’s specifics, and
gracefully guide the user/developer/researcher, if any.

This also greatly facilitates other parts of your build
infrastructure, namely:

• continuous integration,
• continuous inspection.

So keep your build (like your projects) up to date, but make it
stricter over time, and more efficient at reporting violations and bad
practices.

Do not reinvent the wheel, and reuse what you have already done.

Recommended Reading:

• Continuous Integration: Improving Software Quality and Reducing Risk (Duval, Matyas, Glover, 2007)
• Continuous Delivery: Release Sotware Releases through Build, Test and Deployment Automation (Humble, Farley, 2010)

Rigidity in the Choice of Programming Languages

You can’t expect, especially in a research environment, to have all
teams (and even less all developers) use the same language and
technology stack. However, you can identify a set of “officially
supported” tools, and encourage their use. The rest, without a good
rationale, shouldn’t be permitted (beyond prototyping).

Keep your tech stack simple, and the maintenance and breadth of
required skills to a bare minimum: a strong core.

Rigidity of the Coding Conventions and Guidelines

Coding conventions and guidelines are what allow you to develop both
an identity as a team, and a shared lingo. You don’t want to err
into terra incognita every time you open a source file.

Nonsensical rules that make life harder or forbid actions explicity
to the extent that commits are refused based on single simple
violations are a burden. However:

• a well thought-out ground ruleset takes away a lot of the whining
  and thinking: nobody should break under no circumstances;

• and a set of recommended rules provide additional guidance.

Personal Approach: I am aggressive when it comes to coding
conventions, some even say nazi, because I do believe in having a
lingua franca, a recognizable style for my team. When crap code
gets checked-in, it stands out like a cold sore on the face of an
Hollywood star: it triggers a review and an action automatically.
In fact, I’ve sometimes gone as far as to advocate the use of
pre-commit hooks to reject non-conforming commits. As mentioned, it
shouldn’t be overly crazy and get in the way of productivity: it
should drive it. Introduce these slowly, especially at the
beginning. But it’s way preferable over spending so much time fixing
faulty code that you can’t work on real issues.

Some languages even enforce this by design:

• Java was meant to reduce the amount of dull crap you can write with
  it (though no doubt many manage to do it).

• Python’s block structure by indentation is another idea in this
  sense.

• Go, with its gofmt tool, which completely takes away any debate
  and effort (and ego!!) inherent to style: run gofmt before
  you commit.

Make sure that code rot cannot slip through. Code
conventions, continuous integration and continuous
inspection, pair programming and code reviews are your
arsenal against this demon.

Plus, as you’ll see below, code is documentation, and that’s
another area where conventions encourage readability and clarity.

Rigidity of the Documentation

Documentation goes hand in hand with code. Code itself is
documentation. But there must be clear-cut instructions on how to
build, use, and maintain things.

Using a single point of control for documentation (like a WikiWiki or
DMS) is a good thing. Create spaces for projects, spaces for more
random banter and experimentation. Have all spaces reuse common rules
and conventions. Try to make it part of the team spirit.

Most of the advice applying to code and tooling also applies to
documentation.

Rigidity in Code Comments

Code comments, as mentioned above, are also documentation. Developers
like to express their feelings about their code (mostly pride and
frustration, if you ask me). So it’s not unusual for them to express
these in no uncertain terms in comments (or even code), when a more
formal piece of text could have conveyed the same meaning with less
expletives or drama. It’s OK to let a few slip through for fun and
historical reasons: it’s also part of developing a team
culture. But it’s very important that everybody knows what is
acceptable and what isn’t, and that comment noise is just that:
noise.

Rigidity in Commit Logs

Commit logs are not an annoying and useless “step” of your SCM’s
lifecycle: you DON’T skip it to get home on time or get on with the
next task, or to catch up with the buddies who left for lunch. They
matter, and, like (most) good wine, the more time passes the more valuable
they become. So DO them right. I’m flabbergasted when I see co-workers
writing one-liners for giant commits, or for non-obvious hacks.

Commits are done for a reason, and that reason ISN’T always clearly
expressed by your code and the one line of commit log you
entered. There’s more to it than that.

Each line of code has a story, and a history. The diffs can tell
its history, but you have to write its story.

Why did I update this line? -> Because the interface changed.

Why did the interface changed? -> Because the library L1 defining it
was updated.

Why was the library updated? -> Because library L2, that we need for
feature F, depended on library L1.

And what’s feature X? -> See task 3456 in issue tracker.

It’s not my SCM choice, and may not be the best one for your lab
either; but Git gets this right, and tries to force you to write
good logs more than most other SCMs systems, by using short logs and
long logs. Link the task ID (yes, you need one) and a leave a
generic summary for the shortlog, and expand in the long log: write
the changeset’s story.

It is a log: It’s here to keep track and record updates.

Rule of Thumb: If you were searching for something about this
change later, is your log likely to answer your question?

Projects, Documentation and Code Are ALIVE

Keep them in sync, otherwise they do not form that symbiotic entity
anymore. It works wonders when you have:

• clear commits logs in your SCM, w/ links to task IDs in your
  issue tracker,
• where this tracker’s tickets themselves link to the changesets in
  your SCM (and possibly to the builds in your CI system),
• and a documentation system that links to all of these.

Code and documentation need to be cohesive.

Rigidity in Testing

Rules of Thumb:

• Any new code shall come with (at least) unit tests.
• Any refactored legacy code shall come with unit tests.

Of course, these need:

• to actually test something valuable (or they are a waste of time
  and energy),
• to be well written and commented (just like any other code you check in).

They are documentation as well, and they help to outline the contract
of your code. Especially if you use TDD. Even if you don’t, you
need them for your peace of mind. They are your safety net when you
incorporate new code (maintenance or feature) and your watchtower
to guard against code rot and environmental failures.

Of course, you should go further and have integration tests, and
regression tests for each reproducible bug you fix.

Rigidity in the Use of the Tools

It’s OK for the occasional developer/scientist to want to try some new
static checker on the source, generate a graph or model using another,
or implement a new module using a DSL. But it’s best if there’s a
canonical set of tools that all team members are expected to know
and use.

Beyond that, let members use what they want, as long as they are ALL:

• productive,
• NOT regularly requiring assistance
• NOT regularly adjusting to your general infrastructure,
• NOT disrupting your infrastructure (by modifying common
  areas like code, build system, documentation…),
• NOT affecting others’ work,
• ABLE to timely perform any task requested.

If that’s not the case, then enforce that they fallback to defaults.

Rigidity vs Versatility, Adaptability, Prototyping and Emergencies

Flexibility can be good. Letting someone occasionally use a hack, a
quick-n-dirty approach, or a favorite pet tool to get the job done
is fine. NEVER let it become a habit, and NEVER let this code
become the actual codebase to support.

Team Spirit Matters

Develop a Sense of Pride in Your Codebase

• Develop a sense of Pride in Code
  • Use wallboards
    • leader board for a continuous integration game
    • wallboards for issue management and defect counting
  • Use an issue tracker / bug tracker

Avoid Blame Games

• DO use Continuous Integration / Continuous Inspection games: it
  fosters good-mannered and productive competition.
• DO keep track defects: it’s just good house-keeping.
• DO identifying root causes: it’s just future-proofing processes.
• BUT DO NOT assign blame: it’s counter productive.

It’s About the Code, Not About the Developers

Make developers conscious of the quality of their code, BUT make them
see the code as a detached entity and not an extension of
themselves, which cannot be criticized.

It’s a paradox: you need to encourage ego-less programming for a
healthy workplace but to rely on ego for motivational purposes.

From Scientist to Programmer

People who do not value and take pride in code do not produce good
code. For this property to emerge, they need to discover how valuable
and fun it can be. Sheer professionalism and desire to do good is not
enough: it needs passion. So you need to turn your scientists into
programmers (in the large sense).

Someone argued in comments that after 10 to 20 years on a project and
its code, anyone would feel attachment. Maybe I’m wrong but I assume
they’re proud of the code’s outcomes and of the work and its legacy,
not of the code itself or of the act of writing it.

From experience, most researchers regard coding as a necessity, or at
best as a fun distraction. They just want it to work. The ones who are
already pretty versed in it and who have an interest in programming
are a lot easier to persuade of adopting best practices and switching
technologies. You need to get them halfway there.

Code Maintenance is Part of Research Work

Nobody reads crappy research papers. That’s why they are
peer-reviewed, proof-read, refined, rewritten, and approved time and
time again until deemed ready for publication. The same applies to a
thesis and a codebase!

Make it clear that constant refactoring and refreshing of a codebase
prevents code rot and reduces technical debt, and facilitates future
re-use and adaptation of the work for other projects.

Why All This??!

Why do we bother with all of the above? For code quality. Or is it
quality code…?

These guidelines aim at driving your team towards this goal. Some
aspects do it by simply showing them the way and letting them do it
(which is much better) and others take them by the hand (but that’s
how you educate people and develop habits).

How do you know when the goal is within reach?

Quality is Measurable

Not always quantitatively, but it is measurable. As mentioned, you
need to develop a sense of pride in your team, and showing progress
and good results is key. Measure code quality regularly and show
progress between intervals, and how it matters. Do retrospectives to
reflect on what has been done, and how it made things better or worse.

There are great tools for continuous inspection. Sonar being
a popular one in the Java world, but it can adapt to any technologies;
and there are many others. Keep your code under the microscope and
look for these pesky annoying bugs and microbes.

But What if My Code is Already Crap?

All of the above is fun and cute like a trip to Never Land, but it’s
not that easy to do when you already have (a pile of steamy and
smelly) crap code, and a team reluctant to change.

Here’s the secret: you need to start somewhere.

Personal anecdote: In a project, we worked with a codebase
weighing originally 650,000+ Java LOC, 200,000+ lines of JSPs,
40,000+ JavaScript LOC, and 400+ MBs of binary dependencies.

After about 18 months, it’s 500,000 Java LOC (MOSTLY CLEAN),
150,000 lines of JSPs, and 38,000 JavaScript LOC, with dependencies
down to barely 100MBs (and these are not in our SCM anymore!).

How did we do it? We just did all of the above. Or tried hard.

It’s a team effort, but we slowly inject in our process
regulations and tools to monitor the heart-rate of our product,
while hastily slashing away the “fat”: crap code, useless
dependencies… We didn’t stop all development to do this: we have
occasional periods of relative peace and quiet where we are free to
go crazy on the codebase and tear it apart, but most of the time we
do it all by defaulting to a “review and refactor” mode every chance
we get: during builds, during lunch, during bug fixing sprints,
during Friday afternoons…

There were some big “works”… Switching our build
system from a giant Ant build of 8500+ XML LOC to a
multi-module Maven build was one of them. We then had:

• clear-cut modules (or at least it was already a lot better, and we
  still have big plans for the future),
• automatic dependency management (for easy maintenance and updates, and
  to remove useless deps),
• faster, easier and reproduceable builds,
• daily reports on quality.

Another was the injection of “utility tool-belts”, even though we
were trying to reduce dependencies: Google Guava and Apache Commons
slim down your code and and reduce surface for bugs in your code
a lot.

We also persuaded our IT department that maybe using our new tools
(JIRA, Fisheye, Crucible, Confluence, Jenkins) was better than the
ones in place. We still needed to deal with some we despised (QC,
Sharepoint and SupportWorks…), but it was an overall improved
experience, with some more room left.

And every day, there’s now a trickle of between one to dozens of
commits that deal only with fixing and refactoring things. We
occasionally break stuff (you need unit tests, and you better write
them before you refactor stuff away), but overall the benefit
for our morale AND for the product has been enormous. We get there
one fraction of a code quality percentage at a time. And it’s fun
to see it increase!!!

Note: Again, rigidity needs to be shaken to make room for new and
better things. In my anecdote, our IT department is partly right in
trying to impose some things on us, and wrong for others. Or maybe
they used to be right. Things change. Prove that they are better
ways to boost your productivity. Trial-runs and prototypes are
here for this.

The Super-Secret Incremental Spaghetti Code Refactoring Cycle for Awesome Quality

       +-----------------+      +-----------------+
       |  A N A L Y Z E  +----->| I D E N T I F Y |
       +-----------------+      +---------+-------+
                ^                           |
                |                           v
       +--------+--------+      +-----------------+
       |    C L E A N    +<-----|      F I X      |
       +-----------------+      +-----------------+

Once you have some quality tools at your toolbelt:

1. Analyze your code with code quality checkers.

   Linters, static analyzers, or what have you.

2. Identify your critical hotspots AND low hanging fruits.

   Violations have severity levels, and large classes with a large number
   of high-severity ones are a big red flag: as such, they appear as
   “hot spots” on radiator/heatmap types of views.

3. Fix the hotspots first.

   It maximizes your impact in a short timeframe as they have
   the highest business value. Ideally, critical violations should
   dealt with as soon as they appear, as they are potential security
   vulnerabilities or crash causes, and present a high risk of inducing a
   liability (and in your case, bad performance for the lab).

4. Clean the low level violations with automated codebase sweeps.

   It improves the signal-to-noise ratio so you are be able to
   see significant violations on your radar as they appear. There’s often
   a large army of minor violations at first if they were never taken care
   of and your codebase was left loose in the wild. They do not present a
   real “risk”, but they impair the code’s readability and maintainability.
   Fix them either as you meet them while working on a task, or by large
   cleaning quests with automated code sweeps if possible. Do be
   careful with large auto-sweeps if you don’t have a good test suite
   and integration system. Make sure to agree with co-workers
   the right time to run them to minimize the annoyance.

5. Repeat until you are satisfied.

   Which, ideally, you should never be, if this is still an
   active product: it will keep evolving.

Quick Tips for Good House-Keeping

• When in hotfix-mode, based on a customer support request:

  • It’s usually a best practice to NOT go around fixing other issues,
    as you might introduce new ones unwillingly.
  • Go at it SEAL-style: get in, kill the bug, get out, and ship your
    patch. It’s a surgical and tactical strike.

• But for all other cases, if you open a file, make it your duty to:

  • definitely: review it (take notes, file issue reports),
  • maybe: clean it (style cleanups and minor violations),
  • ideally: refactor it (reorganize large sections and their neigbors).

Just don’t get sidetracked into spending a week from file to file and
ending up with a massive changeset of thousands of fixes spanning multiple
features and modules – it makes future tracking difficult. One issue in
code = one ticket in your tracker. Sometimes, a changeset can impact multiple
tickets; but if it happens too often, then you’re probably doing something wrong.

Addendum: Managing Visual Programming Environments

The Walled Gardens of Bespoke Programming Systems

Multiple programming systems, like the OP’s G2, are different beasts…

• No Source “Code”

  Often they do not give you access to a textual representation of
  your source “code”: it might be stored in a proprietary binary
  format, or maybe it does store things in text format but hides
  them away from you. Bespoke graphical programming systems are
  actually not uncommon in research labs, as they simplify the
  automation of repetitive data processing workflows.

• No Tooling

  Aside from their own, that is. You are often constrained by their
  programming environment, their own debugger, their own
  interpreter, their own documentation tools and formats. They are
  walled gardens, except if they eventually capture the interest
  of someone motivated enough to reverse engineer their formats and
  builds external tools – if the license permits it.

• Lack of Documentation

  Quite often, these are niche programming systems, which are used
  in fairly closed environments. People who use them frequently sign NDAs
  and never speak about what they do. Programming
  communities for them are rare. So resources are scarce. You’re
  stuck with your official reference, and that’s it.

The ironic (and often frustrating) bit is that all the things these
systems do could obviously be achieved by using mainstream and general
purpose programming languages, and quite probably more
efficiently. But it requires a deeper knowledge of programming,
whereas you can’t expect your biologist, chemist or physicist (to name
a few) to know enough about programming, and even less to have the
time (and desire) to implement (and maintain) complex systems, that
may or may not be long-lived. For the same reason we use DSLs, we have
these bespoke programming systems.

Personal Anecdote 2: Actually, I worked on one of these
myself. I didn’t do the link with the OP’s request, but my the
project was a set of inter-connected large pieces of
data-processing and data-storage software (primarily for
bio-informatics research, healthcare and cosmetics, but also for
business intelligence, or any domain implying the tracking of
large volumes of research data of any kind and the preparation of
data-processing workflows and ETLs). One of these applications was,
quite simply, a visual IDE that used the usual bells and whistles:
drag and drop interfaces, versioned project workspaces (using text
and XML files for metadata storage), lots of pluggable drivers to
heterogeneous datasources, and a visual canvas to design pipelines
to process data from N datasources and in the end generate M
transformed outputs, and possible shiny visualizations and complex
(and interactive) online reports. Your typical bespoke visual
programming system, suffering from a bit of NIH syndrome under the
pretense of designing a system adapted to the users’ needs.

And, as you would expect, it’s a nice system, quite flexible for its
needs though sometimes a bit over-the-top so that you wonder “why
not use command-line tools instead?”, and unfortunately always
leading in medium-sized teams working on large projects to a lot of
different people using it with different “best” practices.

Great, We’re Doomed! – What Do We Do About It?

Well, in the end, all of the above still holds. If you cannot extract
most of the programming from this system to use more mainstream tools
and languages, you “just” need to adapt it to the constraints of your
system.

About Versioning and Storage

In the end, you can almost always version things, even with the
most constrained and walled environment. Most often than not, these
systems still come with their own versioning (which is unfortunately
often rather basic, and just offers to revert to previous versions
without much visibility, just keeping previous snapshots). It’s not
exactly using differential changesets like your SCM of choice might,
and it’s probably not suited for multiple users submitting changes
simultaneously.

But still, if they do provide such a functionality, maybe your
solution is to follow our beloved industry-standard guidelines above,
and to transpose them to this programming system!!

If the storage system is a database, it probably exposes export
functionalities, or can be backed-up at the file-system level. If it’s
using a custom binary format, maybe you can simply try to version it
with a VCS that has good support for binary data. You won’t have
fine-grained control, but at least you’ll have your back sort of
covered against catastrophes and have a certain degree of disaster
recovery compliance.

About Testing

Implement your tests within the platform itself, and use external
tools and background jobs to set up regular backups. Quite probably,
you fire up these tests the same that you would fire up the programs
developed with this programming system.

Sure, it’s a hack job and definitely not up to the standard of what is
common for “normal” programming, but the idea is to adapt to the
system while trying to maintain a semblance of professional software
development process.

The Road is Long and Steep…

As always with niche environments and bespoke programming systems, and
as we exposed above, you deal with strange formats, only a limited (or
totally inexistant) set of possibly clunky tools, and a void in place
of a community.

The Recommendation: Try to implement the above guidelines outside
of your bespoke programming system, as much as possible. This ensures
that you can rely on “common” tools, which have proper support and
community drive.

The Workaround: When this is not an option, try to retrofit this
global framework into your “box”. The idea is to overlay this
blueprint of industry standard best practices on top of your
programming system, and make the best of it. The advice still applies:
define structure and best practices, encourage conformance.

Unfortunately, this implies that you may need to dive in and do a
tremendous amount of leg-work. So…

Famous Last Words, and Humble Requests:

• Document everything you do.
• Share your experience.
• Open Source any tool your write.

By doing all of this, you will:

• not only increase your chances of getting support from people in
  similar situations,
• but also provide help to other people, and foster discussion
  around your technology stack.

Who knows, you could be at the very beginning of a new vibrant
community of Obscure Language X. If there are none, start one!

• Ask questions on Stack Overflow,
• Maybe even write a proposal for a new StackExchange Site in the
  Area 51.

Maybe it’s beautiful inside, but nobody has a clue so far, so
help take down this ugly wall and let others have a peek!

The very first step would be introduction of a Version Control System (SVN, Git, Mercurial, TFS, etc.). This is must to have for a project that will have re-factoring.

Edit: regarding VSC – Every source control package can manage binaries, although with some limitations. Most of the tools in the market has the ability to use a custom difference viewer and editor, use this capability. Binary source files is not an excuse not to use version control.

There is a similar post on how to deal with legacy code, it might be a good reference to follow – Advice on working with legacy code

When I have to work with spaghetti code, the first thing I work on is modularization. Find places where you can draw lines and extract (more or less) independent pieces of the codebase. They probably won’t be very small, due to a high degree of interconnectedness and coupling, but some module lines will emerge if you look for them.

Once you have modules, then you’re not faced with the daunting task of cleaning up an entire messy program anymore. Now, instead, you have several smaller independent messy modules to clean up. Now pick a module and repeat on a smaller scale. Find places where you can extract big functions into smaller functions or even classes (if G2 supports them).

This is all a lot easier if the language has a sufficiently strong type system, because you can get the compiler to do a lot of the heavy lifting for you. You make a change somewhere that will (intentionally) break compatibility, then try to compile. The compile errors will lead you directly to the places that need to be changed, and when you stop getting them, you’ve found everything. Then run the program and test everything! Continuous testing is crucially important when refactoring.

I don’t know if this is an option to you, but I would start trying to convincing them to hire more professional developers. This way they could concentrate in domain problems (I’m sure they have enough there).

I believe they are very smart people, but becoming a good developer demands a lot of time. Are they ready to spend so much time in a activity that isn’t their main business? IMHO, this isn’t the way to achieve the best results.

Wow. Sounds like you have a really big challenge ahead of you! I’d do something along the following lines:

• First of all: Prioritize. What do you want to achieve first? What is the most important for the current state of the project? What will you get the most from vs how much time it’ll take to get there.
• Ensure that you have a version control system. Git or Mercurial for instance.
• Get some kind of continuous integration system (e.g. Jenkins) up and running.
• Get a bug tracking system up and running. Mantis is quite nice in my opinion.
• Look into static code analysis (if something is available for the language you’re currently working with).
• Try to achieve as much consistency in anything from naming of variables to general code conventions and guidelines in the code base.
• Get the system under test. This is extremely important for a big legacy system like this in my opinion. Use test cases to document existing behavior, no matter if the behavior feels weird or not (usually there’s a reason as to why the code looks a certain why, might be good or bad, or both ;P). Michael Feathers Working Effectively With Legacy Code is an excellent resource for this.

They say that the first step in solving a problem is admitting that you have one. With that in mind, you might start by generating a dependency graph that illustrates the vast tangle that is your current code base. Good tool to generate dependency diagram? is a few years old but contains some pointers to tools that can help create such graphs. I’d go with one great big, ugly graph that shows as much as possible to drive the point home. Talk about problems that result from too many interdependencies and maybe throw in a line from Buckaroo Banzai:

You can check your anatomy all you want, and even though there may be
normal variation, when it comes right down to it, this far inside the
head it all looks the same. No, no, no, don’t tug on that. You never
know what it might be attached to.

From there, introduce a plan to start to straighten out the mess. Break the code into modules that are as self-contained as possible. Be open to suggestions as to how to do that — the folks you’re talking to know the history and the functionality of the code better than you do. The goal, however, is to take one big problem and turn it into some number of smaller problems which you can then prioritize and start to clean up.

Some things to focus on:

• Create clean interfaces between modules and start to use them. Old code may, of necessity, continue to not use those nice new interfaces for a while — that’s the problem you’re starting to solve. But get everybody to agree to use only the new interfaces going forward. If there’s something they need that’s not in the interfaces, fix the interfaces, don’t go around them.

• Look for cases where the same functionality has been repeated. Work toward unification.

• Remind everybody from time to time that these changes are meant to make life easier, not more difficult. Transition can be painful, but it’s for a good purpose, and the more everybody is on board the faster the benefits will come.

After looking into Gensym G2 for a bit it looks like the way to approach this problem is going to be highly dependent upon how much of the code base looks like this:

or this:

versus this, courtesy of 99 Bottles of Beer:

beer-bottles()

i:integer =99;
j:integer;
constant:integer =-1;

begin
for i=99 down to 1
    do
    j = (i+constant);
        if (i=1) then begin
            post"[i] bottle of beer on the wall";
            post" [i] bottle of beer";
            post" Take one down and pass it around ";
            post" No bottle of beer on the wall"; 
        end 
        else begin
            post"[i] bottles of beer on the wall";
            post" [i] bottles of beer";
            post" Take one down and pass it around ";
            if (i=2) then 
                post" [j] bottle of beer on the wall"
           else
                post" [j] bottles of beer on the wall"; 
           end
    end
end

In the case of the latter you are working with source code which is effectively a known quantity and some of the other answers offer some very sage advice for dealing with it.

If most of the code base is the latter, or even if a sizable chunk is, you are going to be running into the interesting problem of having code that likely cannot be refactored due to being extremely specialized, or worse yet, something that looks like it may be removable, but unless it is properly documented, you don’t know if you are removing critical code (think something along the lines of a scram operation) that doesn’t appear to be so at first glance.

Although obviously your first priority is going to be getting some sort of version control online, as pointed out ElYusubov, and it does appear that version control has been supported since version 8.3. Since G2 is a combination of a couple different language methodologies you would likely find it to be most effective to use the version control that is provided with it as opposed to trying to find something else and getting it to work.

Next, although some would likely advocate for starting to refactor, I’m a strong advocate of making sure you fully understand the system you are working with before you start touching any of the code, especially when dealing with code and visual diagrams that developed by developers with out formal training (or background) in software engineering methodologies. The reasoning for this is several fold, but most obvious reason is that you are working with an application that potentially has over 100 person-years worth of work put into it and you really need to make sure you know what it is doing and how much documentation there is in it. As you didn’t say which industry the system is deployed to, based upon what I have been reading about G2 it appears it is safe to assume that it is likely a mission critical application that may even hold a potential for also having life safety implications. Thus, understanding exactly what it is doing is going to be very important. It there is code that is not documented work with the others on the team to make sure that documentation is put into place to make sure people can determine what the code does.

Next start wrapping unit tests around as much of the code base and visual diagrams that you can. I must admit to some ignorance with regards to how to do this with G2 but it might almost be worth creating your own testing framework to get this in place. This is also an ideal time to start introducing the other members of the team to get them use to some of the more rigorous engineering practices involved with code quality (i.e. all code must have unit tests and documentation).

Once you have unit tests in place on a fair amount of the code, you can start approaching refactoring on manner such as that suggested by haylem; however, remember to keep in mind that you are dealing with something that is meant for developing expert systems and refactoring it might be an uphill battle. This is actually an environment where there is something to be said for not writing extremely generic code at times.

Finally, make sure you pay close attention to what the other team members say though, just because the code and diagram quality is not the best doesn’t necessarily reflect poorly upon them. Ultimately, for the time being they are likely to know more about what the application does than you which is why it is all the more important for you to sit down and make sure you understand what it does before making sweeping changes as well.

Usually the complaints you hear upfront has nothing to do with the important problems. After all, it is entirely normal to hear of these complaints in any software projects.

Hard to understand code? Check. Massive code base? Check.

The real problem is that people leave, and when the new person joins the organisation, there is a typical disorientation. Furthermore, there is an issue of unrealistic expectations and code-quality issues.

Here is what I would tackle, in order:

1. Backups, both the server and local version
2. Set up bug tracker
3. Set up versioning system
4. Set up FAQ/Wiki
5. First debriefing of all scientist/programmers
   • Remind them of the 80/20 rule. 20% of the bugs are responsible for 80% of the problems.
   • Focus on the biggest issues, and keep enhancement requests et c off.
   • The purpose here is not to scare people with a big list, but a list of small achievable wins. After all, you have to prove your worth as well.
6. Set up build system
   • Start working on getting reliable builds (this may take a while)
   • identify and name every project
   • identify cyclic dependencies
   • if there are binaries from some open-source projects, try to obtain sources
7. Identify how the G2 code can be modularized, e.g. APIs, services
8. Identify how the G2 code can be tested, documented.
9. Set up code-review system
10. Second debrief
    • What does an ideal module look like?
    • How to cordon off old code
    • How to wrap new module around old code (See the Strangler Pattern http://agilefromthegroundup.blogspot.com.au/2011/03/strangulation-pattern-of-choice-for.html )
11. Identify a crack team of better programmers and work with them to wrap their modules.
12. Code reviews are there at this stage to improve communication, and documentation. Keep it easy at this stage. Sort out any process issues.
13. Roll out the system to other programmers. Let the crack team members become peer mentors to the rest. Remember that scaling is the issue here. You are effectively in a management role.

Questions like these are the whole reason the Software Carpentry project exists.

For the last 14 years, we’ve been teaching scientists and engineers basic software development skills: version control, testing, how to modularize code, and so on. All our materials are freely available under a Creative Commons license, and we run a couple of dozen free two-day workshops every year to help people get started.

Based on that, I think the best starting point is probably Robert Glass’s excellent (short) book Facts and Fallacies of Software Engineering: its evidence-based approach is a good way to convince scientists that what we’re telling them about good programming practices is more than just opinion.
As for specific practices, the two that people are most willing to adopt are version control and unit testing; once those are in place, they can tackle the kind of systematic refactoring that Michael Feathers describes in Working Effectively With Legacy Code.
I no longer recommend The Pragmatic Programmer (lots of exhortation, hard for novices to put into practice), and I think McConnell’s Code Complete is too much to start with (though it’s a great thing to give them six months or a year in, once they’ve mastered the basics).

I would also highly recommend Paul Dubois’ excellent paper “Maintaining Correctness in Scientific Programs” (Computing in Science & Engineering, May-June 2005), which describes a “defense in depth” approach that combines a dozen different practices in a logical, coherent way.

I think first of all you have to clear your situation. What do they want from you?

• It is very unlikely that they want you to learn an ancient language, because this now seems a dead end: there is a decreasing chance to find anyone knowing or wanting to learn G2, so the knowledge will be buried in the collapsing heap of code when the current scientists leave or the all-patched code fails more and more often.
• Are the scientists (or some of them) ready to learn a new language and lots of programming paradigms? Or do they want to separate the programming and scientific activity in the long term, and perhaps have some more programmers if needed? This seems a rational and more efficient separation of expertise.

I think the core requirement here is to “save the knowledge in the system”, so you have to go and excavate it!

The first task is to write a documentation.

Analyze the structure and the requirements as if this would be a new task, but with the help of an existing system. They will be pleased because you ASK instead of TEACHING first – and you will quickly get enough, but more organized background knowledge from a programmer’s point of view: “what’s going on here?” The docs (system static structure, workflow, components, problems) will be immediately valuable for them, and will perhaps show up more relevant information to them than to you (some of the guys may have “AHA!” and start fixing some codes immediately)…

You should then start asking where they want to go?

If they are ready to move away from G2, what system do they want to see (platform, language, interface, general structure)? You might start writing an external wrapper around the system if at all possible, having the target structure, but keeping the original components, thus slowly starting a kind of framework that allows new components to be implemented in a this target environment. You have to find the core services (persistent data connections and “toolkits”: core calculation, drawing, … libraries), and so you provide them a familiar environment in a new platform and language, which allows the transition either by you or them: take the old codes one by one, reimplement (and CLEAN!) them in the new environment. When that is ready, they know the new language; and the service layer (mostly made by you, sorry) is ready to host the new components.

If they don’t move, you must learn G2, and create the modular framework there, into which you or they should move the components (with cleaning). Anyway, the language is just a serialization of data and algorithm tree…

While analyzing and writing the docs, read, use and advertise GoF Design patterns! 🙂

… my 2 cents

I just got done doing a series of presentations on Robert Martin’s SOLID principles for my co-workers. I don’t know how well these principles translate to G2, but since you’re looking for 5-7 core fundamentals, these seem like a well-established set to start out with. If you want to round it up to 7, you could start with DRY and throw in Fail-Fast.

The only production problem sounds like a change management issue. If that is the case and the software otherwise performs it purpose the first advice I would give is to resist the urge to do too much too quickly.

Source control, refactoring, more trained devs are all good suggestions, but if this is the first time you have had to deal with this sort of issue moving slowly and making controlled changes cannot be emphasized enough.

The urge to shred through the mess will be great at times, but until you have reverse engineered enoughof it that you know you can test your replacement version adequately you need to be very careful.

The most important principles for working in such a situation are:

1. Be patient. A hole which took 20 years to dig isn’t going to be filled in in a few weeks.

2. Be positive. Resist the temptation to complain and grumble.

3. Be pragmatic. Look at a positive change you can accomplish in a day, and do that, today. Got a version control system yet? Implement it and train people. Then look and see if you can automate testing (Unit testing or something similar). Rinse. Repeat.

4. Be a model. Show (don’t just tell) people how agile works by being agile. The first three points above are the keys to being a Good Guy, which is the predecessor to being an effective Agile guy. In my opinion, people who are admirable developers aren’t only smart, they’re also good, model employees and colleagues.

5. Map your territory. I have a technique for mapping giant legacy codebases. I clone the repo, make a working copy, and then I try to change something, and see what else breaks. By investigating coupling (via global state, or broken APIs, or a lack of consistent API or any abstractions or interfaces to program against) and by reading the code that breaks when I change things, I discover the cruft, I ask questions that lead to insights from the rest of the team (Oh we added that because Boss X 5 years ago demanded that, it never worked!). Over time, you’ll get a mental map of the territory. Once you know how big it is, you’ll know enough to make your map and get home. Encourage others to map the territory of your giant codebase, and to build the technical knowledge of the team. Some people balk at “documentation” because it isn’t agile. Whatever. I work in scientific environments too, and documentation is king for me, agile manifestos be damned.

6. Build little apps. When working with a legacy codebase, I find I get ground down to a pulp. I get my spirit back by building little helper apps. Maybe those apps are going to help you read, and understand and modify that giant G2 codebase. Maybe you can make a mini IDE or parser tool that will help you work in your environment. There are many cases where Meta-programming and Tool-building will not only help you break out of the giant deadlocks that legacy codebases impose on you, they also give your brain the ability to fly unrestricted by your G2 language. Write your tools and helpers in whatever language you can do them fastest and best in. For me, those languages include Python, and Delphi. If you’re a Perl guy, or you actually LIKE programming in C++ or C#, then write your helper tools in that language. teach the rest of the team to build little helper-apps and tools, and “componentry” and you’ll eventually see that your legacy codebase isn’t so daunting after all.

1. Revision control: show the domain experts the benefit of being able to revert, see who changed what, etc. (This is tougher with all-binary files, but if the contents are indeed code, surely there is some sort of G2-to-text converter that can enable diffs.)

2. Continuous integration and test: get the domain experts involved in creating end-to-end tests (easier, since they must already have inputs and expected outputs somewhere) and small unit tests (harder, because spaghetti code probably involves lots of global variables) that cover nearly all functionality and use cases.

3. Refactor common code into reusable routines and components. Non-software people with no revision control probably copy-and-paste 100s of lines at a time to make routines. Find them and refactor them, showing that all the tests pass and the code has gotten shorter. This will also help you to learn its architecture. If you’re lucky by the time you have to start making the hard architectural decisions, you might be down to 100KLOC.

Politically, if you find resistance from the old timers in this process, hire a consultant to come in and give a talk about good software methodology. Make sure you find a good one whose views you agree with, and get management to buy off on the necessity of the consultant even if the domain experts don’t. (They should agree – after all, they hired you, so evidently they realize that they need software engineering expertise.) This is a money-wasting trick, of course, but the reason is that if you – the new hotshot youngster programmer – tell them they need to do something, they may ignore it. But if management pays a consultant $5000 to come in and tell them what they need to do, they’ll put more faith in it. Bonus points: get the consultant to advise twice as much change as you really want, then you can be the “good guy” and side with the domain experts, compromising to only change half as much as the consultant suggested.

“The program itself is a physical model of a complex chemical processing plant
…”

“Since G2 is like not-code, but rather automated code written by some gadawful GUI…” – Erik Reppen

Assuming your software’s main goal is to simulate (perhaps optimize, run parameter estimations upon) a complex chemical plant, or parts of one… then I’d like to throw out a rather different suggestion:

You might do well to consider using high-level mathematical modelling language to extract the essence, the core mathematical models, out of hand-coded software.

What a modelling language does is to decouple the description of the problem from the algorithms used to solve the problem. These algorithms are generally applicable to most simulations/optimisations of a given class (e.g. chemical processes) in which case they really shouldn’t be re-invented and maintained in-house.

Three commercial packages used widely in your industry are: gPROMS, Aspen Custom Modeller, and (if your models don’t include phenomena distributed along spacial domains) there are Modelica-based software packages, such as Dymola.

All of these packages support “extensions” in one way or another, so that if you have parts of your models that require custom programming, they can be encapsulated into an object (e.g. a .DLL) that can be referenced by the equations in the model. Meanwhile, the bulk of your model remains succinct, described in a form easily readable by the scientists directly. This is a much better way to capture your company’s knowledge and IP.

Most of these programs should also allow you to ‘start small’ and port small parts (sub-models) of your monolithic code into their format, by being called externally. This may be a good way to maintain a working system and validate it one piece at a time.

Full disclaimer: I worked as a software engineer at the company behind gPROMS for 8 years. In that time I saw (and occasionally incorporated) examples of custom software (e.g. coming from academia) which had started small and tidy, implementing some clever solution or algorithm, but then exploded over the years with extensions and modifications – without the helpful guidance of a software engineer to keep it clean. (I’m a big fan of multi-disciplinary teams.)

So I can say with some experience that certain key choices made poorly early on in a software’s development (like a language or key library) tend to stick around and cause pain for a long long time… They’ve already ‘shaped’ the software around them. It sounds to me like you could be facing many many years of pure code cleaning here. (I’m hesitant to use numbers but I’m thinking 10+ person years, maybe much more if you can’t get the code ported from G2 to something that supports good automated refactoring tools like Eclipse/Java quick-smart.)

While my default status is to “refactor and keeping a working system”, I also think once a problem gets “too big”, then a more radical change/re-write becomes faster overall. (And possibly brings additional benefits, like jumping to a more modern technology.) I say that with some experience porting to a new software platform, but from what I gather it’s even more dramatic with a port to a mathematical modelling package.

To give some perspective, you might be quite amazed at the size reduction. E.g. the 200,000 LoC could actually be represented in something like 5,000 lines of equations (OK I’m guessing here, but I could try and get you an actual testimonial from friends in the business); plus a few relatively small function modules written in something like C (for example, physical-property calculations – although again off the shelf packages may exist depending on your chemical process). This is because you literally just throw away the algorithmic solution code and let a general-purpose ‘stack’ of mathematical solvers do the hard work. Once you have simulations running, you can do much more with them, like optimising the process – without changing a line of code.

Finally I would say: if the only reliable documentation of the various mathematical models (and algorithms) is the code itself, you will want the help of the scientists and original authors to help extract those models out, ASAP, not years down the track when some of them may have moved on. They should find that a mathematical modelling language a very natural way to capture those models – they may even (shock horror) enjoy (re)writing it.

Finally, since my answer might be off the mark, I’d just like to add one more book to the list of good books already referenced here: Clean Code by Robert Martin. Full of simple (and justified) tips that are easy to learn and apply, but which could make a world of difference to people developing new code at your company.

I would throw down the following:

1. There is one programmer here. Screw politics. They know their trade. You know yours. Mark that territory even if you have to piss on it. They’re scientists. They can respect that sort of thing or should since they’re pretty much constantly doing the same themselves. Through whatever means you can, mark the boundaries now. This is what I’ll fix. This is what I can’t be responsible for.

2. Scientists write/test the algorithms. Scientists who want to can write their own algorithms in 1-3 languages everybody can agree on for you to convert to core code. That puts testing their stuff on them. Beyond that, they’re going to have to help you isolate the important science stuff vs. the good-god-knows-what they did for architecture. The codebase is hosed. There’s lots of slash and burn that’s going to need to be done. Give them options to hand you working versions of stuff that employs what they know best so you can do what you do best. Stick their knowledge in a box that they’re responsible for but that you can work with. Ideally when things go well on a mega-refactor conversations will be more about what sorts of exciting things you can do with interface rather than what tentacle Z9 of drBobsFuncStructObjThingMk2_0109 did when it pissed off global var X19a91.

3. Use an event-driven-friendly language with first class functions if you can. When all else fails, triggering an event or tossing a callback to some object with an interface and state mechanism that actually makes sense can be a huge time-saver when you’re knee-deep in code that makes no bloody sense and quite possibly never will. Scientists seem to like Python. Not hard to glue lower-level math-intensive C stuff with that. Just sayin’

4. Look for somebody who has solved this or a similar problem. Spend some serious time researching. These guys heard about G2 from somebody.

5. Design Patterns. Adapters. Use ’em. Use ’em a lot in situations like this.

6. Learn what you can of the science. The more you know, the better you can determine intent in the code.

Do the analysis first.

I would do some analysis before deciding what to teach. Figure out where the biggest pain points are. Use those to prioritize what practices to go over.

Introduce only a few changes at a time (in a similar situation I did 2-3 practices every 2 weeks).

I would limit the practices to ~3 depending on level of change to there programming style of SDLC; until they start to get comfortable with them (I would push to introduce 1 new change every ~1-2 weeks as they get more comfortable with the idea of learning new approaches). It’s also a good idea to identify what the criteria for success is. What the practice should accomplish (even if it’s a soft goal like team morale). That way you can show if it’s effective or not.

• Why limit the number of changes?

Even if you assume these people want to be better programmers and are open to learning there are limits to how much and how fast people can learn new concepts and apply them; especially if they don’t have CS foundation or have participated in a Software Development Life Cycle previously.

Add a weekly wrap-up meeting to discuss how the practices affected them.

The meeting should be used to discuss what went well and what needs work. Allow them to have a voice and be collaborative. Discuss and make plans to address problems they are having and to preview the next changes coming up. Keep the meeting focused on the practices and their application. Do a little evangelizing on the benefits they should start to see from applying the practices.

Certain practices take precedence.

Proper use of a version control system (IMO) trumps everything else. Close behind are lessons in modularization, coupling/cohesion and feature/bug ticket tracking.

Remove practices that don’t work.

Don’t be afraid to get rid of practices that don’t work. If there is a high cost and little to no benefit, remove the practice.

Improvement is a process.

Convey that sustained, consistent improvement is a process. Identify the biggest pain points, apply a solution, wait/coach and then repeat. It will feel agonizingly slow initially until you build up some momentum. Keep everyone focused on the improvements that are coming and the improvement that are already successful.

Sounds like the first step you have is to sell to the team the need to invest in new software methodology. Per your statement, there is no concensus in the team, and you’ll need it to be able to plow ahead with a slow “upgrade” of the code.

I would (if I can) personally take the hard lessons learned, and introduce each of the key concepts that you want to as the solution to the problem in the software industry.

E.g. two developers had different copies and ended up deploying a hybrid untested release -> Introduce version control, branching, and testing.

Somebody removed a few lines of code that they didn’t understand and caused an outage -> introduce DDD.

If the hard lessons aren’t being shared with you in sufficient detail, then just show your own examples of how things went wrong when this discipline wasn’t adhered to.

Source code control is step #1 as already stated many times. While the people you work with may not be professional developers and they won’t respond to a lot of enterprise or agile mumbo jumbo. They’re not low level code monkeys either and trying to treat them like that by forcing them to do things ‘your way’ will not fly.

You’ve got to survey whats out there. If they haven’t used source code control then just identifying the right versions of the code (if possible) and what all the possible deliverables are is going to take a long time. Then you’re going to have the task of teaching your colleagues how to use source code control and convince them that its worth their time. Start with the benefits!

While you’re doing that, find other low hanging fruit and fix those problems.

Above all else, listen to what they have to say and work on improving their situation. Don’t worry about trying to put your stamp on what they do.

Good luck!