We have an issue of maintainability for our pipelines at the organization where I work, whereby making a change and ensuring the change is tested and bug-free for our customers (other developers in the organization) can take at least a month or more in some cases for seemingly simple enough changes.

We provide standard pipelines for our code templates (Java for ECS and NodeJS for Lambda, Angular for SPA) that developers in the org use. The pipelines are somewhat straightforward (build, unit test, deploy, health check). But do use other techs like terraform, docker, serverless etc within the pipelines.

We also have other pipelines that handle consumer creations (an API wanting to call another API in our org), which handles the usual delete/update create operations on two different pipelines that handle all of these requests. There can be some other business logic-esque (but not many) parts in the code template pipelines.

To break it down further, I provide the pain points of making and testing a change below. For reference, these pipelines are pretty large and do a bunch of custom logic, all in Groovy using Jenkins.

1. No static typing; we have shared global objects, but these have become quite monstrous in size and are just a hashmap where the maps are hierarchical, but still some property names seem to change. When creating any change whatsoever, I need to commit my change, run the pipelines (depending on the stage, this can take 10+ minutes) and wait for a failed run because I am calling an undefined property or variable, and the cycle begins of fixing.

2. There is no automated testing for testing the pipelines themselves and QA will have to manually test the pipelines, if the change involves one of the starter code templates, the QA will have to go through the whole flow of creating a new project, registering it on our in house full-stack app (we have control and can make changes to this as we please), creating a pipeline and then run differing cases which can take around 30 minutes each for the pipeline to complete (the caching of builds and etc makes this slower, a WIP to make this faster is underway). The QA then manually checks AWS and other services to ensure it was created correctly

3. Because of the large amounts of code in this repo, developers often have to re-learn what is happening under the scenes in the set-up stage when making small changes.

Overall, it is unpleasant to work on these, but we have to maintain and upgrade parts of the pipelines quite regularly. So my questions are below if someone has encountered the same sort of problems.

How do you ensure pipeline code is tested through automated test suites?
How do you get fast feedback on code changes when altering CI/CD pipelines?
Are there alternative products or techniques with Jenkins which were more maintainable/”fun” to work with?