how to convince team and project manager that post-launch deployments are okay for new features

We have a site that launched and is “feature-complete”, we’ve done all that the client has asked for and fixed a few bugs post-launch. However, I noticed that our site’s home page is using 100 SQL queries, and there’s no caching. I took half a day to add caching and did it in a branch.

Now I’m wondering how I can convince the project manager and other team members and gain support for a post-launch deployment of this optimization.

What reasons can I give that it won’t harm the project to do this? Or do I just wait until the next project launches to make these sorts of improvements?

More details:

I can’t very well tell future employers that I implemented caching and saw an increase in performance without actually deploying it and capturing the improvement (through google analytics or Page Speed or some other method), or can I?
This is an agency environment, none of our projects use any caching or any other performance optimization
The site has 50k visitors
The site is only live with full functionality for 1 month, around the end of May it will be replaced with a static page (marketing campaign site)
local testing reducing page load from ~3 seconds to ~700ms

While what you have done is sure to be a good thing, the project manager has to worry about a number of things:

testing – what if you break something. Either something in the home page, or some other unintended consequence? The system needs to be retested. Depending on how far your company goes, this could be a large cost.
Scheduling user downtime. May be trivial, may be costly.
The PM may worry about any political reasons to do this outside a normal release. Telling the customer so soon after release that you want to do an update purely for performance issues that have NOT been raised by users may decrease confidence in the system. Maybe he would prefer this was rolled into a normal release cycle.

Additionally, if no-one has reported performance as an issue, then why should the project waste money fixing an issue that to them, does not exist?

Well first and foremost, it has to pass all your regular Q/A steps AND show a real performance improvement.

Now you’re faced with a business problem: the client has what they paid for. They’ve accepted it, and are using it in production. They think the project is finished.

It is possible that they don’t want to risk ‘rocking the boat’ for an update that would bring little or no visible changes to them (this is why pure refactoring changes are sometimes resisted by the business).

It is also possible that your sales folk would want to get paid for this improvement.

Old joke: a sales guy & a programmer are meeting with the customer. The customer asks for a difficult feature & the sales guy promises it immediately. The programmer kicks the sales guy under the table. The customer then asks that the main page’s color be changed. When the programmer says ‘no problem’, the sales guy kicks him under the table. Later he says, ‘you idiot – the customer would have paid $10k for that change’.

TL;DR

Test your changes to ensure they are truly improvements. Run them through your Q/A process. After that, introduce the improvements to the project manager and have her run it past the business folk. Maybe the changes will be promoted, maybe they won’t. Either way you did some good work.

Has the client approved the cache expiration period?
Have you run the same tests as for the last release against your
branch?
Will redeploying result in the site being unavailable? If yes, will the client provide a variance to your service level agreement regarding up-time?

If the answer to any of these is “No”, then it isn’t ready for deployment. If the answer to all is “Yes”, then it should be a clear win for integrating back into the trunk/main branch.

Typically one would refer to an in-between mini release as a patch. The line between a patch and a full software release however is not a settled matter by any stretch of the imagination. It is sometimes hard to make that determination.

How much development work went into the patch?

It is not always about how much development work went into the patch. A patch might comprise less than an hour of easy development but the impacts these changes make can be monumental.

How much risk am I introducing?

Above all else a patch must be stable. Stability of a patch is directly correlated with the number of components, features and deployment complexity that are impacted by the patch changes. There are a number of different risk scoring systems to choose from but they all tend to revolve around the number of impacts and the importance of these features to the client.

High risk is okay as long as it is met with an equally high amount of quality assurance and testing effort.

Does your system have repeatable unit tests covering components or functionality that might be impacted by caching?
Does there exist any kind of automated UI or front end testing (Eg. Selenium )
Do you have load testing scripts that can verify the cache holds up to production level + demands?

These are all a minimum requirement in my mind just to even consider something like caching being introduced into software in the form of a patch.

Caches are no small thing. Small mistakes can have profound show stopping issues so you must be sure that you get it right.

I can’t very well tell future employers that I implemented caching and saw an increase in performance without actually deploying it and capturing the improvement (through google analytics or Page Speed or some other method), or can I?

You can and you should. It is important that you have a seperate test environment for your web site that is self contained, has quality data, and attempts to mimic the production environment in as much a way as possible. How can this be done?

Seperate public hosting for your test environment, or unutilized or under utilized in house server that can be used to host your web application.
Database can be established with the same exact schema and a snapshot of real production data (De-identified of course if it contains sensitive user information).

Now with automated UI testing tools as well as load testing tools against your test environment, you should be able to mimic the 50k visitor load against a test environment that is a copy of the production. This will help flesh out any hidden problems that you would otherwise unfortunately discover in production post deployment.

Filed under: softwareengineering - @ 10:57

Thẻ: agile, deployment, management, project, release

Thiết kế website giá rẻ

Danh mục

how to convince team and project manager that post-launch deployments are okay for new features

How much development work went into the patch?

How much risk am I introducing?