User Acceptance Testing Defect Classification when developing for an outside client

I am involved in a large development project in which we (a very small start up) are developing for an outside client (a very large company).

We recently received their first output from UAT testing of a fairly small iteration, which listed 12 ‘defects’, triaged into three categories : Low, Medium and High.

The issue we have is around whether everything in this list should be recorded as a ‘defect’ – some of the issues they found would be better described as refinements, or even ‘nice-to-haves’, and some we think are not defects at all.

They client’s QA lead says that it is standard for them to label every issues they identify as a defect, however, we are a bit uncomfortable about this. Whilst the relationship is good, we don’t see a huge problem with this, but we are concerned that, if the relationship suffers in the future, these lists of ‘defects’ could prove costly for us.

We don’t want to come across as being difficult, or taking things too personally here, and we are happy to make all of the changes identified, however we are a bit concerned especially as there is a uneven power balance at play in our relationship.

Are we being paranoid here? Or could we be setting ourselves up for problems down the line by agreeing to this classification?

You can say “Reported Customer Issue” or anything else if it makes people happy. I’m going to say “Bug” because it has such a grand history and is the industry standard term. In a brand new bug list, you might expect to fix near 100%. But the more bugs that are added to the list, the fewer you expect to fix. In fact, most bug tracking tools let you close bugs for various reasons: duplicate, feature-request, unable-to-reproduce, on-hold, additional-feedback-needed, etc.

The important thing is to capture the feedback and the actions taken on that feedback. We make it very clear to our customers that we get to choose what bugs we fix and what features we implement. We do it on the basis of what we think will make the most clients most happy, but it’s our choice.

Be respectful in providing feedback on these bugs. If you aren’t going to fix something, tell them why. Sometimes it’s OK to say that you aren’t ready to commit to a particular solution to a given bug. Or that a bug is out of scope for this release (or this product). Or that a bug in how the system works might be “fixed” through improved documentation (such as tweaking the UI or documentation so that users don’t do what caused the bug). Obviously, if you fix some bugs, your client will be more open to your not fixing others.

If the customer has major problems using your software, no bug naming system will make them happy. If they love your software, the name won’t matter either. Make great software, be respectful to your client and don’t get hung up on bug categories.

What you need to understand is that in large corporations, there needs to be a clearly defined process, or else everyone’s off doing their own thing, and team A doesn’t know what Team B’s definition of “defect” is. So, there’s probably a pretty strict classification system, so that teams can instantly know what other teams are talking about.

However, this doesn’t mean that those terms should have any meaning at your organization. In fact, just because it’s a high priority for them doesn’t necessarily mean it has to be a high priority for you (I know that’s scary to think about because they’re probably a huge customer). In their eyes, whenever it does something unexpected, it’s a defect, whether that unexpected is as designed or not. Also, if you have complex software, it can be easy to get confused about how things are supposed to work. I wish our bug tracking software had a closing reason of “You’re not doing it right.”

For instance, at my company last week (we’re not a custom development shop, we have actual software products), a customer submitted a bug through their account rep, and acted like it was an actual bug, i.e., the product didn’t work correctly. I explained that the product worked as designed and as in the contract, what they wanted was an enhancement. That’s not as scary for management as “bug” or “defect”.

Just make sure that you don’t “fix” defects that are out of the scope of the project. That’s how you get mooching customers that assume you’ll do everything for free.

Yes, you could set yourselves up for trouble by blindingly accepting the client classifications of bugs/defects/features/they-don’t-know-what-they-want/requirements. As Greg Bair points out, there are cases where “You are doing it wrong” is the issue not the actual software.

These most often come up because the persons who define the requirements don’t use the software day-to-day. One thing to help mitigate this would be to establish a channel to get direct user feedback instead of through the Sustainment/Support teams.

Ideally, your team would have a chance to talk to the individuals who entered the defects so you can extract specifics. Often times simply talking with the individual and explaining why something does what it does(even saying the requirements say it must) can get them to back of a defect.

It would be nice to think there is a way to resolve this, but you’ll find that as a very small supplier to a large company, they pretty much dictate terms. I’ve been in this situation for a long time with a huge multinational, and we have to learn to roll with their way of thinking.

The most important mitigation is to have good relationships with the people involved, especially the decision makers. Then you can gently influence, and if things get ugly, they will speak up for you.

Of course you can make sure you have good contract/legal terms that protect you, but they won’t help much if the customer really decides they don’t like you and is going to make life difficult. Which is why the personal relationships are so important.

Are we being paranoid here? Or could we be setting ourselves up for problems down the line by agreeing to this classification?

This sounds to me as Client management topic/issue.

It seems that your development approach is using Agile/Scrum methodology which is the very correct for this type of client. However, your product owner may better know what is the client’s real need and essential expectations – which should be the high priority to be included in the development and sprint planning.

Your approach to log all medium and lower level defects is important, because you may include them in other iterations or deliverable phases of the product. Thus, it comes to my previous point that product owner’s clean understanding of essential business requirements is the key factor that you should take into account.

Filed under: softwareengineering - @ 04:29

Thẻ: acceptance-testing, defect

Thiết kế website giá rẻ

Danh mục

User Acceptance Testing Defect Classification when developing for an outside client