User Acceptance Testing (UAT)
Some patterns pop up again and again. On many, actually most, of the products that I’ve managed, I’ve had to spend huge amounts of time doing manual testing, finding what’s broken, filing issues, tracking issues, chasing down engineers, apologizing to stakeholders and clients, and then following-up and re-testing to make sure that the issues were resolved… only to find that the same issues pop back up a week or two later, and force me to relive the same nightmare over again.
It’s common to refer to this process as User Acceptance Testing (UAT). Let me make it clear that this is a grotesque violation of the acronym and meaning of “UAT.” If you’re working on a small team and doing Agile development, you may very well NOT have a dedicated (or even part-time) QA person/team. Moreover, unit and feature tests, and peer-review of code oftentimes do not catch bugs and issues that can easily be seen in the live application. An obvious example, and one that I’ve seen countless times in the past, is when two engineers are working on different features and they both make conflicting changes to the CSS or JS that breaks something in the UI or renders specific interaction elements dysfunctional.
Preventable issues like these that get past the engineering team to be discovered by the Product Manager (or the user) add unnecessary overhead and operational inefficiency to the team as a whole. Reason being, as a PM, I now have to create a new issue in JIRA, prioritize/schedule it, assign it, track it, and retest it. Moreover, the engineering team has to create a new branch, go back into their code (after it’s become a bit less familiar), fix it, and redeploy it in another sprint. Even if you choose not to attach story points to bugs, which I believe is a bad practice, it will impact your velocity; it must, because it detracts from the team’s ability to work on value-added features. Bottom line, there is a huge yet unrecognized cost inherent in this process when it’s performed the wrong way.
How do we do it better? Ninety percent of the issues that I find when doing “UAT” could have been easily found by the engineers themselves. In Agile, the onus of developing “shippable” features is on the Engineering team. They should be responsible for, at very least, doing a sanity check on the feature that they worked on and any other existing features likely to be impacted by their code. DevOps process improvements can also be helpful. Oftentimes, code conflicts don’t exist on a developer’s local machine because they haven’t merged upstream changes into their feature branch. A best practice is to ensure that this occurs prior to the engineer merging their code (or pull request if CI is setup to auto-deploy to Dev).
It also helps A LOT to implement product demos (performed by the engineers) prior to, or at the start of, the sprint planning meeting. This is built into the Scrum process but, unfortunately, its importance is often overlooked by organizations, and thus, it doesn’t get included. Here’s the reason it’s important: when an engineer knows that they have to publicly demonstrate (to their peers, team, and Product Owner) a feature that they’ve developed, they make sure that it’s working ahead of time, they make sure that there aren’t bugs that will evidence themselves during the demo, and they make sure that it meets the Acceptance Criteria that was created by the Product Owner. Nobody wants to be embarrassed, and engineers typically take great pride in their work. Furthermore, the product demo provides an opportunity for engineers to be publicly acknowledged for their work and the effort that went into it.
In closing, the bottom line is that it’s crucial to establish a culture of ownership and accountability in your Engineering team. Code commits shouldn’t just pass the tests that the engineer wrote (assuming TDD) -that’s not enough -the end-user doesn’t give a damn about tests; they care about whether or not the product/feature works as it’s intended and provides the value that they expect.