I work in the Veteran Relationship Management program office in the Veterans Benefits Administration (VBA) and we manage the delivery of about 25 veteran-facing enterprise software applications. The vast majority of the development work is contracted to outside developers. As is good practice in larger scale software development – especially when there are multiple companies involved – there is a focus on “requirements trace-ability”. This helps ensure that the software that is ultimately delivered to a business stakeholder (the VA) by a contractor can be traced back to the original requirements the business and stakeholders agreed on. But how do we know that the requirements that have been put on paper actually meet the real needs of veterans? I’m proposing that for user-facing software solutions along with a “Business Requirements Document” (which is the current artifact for all projects at the VA that detail what a solution needs to do in non-technical language), a “User Research Document” is required as well that uses artifacts from Human Centered Design to support a real understanding of users.
How do we know that our requirements are any good for our end-users (veterans)?
From my my background in startups and product development I’ve been most concerned the past five years with how to do a better job of building products and services people actually need. In my Presidential Innovation Fellowship at the VA I’m trying to answer the same question but in the context of the federal government delivering services to 21 million veterans in many cases through technology solutions such as websites and mobile applications.
In the startup world or in the private sector it’s fairly straightforward to see how latching onto user needs is important. If you don’t build things people truly want or need then no-one will buy your product and you’ll soon go out of business. So many startups have built well-crafted technical products that users didn’t really need and then went out of business (taking their investors millions with them) that the Lean Startup movement was basically born to address this problem. But in the federal government our users (citizens, veterans) don’t directly pay for our services and there is mostly no competition. In other words, when we have a website with a low user satisfaction rate (such as one I work on with a customer satisfaction rate of 55%), there are few direct consequences for that – unless veteran complaints go through congress or somehow reach the Under Secretary etc. The the natural “nerve-endings” that wire dangerously low user-satisfaction to the product owner “brain” in a private company don’t really exist in the federal government.
So we have to work extra hard in the government to intentionally wire-up a nervous system that makes sure for user-facing services delivered through technology we are paying attention to user needs. I have some thoughts on how to do this, but that is for another blog post. For this post, I’d like to propose the idea of “User Trace-ability” to exist with equal importance with “Requirements Trace-ability” and use existing tools from Human Centered Design and Lean Startup to implement this. I propose To complement the BRD (Business Requirements Document) all business users will have to include a URD (User Research Document).
How software is currently built at the VA (and most of the federal government) in the immediate year prior to a development contract being awarded.
- Business stakeholders with the help of a program office like VRM or OBPI write up high-level business requirements for a solution. This is a document that reads like “The system shall allow veterans to log into the site with DSLogon credentials”, “The system shall allow veterans to search on mental health services near them and receive a set of results of nearby facilities”.
- This document gets fed into the procurement process – IT comes up with a “level of effort”, there may be some back and forth with the business (but in most cases not), a contracting officer writes up the Statement of Work (or Performance Work Statement in the VA) and it goes out for bid to contractors.
- When contractors come on board many months later, they dig into more detailed requirements with the business stakeholders. The output of this is a more detailed business requirements set that is to be used to measure software delivery.
From what I’ve seen, however, there is no evidence offered to anyone downstream of how these business requirements were identified, vetted or validated with end users (veterans). Basically, everyone largely takes the word of whoever put the BRD together. I’ve been told in some cases BRDs are put together last minute in 48 hours. The many choices of what goes into a website that delivers a service should be based on actual work done with users rather than a smart business stakeholder putting on paper their best effort at what the solution should look like based on their experience. In the best private companies and startups, this practice is seen as very flawed and the source of a tremendous amount of wasted effort and mediocre products. In Lean Startup you learn that it really doesn’t matter what a room of smart people think their users need – they’re almost always wrong. You need to check your assumptions and validate your choices rigorously with direct user research and contact. At the VA and the federal government in general we almost always embark on building very expensive solutions based on business requirements that have little evidence of user research to back up the choices made.
The User Research Document
I propose that when a BRD is delivered to IT, the business must also product a URD (User Research Document). This would consist of artifacts from the Human Centered Design discipline:
- A stakeholder map – of all of the players involved in the service being rendered to the user.User Personas (or “mind-sets”) of the main types of users
- A journey map of how the user currently goes through the system
- Key insights derived from 8-15 user interviews with stakeholders in the system.
- These are not boxes that can be checked in a hurry. You need to go through some effort to do this.
But we already do some of this in the BRD!
Sometimes, some of this is included in a BRD. But its not complete, is in Lean/Six Sigma process speak (not user-focused). But its generally not clear where it came from and I’ve never seen a BRD where there is proper user-based research and evidence to back up the hundred items “The system shall do …”.
But we don’t have the time to do that!
Well, tough. You have the time to ask for $5 million of tax payer dollars to build a solution that you made up in three days? Make the time to do a clear, focused user research up front. Hire fewer people in the the army of PMI-certified contractors and make some of them design researchers – its money very, very well spent. Then the folks managing the implementation will actually be managing the delivery of something good that veterans will be happy with.