Customer Experience, Development Operations and Agile Releases (CEDAR)

Our guest speakers will further describe how viewpoints of Veterans are kept front and center throughout the lifecycle of a product by recruiting Veterans for user experience studies to help the VA make its digital tools more user-friendly.

What is TAC Talks?

Come for a peek behind the federal acquisition curtain as we gain insights from acquisition professionals at the US Department of Veterans Affairs and dissect varying relevant topics. In this five-episode series we will explore topics such as proposal evaluations, innovation, debriefs, and more!

TAC Talks is premiering Tuesday, September 29th!

The Department of Veterans Affairs does not endorse or officially sanction any entities that may be discussed in this podcast, nor any media, products or services they may provide.

[Music]
Chuck Ross: Welcome to the Technology Acquisition Center Podcast, which we affectionately call TAC Talks. Join us as we discuss highly relevant and compelling acquisition topics with highly esteemed industry professionals and attempt to share information with you, the 1102 workforce, program officials and our contractor friends. We hope that you find these topics and discussions helpful. So turn up the volume on your earbuds, and get ready for TAC Talks.
[Music]
Chuck Ross: Hello, friends, and thank you for joining me today for TAC Talks. My name is Chuck Ross, a Service Director at the Department of Veterans Affairs Technology Acquisition Center. We have an exciting episode for you today regarding how the Veteran stays in the forefront of the Requirements Development Process under a recently awarded contract here at the TAC. It is called the �Customer Experience Development Operations and Agile Releases Indefinite Delivery Indefinite Quantity Contract�, or as we affectionately call it here at the TAC �CEDAR�. On March 29, 2021, the Technology Acquisition Center, awarded four spots to Serviced-Disabled Veteran-Owned Small Business vendors on the 247.3 million dollar CEDAR IDIQ contract for commercial and application development. The contract was a 100% set aside for Serviced-Disabled Veteran-Owned Small Businesses and served as a streamlined enterprise-wide contract vehicle that will provide VA access to vendors specializing in Agile design and delivery services, including User Centered Design, User Research, Product Management, Agile Software Development, application programming interfaces or API development, Automated testing, cloud infrastructure design and management, continuous integration, continuous delivery, and DevOps. The CEDAR contract vehicle allows for Agile delivery services to be delivered in a rapid cycle fashion, in accordance with industry best practices. In addition, it supports the execution of multiple VA digital modernization efforts by providing VA a simplified method to add specific and specialized commercial capabilities to these efforts as needed.
(CR) So, I'm here today with Mr. Jeffrey Barnes, the Deputy of the Digital Experience product line in the Office of the Chief Technology Officer, Mr. David Melton, who is a Contract Specialist here at the Technology Acquisition Center and Mr. Joshua Cohen, who is the Contracting Officer at the Technology Acquisition Center, responsible for awarding the CEDAR contract. So, thank you for joining me today gentlemen and Welcome to Season 3 of TAC Talks.
(CR) Jeff I'd like to start with you. So, we understand that the Cedar IDIQ is primarily focused on building digital services and products for Veterans and their families, so can you please talk to us about the concept behind the CEDAR contract?
Jeffrey Barnes: Sure, thank you for having me on Chuck, I'm glad to have a chance to talk about CEDAR as it was a long road getting to this point and we have a lot of optimism about what we're going to be able to accomplish here. So, as many listeners are probably familiar there had been some difficulties with building digital services in the Government space and in the Obama Era the U.S. digital service was founded in response to some issues with Healthcare.gov so the idea was that we could find a way to continuously rotate private sector talent into the Government to help keep us up to date. So, we had a digital service team at VA trying to solve some tough Veteran facing challenges, including applying for benefits and seeing your claim status and our government contractor teams, they try to follow the digital services playbook, which focuses on software development using the Agile methodology user centered design and dev OPS delivery model. So, we wanted to create a niche space that gave those vendors the opportunity to really narrowly focus on that way of working. And we were looking for vendors that focused on user outcomes, on quickly shipping minimum viable products, instead of a more traditional IT model of building to requirements with longer development phases before anything goes live to the public. So, CEDAR was intended to be a consistent way for the Government to ask those vendors to follow Agile principles and allow that user feedback to influence and shape the final products.
(CR) CEDAR�s all about Veterans, right? I mean, you have your SDVOSB set-aside so the companies � our Service-Disabled Veteran-Owned Businesses � and it's my understanding that you take the viewpoints of Veterans and keep them in the center and forefront as you're developing these products under this contract. Is that right? Can you expound upon that?
(JB) Yeah, absolutely, and you're correct about this being an SDVOSB set-aside, and one of the things that we specifically focus on at VA is what we call our digital modernization vision, and that basically says that we want to provide Veterans and their families self-service tools that are on par with some of the top private sector companies, and we want to have the best online experience in the federal government. Now, as you can see, that's an ambitious vision, but we've kept some principles in mind that we follow on a more daily basis to make sure that we're shipping products that Veterans want and need. So, for example, every VA service, we wanted to have a single high-quality digital version designed with direct input from users. We want those services to be accessible, to be reliable, and to be secure, and whenever possible we want those services to actually be personalized to the individual that's using them so that they can see what VA knows about them already. We also want to make the account holder experience good by having them be able to access multiple services without having to sign in again or create new accounts. So, in order to accomplish this, we follow a robust user research practice that gets early designs into the hands of the Veterans to test out and then we iterate on those designs, we build functional prototypes, and then researchers like myself can take that to the medical center or wherever else and test it on a phone or a laptop with a real user. So finally, when we launch those products, we send them quickly to live sites so that only a small percentage of our total population sees them. That allows us to experiment on the approaches and figure out which ones lead to the best outcomes. One example of this approach that we've applied in the past was, we had a small team that built a product called Vets.gov that allowed Veterans to apply for a few benefits, and then we decided to launch it to all Veterans on VA.gov, and we simultaneously allowed them to sign into that tool with their other existing VA credentials. So VA is a big place, but Veterans, they think about it as one organization, so we strive to make their experiences with the digital tools reflect that.
(CR) That's great, and when you go out and you do the user centered design, are you going out into the open public with that and getting Veterans, or are you going to facilities, or if I'm a Veteran listening here today and I am interested in taking part in some of these type things is there an opportunity for me to do that?
(JB) Yes, absolutely, we have ways for Veterans to reach out and to join. We actively do recruiting on a lot of social media platforms for user research, and we also have teams of researchers that are employed by the VA as either staff or contractors that go to medical centers and other locations owned by VA in order to find and interact with users.
(CR) Thanks, Jeff. We have a large Veteran listening audience, so I think they'll find that information useful. Now, switching over to the contracting side of this contract, Josh Cohen was the contracting officer that awarded this, and as part of the contracting approach, Josh, I understand that you used several novel or innovative approaches to this contract. Can you describe some of those for us?
Josh Cohen: Great, thank you, Chuck. And yes, absolutely. And I think the first point I'd make was any novel or innovative approach we used was specifically intended to solve a kind of known challenge. So it wasn't necessarily that we kind of ventured off on this with the intent of just trying a bunch of stuff for the sake of trying it. I think we looked at anything novel, or if you want to use the word innovative that we did as a means to try to solve some of the challenges that Jeff had just discussed related to building digital, you know, modernized digital products in the government space so some of those issues, as he mentioned were kind of the uniqueness of the work to the government that kind of smaller companies that tend to work on it, and that are fairly new to the government market and that kind of normal barriers to entry and challenges that a company like that would face when trying to do business with the government and then obviously the kind of technical difficulties that come along with the work in general and just making sure that that if we were going to put the time and effort into looking for firms that could do this work well that we came up with you know evaluation techniques that allowed that to be made clear during the process and obviously heavily weigh on your work. So how do we go about doing that? The first thing we did was to utilize a fairly unusual basis for award called the highest technically rated with a fair and reasonable price, which had been used a few times by the GSA and by a few other agencies, but normally with something they called the self-service model or self-scoring model, whereas we, we did it a more traditional way with adjectival rating. So, we, we did do it a little differently, but the basics of that is, as opposed to making a trade-off between price and technical we made a ranking based solely on technical and non price factors since obviously because we discussed this is very technically challenging work and we really wanted to make sure we have the right kind of vendor and then handle price as a second step based on that ranking so once an apparent set of winners had been chosen and then pricing was checked and then only to review it as fair and reasonable and how that was done was by comparison to rates that were provided with the solicitation so there was no guessing on the part of industry as to what a fair reasonable rate would be and that it helped us avoid some of the gaming that I think can happen on IDIQ is where prices are bid without there necessarily being a promise of delivery at that level and that can sometimes skew an award decision to a lower price as opposed to the technical chops. After that, I think the next most important thing we did was we call an advisory down-select. So, obviously we're talking smaller firms that are kind of new to the process maybe wouldn't have the type of proposal and legal staffs that some companies have and we wanted to try to use a process that helped them either, either move forward if it looked like they had a reasonable chance of winning. But if not, give them that information quickly and let them make it a kind of informed decision as to whether they wanted to move on and that is referred to as an advisory down-select and there's a little different from like a competitive range where we're not telling them there's a cut. We're giving them information on their chances moving forward and then letting them make a decision and obviously the advantage of that is there choosing to remove themselves were not actually kicking anybody out and that worked well and again saves a lot of smaller firms that really didn't necessarily have a reasonable chance after the first round of case studies, which we'll discuss in a minute and and I again, I think really supported small businesses and helped them avoid some unnecessary costs. We also allowed for in that first round there was something of a past experience evaluation that was done via case studies and we allowed for commercial experience so again these are vendors that maybe haven't had a ton of experience in the government this is still work that is fairly novel here and that allowed for them to use the work, they have, maybe done in the commercial marketplace that was very similar and and still get the same credit as if it had been done in the government and I think the last important point to make that we really intentionally went out with to try to incentivize like a smaller, a smaller Veteran owned small businesses that that are a little newer to the market, which was to limit the size of the orders to only $10,000,000.00 and that was based on a lot of market research and conversations with small businesses about like how much work would be too intimidating for them to take on I think they were a little worried about having to take on a huge thing that was just outside the scope of a company and that would have dissuaded them from bidding by limiting the size of the orders you know, I think that that puts a certain cap on the risk to them of what they would be asked to take on. And again we were we were told through some informal channels that that that certainly helped and there were definitely firms who probably would have been a little intimidated by larger contract and then that aspect of it was a big, big part of them coming in on it, so.
(CR) So you had, I mean, you had a lot of stuff going on here. I mean, just to recap you had the highest technically rated proposal that was determined fair and reasonable, reasonably priced that is a novel concept in and of itself. I don't even think that it would be new to business, that's traditionally done business with the federal government as well so that's really an interesting evaluation technique that you used and the advisory down-select-- We're seeing more of that at least at the VA TAC, so that that's very interesting. Now was there any lessons learned or growing pains from that or you know for our other agency contracting officers that are out there, you know is there any tips that you could offer them? With respect to that technique because it is, it is rather innovative?
(JC) Absolutely and I think you could probably make a whole episode just on kind of lessons learned and and like anything you kind of do for the first time there's lots of stuff you probably would have done differently. But I think the big lesson learned is make sure you're careful with what you ask for that you, you know remember that, you don't need to ask for the whole world but you wanna ask for enough that it's meaningful so there's kind of a balance you strike between those two things and just being really clear like through your market research phases and through your requirements documents. What are we looking for in terms of a technical ranking because again it's a different mindset for the vendors between, you know it's not like we're not looking for the best value for the government but we're defining that purely based on technical. So they then have to go out and and think that same way of wow, it's what's the best foot I can put forward technically as opposed to really like focusing on how can I get the price down, right? We expect that to happen at the task order where level where we drive competition and we ask for price discounts. I mean, obviously this is just to establish the vehicle, base vehicle and then there are orders that are then negotiated where that happens so, we did a good job at it but I think more and more of that would be good in the future just to really hammer home those points.
(CR) Right and there's often this misconception out there that the only fair and reasonable price is the rock bottom price offered and you know, Jeff you probably were happy to see an evaluation approach where it was considering the highest rated technical proposals right off the bat. Because you know you as a Program Manager want to see success for your program and you know you wanna, you want to be have the opportunity to award to those types of companies right.
(JB) Yeah, absolutely just getting to a point where we can really use technical qualification as the biggest discriminator when we select a company is very challenging almost everywhere in the government. And being able to get to that point on this acquisition made all the difference in terms of being able to select those vendors that really met the highest standard across the board.
(CR) So now let's shift to Dave. Now Dave's the Contract Specialist and all of us in contracting know that the contract specialist is the one that really makes all this, this work and that's where all the works done right, Dave. (chuckle)
David Melton: Thank you. Yes, no, I'm just kidding.
(CR) That's right! So Dave, as the contract specialist you were intimately involved in with assisting the technical evaluation team, I'm sure through the evaluation process so can you describe the overall evaluation approach and how the process considered the Veteran experience I believe I am of the understanding that you guys concentrated on the Veteran experience a bit in your tech eval is that right?
(DM) So yeah, Chuck, so for our first technical factor we requested case studies be submitted. Each offeror was to provide up to three case studies, covering three of the thirteen CEDAR functional areas, including Agile software development, User Research and User Story collaboration and DevOps. Offerors were also required to provide up to nine artifacts and each of the three functional areas needed to be covered in at least one artifact. The case studies evaluated the government's confidence in the offerors' ability as evidenced by the past experience and expertise identified within each case study as well as all artifacts provided with the case studies to perform the work in the Performance Work Statement. And as Josh previously discussed each offeror was sent an advisory notification with their respective confidence rating and given the opportunity to remain or withdraw from the competition.
(CR) You then provided them with your advice as to the likelihood that they would be selected for an award or not, and advised them accordingly. I mean, personally, I think that�s a great thing for industry so that they�re not sinking costs into chasing a requirement that there�s a high unlikelihood that they would be selected for. So that was step one. Now, was that the end of step one?
(DM) Yeah, so once they received their advisory notification, they had the option to either stay in the competition or withdraw as previously stated. So then the second technical factor was a six business day remote design challenge based around a fictional but potentially real-world scenario for the creation of a pre-proof of concept design for Veterans to check into the COVID-19 vaccine clinic. The goal of the scenario provided was to conduct a single design sprint consisting of a non-evaluated scenario and test session on day one, sprint planning session on day two, dialogue with the product owner and sprint demonstration and team retrospective on day six, to present their proof of concept design and interactive live retrospective.
(CR) So step two then was the challenge. Now, that was kind of, you basically said, OK, go, you�re now in the challenge, and then they had six days to work independently, or were there different check-in points after those days where the government kind of followed the progress?
(DM) So there was the initial day one test step, which was an unevaluated step, and then there was the second day�s step, which they were able to dialogue with the product owner. And then, on the sixth day, then they came back and presented their proof of concept design and the interactive live retrospective. And the successful offerors conducted user research by engaging with actual Veterans and other key stakeholders as part of the process. Based on your question although Veterans� experience was considered throughout the process, it was front and center in the design challenge.
(CR) Very good. I understand that now much better, having you walk me through that, and that was very complex, but it does make sense, and it has a very good common-sense evaluation approach to it. Now, so Josh, can you further discuss the remote design challenge and how it helped the evaluation team ensure contractors are able to incorporate Veteran feedback into their design?
(JC) Absolutely, and I would say that, I would start off by saying, I think of all the things we, we tried a bunch of stuff and I definitely think the remote design challenge was one of the more successful aspects of that since obviously we've spent a lot of time talking so far, about how you kind of bring the Veterans view into our work and how you let it inform the development of these processes and products that are going to be so important to their lives and their health care and their you know Financial health and all these other aspects, using a design challenge that fundamentally asked the vendor to start with a conversation with the vendor and then build something, you know, in a technically sound way based on that feedback from start to end, document the whole thing, tell us about it, and then give us the documentation was a very powerful way to do that. I think to really get into how it specifically incorporated the feedback, so as Dave said, and again there was no requirement, you know it was suggested you certainly speak to a Veteran, but I guess you could try to do it without it, but obviously a vendor that was going to do well on this was not only going to talk to a Veteran; Talk to them regularly and throughout the process and do so well. We obviously had a lot of experts on the call, it's not like that just means sitting in a lobby and just chat with people. There's obviously kind of a science behind this is as well. Using the process we did, we were allowed to, over the course of the couple of interactions that Dave described, kind of give them an opportunity to digest the challenge and talk to some Veterans, come back to us on a on the next day and speak to an actual you know a fake product owner but a real VA person that works in this area could have really spoken to some of the requirements testing assumptions and things like that which would be normal for that step of the development process that they then have the ability to take that information go off for 5 days, design a prototype of whatever they decided to design, usually a clickable prototype develop the kind of standard kind of proposed documentation process they would have use for it and then show it to us and that again really allowed us to see, again have them tie back the conversation with the Veteran and a need and a thing they wanted and how they wanted it, to how that was then reflected in a design; What assumption did they make? What tradeoffs did they make? maybe that wouldn't work but we did this and here's why, and then see the end product� some kind of low tech, so there was no coding to this required of anybody no one was actually asked to write code. It was more about designing it and there is something called the clickable prototype, which is kind of a low fidelity type thing, where they could show us the design and it's clickable but again there's no code behind it, so that was incredibly powerful to see, and again, to see that end to end process, from the conversation with the Veteran all the way through and how that was taken into consideration what they've built and the last thing I think we put a huge emphasis on this was accessibility, which I think Dave and Jeff both mentioned a little bit, but I didn't want to spend a second focusing on that. VA has an enormous population of Veterans that are accessibility challenged and since we have to provide Veterans, all Veterans services, regardless of their status, it's very important that we take the unique considerations of accessibility challenged Veterans into the work we do, and again making sure, part of that is, making sure we have vendors that do that well and also take that into consideration and again I think having that mentioned in our solicitation that's something they were asked to consider, again once they did really well on this and really took that seriously. Not only kind of the basic level checkbox of yes we did the required VA 508 checks at the end of the project, but I think you saw some vendors really take it throughout the entire process and design products almost end to end with accessibility in mind, and that was a very powerful thing and again I think something that would have been much harder to evaluate under different circumstances.
(CR) So the contract was awarded I think back in March 2021, has it been getting some traction. A lot of action on the contract to date?
(JC) I mean, I think so. We certainly are up to 6 or 7 task orders and counting and there's a few more in the pipeline. So yeah, everything's rolling along pretty well so far.
(CR) So Jeff would you be able to share with us some success stories out of those 6 or 7 task orders to date? or any user stories that you're most proud of to date? that are directly benefiting our Veterans.
(JB) Yeah, sure there's one that we're currently working on that is a Veteran check-in system that is in functionality in some ways similar to what was described as the code challenge, but a different application, but this is a mobile friendly experience that allows Veterans to complete preappointment forms at their home and then check in for their health appointments when they're at a Medical Center. The teams did extensive research with Veterans to optimize the experience and specifically to reduce the stress and provide an intuitive interaction for check in that they can do at their own convenience. So, it's currently being piloted in the St. Louis area and it's going to begin a nationwide rollout over the summer. So now at St. Louis, the Veteran when they arrive for their appointment they can check in through a text message on their mobile device and they get a link back and they can complete the check in there with a few simple prompts from their phone and then they don't have to wait in line at a kiosk or talk to a staff member they can start that check in process as soon as they're ready and it also integrates with other existing VA systems to let the staff know that the Veteran has checked in and so far we've had 2000 successful check-ins in the past month and I think that this is just a good demonstration of how we can take this type of interaction that's common in the private sector to make a small improvement to the Veteran experience with VA healthcare and we can do that by piloting products and making sure that they received well before we make a big commitment to that solution as an organization.
(CR): Well, that's great. So Veterans that are listening out there can expect to hopefully see this being rolled out nationwide around the summer timeframe and it will help with their check-in experience. I assume you know, we often talk about user centered design and having our user base test these things of that nature, after you've rolled it out nationwide. I'm sure you also have surveys and other ways of tracking satisfaction with the overall Veteran population as well, right?
(JB): Yeah, absolutely. We follow-up with all of our products in this way. It sort of gets out of that traditional you know development and then sustainment model. You know, when we build a product, we say if you build it, you own it. You know? So, we iterate on our products continuously once they're live and we have application monitoring where we are tracking the data, we're tracking user satisfaction. We are continuing to test that product after it's live in production and continue to sort of iterate and to build additional new features in response to the changing Veteran needs or other changing circumstances. Each product that we own has a life of its own and an existing user base who we want to ensure we�re continuously checking in with and make sure they�re still satisfied with that experience.
(CR): Well, that's great! So, I think our time is almost up here for our session of TAC Talks. So I wanted to thank Jeff, and Dave, and Josh for coming on to TAC Talks season 3 with me today. I know I found this super informative, this discussion on the Cedar Contract and all the cutting-edge acquisition approaches and techniques that you used on this contract to deliver cutting edge technology to our Veterans. So I think it's a great success story and hopefully we'll see more of this kind of innovation with this kind of niche contractor capabilities being deployed here throughout the VA and other agencies. So thank you very much guys for joining me today.

[Music]
(CR): As always, we must remind you the Department of Veterans Affairs does not endorse or officially sanction any entities that may be discussed in this podcast, nor any media, products, or services that they may be providing. We thank you for listening to this episode of TAC Talks and hope you found it helpful as well as enjoyable. You may direct any questions or feedback to me, Chuck Ross at [e-mail]. And remember, if you are passionate about government acquisition; you are a continuous learner and enjoy fruitful dialogue then keep tuning into TAC Talks.