Transcript
Transcript: Innovate on Demand, Episode 19: A Culture of Experimentation
Todd
I'm Todd Lyons.
Natalie
I'm Natalie Crandall.
Pierre-Olivier
I'm Pierre-Olivier Bedard.
Sarah
And I'm Sarah Chan.
Todd
And this is the Innovate on Demand podcast.
To paraphrase Stefan Thompke writing in Havard Business Review, to innovate successfully, organizations must make experimentation an integral part of business. Why? Because it's the approach that's allowed startups to derail market leaders. But if it's so vital, why aren't more organizations doing it? Culture, he says. It's not lack of tools and tech that hold organizations back so much as mindset. Nurturing curiosity, empowering every employee to spearhead change, and embracing failure can seem like waste and risk. How are we faring in the federal public service?
Todd
Welcome.
Sarah & Pierre-Olivier
Thank you. Thank you.
Natalie
Thanks for being with us today. So you guys are joining us today from the GC experimentation team. Tell us a little bit about what it's like to work on that kind of a team, that kind of an environment, and have a mandate to experiment in the federal government.
Sarah
It's great. We've been around for just about three years, in our current form and team structure. We're a very small team, but very diverse within that. So sitting here beside me is my colleague Pierre-Olivier who brings deep expertise in experimental design and public admin contexts. I do not have a PhD in that, but [I have] 10 years in government, most of that in the policy innovation community. [I'm] very passionate about building community [and] new ways of working. How do we work better? How do we build evidence into what we do? Whatever you're doing, there's always a need to manage things well and let people know what's happening and hear what they're working on, make connections, share examples. So yeah, we really enjoy our work and are happy to talk about it.
Pierre-Olivier
Yeah. And I feel like given that we're working on experimentation, there's some connection to other business lines of innovation and these other things. So I feel like the way we work also reflects that broader engagement with these ideas. So it would be a bit strange or unusual to be advocates for experimentation and innovation and not embed that in our own ways of working. So I feel like we're trying to make sure that the ways of working reflect our own ideas and principles.
Sarah
We also recognize from the very beginning [that] this team literally started as 1.8 FTEs and even at the moment, at the very moment--
Nathalie That's very "Treasury Board stat", by the way.
Sarah
--we're 2.3 FTEs. We are excited to grow to about five FTEs in the coming year, but we've always been a small team. And so I think from the beginning, it was clear to us: how do we promote experimentation? It's probably not going to [be] us running all of the experiments. It tends to be departments who have the opportunities to experiment on the programs that they deliver on [and] the policies that they are developing that have the linkages to the programs to the services to Canadians. And so I think one of the best things that we've been able to build is this initiative called Experimentation Works. And so the whole idea is that we try and identify, through a selection process that we're just finishing, these promising areas of experimentation across the federal government. [We're] particularly [looking] with a view towards: do you think you might have something that is ripe for an experiment, but maybe you don't know how to fully implement that, because that's kind of [difficult]. It does require deep expertise. Or, maybe you do have some of the expertise but you could use some more. So this is what we are currently working on as part of other things we do. And it is a way that we're hoping that TBS can be that matchmaker [and] can identify experts that already exist within the federal government, especially in our science based departments [and] get them to come on board. And so we've already got well over 20 experts who have said, Yeah, I'd love to help out another department, or someone in my department in another team that wants to run an experiment, but maybe lacks some of that expertise. But they have such an interesting area and such a good potential to measure something or test something or to implement two different versions and see what works better.
Pierre-Olivier
Yeah, because we saw that there was some appetite and some demand for support for these types of projects. So departments were responsive to some extent to the previous selection, the mandate letters that we're highlighting in evidence based decision making, the need to testing new ideas, and so on. And there was a direction to deputy heads following that, so we're finding that understanding of experimentation. So a lot of departments, got on board with those ideas, but lacked the capacity to implement these things. So that's where Treasury Board started to play a role in this to guide departments, not just by saying, Here's how you should experiment and go and do it, but by building capacity and being there alongside departments to develop these things. So it's been an interesting experience, at least from my perspective because I was involved in this even before joining TBS, and I started to work in formally supporting TBS and other departments and running those types of experiments. So I could I could see the the added value for me as an expert, learning what people are doing and learning about how you can implement these types of complicated projects and for them to benefit from my advice and my guidance at a very low cost to no cost at all.
Sarah
Yeah, we essentially run Experimentation Works on no fixed budget. We don't have a budget to run it. We've designed it our heads as, Wouldn't it be great if we could find experts and they might be interested in supporting projects across government. So that's been interesting. And I think everything we've done has been from the perspective of, Okay, well, we don't have a lot of resources, and there's maybe not appetite right now to commit more than what they've given us. So, what can we do with what we have? And how can we keep signalling back to others in positions of decision making to say, Hey, look at the demand for this. Look at what it actually takes to do some of this. It's not enough to say, Thou shalt experiment. What do we mean by that? What kind of methods are we talking about? What kind of expertise do you need, what considerations need to be taken into account. Therefore, how would you practically do this? And so, another kind of driver for EW -- Experimentation Works -- was that when we started in 2017, [we said], Okay, well, so what experiments are happening in government? So you go out [and] chat with departments. And so either you find, not much, or maybe we don't yet have the networks to understand where [experiments] are happening. Or, hey, we found some great ones but they didn't really have the extra resources to document the journey along the way. We only had the resources to actually run the trial and here's our outcomes report. Here's what we tested and here's what we learned, but [we didn't document] the rich journey of what kind of approvals do you need and all of that. And so we [thought], what if we were involved in experiments from the beginning, and we brought the resources to do the storytelling and some of the documentation and some of the lessons learned as a way to broadcast that out to the system and say, This is what it looks like to do experiments. So that was also some of our motivation.
Pierre-Olivier
And so we were very public about [the fact] that we have canada.ca page, we also have a blog page that's just public facing -- open to anyone. A participant from last year talked about their journey, experts in last year talked about how they helped support the whole thing. So, [it's] very public. Again, this renewed cohort we want to be even more public by having participants register their projects on a public facing page [and] eventually report back on the results. So for us the added value of this is to generate robust evidence to inform decisions, but also to have this wider community where learning is common and there is exchange of information. People know about what's happening. I might start a project and learn that department X has done one last year, so I might inspire myself by looking at their project. So you want to foster this kind of learning and continue building this community of practice because that's what we are doing in a way -- trying to maintain this active space of passionate people [and] highly skilled people that really want to pursue this agenda.
Sarah
We also want to be the face of... I don't know... I don't have senior level approval to say it this way, but just in my own head every day I go into TBS [and think] I want to be the TBS that I wished I had when I was in a line department. Well, come on, TBS! Why don't you roll up your sleeves and get involved! And it also can be the view of a citizen towards government. Who are you in your ivory tower to ponder and think of these things, and you haven't actually experienced [them] or gotten on the ground. So I think we can also have this inner dynamic in government where in TBS, we can set [policy]. Oh, here's the new policy direction, here's the new directive, here's the new blah, blah, blah. But have we actually experienced what it is to set those up? And are we actually trying to help departments? Are we just telling them what to do, or can we also help them? And can we also be open to [changing]. Oh okay, so the way we wrote that is really confusing. Oh, you're right. We didn't think about that. And, oh yeah, this conflicts with another rule we made 10 years ago. So we're trying to bring that spirit to EW to say, how can we help you? And if actually it's [the] case [that] you're encountering some resistance in your department because some part of your department doesn't want to be open about this. Well, then we can also tell you that TBS is now saying to be open. Even if [your experiment] didn't work, share that because that's part of our new defaults, to be open by default, to share what didn't work as much as what did work.
Natalie
This is very interesting. I'm really glad that a team like yours exists. I've been very interested by the Experimentation Works platform or the program, I should say. Because we have that conversation, that dialogue in government all the time about how we need to create an environment where it's safe to fail fast and all that. But the truth is, most public servants don't really understand what that means. Because that's not the culture and the mindset we've had, or that we've been in for a very long time. So it's really nice to know that there are supporting groups and corporate enablers who can actually help make that happen in a different way. [We'd appreciate it] if you talked a little bit about some of the ways in which you guys work in a different way that are innovative and different? What will be some examples of those and how do you think that's enabled you to do better work, I guess.
Pierre-Olivier
I can speak to it a little bit, but I would say Sarah is probably the the lead expert on managing our own team. So we use things like Trello. It's a tool that you can use to structure your work and split it into different cards and tasks and have a list of how complete those tasks are, and so on. So we use that as a way to really make sure we write down everything we want to do. And we have that chunk of work that spans over two weeks. And then we have regular meetings. It's very procedural. We have meetings to set our targets, meetings to check midway if we're doing okay, and then a meeting the end to see if we reach our target or not. And then we include our manager in some of those to have a signal check. So we're really on top of our files and what the tasks are, and where we are at any kind of moment in the process.
Sarah
Yeah. We go above and beyond anything that anyone else is asking us to do, just because we want to be really logical about our work and really clear as well and have that full transparency. Pierre-Olivier and I had this moment, one day earlier this month, where we had just come out of a meeting or something about we learned some new information about projects that might be coming down the pipe for EW. We each went back to our desk and then we realized we each started updating the same information in the same document. Now, thankfully, we work on real time collaborative documents, so we actually noticed that it was happening and we laughed. But, oftentimes this happens, right? We all know [about] duplication [and] not really knowing what's happening. And so we just wanted to bring that out into the open. And also, it's a way that we self-manage. We have this high level of autonomy. A shout out to our director Nick Chesterley and DG Kalli Levesque for really supporting that. They trust us and anytime they want to know anything, we can tell them or they can check out our board. We even make lists of everything that we want to talk about with them ahead of our check-ins with them just because we recognize their time is really limited. These tools are just a way to work in the open. So if I have a thought, Oh, I think we should talk to Kalli about X, well, I just put it on the list so that if P-O goes into the card, oh, he can see that I already put that thought there. It also gives us the freedom to work from wherever, because we don't need to be sitting at our desks to know what each other are doing. We've already mapped it out, we've already decided, we've already agreed on it. And of course, it can be flexible to change. But I think that then leads into our strong commitment to work life balance, and alternative and flexible work arrangements. So we sometimes joke that you need to have 1 to 3 kids to work in our team, not because that's real, but because it just so happens that we have a very high percentage of young parents with young kids on our team. And I think it's for everyone to had the freedom to choose the locations and the times and the percentage of work that is optimal for their life right now, based on anything else you might want to do in your life.
Natalie
Having 1 to 3 small children at home, I know exactly what you're saying.
Sarah
Pierre-Olivier works maybe one to two days a week from home and I work 80%. We also work out of different locations. We're really passionate about that, because we want to enjoy our work. And we know that we can't go home at the end of the day exhausted and frustrated, because of toxic work environment, or you've taken on way too much. There's just no room for that. This has to be sustainable and it has to be enjoyable.
Natalie
This has been very informative on a personal level, because when I met your DG the first time I became instantly convinced that I would work for her one day. So this whole session is just solidifying my determination on this.
Sarah
They are looking to hire.
Todd
So, if you're listening...
Pierre-Olivier
But yeah, but it's been it's been really great to work in that environment and have this approach to our work. We can see [it], we can track it, we can evaluate it. But just on a personal level, I feel like it's been fundamental to really make sure we hit our targets, and that we really maximize the way we work. And especially on a file like this where it's really self directed. I mean, we get approval for things, Okay, let's run Experimentation Works. But there's 75 micro-tasks under there that you need to explore and figure out before you even get to that stage. So we need we need to have a system in place to capture all of this and organize all this. So it's been working great for us. And actually, Sarah is doing a lot of [promotion]. [She's] sometimes got a roadshow to explain these things and bringing it to other teams. She's really passionate about these things. And she got me convinced as well.
Natalie
So it's been a successful experiment?
Sarah
Yeah. You know, another thing. Ever since Dan Monafu and I started the team in 2017, I know I credit him for really bringing this Fail Friday ritual to our team, amongst many, many, many other things. But I think that's also a really important one, because it ensures that pretty much every week, we sit down as a team, we book a room so that we can just have that time -- a closed door session where you just spend 20 [or] 30 minutes and everyone can go around. Sometimes we just all talk openly as things come up into our heads, or each person takes a turn. And I mean, it's kind of like the good, the bad and the ugly, right? It's the what's going well, what's not, and what do you have questions about. And, I've been thinking a lot about these things and analyzing them from so many different perspectives. And I think one of the powerful things about these rituals is that it ensures that they happen, whether you feel like doing it or not. Because often, most people would probably be open to having [these discussions] -- even most managers. That's a good idea to check in as a team. But it's probably not going to happen if you just don't build it in by default. It's about defaults. So when you have it there by default, it reminds everyone, Oh, yeah, like, what am I enjoying about work this week? And what am I not enjoying? And why? And then having that touchpoint every week as a team, hopefully you'll surface things [so] if there are problems, hopefully you're starting to talk about them when they're just little problems, as opposed to big problems.
Natalie
In a conversation I was having this week, someone told me that the measure of success -- [whether] something is a been a successful experiment or a successful innovation in government -- is that it's scalable. Maybe that actually applies more to innovation, than experimentation. But what do you think about [that]? What is the measure of success? What is a successful experiment look like? We all know, we can succeed even though we fail.
Pierre-Olivier
I think ultimately, it's really about learning something from from what you've tried. I mean, if you find that You know, program X didn't deliver on the targets, then you could frame that as a failure. And some people do sometimes even the media. But that means if you learned that at a small scale, it didn't work, maybe the best course of action is not to replicate or scale up, then you focus on something else and you move on. So you learn from that. If you find that your program is delivering, as expected, or even more, then you have evidence to again move forward with something and you have confidence in what you're doing. So [the] measure of success is being able to get to that stage of learning. So another kind of intermediate indicator is making sure that the results get highlights from from senior management. So if you just have a report that's shared among colleagues, I'm sure people can learn. But if there's no formal process to bring it to the attention of people who are in a position to make these decisions, then if you didn't get to that stage then the experiment didn't deliver that much in the end. So when we talk about experimentation it's just one specific approach that supports evidence based policymaking or evidence informed policymaking. So we really want to see robust designs, robust measures. But at the end of the day, it's really about going back to that problem that you're trying to solve and informing decisions based on the evidence generated. If you got if you don't get to that stage, it's a bit pointless, I think.
Sarah
I think another indication of scale and success in experimentation is building that culture of experimentation. The culture and the mindset that supports it, but also the embedded skills and knowledge and know-how so that success would [mean] you get to a place of some form of continuous experimentation. And it's pretty clear in our call for proposal process because we have options whether you just want to come and observe and learn -- like auditing the course but not running a project. Or, you want to run a pre-experimental project because you're not ready to experiment [because] you need to gather his foundational understanding and knowledge of that intervention. Or, are you ready to experiment? So, we see the whole range. We see the departments that know that where they're at in the stage on this particular topic is pre-experimental. They say: we know we can't experiment yet. We know that we haven't yet figured out what would be the promising areas to test. We haven't talked to our users we've got to reach. So, they're pre-experimental and they know it. And then you get the groups that think they're ready to experiment, but actually they don't have any data. They've never done this before. [We realize this and respond,] I think you got to start over here, right? And then we've had the groups come in that can show us like everything they've done to date. And they [say], we've already run experiments, or we've done two to three years of exploratory research [and] talking to our users. And so they [say], We're ready. And we [respond], Yeah, we can see that. So, I think that scale and success [means] building that foundation and, and getting yourself in a position that allows you to work in this way.
Natalie
Very, very interesting. Thank you very much.
Todd
Thanks for coming in today.
Sarah
Thank you.
Todd
You've been listening to Innovate On Demand, brought to you by the Canada School of Public Service. Our music is by grapes. I'm Todd Lyons, Producer of this series. Thank you for listening.
