One of my favorite parts of my job is that I am allowed to fail. As you may know from reading some of the other posts on our blog, one of the things that make our work unique is that we are tasked with investigating problems and trying out new and different ways of tackling those problems. The concept of “trying things out” and “failing” has been on my mind quite a bit lately as at least half of my current projects are approaching their pilot phase -- which means that within the next few months, I’ll have a pretty good idea of what I did well and what I failed to think about or plan for.
Some people reading this might be thinking -- why don’t we just spend more time at the beginning planning really well for the program or initiative we’re going to launch to make sure it works out? Isn’t starting over a waste of time? Although it might not always seem obvious, there is a special place in the world for pilots, and they are especially helpful in organizations that don’t have a large number of expendable resources required to take on big financial or public-facing risks. Whether you’re a curious resident or a fellow local government worker, understanding the value in not only trying new ideas but also testing them out, is important.
‘Piloting’ is what we call the period of “testing out” an idea, a program, or a new service, usually on a small scale with relatively low impact before we launch it for use by a wider audience (otherwise known as scaling up). We can have many different reasons for wanting to try an idea out before fully implementing it -- maybe we want to see if a new idea is worth the money it will eventually cost to fully implement it; maybe we want to make sure a new program will impact people as positively as we think it will before offering it to more people, or maybe there are a lot of unknowns in a new program and we’d like to allow space for those unknowns to present themselves in a low-stakes environment. If you’ve ever been at a Wegmans and stopped at one of the live cooking booths to sample a product, you’re piloting the new food -- that is, you’re spending zero dollars making sure that you like the new food before you invest in it any further.
While piloting allows some flexibility in the design of a new idea, program, or service and we may welcome failure in this stage, it must be done with a high level of care and attention. Before entering into a pilot, we need to know:
How we will ensure that we have enough information to make an informed decision at the end of the pilot;
This includes decisions like how long the pilot needs to run for and what key milestones we will need to reach in order to gather information on them.
What our objectives are for the new idea, program, or service;
This question helps us narrow down why we are conducting a pilot and what we need to pay attention to in order to decide whether the idea, program, or service is worth investing in or expanding. If the pilot shows us that the fully scaled-up program is unlikely to achieve what we hoped it would, that’s a good indicator that we should cut ties before spending any more resources.
What our indicators of success or failure are;
Before launching a pilot, we need to decide how we will know if we succeeded or failed. This helps us tune in to specific pieces of information that can help us understand whether we met our benchmark of success or if we fell short.
What the results of the pilot will mean;
Depending on how well the new idea, program, or service performed, will we continue to scale it up anyway, or if things don’t go as planned, will we scrap the idea as a whole? What information will we use to improve on our idea and how will we go about improving it? Being clear about what the results of the pilot will mean is important for setting the stage for evaluation and decision-making regarding the longevity of the initiative later on.
These considerations are important to think about whether you are someone who works directly on implementing policy or programs at the government level or if you are a city resident who is curious about why local leaders make the decisions that they do. Regardless of one’s proximity to this work, having clear ways of measuring the success of our work helps make sure that local government stays accountable to the public in making well-informed decisions.
Below are a couple of examples of pilot programs that our office has run in the past as well as one that we are preparing to launch now:
Tenant-Owner Proactive (TOP) Pilot: Launched in 2017, the TOP pilot-tested a comprehensive set of approaches centered around the goal of improving housing quality in Syracuse neighborhoods. The key components of the program (proactive code enforcement, tenant support, and landlord resources) all seem like no-brainers in the realm of city investments, but the six-week pilot provided an opportunity to test whether these strategies were as impactful as the team thought and if they were, how the city could make them even more impactful in the future. The criteria the pilot was evaluated on include:
The number and percentage of new violations that result from proactive enforcement;
The number and percentage of violations within the catchment territory that are resolved by the comply-by date;
Number of communications with tenants or landlords; and
Customer service scores.
The pilot resulted in a 229% increase in housing violations that were proactively spotted by code inspectors and improved compliance of code violations by over 18%. Piloting these strategies allowed the city to be more confident in the efficacy of the program and in the investment it would take to expand the program.
Illegal Setouts Behavioral Insights Trial: In the spring of 2021, the City will launch a trial to test the efficacy of proactive and behaviorally-informed communication on the rate of illegal setouts in the City of Syracuse. This means that the City will be investing a small amount of money to test whether we should continue to invest in the following techniques that we predict will help residents follow the rules of setting out the trash for pickup:
Behaviorally-informed messaging: This means taking an understanding of what impacts human behavior and using it to make messages more meaningful.
Proactive communication: Providing information and resources to community members in order to prevent or encourage certain behavior, rather than waiting to educate community members after a rule has been broken.
Accessible communication: Offering public communication in multiple languages so that community members who do not speak English or speak English as a second language have the same opportunity to access information as English-speaking community members do.
In order to test whether this combination of strategies works in reducing illegal setouts (when someone sets their trash out at the wrong time or sets out something out for pickup that is against the rules), we are compiling what is called “baseline data.” Baseline data is data that gives us a picture of how things exist right now without any of the strategies we will be testing. In this case, our baseline data is the number and location of illegal trash setouts in the City of Syracuse. In a few weeks, we will send out postcards to city residents outlining the rules for setting out trash in the city. After we do that, we will keep an eye on the number and location of illegal setouts over time, compare that with our baseline data, and see whether the postcards seemed to have an impact on residents’ behavior. After we know that, we can decide whether these strategies are ones that work and are ones that we should invest further in not only for preventing illegal trash setouts but with other city services as well.
Piloting new ideas, or testing them out before fully investing in them, is a valuable tool that can be used across many industries - in fact, we even pilot many things like new foods on a regular basis in our personal lives. For local government, piloting new ideas can help us make responsible and well-informed investments in new programs, and even helps encourage innovation and (responsible) risk-taking. When we have the ability to test new, out-of-the-box ideas on a small scale where the impact on residents and on the budget will be low, we are able to quickly turn a bold idea into a living thing, see if it works, and learn from it. If you are a resident, I hope this gives you some ideas on what questions to ask your elected officials the next time they announce a pilot of a new idea, policy, or program. And if you are a fellow local government employee, I hope this gives you a glimpse into what types of ideas we test and how we do it. We encourage anyone who is curious about what we do and how we do it to reach out by email at ndiaz@syrgov.net or find one of our staff members on the “Our Team” page.