Imagine you are assigned the task of evaluating a new PB process in North America.

What questions would you want to answer? What tools and instruments would you need? What challenges would you expect to face in the field? Luckily for you, you don’t have to completely reinvent the wheel. While new challenges and questions are always arising in the field, many PB evaluation teams in North America want to share their experiences and tools.

At the 3rd International Conference on Participatory Budgeting in 2014, PBP and Public Agenda organized an “Evaluation Challenges” workshop in which researchers, practitioners, academics, and elected officials raised their questions about PB evaluation.

The expert panel

Four professional researchers engaged directly and indirectly in PB evaluation spearheaded the session:

Thea Crum – Great Cities Institute, University of Illinois-Chicago.

Doug Hall – National Priorities Project

Erin Markman – Community Development Project at the Urban Justice Center

Sonya Reynolds – New York Civic Engagement Table

The insights

This session generated so much interest that many more questions were raised than could be answered by the panel.

People in the room were interested in learning not only about how to conduct effective evaluation of PB, but also about how to:

• Connect PB research with larger civic engagement data collection projects in the US
• Find data that will convince elected officials to implement PB
• Involve participants and local residents in the evaluation process

Our four panelists shared what they have experienced as the key challenges for research teams evaluating PB:

• Scale of PB. When there are huge numbers of paper surveys, someone has to manually enter that data. What are alternative research designs that could make the process more manageable, particularly as PB expands in cities like NYC?
• Baseline data. What is an appropriate baseline against which to measure participation in PB, so that we can measure the extent to which PB is expanding civic and political participation?
• Relationships with elected officials and their staff. How do you get buy-in from elected officials and build strong relationships? Research needs to be embedded in the PB process from the beginning and buy-in can make evaluation easier from the get-go.
• Making research participatory. How can PB participants and the broader community be empowered to use data to answer their own questions and advance their own interests?
• Research fatigue. How do you keep PB participants engaged with the research process?
• Key audiences. What data is relevant to whom? Whose questions are being asked?
• IRB approval. For researchers associated with a university, their research plan and instruments must meet the requirements of the university’s research ethics committee.

Next steps

Thanks to the collaboration between PB researchers and organizations like PBP, many of the lessons learned in the field can be passed on to new research teams, even if new questions continue to arise and some remain unanswered.

PBP has developed standardized instruments with input from researchers in NYC, Chicago and Vallejo. We are also developing a research framework that can help new researchers plan a successful and comprehensive evaluation effort.

The North American PB Research Board will continue to support local research teams and develop the infrastructure for effective collaboration among researchers.

Subscribe to our newsletter and keep checking our blog for more resources and updates regarding PB evaluation.

If you have any questions about this conference session or about the tools being developed, please contact PBP Research Associate Madeleine Pape at madeleine@participatorybudgeting.org