In my role as President of PIE Consulting, Inc., I work with many nonprofits on every aspect of program evaluation. Oftentimes, an organization will contact me to design a survey, and my response is always, What do you want to know? You see, a survey is not always called for, nor appropriate. The method to collect information must be driven by the type of information needed. It is absolutely critical that nonprofits determine exactly what they want to know before designing and implementing a program evaluation plan.
Through my experience, I have found that the most efficient way to answer the question, "What do you want to know?" is to complete a logic model. A logic model is a tool used by program managers, funders, and evaluators to outline and align the program population, philosophy, inputs, activities, outputs, outcomes, and measurement tools.
While there are many different templates, a logic model often looks like a program matrix with boxes and arrows, leading the reader from one component to the next. This not only develops a theory of change, but a logic model also helps determine the information needed to make programmatic improvements. Logic models encourage and allow for us to start with the end in mind to determine what we want to know.
When we complete the logic model, we might be surprised to find out that what we want to know is not best measured via a questionnaire. For example, a new client recently contacted me to design a satisfaction questionnaire for professional development sessions. After working through a logic model, they determined that merely measuring the participants' satisfaction would be missing an opportunity to measure additional knowledge and skill development.
Instead, we developed an innovative strategy using a rubric to measure knowledge and skills as components of the program. The rubrics we developed not only provided information for an external evaluation, as required by the funder, but also provided information to the instructor and feedback to the participants.
I encourage you to align your data collection tools with your data needs. For example, you would not want your questionnaire responses to determine the attendance levels over time. If you need to know attendance levels over time, you will need to use aggregate attendance sheets as a measurement tool. Below are some common data needs that my clients often determine after completing their logic model:
- # of Participants
- # of Meetings
- Development of Curriculum
- Use of Curriculum
- Program Fidelity
- Increased Skill
Tools for Collection
- Self-Report Questionnaire or Document Analysis of Records
- Attendance Sheets
- Count of Agendas or Minutes
- Confirmation of Existence
- Self-Report Questionnaire or Documented Repeated Use of Services
- Report by Users or punt of Users Observations of Use
- Report by Users or Count of Users Observations of Use
- Self-Report Questionnaire Retrospective or Self-Report Questionnaire Pre-Post or Assessment
In summary, before implementing a questionnaire, I would like for you to consider that the information you need to collect should dictate the mode in which you collect that information. We should not simply default to administering a questionnaire, because it is easy, or efficient, or what we've always done.
Tania Rempert, Ph.D., is the founder and director of Planning, Implementation and Evaluation Org. Recognized as a dynamic facilitator and leader in evaluation coaching, Dr. Rempert is at the forefront of helping organizations gather, understand and use their own research data to build organizational capacity.