Since there are many different data collection methods, selecting the best method is an important step when creating an evaluation plan. However, there are no clear cut “right” methods. While our past posts have presented four data collection methods and the pros and cons of each, this blog will follow a fictional non-profit, Organization X, as they use multiple data collection methods to evaluate their programs.
Imagine that Organization X is a non-profit that helps families and children develop interpersonal skills, build emotional maturity and connection, and resolves family conflict. Organization X provides a family engagement program at their center and an afterschool program in six different schools. They are conducting a full-scale organizational evaluation of both programs to meet their funders’ requirements and report on their progress. Five different data collection methods are being used to address different aspects of their four evaluation questions:
- What are the family engagement program’s successes and challenges?
- What are the afterschool program’s successes and challenges?
- How are the programs being implemented by the facilitators?
- How are the programs meeting leadership’s expectations?
The five data collection methods used for this case study:
Data Collection Method #1: Surveys
Organization X uses surveys to gain perspective on what parents think of the family engagement program and their families’ involvement. They create an online survey that they distribute to participating parents. The online format reaches parents in a timely manner, making it easy to share their feedback with Organization X so they can draw insight from a defined target audience.
Data Collection Method #2: Focus Groups
To better understand the experiences of program facilitators, Organization X contracts an external evaluator to conduct focus groups. Three focus groups are conducted with afterschool program and family engagement program facilitators. By collecting program-specific data and comprehensive data across programs, Organization X identifies strengths and areas for improvement with the program.
Data Collection Method #3: Interviews
Organization X wants to better understand the views of their board members. Because they want to encourage the board to share honest feedback, they consult with their external evaluator on whether to use interviews or focus groups. Organization X decides that interviews with their external evaluator would be the best approach because it maintains confidentiality. Interviews also allow board members to fully describe their thoughts, experiences, and provide feedback (both positive and negative).
Data Collection Method #4: Internal Data
Organization X also wants to examine whether participating youth continue in the afterschool program across time and school. Using internal data, Organization X is able to leverage existing data they collected from parents as part of the enrollment process. Further, program facilitators maintain attendance records, another data source that can be utilized. Using this method, Organization X can determine student retention rates across time. Additionally, Organization X can use this data to group schools and identify if any of the 6 partner schools have lower attendance or retention rates.
Data Collection Method #5: Observations
Finally, Organization X is interested in understanding how facilitators and kids are learning and using skills in both the afterschool and family engagement programs. They decide to have board members collect data through planned observations. Organization X creates a list of outcomes they hope to see (e.g., emotional growth) and behaviors (e.g., kids talking about how they were feeling) for their board members to use as guideposts. They visit each site to observe and record these planned aspects of program implementation as well as kids’ real-life experiences with program content. Collecting this data provides new information regarding the success of the program.
After completing these data collection efforts, Organization X identifies successes and challenges within their programs. Using different approaches gives Organization X a broader understanding of their programs and organizational priorities while answering several specific evaluation questions.
Still have questions about data collection methods?
Contact REC! We can provide a free 30-minute consultation and discuss what data collection methods are most feasible and useful for your organization.
Sources
Bullen, P. B. (n. d.). Should I Use Interviews or Focus Groups? Tools4dev. Link: https://tools4dev.org/blog/use-interviews-focus-groups/
Evaluation Research Team. (2018). Data Collection Methods for Program Evaluation. Centers for Disease Control and Prevention. Link: https://www.cdc.gov/healthyyouth/evaluation/pdf/brief13.pdf
McLeod, S. (2015). Observation Methods. Simply Psychology. Link: ://www.simplypsychology.org/observation.html
Shtivelband, A. (2017). 4 Data Collection Methods. Research Evaluation Consulting LLC. Link: https://researchevaluationconsulting.com/4-data-collection-methods/
Shtivelband, A. (2017). 4 Data Collection Methods – Pros and Cons. Research Evaluation Consulting LLC. Link: https://researchevaluationconsulting.com/4-data-collection-methods-pros-cons/
Survey Planet. (2020). Surveys as a Research Method – When Are Surveys Used and Why Should You Use Them? Survey Planet, LLC. Link: https://blog.surveyplanet.com/when-are-surveys-an-appropriate-method-of-research
Ventiv Technology. (2022). What’s the Difference Between Internal and External Data? Ventiv. Link: https://www.ventivtech.com/blog/whats-the-difference-between-internal-and-external-data
Related Posts: