Course Objectives Students will develop skills in: 1. selecting and using appropriate methods for evaluation of interventions and program processes and outcomes; 2. applying knowledge of human behavior and the social environment, person-in-environment, and other multidisciplinary theoretical frameworks in the evaluation of processes and outcomes; 3. demonstrating how to critically analyze, monitor, and evaluate intervention and program processes and outcomes; 4. applying evaluation findings to improve practice effectiveness at the micro, mezzo, and macro levels. Required Text(s) Grinnell, R. M., Gabor, P. A., & Unrau, Y. A. (2016). Program evaluation for social workers: Foundations of evidence-based programs (7th Ed.). New York: Oxford. Locke, L. F., Silverman, S. J., & Spirduso, W. W. (Eds.). (2010). Reading and understanding research (3rd Ed.). Thousand Oaks, CA: Sage. Grading ASSIGNMENT PERCENTAGE OF TOTAL GRADE SUBMISSION DATE Common Assignment: Research Proposal* 40% Dec 4th Required Assignment: Oral or written presentation of research findings 40% Dec 4th/11th Other: participation, and other assignments (e.g., discussion board, quizzes, exercises, etc.) 20% *See Appendix A for common assignment and/or grading rubric COURSE OUTLINE Module 1 Overview of the Research Process Module Topics 1. Review of concepts and methods of research 2. Introduction to evaluation and intervention research 3. Importance of evidence-based practice Readings Cheung, M., Ma, A. K., Thyer, B. A., & Webb, A. E. (2015). Research-practice integration in real practice settings: Issues and suggestions. Research on Social Work Practice, 25(4), 523-530. Drisko, J. W., & Grady, M. D. (2015). Evidence-based practice in social work: A contemporary perspective. Clinical Social Work Journal, 43(3), 274-282. doi:10.1007/s10615-015-0548-z Module 2 Overview of Intervention Research Module Topics 1. Definition of intervention research 2. Overview of intervention research 3. Manualized evidence-based practice 4. Common factors Cabassa, L. J. (2016). Implementation science: Why it matters for the future of social work. Journal of Social Work, 52(S1), 538-550. doi.org/10.1080/10437797.2016.1174648 Fraser, M. W., & Galinsky, M. J. (2010). Steps in intervention research: Designing and developing social programs. Research on Social Work Practice, 20(5), 459-466. doi/pdf/10.1177/1049731509358424 Goldstein, N. E. S., Kemp, K. A., Leff, S. S., & Lochman, J. E. (2012). Guidelines for adapting manualized interventions for new target populations: A step-wise approach using anger management as a model. Clinical Psychology, 19(4), 385-401. doi:10.1111/cpsp.12011 Module 3 Designing and Conducting Intervention Research Module Topics 1. Designing and refining an intervention 2. Theory of change 3. Preparing a logic model 4. Conducting an intervention research study 5. Testing efficacy 6. Testing effectiveness in practice settings Fraser, M. W., & Galinsky, M. J. (2010). Steps in intervention research: Designing and developing social programs. Research on Social Work Practice, 20(5), 459-466. Rodriquez, M.N, Baumann, A.A., & Schwartz, A,L. (2011). Cultural adaptation of an evidence-based intervention: From theory to practice in a Latino/a community context. American Journal of Community Psychology, 47, 170-186. Module 4 Overview of Program Evaluation Module Topics 1. Definition of program evaluation 2. The importance of program evaluation 3. Ethical issues in program evaluation Readings Grinnell, Gabor, Unrau Chapter 1. Introduction Chapter 2. Approaches and types of evaluation Chapter 3. The process Chapter 5. Ethics Kerns, S. E. U., Pullman, M. D., Negrete, A., Uomoto, J. A., Berliner, L., Shogren, D., Silverman, E., & Putnam, B. (2016). Development and implementation of a child welfare workforce strategy to build a trauma-informed system of support for foster care. Child Maltreatment, 21(2), 135-156. doi: 10.1177/1077559516633307 Morris, D. A., & Galicia-Castillo, M. (2017). Caring about residents’ experiences and symptoms (CARES) programs: A model of palliative care consolation in the nursing home. American Journal of Hospice & Palliative Medicine, 34(5), 446-469. doi: 10.1177/1049909116641606 Agency for Healthcare Research and Quality. (2012). Chapter 3: Getting involved in the research process. In The effective health care program stakeholder guide. Rockville, MD: AHRQ. Content last reviewed April 2018. https://archive.ahrq.gov/research/findings/evidence-based-reports/stakeholderguide/chapter3.html Module 5 Process and Outcome Program Evaluation Module Topics 1. Steps in the evaluation process 2. Developing logic models for program evaluation 3. Process and outcome evaluations 4. Client satisfaction studies Readings Grinnell, Gabor, Unrau Chapter 11. Process evaluations Chapter 12. Outcome evaluations Darnell, D., Dunn, C., Atkins, D., Ingraham, L., & Zatzick, D. (2016). A randomized evaluation of motivational interviewing training for mandated implementation of alcohol screening and brief intervention in trauma centers. Journal of Substance Abuse Treatment, 60, 36-44. doi:10.1016/j.jsat.2015.05.010 Developing an effective evaluation report: Setting the course for effective program evaluation. (2013). Atlanta, GA: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health, Division of Nutrition, Physical Activity and Obesity. https://www.cdc.gov/eval/materials/developing-an-effective-evaluation-report_tag508.pdf Fraser, M. W., & Wu, S. (2016). Measures of consumer satisfaction in social welfare and behavioral health: A systematic review. Research on Social Work Practice, 26(7), 762-776. doi:10.1177/1049731514564990 Huisamen, A., & Weyers, M. (2014). Do social workers really make a difference? Measuring client satisfaction in an occupational setting. Social Work, 50(1), 1-18. https://dx.doi.org/10.15270/50-1-13 Module 6 Selecting and Evaluating Outcome Measures Module Topics 1. Selecting outcome measures 2. Reliability and validity 3. Culturally appropriate measures Readings Grinnell, Gabor and Unrau Part V: Evaluation Toolkit Tool H. Data collection and sampling procedure Tool L. Measuring variables Tool M. Measuring Instruments Church, A. (2010). Measurement issues in cross-cultural research. In G. Walford, E. Tucker, & M. Viswanathan (Eds.), The SAGE handbook of measurement (pp. 151-176). London: SAGE Publications Ltd. doi: 10.4135/9781446268230 [Access via Fordham Library’s Sage Research Methods Online database] Module 7 Reading and Understanding Quantitative Research – Descriptive Statistics Module Topics 1. Understanding descriptive statistics 2. Critically reading and evaluating published reports of quantitative research Readings Bates, L., Luster, T. and Vandenbelt, M. (2003), Factors Related to Social Competence in Elementary School Among Children of Adolescent Mothers. Social Development, 12: 107-124. Gelman, A. (2011). Ethics and statistics. Chance 24(4), 51-54. http://www.stat.columbia.edu/~gelman/research/published/ChanceEthics1.pdf Lane, D. (n.d.). Online statistics education: A multimedia course of study. Houston, TX: Rice University. Access at http://onlinestatbook.com/ Locke, Silerman, and Spirduso Chapter 8: Explaining as a tool for learning to read reports Chapter 9: Reading reports of quantitative research-critically: Things to notice and questions to ask Oleś, M. (2016). Dimensions of Identity and Subjective Quality of Life in Adolescents. Social Indicators Research, 126(3), 1401-1419. Satre, D. D., & Knight, B. G. (2001). Alcohol expectancies and their relationship to alcohol use: Age and sex differences. Aging & Mental Health, 5(1), 73-83. Module 8 Reading and Understanding Quantitative Research – Inferential Statistics Module Topics 1. Understanding inferential statistics 2. Critically reading and evaluating published reports of quantitative research Readings Bates, L., Luster, T. and Vandenbelt, M. (2003), Factors Related to Social Competence in Elementary School Among Children of Adolescent Mothers. Social Development, 12: 107-124. Gelman, A. (2011). Ethics and statistics. Chance 24(4), 51-54. http://www.stat.columbia.edu/~gelman/research/published/ChanceEthics1.pdf Lane, D. (n.d.). Online statistics education: A multimedia course of study. Houston, TX: Rice University. Access at http://onlinestatbook.com/ Locke, Silerman, and Spirduso Chapter 8: Explaining as a tool for learning to read reports Chapter 9: Reading reports of quantitative research-critically: Things to notice and questions to ask Oleś, M. (2016). Dimensions of Identity and Subjective Quality of Life in Adolescents. Social Indicators Research, 126(3), 1401-1419. Satre, D. D., & Knight, B. G. (2001). Alcohol expectancies and their relationship to alcohol use: Age and sex differences. Aging & Mental Health, 5(1), 73-83 Module 9 Reading and Understanding Qualitative Research Module Topics 1. Understanding qualitative findings 2. Critically reading and evaluating published reports of qualitative research Readings Chan, C., & Holosko, M. J. (2017). The utilization of social media for youth outreach engagement: A case study. Qualitative Social Work: Research And Practice, 16(5), 680-697. Meyers, A. (2017). Lifting the veil: The lived experience of sibling abuse. Qualitative Social Work, 16(3), 333-350. Locke, Silverman, and Spirduso. Reading and understanding research (3rd ed.). Chapter 11: Staying organized when reading a qualitative report Chapter 12: Reading reports of qualitative research—critically: Things the reader should expect Paul, E., & Hayes, K. (2002). The casualties of ‘casual’ sex: A qualitative exploration of the phenomenology of college students’ hookups. Journal Of Social And Personal Relationships, 19(5), 639-661. Module 10 Implementation Science Module Topics 1. What is implementation science 2. Key characteristics of implementation science 3. how implementation science impacts the field Readings Cabassa, L.J. (2016). Implementation science: Why it matters for the future of social work. Journal of Social Work, 52 (S1), 538-550 Module 11 Dissemination: Oral and Written Presentation Module Topics 1. Presentation to community stakeholders 2. Presentation at a professional meeting Readings Developing an effective evaluation report: Setting the course for effective program evaluation. (2013). Atlanta, GA: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health, Division of Nutrition, Physical Activity and Obesity. https://www.cdc.gov/eval/materials/developing-an-effective-evaluation-report_tag508.pdf Appendix A. Common Assignment. Research Proposal Students will apply the concepts of this course by developing a research proposal for an intervention study or a program evaluation. The proposal will include the sections listed below. (8-10 pages) Program that will be evaluated is CAMBA.org; non profit Evaluation type: change or outcome HYPOTHESIS: a lack of staff impedes providing good customer service? 1. Introduction. a. Statement of the problem that the particular program or intervention is addressing. b. Description of the program or intervention to be evaluated, including a statement of the underlying explicit or implicit theory of change for the program or intervention (i.e., the mechanisms through which the intervention is expected to produce its intended outcomes). c. Description of the agency environment: including the receptivity of the administration staff, and other stakeholders d. Client and organizational factors that influence the evaluation e. Logic model 2. Literature Review (8-10 articles, summarized, listed,cited, objective,N, sampling type; probability or no probability, design, variables measured, measure instrument used, finding, one page ) a. Critical review of the literature on the current evidence for the program or intervention effectiveness b. Gaps in the literature 3. Study aims a. Aims and objectives b. Hypotheses 4. Methods a. Study Design i. Type of study (e.g., process, outcome, client satisfaction study) ii. Study design (e.g., survey, pre-experimental, quasi-experimental, experimental) b. Sample i. Study population ii. Eligibility criteria iii. Sampling strategy iv. Recruitment strategy c. Measurement i. State the independent, dependent, and other variables ii. Identify and describe at least one measure of a study outcome iii. If relevant: specify how program fidelity will be assessed d. Data Collection Plan: what steps will be taken and by whom to collect data and provide informed consent e. Data analysis strategy References