Number 6 OCO FACT SHEET March 2001

Evaluating Programs for Students with Significant Disabilities in Postsecondary Settings

This fact sheet is designed to provide information to teachers and personnel who are interested in evaluating programs for students with significant disabilities, ages 18-21, located within alternative settings such as, community colleges, four year universities, and community facilities.

What is program evaluation?

As school systems plan for and develop alternative programs for students with significant disabilities, ages 18-21, on college campuses or in the community, it is important to incorporate program evaluation into the overall design. Program evaluation is generally used to judge a program’s success. Program evaluation will assist you: (a) document the success of a program; (b) determine student, family, and teacher satisfaction with the program; (c) identify how and when a program should be expanded or reconfigured; and (d) convince school boards that these efforts not only promote inclusion in age-appropriate environments but are also cost-effective. Dooley (2001) identified five stages of program evaluation which are discussed in this fact sheet including:

• Identify if there is a need for the program and the goals of the program
• Determine how the program will be monitored once it starts
• Determine the impact the program has on student outcomes
• Determine the program’s success by its costs
• Judge how others use the evaluation results you’ve collected

Needs Assessment

Many people begin their program planning with a needs assessment. It is most helpful if you have a small advisory committee to complete the needs assessment. OCO has developed a Needs Assessment Guide to help you identify members for your advisory committee and to begin your needs assessment process. This document can be found on our website under OCO Resources. In addition to having school personnel, students, and parents as part of the advisory committee, remember to include representatives from outside organizations such as college personnel, adult service providers, and employers.

Here are some questions that a committee should consider :

• How will this program differ for students ages18-21 than the program(s) currently available in the high schools?
• Which students should be targeted for this program? (e.g. students who will exit with a certificate, those with previous experience in unpaid employment sites).
• How will families and students find out about the program?
• Where will the program be located and how will it be staffed?

Identify Clear, Measurable Goals

Once you have completed a needs assessment, it is important to establish clear, measurable goals before you pilot the program. If you develop clear goals for the program, you will be able to determine if you are meeting the needs of students and accomplishing the intended goals of the program. Here are some examples:

"Students in the program will participate in community-based employment experiences"

"Students in the program will participate in recreational activities on the college campus".

Program Monitoring: Is the Program Operating as We Planned?

Whether you have an established program or are piloting a program, you need to determine if the program is operating as planned. This involves keeping track of the activities you do in the program and determining if you are providing the services and activities that are targeted in your goals. You may also want to keep track of activities staff undertake, the hours staff work, and the types of interagency efforts that are made. This information will help you determine at the end of the year if the program is operating as you originally planned. This information will also help you determine if you need to change the goals of the program, add more staff, or identify more college or community linkages.

Keeping track of this information can also help you determine if you have met your goals. For example, if one goal was to have all students participate in community-based employment sites, document how many students were employed (paid, unpaid) in the community. Have you met your goal? If yes, that’s great! If not, do you need to change the goal(s) of your program? Do you need extra personnel to develop job sites in order to reach your goal? Do you need to work with the high school teachers to make sure students have unpaid employment experiences before entering your program? Reviewing your activities each year and determining if you have met the goals you established for the program can help you monitor your successes and needs, and prioritize the goals of your program.

Program Impact: How Does This Program Impact Student Outcomes?

You probably collect lots of information on students in your program that you can use in program evaluation. The key is to develop a system that helps you track the information on an on-going basis and allows you to compile it when needed. Creating a database or organizing a file for each student may make it easier for you to input and retrieve this information. You will also have to consider who is going to keep track of the information and a method for making sure the information is collected on an ongoing basis if several people collect data. Here are some examples of the type of information you want to collect:

• How many hours do students work?

• Are work experiences paid/unpaid?

• How much support do the students need?

• Have functional skill levels increased?

Develop a System to Monitor Satisfaction

In addition to identifying students’ outcomes, it is important to consider other benefits and limitations of the program. Most program evaluations include information from a variety of people, including students, their families, community personnel, employers, college instructors, and adult service providers. In general, you are looking at their satisfaction with the program, what they would like to see changed, and other issues that should be considered during the next year of the program. You may want to develop a short questionnaire or ask participants in small groups the following questions:

• What do you like about the program?
• What would you change in program?
• Would you recommend this program to others? (Why or why not)
• What do you perceive as the barriers to achieving the goals of the program?

Cost-benefit: Are We Getting our Money’s Worth?

At some point you may need to expand the program, request additional staff or funds for materials, or justify continuation of the program. One way to approach this is to determine if the program is cost-efficient to operate. Start by simply documenting the type and the amount of resources used in your program for the year. Include staff salaries, monies for instructional materials and equipment, rental space, and transportation costs. Identifying the cost per pupil will most likely require expertise from administrators or researchers within your school system so you may want to include them on the advisory committee from the start. This is an issue that should ideally be discussed during your needs assessment. There may be someone in your school system that is familiar with cost-benefit analyses and be able to undertake this task for you. However, you may still want to keep track of the following information:

• How much time do you and your staff spend on program-related tasks such as planning, student support, and instruction?
• How much time do you and your staff spend contacting parents, employers, and college instructors?
• How much of your own personal resources have you used to support the program? (e.g. cell phones, pagers, car, travel expenses)
• How much of your budget did you spend on materials?
• How much of your budget is contributed by the college or community agencies?

Using the information from Your Program Evaluation Efforts

Obviously, including program evaluation strategies within your program takes time and talent! You want to make sure the information that you gathered on the program will be used to document the successes of the program and to refine or expand program efforts.

If you have an advisory committee, you already have a group of people who are invested and interested in the outcomes of the program. Have the advisory committee review the information and determine what (if any) changes should be made to the goals or operation of the program. Your program evaluation results should also be shared with key school administrators and other stakeholders (e.g. parent groups, funding agencies, etc.) to demonstrate the successes of the program. Document not only the goals and objectives that were achieved, but also the ones that were not accomplished and why. Providing school administrators and college personnel with both positive and negative feedback can provide then with a more realistic picture how a program actually operates.

Remember to provide feedback to college instructors, employers, students and parents. They can use this information to make improvements, and it communicates to them that their input is important, which fosters positive future working relationships. Some teachers in programs on college campuses have also publicized their efforts features in newspaper articles and local newscasts, during in-service workshops for high school and college personnel, and in presentations to parent groups, student organizations, and professional organizations (e.g., TASH, Council for Exceptional Children).

It will be important to keep track of how and why program evaluation results are used. When new programs or ventures are undertaken in education, it is important to identify why these efforts work, whom they serve, and if this is the most efficient way to use resources. Providing administrators and policy makers with this information is vital in securing more resources and staff for alternative efforts for students with significant disabilities during their final school years.

References

Dooley, D. (2001). Social research methods (4^th ed.). Upper Saddle River, NJ: Prentice Hall.

This OCO Fact Sheet was developed by Debra Neubert and Vanessa Redd.

For more information please write to:

Sherril Moon & Debra Neubert,

Co-Principal Investigators

Meg Grigal, Project Director

On-Campus Outreach, Department of Special Education, 1308 Benjamin Building,

University of Maryland,

College Park, MD 20742

Visit our website: www.education,umd.edu/oco

Email us at ocomail@umail.umd.edu.

On-Campus Outreach is funded by the

US Department of Education,

Office of Special Education Programs,

Grant # H324 R990032

Copyright „ 2001

On-Campus Outreach,

Department of Special Education,

University of Maryland