Completer effectiveness is vitally important, yet an especially challenging standard to measure. The Commonwealth of Virginia does not make teacher or student performance data available to the institutions at which the teachers were trained. Accordingly, Virginia’s EPPs have had to find different and innovative ways to address completer effectiveness. To counteract this data gap, the Virginia Education Assessment Collaborative (VEAC; https://projectveac.org), a collaborative team of assessment and accreditation professionals from EPPs across Virginia, formed to begin coordinating completer data collection. The purpose of VEAC is to provide a centralized assessment structure for Virginia EPPs that standardizes and reduces the complexity of data collection for both the Virginia Department of Education (VDOE) and CAEP. VEAC administers common surveys to completers of nearly all of the 36 EPPs in Virginia.

VEAC’s annual Completer Survey was designed to satisfy the CAEP sufficiency criteria, and has the benefit of facilitating benchmarking of EPP Completer ratings to VEAC-wide data. The Completer Survey consists of 14 items aligned to the 10 Interstate Teacher Assessment and Support Consortium core teaching standards (InTASC, 2013) and the 8 Virginia Uniform Performance Standards on which inservice Virginia teachers are evaluated (VUPS, 2021). Table 1 shows the alignment of the VEAC Completer Survey items to the InTASC and VUPS standards. The Virginia Uniform Performance Standards are:

Performance Standard 1: Professional Knowledge

The teacher demonstrates an understanding of the curriculum, subject content, and the developmental needs of students by providing relevant learning experiences.

Performance Standard 2: Instructional Planning

The teacher plans using the Virginia Standards of Learning, the school’s curriculum, student data, and engaging and research-based strategies and resources to meet the needs of all students.

Performance Standard 3:  Instructional Delivery

The teacher uses a variety of research-based instructional strategies appropriate for the content area to engage students in active learning, to promote key skills, and to meet individual learning needs.

Performance Standard 4:  Assessment of/for Student Learning

The teacher systematically gathers, analyzes, and uses all relevant data to measure student progress, guide instructional content and delivery methods, and provide timely feedback to students, parents/caregivers, and other educators, as needed.

Performance Standard 5:  Learning Environment

The teacher uses resources, routines, and procedures to provide a respectful, positive, safe, student-centered environment that is conducive to learning.

Performance Standard 6:  Culturally Responsive Teaching and Equitable Practices (New in 2022; will be mapped to VEAC surveys in 2022-23)

The teacher demonstrates a commitment to equity and provides instruction and classroom strategies that result in culturally inclusive and responsive learning environments and academic achievement for all students.

Performance Standard 7: Professionalism

The teacher demonstrates a commitment to professional ethics, collaborates and communicates appropriately, and takes responsibility for personal professional growth that results in the enhancement of student learning.

Performance Standard 8:  Student Academic Progress

The work of the teacher results in acceptable, measurable, and appropriate student academic progress.

Table 1. VEAC survey items and VUPS / InTASC alignment

Survey results from 2021-22 JMU program completers indicated that JMU graduates who are now in their first three years of teaching feel generally well-prepared to make a positive impact on their students. Completers respond to all items on a scale ranging from 1 (Unacceptable) to 4 (Exemplary), following the general prompt, “Based on your preparation at James Madison University, how would you rate your performance in each of these teaching areas:.” The expected performance for each item is an average score of 3, or a rating of “Proficient.” Of the 14 items on the VEAC Completer Survey, JMU completers scored at or above an average of 3/Proficient on all, and scored at or above the VEAC mean on all but one. The one item on which JMU completers rated themselves slightly below the VEAC-wide average was, “Uses assessment results to inform and adjust practice” (JMU M = 3.17, VEAC M = 3.23, not statistically significant). One item particularly addresses completers’ impact on student academic progress: “Work results in acceptable, measurable, and appropriate student academic progress.” The JMU average on that item was 3.24, equal to the VEAC-wide average, suggesting a favorable impact of JMU completers on their students. Scores for all VEAC Completer Survey items for JMU and VEAC-wide are reported in Table 2.

Table 2. 2021-22 VEAC Completer Survey Results for JMU and VEAC (Completer Effectiveness)

Of course, the VEAC surveys have limitations in addressing student effectiveness. Most significantly, the surveys offer an indirect measure of perceptions around effectiveness without using true empirical assessment data. In addition, the sample of respondents to the VEAC surveys is limited, which impacts representativeness and generalizability. Accordingly, to buttress information about completer impact, JMU is pursuing a variety of different avenues to gather additional data. In Spring 2023, we will pilot a VEAC-based focus-group protocol with new teachers (i.e., recent JMU graduates) to gather more information about their impact.

We are also exploring new avenues for acquiring teacher evaluation data from recent graduates who are now inservice teachers. First, we have partnered with JMU’s chapter of Kappa Delta Pi (KDP), the international honors society in education, to communicate with graduating preservice or recently-graduated inservice teachers. KDP operates a fully student-run conference each spring for JMU preservice teachers nearing graduation. Presenters at the conference include recent JMU graduates as well as teachers and staff from local school divisions. The JMU Director of Assessment, Accreditation, and Accountability and the Associate Dean of Internal Supports and Accountability started attending the conference in 2023 to support the students and to establish friendly contact with recent graduates and soon-to-be graduates. Attending the KDP conference in Spring 2023 yielded contact information for five recent graduates who are now inservice teachers willing to share more information about their teaching experience and impact, as well as contacts for eight current teacher candidates who will graduate soon. We plan to work with these students in Spring 2023 to obtain more information about their teaching impact and evaluations.

We are pursuing a second avenue for obtaining teacher evaluation data for our recent graduates through the Mid-Valley Consortium (MVC), our consortium of four locals EPPs (JMU, Bridgewater College, Eastern Mennonite University, and Mary Baldwin University) and seven local school divisions (Augusta County, Harrisonburg City, Page County, Rockingham County, Shenandoah County, Staunton City, and Waynesboro City). Because the MVC includes personnel from local school divisions, it provides a different potential avenue for gathering evaluation data. In future years, we hope to include more direct empirical impact data as a result of these additional avenues.

JMU gathers information about employer satisfaction in a variety of ways. One method is the VEAC Employer Survey, which asks employers of JMU program completers to rate inservice teachers on key knowledge and skills. For initial licensure programs, prinicipals typically complete the surveys to evaluate new inservice teachers. Initial licensure items mirror the items on the Completer Survey, aligned to InTASC standards and the Virginia Uniform Performance Standards. An advanced licensure program Employer Survey was piloted in Fall 2022. For advanced licensure programs, employers completing the survey might be principals or district administrators (e.g., superintendents). Advanced licensure survey items align to the general CAEP competencies for advanced programs and are mapped to professional standards for each program. Both initial and advanced Employer Surveys are designed to satisfy CAEP sufficiency standards for surveys.

Results from the VEAC Employer Survey indicate that JMU program completers are performing well. The Fall 2022 pilot of Advanced Licensure employers was very small, resulting in responses for just two program completers. Those employers rated both JMU completers as Proficient or Exemplary on all indicators, and reported that both completers were fully ready for employment at completion. The Initial Licensure Employer Survey provided more robust data, with ratings for 118 JMU program completers. Employers respond to all items on a scale ranging from 1 (Unacceptable) to 4 (Exemplary), following the general prompt, “Please rate [Completer Name’s] performance on each of the following:”. The expected performance for each item is an average score of 3, or a rating of “Proficient.” Of the 14 items on the VEAC Employer Survey, JMU completers scored at or above an average of 3/Proficient on all items, ranging from a low of 3.17 on Item K (“Integrates diverse language and cultures into instruction to promote the value of multilingual / multicultural perspectives”) to a high of 3.39 on Item F (“Maintains a commitment to professional ethics, communicates effectively, and takes responsibility for and participates in professional growth that results in enhanced student learning”). Employers rated JMU teachers at or above the VEAC average on 11 of the 14 items. The three items on which employers rated JMU completers slightly below the VEAC-wide average were, “Selects technologies, informed by research, to promote learning for all students” (JMU M = 3.25, VEAC M = 3.26, not statistically significant); “Integrates technology into instructional materials” (JMU M = 3.28, VEAC M = 3.32, not statistically significant); and “Uses assessment results to inform and adjust practice” (JMU M = 3.20, VEAC M = 3.21, not statistically significant). Although these differences are small and not statistically significant, JMU faculty are considering these areas as potential growth areas and discussing how best to address them. Scores for all VEAC Employer Survey items for JMU and VEAC-wide are reported in Table 3.

Table 3. 2021-22 VEAC Initial Licensure Employer Survey Results for JMU and VEAC

Additionally, on the VEAC Employer Survey, employers are asked to respond to the following question for each completer: “Based on your experiences with this teacher, what best describes the extent to which they were ready to meet the needs of students in your school?” Responding employers were presented with the options:

• 5 – Fully ready (able to have an immediate impact on student learning)
• 4 – Mostly ready (able to successfully meet the needs of most students
• 3 – Moderately ready (in order to be successful, needed additional training, support, and coaching beyond what is typically provided to beginning teachers)
• 2 – Minimally ready (limited success meeting the needs of students and improving outcomes even with additional supports)
• 1 – Not ready (unable to meet the needs of students even with additional supports)

Table 4 provides descriptive statistics on this scaled version of the overall satisfaction item. James Madison University’s mean is above the average rating for the other VEAC partner EPPs, indicating employers observed that their JMU-prepared teachers have a higher average readiness level to impact student learning than teachers prepared at other VEAC institutions.
 

Table 4. 2021-22 VEAC Initial Licensure Employment Readiness Ratings for JMU and VEAC

Data are provided here for teachers who completed initial licensure programs. VEAC piloted a parallel survey for teachers and other school personnel completing advanced licensure in 2021-22, and future administrations of the VEAC employer survey will include advanced licensure data.

JMU also ensures employer satisfaction through regular and frequent communication with local school divisions who employ our graduates. This communication happens in both formal and informal ways. JMU faculty prioritize strong relationships with local school divisions and ensuring that JMU is meeting their needs.

Stakeholder involvement is a key element of ensuring that JMU’s programs are designed to prepare program completers for success. There are many layers of stakeholder involvement at JMU:

Stakeholder Involvement

Professional Education Coordinating Council (PECC)

Stakeholders serve on the JMU’s Professional Education Coordinating Council (PECC), a group that meets monthly to review policies, needs of school partners, and changes in curriculum, assessment, and program/professional requirements. PECC includes program coordinators and/or assistant academic unit heads from every initial and advanced licensure education program at JMU. It also includes representatives from local school divisions.

More information about the PECC is available in the Professional Education Handbook 

MidValley Consortium (MVC)

JMU is part of the MidValley Consortium (MVC) that collaborates with three other area EPPs (Mary Baldwin University, Bridgewater College, and Eastern Mennonite University) and seven partner school divisions. The MVC Mentorship and Clinical Experience Team (MCET) representing teachers, administrators, and teacher educators meets monthly to plan consortium activities. The MCET and MidValley Consortium Advisory Council meet to evaluate consortium projects, set policy, and approve the annual budget. Each school division and college or university supports consortium activities. In addition to maintaining regular collaborative workflows and relationships, the MVC engages in timely special projects. In 2022-23, the MVC (a) revised the collaborative Student Teaching Performance Assessment to align to the updated Virginia Uniform Performance Standards (to reflect the additional of Culturally Responsive Teaching as Standard 6); and (b) convened to discuss opportunities for innovative field placement formats, including the possibility of leveraging field placement experiences to meet long-term substitute needs in local schools facing teacher shortages.

More information about the MVC is available online and in the MVC Handbook

Dean’s External Advisory Council (EAC)

At the highest level, JMU has an External Advisory Council (EAC) comprising alumni members who guide the College of Education generally and serve the profession around the Commonwealth in a variety of ways to support JMU students in becoming effective teachers and leaders.

Advanced Programs and Stakeholder Involvement

Although the PECC, MVC, and EAC involve advanced programs in some ways, they are often more targeted to the immediate needs of initial licensure programs. Most candidates in advanced licensure programs are already employed in schools; typically, candidates in these programs enter in cohorts of colleagues already holding teaching and other academic roles. Accordingly, the needs and nature of stakeholder involvement in advanced licensure programs differ. 

In JMU’s Educational Leadership program (certificate and M.Ed.) JMU faculty are retired school superintendents with rich networks of professional colleagues. The Educational Leadership program also utilizes approximately 20 adjunct professors who all hold high-level positions in school divisions across Virginia. This model ensures that local school divisions are directly involved in the creation and dissemination of program content and structure. Having such extensive buy-in from Virginia administrators strengthens the program and ensures it is responsive and proactive to current issues of practice in the schools. Educational Leadership program faculty also meet with a network of superintendents informally at twice-annual meetings of the Virginia Association of School Superintendents.

The Literacy Program has an Annual Stakeholder Meeting to discuss the program structure, assessment, trends in literacy education, policy, and other relevant topics. Participating stakeholders in this advisory group include current students, alumni who are working as Reading Specialists, and other faculty and administrators from Virginia schools. In 2022, the Literacy program adjusted its process for acquiring stakeholder input. In even years, they will hold synchronous meetings, and in odd years they will use surveys to gather stakeholder input asynchronously. The first stakeholder input survey will be piloted in 2023. The Literacy program also initiated an annual event, the JMU Literacy Leader Awards, in 2022. This annual event convenes local reading specialists for networking and learning, and helps publicize the JMU Literacy Program while celebrating high-achieving literacy educators in Virginia schools.

James Madison University triangulates data from several sources, both internal and external, to ensure candidate competency at completion. These measures build from internal, program-level gateways and key assessments during the program, to student teaching data gathered during final initial licensure field placements, to external proprietary assessments necessary for initial and advanced licensure.

Each education program at JMU uses a variety of program-level gateways and key assessments to ensure candidates are making sufficient progress throughout the program and well-prepared for employment at program completion. Programs design and administer their own program-level measures and evaluate student progress as part of program-level meetings as well as university-level annual program assessment requirements.  

There are also unit-wide assessments that measure candidate competency across programs. The Student Teaching Performance Assessment (STPA), created collaboratively through the MidValley Consortium of four local EPPs and seven school divisions, is administered for all students in their final field placement. Midpoint and Final assessments of student teaching performance are completed by cooperating teachers and university supervisors. A final Dispositions assessment is also administered to cooperating teachers with the final STPA.
More information about the STPA and Dispositions measures are available online:
https://www.jmu.edu/coe/esc/fieldexperience.shtml

Student Teaching Performance Assessment (STPA) indicators and final Dispositions are rated on a four-point scale, where 1 = Does Not Meet Expectations (DN); 2 = Developing Towards Expectations (DE); 3 = Meets Expectations (ME); and 4 = Exceeds Expectations (EE). Each indicator has its own specific criteria for each scoring level, reflected in the measure. The target score on each indicator is Meets Expectations. Accordingly, JMU hopes to see an average score of 3 or higher on each indicator by the time candidates complete student teaching. Table 5 shows midblock and final STPA scores and final Dispositions scores for the Fall 2022 cycle, the most recent available data. Dispositions are rated only at the end of the student teaching placement and only by cooperating teachers. This table shows a snapshot of EPP-wide initial licensure data across programs. Average final ratings for all performance indicators and dispositions were greater than 3, suggesting that initial licensure teacher candidates are meeting expectations for program completers at the conclusion of their field placements. The lowest-scored indicators relate to facilitating students’ higher level thinking skills (M = 3.01, SD = 0.45) and providing specific evidence to document student learning (M = 3.05, SD = 0.45). The lowest-rated disposition was also related to student assessment: “Values the use of self and peer assessment and uses data ethically to identify learner strengths and to promote learner growth” (M = 3.11, SD = 0.51). Faculty have discussed these areas of relative low performance and will monitor trends over time and consider possible interventions. The highest-rated indicators were, “Establishes a climate of trust and teamwork” (M = 3.48, SD = 0.53) and “Communicates effectively through oral and written language” (M = 3.42, SD = 0.53). The highest-rated disposition was, “Promotes inclusive learning environments that enable each learner to meet high standards by valuing different backgrounds and abilities” (M = 3.41, SD = 0.54). The high performance on this disposition reflects the hard work of college faculty to focus on inclusive and culturally responsive teaching.

Table 5. Student Teaching Performance Assessment and Dispositions Scores for Initial Licensure Programs in Fall 2022

In addition to these unit-wide measures, JMU recently developed unit-wide practicum-level field placement performance assessment and dispositions measures. Both measures will be piloted in 2023, allowing another level of triangulation for measuring unit-wide candidate progress and ensuring candidates are on track to be well-prepared and highly competent at program completion.

JMU annually completes reports documenting its candidates’ program success and ability to meet licensure requirements. The U.S. Department of Education annually gathers data for Title II of the Higher Education Act from all institutions of higher education with teacher preparation programs. These reports provide publicly-available data about teacher preparation and certification, including licensure pass rates. JMU also provides data to the Virginia Department of Education (VDoE) for biennial reports of teacher readiness.

All of our completers have achieved all of the state’s licensing requirements prior to leaving their education program. As the most recent (2021) VDoE Biennial Measures report indicates, 100% of our initial completers passed their licensure assessments. A small percentage (7%) of the Administration and Supervision candidates took the licensure assessment and at the time of the report did not pass. Following that report, the program decided to adjust its requirements so that individuals will pass their assessment prior to graduation; in previous years individuals were given greater discretion as to the timing of their assessment (due to the price of the exam and individual career goals).

JMU’s Title II annual reports and VDoE Biennial Measures reports are available the College of Education assessment reports site:
https://www.jmu.edu/coe/about/accreditation-and-assessment/state-federal-reports.shtml

All of our completers have achieved all of the state’s licensing requirements prior to leaving their education program. Accordingly, JMU’s program completers typically enjoy success finding employment upon graduation.

The Commonwealth of Virginia does not provide public P-12 employment data to EPPs, and obtaining complete and comprehensive data for program graduates is not currently possible. However, the JMU Career Center reports annually on career outcomes for graduating students with known data. The most recent available report for the College of Education, for 2021 graduates, reports outcome data for Bachelor’s-level and graduate-level students. For Bachelor’s-level graduates from the College of Education, only 1.3% of students with known data were seeking employment 6 months after graduation; 25% were employed, and 73% were pursuing continuing education [at the time of the reported data, most education programs required a Master’s degree]. For Master’s-level graduates from the College of Education, 98.1% of students with known data were employed; 1.4% were pursuing continuing education, and 0.5% were still seeking employment.

More information about career outcomes for graduates from the College of Education is available at the JMU Career Center website: https://www.jmu.edu/career/careeroutcomes/_files/2021/coe2021.pdf