8.2a Student Outcomes: Educational Programs

Assessment of Student Learning Process

As stated, the assessment of the student learning process is comprised of two ongoing processes: annual Academic Program Viability Reports and triannual Program Assessment Reports.

Institutional Effectiveness uses the Educational Outcomes Assessment website (shown below) to manage these assessment processes. This site provides an online database for completing educational assessment reports, aggregating student learning outcomes across educational programs, and establishing a repository for additional program-specific information, such as viability reports, employer surveys, and recent alumni surveys. The site also allows public access (shown below) to all “completed” assessment reports for active programs dating back to 1999. The public nature of the site further encourages the use of assessment data across the College by allowing programs to review completed reports and best practice utilization, not only by their department but from other programs as well. The following program assessment reports are captured by this system: Academic Program Viability Reports (VR), Academic Program Assessment Reports (APAR), and follow-ups to both reports. Currently, the system stores 93 active programs and 56 inactive programs. A quick view of what can be found for academic programs in regard to the assessment reports status can be seen in the academic assessment status document, which is maintained by the IE department.

Figure 2 – Educational Outcomes Website

Academic Program Viability Reports (VR)

Annual Academic Program Viability Reports (VR) are grouped by academic organization, which include all academic degrees and certificates within each degree.  Though not specific to program learning outcomes, the VR provides key College stakeholders, such as the President, program advisory boards, Deans, and program administrators with a comprehensive report that summarizes all elements of the program’s viability from a 360-degree perspective. This helps College leadership make critical decisions regarding the continued sustainability of a program and provides program leadership a vehicle to support and document actionable change for the purposes of performance improvement, as well as serves as a way to continuously review key student success metrics.

Most metrics within the VR utilize data from POWER BI and PULSE BI, SPC’s real-time business intelligence systems accessible and verifiable by anyone with login credentials (shown below).  The systems filter management and student success data to address questions and identify potential areas for improvement. Measures (such as student course-level success rates) can be viewed at the aggregate institutional level or at the campus or program level.

Figure 3 – Viability Report PowerBI Dashboard

This dynamic system allows Deans and program administrators to gather the necessary data for the reports themselves and to monitor changes to these reports throughout the year to gauge the success of action plan implementations and improve student success. Moreover, 9 filters are available to allow for further data disaggregation, including term, program, plan, college/school/ department, academic organization (default for VR), community, first time in college flag, part/full time. and gender. 

The VR includes twelve measures focused on the College’s mission of student success:

Academic Program Assessment Cycle

Once the program-specific data are pulled, Deans meet with program faculty, program administrators, Career Connections staff, Advisory Committee members, and IE to review statistical trends, create follow-up reports on past year goals, evaluate the impact of the goals on program quality, create the next year’s program-specific goals, identify special resources needed, and note area(s) of concern or improvement. All Viability Reports, upon completion, are printed and reviewed by the Vice President of Academic Affairs and Cabinet. The 2021-22 completed reports can be seen here:  Bachelor’s Degree Programs, Associate Degree Programs.

Academic Program Assessment Reports (APAR)

SPC uses academic program assessment reports as a vital tool in achieving its commitment to continuous improvement. General education and program-specific academic assessments are conducted according to a three-year cycle (shown to the right). Although program assessment reports are developed on a three-year cycle, programs are required to collect assessment data on an ongoing basis.  SPC’s three-year reporting cycle ensures that assessment reports include multiple data points and that programmatic changes are made based on trends as opposed to a single year. The three-year cycle also allows time for programmatic changes to take effect before the next report is completed.  As of 2020-21, any program that reaches the end of its three-year cycle without collecting sufficient data is placed upon an annual cycle until a minimum of three years of data has been collected.  This process ensures the integrity of the process, and its intent to support program improvement remains intact.

The Academic Program Assessment Reports consist of the following seven sections:

1) Use of Past Results – captures the effects of previous action plans

Figure 5 – Sample Past Results Language – Business Administration, BS

2) Program Learning Outcomes – lists the program PLOs

Development of Measurable Program Learning Outcomes

The development and assessment of student learning is a systematic, unified process across all programs at SPC.  Identified and developed with input from faculty, advisory board members, and program administrators in conjunction with the Curriculum Services Department, SPC Program Learning Outcomes (PLOs) are developed to clearly articulate the knowledge, skills, and abilities of their graduates through demonstratable student learning. PLOs also incorporate general education outcomes, where appropriate, as well as identified best practices in their fields.

Each year, as part of the Annual Viability Report, Institutional Effectiveness (IE) meets with Deans and program faculty to review the Program Learning Outcomes as part of the supplemental Viability Report Checklist. This ensures not only regular review but provides a venue for continuous improvement of these outcomes. For programs that are externally accredited by discipline-specific professional and/or state agencies, programmatic learning outcomes are closely aligned with those established by the accrediting bodies and, in some instances, they are stated verbatim.

The following examples demonstrate PLOs for a baccalaureate program and an A.S. degree program at SPC:

Paralegal Studies B.A.S

1. Acquire and implement advanced critical thinking and analytical skills needed to effectively problem-solve.
2. Demonstrate the appropriate use of communication and organizational skills within and outside the legal system.
3. Execute the technical skills, including legal research and writing, needed to succeed in the legal profession.
4. Exhibit the ability to deliver professional and ethical legal services in a culturally and locally appropriate manner.
5. Demonstrate an advanced knowledge and understanding of the American legal system.

Environmental Science Technology A.S.

1. Scientifically interpret and apply the concepts, principles and theories that constitute aspects of environmental science.
2. Evaluate environmental risks and develop plans to address their effects using current methodology and technology.
3. Evaluate the role of environmental policies, laws and management practices on the changes in a local ecosystem over a defined period of time.
4. Systematically apply field testing and field measurement collection practices in an ecosystem.

Development of General Education Outcomes

SPC’s general education outcomes (shown below) closely align with the State of Florida’s recommended student general education outcomes and are designed to assess broad soft-skill proficiencies attained by students through the general education curriculum, complementary to specific program learning outcomes. As General Education courses are a part of all academic programs at SPC, IE staff, faculty, and program administrators review the general education curriculum to produce assessment measures that provide evidence that the courses and curriculum meet institutional goals for student learning. General Education Outcomes are reviewed as part of the 3-year assessment cycle.

SPC General Education Outcomes

1. Critical Thinking: Analyze, synthesize, reflect upon, and apply information to solve problems, and make decisions logically, ethically, and creatively.
2. Communication: Listen, speak, read, and write effectively.
3. Scientific and Quantitative Reasoning: Understand and apply mathematical and scientific principles and methods.
4. Information and Technology Fluency: Find, evaluate, organize, and use information using a variety of current technologies and other resources.
5. Global Socio-Cultural Responsibility: Participate actively as informed and ethically responsible citizens in social, cultural, global, and environmental matters.

3) Assessment Methodology – defined per PLO

The assessment methodology section describes the means by which each PLO is being assessed, using a multiple methods approach that includes direct and indirect metrics. Direct metrics provide quantitative data and may include course examinations, assignments, capstone projects, and industry certification exams. A minimum of one direct metric is required to assess each PLO.  Quantitative data provides evidence to ensure the knowledge and skills necessary to address the PLO were met.  Indirect metrics provide qualitative data and are collected through the administration of institutional and national student surveys, at specific points throughout their tenure at SPC.  The section also describes the timeline and population associated with the data collected; the instrument used to collect quantitative and qualitative data, and rubric utilized.

4) Criteria for Success – defined per PLO

The Criteria for Success section specifies the evaluation standards or benchmarks by for each metric used to assess a PLO, and determines to what extent each metric was successfully met.

5) Summary of Assessment Findings – defined per PLO

The Summary of Assessment Findings section displays the actual data collected, the number of students assessed, and whether the criteria for success were met, for each metric listed in the Methodology section, for each PLO.  Data are provided by semester or annually and are disaggregated by delivery modality (face-to-face, online).

6) Discussion and Analysis – defined per PLO.

This section details the comparison between the current year’s assessment results and the established thresholds set in the criteria of success section of the document. This information is used to drive change and improvement in the program by highlighting the learning outcome areas that do not meet established thresholds and establishing appropriate action plans.

7) Action Plan and Timeline – defined per PLO and includes budgetary and planning implications.

The action plan and timeline section is created by the program administrator to address any issues identified in the assessment findings and drive program improvement.

In the year following an APAR, program administrators submit a follow-up report to ensure that identified changes from the action plans are implemented and will be measured in the subsequent reports. For programs that meet their goals or report no findings, an action plan will be implemented to review criteria and determine what/if new goals or targets are appropriate. This supports a continuous improvement process of review and refinement.

Examples of Academic Program Assessment Reports

Access to all Academic Program Assessment Reports is available to the public on the Educational Outcomes website, found on the Institutional Effectiveness website (see Instruction Guide). Due to the ease of public access to all assessment reports, the College is not providing sampling, but to ease review, below are some examples including off-cycle assessments that have been placed on annual review.

Environmental Science A.S

• 2021-22 Environmental Science assessment report;
• 2018-19 Environmental Science follow-up report;
• 2018-19 Environmental Science assessment report

Funeral Services A.S.

• 2021-22 Funeral Services Assessment Report;
• 2018-19 Funeral Services Follow-up Report;  
  
• 2018-19 Funeral Services Assessment Report

Nursing A.S. (off-cycle)

• 2022-23 Nursing Assessment Report;
• 2017-18 Nursing follow-up report;
• 2017-18 Nursing Assessment Report

Paralegal B.A.S (off-cycle)

• 2021-22 Paralegal BAS Assessment Report;
• 2017-18 Paralegal BAS Follow-up Report;
• 2017-18 Paralegal BAS Assessment Report

Public Safety B.A.S. (off-cycle)

• 2022-23 Public Safety Administration Assessment BAS;
• 2018-19 Public Safety Administration Follow-up BAS;
• 2018-19 Public Safety Administration Assessment BAS

General Education Assessment

In addition to the programmatic assessment process described above, the APAR captures the general education outcomes assessment process and follows the same format.  Broken down by each General Education area, each report uses a wide variety of methods to assess overall student learning, including survey data, exam data, and course assessments.  Within the Use of Past Results section, trend data is included to allow Deans, Program Administrators, and Faculty to provide greater insight on the impact of their action plans.

Association of American Colleges and Universities (AAC&U) Value Rubrics have been utilized to enhance the general education assessment process at SPC since 2015.  The inclusion of these Value Rubrics allows faculty to evaluate the quality of the general education curriculum and identify areas for performance improvement, based on globally accepted discipline-specific rubric categories.

In 2015, IE assisted Deans and faculty to review student competencies in the areas of Critical Thinking, Ethics, Mathematics, and Humanities/Fine Arts.  Student competencies were then compared to various AAC&U Value Rubrics for alignment.  Ultimately each discipline selected the rubric which best aligned to the respective student competencies and designed PLOs and assessments to incorporate those best practices.

Example: Mathematics–Scientific and Quantitative Reasoning

The Mathematics Department selected a revised version of AAC&U’s Quantitative Literacy VALUE Rubric to assess Mathematics student competencies within SPC’s Scientific and Quantitative Reasoning general education outcome.  Elementary Statistics STA 2023 was chosen as the course that best aligns with all six competencies. Questions from the common online midterm exam were mapped to the appropriate competency. Instructors teach the course focusing on state and department mandated course objectives. Students are assessed automatically when they take the online midterm.  Data is gathered at the end of each semester and analyzed by the Mathematics Dean and faculty and reported under the program learning outcome.  The process was designed to allow for a non-interruptive and efficient way to assess students.

Example: Humanities and Fine Arts–Global Socio-Cultural Responsibility

The Humanities/Fine Arts department selected a revised version of the AAC&U Global Learning VALUE Rubric to assess student competencies in social, cultural, and global matters, within SPC’s Global Socio-Cultural Responsibility general education outcome.  The focus is on various topics, relationships, and themes of human expression from a global perspective.  Instructors design and grade individual assignments using a modified version of the Global VALUE Rubric as an additional competency measure. The revised Global VALUE Rubric data are gathered at the end of each semester and analyzed during the Global Socio-Cultural Responsibility assessment processes. The assignment requires students to incorporate theoretical concepts and vocabulary into an essay.  The rubric focuses on students’ knowledge of aesthetics and cultural expressions of both Western and non-Western origins.

Results from these assessment activities are reported on the general education program assessment reports.

Examples of General Education Assessment Reports

Access to all General Education Assessment Reports is available to the public on the Educational Outcomes website,  found on the Institutional Effectiveness website.  Due to the ease of public access to all assessment reports, the College is not providing sampling, but instead to ease review below are some examples, including off-cycle assessments, representing areas that have been placed on annual review.

Critical Thinking

• 2021-22 Critical Thinking assessment report;
• 2018-19 Critical Thinking follow-up report;
• 2018-19 Critical Thinking assessment report

Global Socio-Cultural Responsibility (off-cycle) (Note: the assessment and follow up were completed simultaneously due to significant rubric changes that caused the assessment focus to change.)

• 2019-20 Global Socio-Cultural (Art Appreciation) assessment report;
• 2019-20 Global Socio-Cultural (Art Appreciation) follow-up report;
• 2018-19 Global Socio-Cultural (Art Appreciation) assessment report

Use of Results to Drive Institutional and Program Improvement

Each of the two components of the assessment cycle, viability reports (VR) and three-year program assessment, as well as their associated follow-up reports, include specific goals and action plans focused on making programmatic improvements based upon the results being analyzed. Below are several examples of action plans and/or follow-up reports that track these desired and implemented improvements.

Within the Action Plan Follow-Up reports, the final section requires an explanation about the intended future impact of the change being made on program quality. These explanations are used to determine next steps in future assessment planning.

General Education Assessment Reports, like program assessment reports, also include action plans and follow-up reports, as demonstrated by the example below from the Global Socio-Cultural Responsibility–Effective Citizenship Follow-Up report.

Depending on the analysis of the data, action plans created during the APAR process can focus on a variety of improvements: curricular changes, assessments used, course or assignment revisions, etc. To support the implementation of these proposed changes, the Dean and Faculty can reach out to a variety of academic and student support departments.  Action plans that align to the Enhance Curriculum category—such as course additions/deletions, course content, and pre-requisite/co-requisite revisions—are completed with assistance from the Curriculum Services department and/or Instruction, Design, Education and Support department.  Action plans that require Faculty Professional Development are completed with assistance from the Center for Excellence in Teaching and Learning.  Action plans aligned to the Assessment Methodology category, such as revising the assessment plan; data collection and analyses; and assessment instruments are completed with assistance from the Institutional Effectiveness department.  Generic action items that align with the overall Student Success category are completed with assistance from any of the aforementioned departments as well as other collegewide teams of faculty and administrators.

Review and Approval Process

Academic Program Assessment Reports and General Education assessment reports undergo an extensive review and approval process to ensure quality and provide feedback to the program before the report is published. Program administrators submit completed assessment reports within the Educational Outcomes website where they are reviewed by IE staff, the associated Academic Dean, and the Associate Vice President for Institutional Effectiveness and Academic Services and/or the Vice President of Academic Affairs in succession. Assessment reports are approved and continue through the approval chain or are returned to the program administrator for revision. This process ensures that assessment results are communicated and reviewed by key stakeholders at each level of the institution. 

Distance Education and Dual Enrollment Assessment of Program Learning Outcomes

All academic programs, regardless of modality, follow the same assessment process.

The Academic Program Assessment Report (APAR) process addresses learning outcome attainment for distance education. Academic programs have historically identified course(s) where students demonstrate attainment of program learning outcomes. The Summary of Assessment Findings section within the APAR is subdivided to allow academic programs to disaggregate outcome data between face-to-face and distance learning instruction. Several academic programs offer instruction via a single modality. For selected courses that include both traditional lecture and distance learning sections, assessment data are disaggregated by modality. Deans and faculty can then review disaggregated results to determine whether there are differences in student learning outcomes and address any differences in a follow-up action plan.

Like distance education students, Dual Enrolled students are assessed together with all other students in their program. However, APARs do not disaggregate data based on Dual Enrolled student status.