Indirect assessment is when you collect information from students about what they think they’re learning. Direct assessment is when you look at actual work products produced by students to determine what students genuinely demonstrated they knew or could do in relation to your course goals. Both kinds of assessment are always in relation to specific goals for learning, so for Essential Studies (ES) revalidation, you’ll want to conduct direct and indirect assessment in relation to the ES goal that your course addresses.

Grades are NOT a direct assessment for a goal because a grade is never based directly and exclusively on the student demonstrating attainment of a single, specific ES goal. However, the work students are doing in your class, for which they’ll undoubtedly be earning grades, can also be analyzed for assessment. To do that, you’ll identify the ES goal that you’d expect to see demonstrated within the piece of work, segment out the portion of the work that’s directly connected to that goal, and then make sure you score in a way that allows you to look at the portion of the score that relates only to that part of the work (e.g., look only at the portion of the grade that relates to thinking and reasoning, if you’re assessing for that goal, or to familiarity with diverse cultures, if you are assessing that goal).

If you have questions about this, contact the chair of the ES Committee or the Assistant Provost (Joan Hawthorne).

The vast majority of revalidation petitions include analysis of data taken from exams (whether essay, multiple choice, or some other format) or analysis of data taken from papers. A few revalidation petitions include pre- and post-tests – that is, an analysis of data collected early in the semester compared with an analysis of data collected later in the term. However, almost any work students do in your class is potentially a source of revalidation data as long as the following conditions are met:

• there is a clear alignment between the ES goal you’re seeking to assess and the work product that you’re using;
• the students’ work is being systematically analyzed specifically in relationship to the relevant ES goal;
• you explain the alignment, the analysis process, and your conclusions/future plans in the revalidation petition.

The ES Committee is looking for evidence that students are doing effective thinking and reasoning, effective written communication, etc. (pick the goal of your choice) in your class. It is great to see evidence of progress. However, documenting significant gains, during a single semester, in the development of complex skills and knowledge areas (such as those specified in ES goals) is pretty difficult. So if you want to do a pre/post test and are able to show an increase in knowledge or skill, we’re happy to see that. If you don’t do a pre/post test but you do show that students are practicing the knowledge or skill in question and demonstrating a level of facility that’s reasonable and appropriate for your course, that’s good too.

In fact, if you find that students aren’t achieving the desired outcomes or seem to be actually slipping (as documented by a pre/post test), you won’t lose validation as a result of that. But we will want to see your analysis of your findings and what changes you intend to make to improve student outcome next time (if you think your methods were sound) or to improve data collection and analysis (if you don’t trust your own findings).

One way to look at it is to think about how large of a sample would be necessary for YOU to find the evidence persuasive. If you find the evidence persuasive and you explain to us why you think it’s persuasive, we’ll probably agree with you. Most of you probably wouldn’t be persuaded on the basis of looking at the work of a couple of students, even randomly selected students, that you had credible findings. What if you selected 10 students randomly? What if you selected 8 students, two of whom had received each grade (A, B, C, D/F)?

Another way to think about credibility with this kind of analysis is whether you reach a point where more data are just reinforcing what you’ve already seen. Or consider whether the direct assessment data fit with your indirect data and other information you might have collected. All of these things influence data credibility. So make a decision about sample size, plan to explain your rationale so that we can see the logic of your choice – and do a better job next time if the data you collect end up feeling something less than totally credible by the time you analyzed it.

The basic answer is no, you don’t need to collect data in every last section. Approach this as a sampling problem, like the question above. Collect data from enough sections so that you’ll be persuaded that your results apply generally to the course. Don’t “cherry pick” sections. And plan to rotate data collection among sections and faculty so that, over time, you get a very full view of the learning, in relation to your chosen ES goal, that occurs in your courses.

Or take a more thorough approach and treat it as a faculty development opportunity. Having conversations about goals and outcomes among all faculty who teach in the class will probably turn out to be wonderfully informative – and may result in more learning in sections across the board.

In most cases, the department chair or another faculty member who has taught the class will be able to submit the paperwork, especially if you have collected data in advance. Or a permanent faculty member can work with the visiting faculty person to collect and analyze outcomes data.

It is occasionally impossible for a department to collect assessment data. We don’t revalidate without data, but please contact the ES Committee chair to explain your situation and request an extension on specific courses if individual circumstances mean that a course in your department simply cannot be revalidated on schedule. We will expect paperwork on those classes to be filed within two months after the completion of the next semester in which the course is taught by a regular faculty member. These courses will stay on the regular revalidation schedule assigned to your department, which means that you won’t have a full three years until they need to be revalidated again. So we strongly recommend that you don’t wait to collect data, next time around, until the last semester before the revalidation is due.

See the comments above, which apply to this question as well. Plan ahead next time so that you don’t run into this problem when you revalidate!

A collection of raw data doesn’t mean much by itself. You have lots of context for making sense of the data – like insight into where and how you collected it, why you chose to collect that particular kind of data, how you analyzed the data, what you learn from considering the data, and what kinds of changes you might want to make based on what you’ve learned. We want to understand this thought process.

It’s not so unusual to find that students just don’t seem to be making the progress we’d expect to see on the ES goals. We all love to see evidence of success, but more problematic findings can still be interesting and important. Why do you think your results turned out that way? Did you collect data that turned out to be inadequate to address learning in relation to the goals? Do you think there’s something unusual about that year’s students? Do you think the data suggest trying a different approach? Talking about what you do with your findings becomes especially important when the findings themselves are disappointing.

No. In fact, we ask that you focus revalidation efforts on the ES goal that is most important to your class and most directly addressed in the teaching and learning.

We ask that you revalidate every four years, which seems often enough to keep the list of ES courses relatively current, often enough to provide us with a current understanding of the learning that’s going on relevant to the ES goals, and not so often as to burn out faculty teaching ES courses. Any assessment process needs to be on a cycle to be useful, and the same is true for Essential Studies. Revalidation is part of OUR process of assessing the ES program, as well as YOUR process of assessing learning in your class that relates to your chosen ES goal.

This is totally a departmental decision. UND doesn’t have a policy of attempting to limit the number of courses available for ES credit, but we also have a lengthy list of courses – so we aren’t actively seeking to add additional classes to the list either. If your class addresses at least one Essential Studies goal, it’s a candidate for the list. If a class meets one of the ES goals and you believe being on the list would help you keep student numbers where you want them, definitely consider validating. If your class is more than sufficiently enrolled with students taking it for other reasons, you might want to consider if it’s worth validating. Also remember that most students taking upper division classes in a field like Art or Biology are often students with a major in a related discipline – in other words, students who have almost certainly already fulfilled your discipline’s share of the ES distribution requirement credits. If you teach a course like Military History, Abnormal Psychology, or Human Sexuality, which may well attract significant numbers of non-majors and may not have extensive pre-requisites (despite being potentially at the upper division level), students may be much more interested in being able to earn ES credit for your course.

Our aim is not to pull courses from the approved list, but we do need to uphold program standards. Any delays in submitting revalidation paperwork should be communicated to the ES Committee chair so that a reasonable compromise can be determined, when appropriate, by the full committee. If you really don’t want to revalidate, please let us know right away.

The first step for any department with courses up for revalidation should be attending a Revalidation Workshop sponsored by the Essential Studies Committee (you’ll be informed of the date in advance), and the second step should be meeting with your departmental liaison, who should have already been in contact with your department. This person is a member of the ES Committee who is assigned to work with you throughout the revalidation process. Your liaison can meet with faculty, individually or collectively, to answer questions or provide additional explanation and information.

What you’ll learn by attending the Revalidation Workshop, in very brief form, is that you will need to submit direct assessment data as part of the revalidation process. In order to collect direct assessment data, you’ll begin by identifying a goal for learning within the class that aligns with one of the ES goals. Then collect data (see “Data Collection” above) related to the selected goal, analyze the data, and submit required revalidation forms including a summary of the assessment process and your conclusions about learning.

Members of the ES Committee are divided into three sub-committees for review of revalidation petitions. Sub-committee members read your paperwork thoroughly and, in most cases, recommend action to the full committee. If there is something particularly unusual about a particular course in your department, the full committee might be asked to read your petition prior to official action. (And if you want to be present at meetings where the official vote is taken on your departmental revalidation petitions, you are always welcome.) You receive official notification of the outcome from the Registrar’s Office after the vote has been taken.

February 1 is this year’s deadline. Please work with your liaison to get the paperwork in on time – or early if possible – so that any last minute questions or problems can be addressed. But, as usual, our concern is about working with you to get the revalidation done, not about looking for reasons to kick your courses off the approved list.

Your liaison for the ES Committee chair (Brett Goodwin brett.goodwin@und.edu, 777-2621) can answer many of your questions. The assistant provost (Joan Hawthorne joan.hawthorne@und.edu, 777-4684) is a member of the ES Committee and works with faculty on questions about any assessment issues, whether related to revalidation or not.