Assessment at the Course Level: Should I "Do Sotl"?

• What is SoTL?
• Getting approval for human subjects research
• Approaches to SoTL research
• Using assessment to answer your own questions about student learning
• Course level assessment for Essential Studies revalidation
• Next steps

What is SoTL?

The Scholarship of Teaching and Learning (usually called SoTL) is about doing research on your own teaching. It's not just about telling teaching stories or explaining a best practice that you've been using successfully. SoTL is a scholarly practice, and projects are designed to meet standards for academic presentations (at conferences on teaching and learning, or within your own discipline) and, in many cases, for academic publications as well. They usually grow out of questions about teaching and learning, but "doing SoTL" means taking a scholarly approach to answering your questions.

So perhaps you're teaching a research methods class for social science majors and despairing of their ability to apply quantitative methods purposefully rather than formulaically. One solution, you think, might be to devote 15 minutes of every class period to a hands-on activity done in small groups. If you make that change and notice all kinds of good results, that might make a great teaching story. But if you want to make it a SoTL project, you'll plan your project in the same way as you might plan some other kind of scholarship. You might review the literature on the subject, since the problem you're seeing is likely one that other faculty have also encountered. You'll learn something by looking at what they've tried.

Perhaps you conclude that what you have in mind is a bit different than what you find in the literature. SoTL projects usually have a research question behind them, so a next step might be to see if you can frame an interesting question that your project could address. Once you have a question, you'll have a clearer idea of what kinds of data would be useful and feasible to collect – and you'll be well on your way to a SoTL project.

Framing a good question and developing an appropriate method are not difficult tasks, but they can seem formidable when you're new to them. So don't hesitate to seek help. UND has many faculty who have experience in SoTL research, and the Director of Instructional Development or the Director of Assessment and Regional Accreditation will be able to help you make contact with someone who'd be willing to talk with you about your research ideas.

top 

Getting approval for human subjects research

At some point, you'll want to download UND's Institutional Review Board (IRB) forms and request their permission for your data collection. Since your research will presumably involve human subjects (usually your students – and possibly other faculty in your department, depending on the project), you'll need the IRB's approval. Board members are quite supportive of SoTL research, but the IRB process can be perceived as an obstacle when the steps are unfamiliar. Going through the IRB will be useful to you in the long run, however, not only because it'll ensure that your research meets appropriate ethical standards, but also because you'll need to think through answers to questions that might not have occurred to you (but are important considerations for your project) in order to submit your request for approval.

top 

Approaches to SoTL research

Once approval is granted, you're ready to carry out your project. It can be semi-experimental, yielding statistics and numbers and pre/post comparisons, if that's the kind of research approach that feels most meaningful to you and most appropriate for your question. Or it can be more anthropological and qualitative if that's your preferred approach to scholarship and likely to answer your question. The point is to plan your project using methods that will help you in your own thinking about your teaching and learning question, and that other faculty in similar disciplines (here or elsewhere) would also find instructive.

And, of course, once you've followed through with the process, there is a very good chance that you'll have a study that can be presented and published in venues of interest to scholars in your field and beyond. You'll be contributing to the knowledge base about teaching. Presentations and publications about your study will "count" in your tenure and promotion file in most colleges at UND (although every department, of course, has its own standards for valuing different kinds of scholarship).

top 

Using assessment to answer your own questions about student learning

You can carry out an assessment project to answer your own questions about learning whether it's a thoroughly planned "SoTL project" or not. You might not be interested in contributing to the literature on teaching and learning. Maybe you just want to know how well students are learning in your class, so you'd like to see how they're doing on each of the three or five intended learning outcomes you've identified. Or maybe you're curious about the effectiveness of a particular assignment, or interested in understanding what's happening with one specific outcome that's a concern for you.

Course-level assessment can be carried out exactly as would program-level assessment. What is it that you want to know? If it's about learning outcomes (i.e., "have they learned what I intended?"), begin by naming the outcomes you want to find out about. Figure out what assignments or tests you are using that provide them with opportunities to demonstrate those kinds of learning. Maybe you're interested in how well they understand a particularly complicated theory in your physics class, so you will want to include some number of questions related to that theory on a final exam. Then look at how scores on those questions compare with scores on the test overall. The discrepancy (if you find one) will provide you with insight about that part of their learning.

Or maybe it's specifically the application issue that concerns you. Students always do well at remembering the formulas taught in your chemistry class or recalling the incidents from your history lectures. But you want to know if they can figure out which formula applies where and why. Or you want to see if they can use what they learned about a particular historical event to similarly analyze a different event. You probably already have assignments (test questions, essay assignments) that ask them to do this more advanced skill, and you'll want to look at how successfully students carry out the application tasks compared with their ability to do the memory tasks.

If you're teaching a class of 200 students, you won't need to analyze the work of every student to answer your question. Think about how you could sample in a way that you'd find meaningful. What if you scored 20 final essays that address the application task specifically – would that persuade you? What if you pulled out 5 tests that earned A grades, 5 with B grades, etc.? Would that be enough to give you confidence that you were seeing the big picture of their learning? Devise a sampling strategy that will feel like "enough," and try it. Perhaps you'll end up with additional questions, and then you'll have a clearer idea of what else needs to be done.

In some cases, you might find that there's really no existing assignment that asks students to demonstrate the kind of learning you're interested in looking at – and recognizing that is useful in itself. If a particular learning outcome is important, surely a first step is ensuring that students actually "do" (and demonstrate) that kind of learning at some point in the class. And if you don't want to build it in as a formally graded activity, you can think about ways to do it informally: you want to know whether they've come to understand why they learn those formulas. So give them five minutes in class one day and ask them to explain in writing the value of learning them. They can turn their answers in anonymously. You'll probably find that more than 75% of your students have written serious answers. You'll learn something by reading them.

top 

Course level assessment for Essential Studies revalidation

These strategies are also applicable for the assessment section of your Essential Studies (ES) revalidation request. Your basic question, in that case, will be "how well are students doing on the ES learning outcome that I've identified for my class?" In most cases, you will want to draw from existing assignments to collect study work products that can be reviewed to answer that question.

Most faculty like to collect information for revalidation at the same time as they review student work to assign grades. If you're using a rubric to grade, make sure that two or three categories on your rubric are written to align with the ES goal you've selected. You can draw directly from the ES rubrics but you can also develop your own criteria and rubrics. However, since your course is part of a program that belongs to the university at large, you'll want to make sure that your definition of "information literacy," for example, matches with the definition (or some portion of it) found in ES documents about the goal, including the rubric that establishes a sort of "extended definition" via the list of criteria included.

Some faculty assess ES learning outcomes by examining scores on specific exam questions that align with the goal, and, perhaps, comparing those scores with scores on other questions. Are students doing as well on the critical thinking outcome for your class as they are on other parts of the test? Collecting scores from the questions that directly align with your ES outcome (and explaining that alignment to the ES Committee) will allow you to examine student learning related to the ES goal at the same time as you score the rest of the test.

There are many other strategies that can be used to collect assessment information for revalidation of ES courses too, but the basic point is to use a strategy that allows you to look explicitly at the ES outcome, rather than at overall student performance on any particular assignment or on the course as a whole. Grades, for example, won't serve to answer your ES assessment question.

Finally, assessment for ES revalidation poses special challenges when courses are taught in multiple sections or formats, or if the course itself is large (typical of many first-year courses, e.g.). If that is your situation, you likely will want to sample rather than closely examining the work of every enrolled student in every section. As always with sampling, think about what kind and size of sample will allow you to see "enough" to draw meaningful conclusions about student learning in the course overall. And explain on your revalidation request the rationale for your sampling decisions. Plan to specifically address sampling done in courses taught both face-to-face and online or in many different sections. ES Committee members will want to understand what you were thinking but, assuming you had thoughtful reasons for your sampling approach, they are not likely to second-guess your decisions. They will, however, want to know whether your final results were ultimately persuasive to those teaching the class and, if not, how you will generate more useful findings next time. 

top 

Next steps

Whatever you want to know about learning in your courses, there's probably a way of finding the information. And, as always, there are people on campus who would be happy to talk with you further if you have questions. A good place to begin, depending on the question, is with Joan Hawthorne, Director of Assessment and Regional Accreditation, Anne Kelsch, Director of Instructional Development, or Tom Steen, Director of Essential Studies. You may be referred to other colleagues if you want to talk with someone who's studied a similar kind of question. The Office of Instructional is also home to a library of publications on pedagogy, including both books about SoTL and assessment and articles based on SoTL projects.

top