Fall semester is such an energizing time on campus! It is also a perfect time to begin thinking about your assessment plan for the year. Specifically, what instruments you want to use for measures. It is unfortunately common to gather data, only to discover it does not tell us what we set out to discover (see Dr. Finney’s webinar on Implementation Fidelity Data for a solution to this!). Whether you’re tweaking what you’re currently using or if you want to try something different, rubrics, surveys, and focus groups are good options to consider. This summary article from the “Using Assessment Tools Effectively” webinar by Dr. Tisha Paredes, Dr. Tancy Vandecar-Burdin, and Dr. Sheri Popp will walk you through getting started with each!
Rubrics
What is a rubric?
Shared terminology is always important in IE work – assessment literacy promotes collaboration. A rubric is a teaching and assessment tool designed to clearly describe the characteristics and qualities of a specific learning or assessment task. It establishes criteria for how something is to be evaluated or judged.
Why should I consider using a rubric?
There are advantages for both students and faculty. Faculty can even build the rubric with students, which helps them think critically about assignments, outcomes, and quality.
For students, a rubric provides clarity on expectations, and articulates what “good” means. It can tell AND show how students can succeed. A rubric also provides useful and targeted feedback, allowing students to gain control of their learning. And lastly, it encourages metacognitive reflection, as students can self-assess with a rubric.
For faculty, a rubric forces clarity of instructional decisions, as you articulate the knowledge and skills students need for a task and to accomplish the goals of the project. It is a map that points to component parts of the performance. This in turn informs areas of needed instruction, tying everything together.
A rubric also encourages more consistent scoring. While all grading is subjective and a rubric won’t remove that completely, it will help the process be more objective. And probably one of the best reasons, a rubric saves time! While it takes time to create a rubric and calibrate, it should save time in the long run, especially when you score many items.
Characteristics of quality rubrics
There are five attributes of an effective rubric:
- Linguistic clarity: Avoid using general terms like good/bad. Instead, use very specific language. “The tone is appropriate for the audience.”
- A focus on quality (not quantity): While projects must follow the directions regarding length, etc., a rubric takes quality into account.
- Simple (four point) scale: An odd scale gravitates to the center, and more than 4 can be too complex.
- Manageable length: The rubric should be thorough but manageable – no more than one page.
- Essential criteria: What do you REALLY want to know from this tool? Some common themes might be quality of thinking, quantity of content knowledge, and organization of the product. If something you want to include is hard to define, don’t skip it, but clarify it (your students need that).
How do I create a rubric?
Fortunately there are many resources out there. You can consult the literature and professional organizations, which often have examples and instructions. Often a search for the type of project or discipline you need the rubric for will yield a wonderful place to start or borrow from. You could also create your own from scratch! Gather sample student work, sort and group them by good/better/best, then classify the attributes of those and condense those reasons into clusters, which makes the rubric categories. This is an exercise that even works well in class.
Surveys
What is a survey?
While we have all taken surveys, most of us have never designed one. The word ‘survey’ is used most often to describe a method of gathering information from a sample of individuals. A survey provides a quantitative or numeric description of people’s attitudes, perceptions, preferences/opinions, or actual behavior/ experiences by studying a sample of that population.
Why use a survey?
Surveys are most often used when existing data sources are not adequate and you need information about people’s views, perceptions, beliefs, or experiences. They can provide data that can be used to make important decisions or changes to courses, programs, policy, etc.
Specifically in relation to assessment, surveys can help measure:
- Program evaluation/effectiveness
- Student learning outcomes
- Student satisfaction
- Employer satisfaction
- Alumni experiences
How do I create a survey?
First, identify your “big” question – what do you need to know? For assessment, it should be related to your goals and objectives for your program, course, or unit.
Then you’ll want to focus on that question in order to conceptualize and operationalize what you need to measure. What are the dimensions and indicators that will help you answer your question? Dimensions might be tasks or activities, frequency, value, or skills. Indicators could include satisfaction, plans to continue using a service, if one would recommend something, or ratings.
Finally you’ll need to plan for conducting the survey. It is critical to know your population. How do you reach them? Once you have access to contact information you can determine your survey mode(s) (phone, in-person, web, mail, or multi-modes). As you consider your recipients, think through other aspects that might affect the survey, such as literacy issues, vocabulary and jargon, access to technology, and incentives
Keep in mind you do not always need to reinvent the wheel! It is common to use existing surveys and questions, as well as consult with your colleagues and professional associations.
How do I write good survey questions? This could be an entire article on its own! But in short, questions need to be reliable and valid, and they should yield results that answer your big questions (if not, delete them!).
As you draft, consider asking yourself the following:
- Are there questions that are unclear or difficult to understand?
- Are you asking more than one question in a question?
- Do the response options match the survey question “task”?
- Are response options exhaustive and mutually exclusive?
- Does the survey flow well?
- Are there questions that are redundant or unnecessary to the survey purpose?
- Are there questions that might be missing?
Focus Groups
What is a focus group?
One lovely definition is “a group discussion that resembles a lively conversation among friends or neighbors” (Morgan, 1988). More practically, it could be defined as a confidential group discussion with trained and skilled moderators using open ended questions that promote interaction and explore participants perspectives and experiences in a structured but relaxed atmosphere designed to generate qualitative data.
Focus groups are rigorous and relevant where surveys are reliable and valid. Surveys are very quantitative, but it can sometimes be unclear what those numbers always mean. Focus groups can help answer the why behind quantitative data.
Why use focus groups?
Focus groups are a good way to gather data on how participants make meaning about a topic of interest (sole source), how to construct a survey (exploratory), or to help explain the results of a survey (follow-up).
More generally, focus groups can support professionals in answering increased accountability (external and internal). Additionally, colleges and universities are awash in quantitative data – everything is counted. Focus groups can help that data tell a more complete story because the experiences of stakeholders are less clear in those instances, as these data tell us nothing about the meanings behind the numbers. The qualitative data from focus groups address the elusive “Why?” question in order to further the understanding of what really counts.
Here are some potential focus group ideas:
- Gain employer feedback
- Determine how well students learned and how well they apply skills
- Develop university strategic plan
- Assess student services
- Evaluate faculty, staff and students’ sense of belonging
- Develop quality enhancement plan
- Examine quality of university life (work/life balance)
How do I conduct a focus group?
There are six steps to conducting a focus group, though please note entire books have been devoted to the topic!
- Develop a proposal, define the purpose, and write the research questions (the questions are the most important).
- Select participants (typically 5-12 participants, but you will also likely run more than one group to get good data), including two moderators.
- Design the Moderator’s Guide.
- Select and train the moderators.
- Conduct the focus groups. Typically these are recorded and moderators also take notes on non-verbal communication, such as nods, hand raising, expressions, etc.
- Analyze the data and report the results.
Again, each step could have detailed articles and chapters devoted to it! For the purposes of getting started, here are some tips for writing good focus group questions related to your research goal.
Do
- Align questions with project goals or research questions
- Use clear, unambiguous language
- Use open-ended questions
- Use questions that ask people to reflect or “think back”
Don’t
- Ask “why?” – instead, say tell me more or elaborate
- Ask leading questions (or if you must, at least use them sparingly)
- Use double-barreled questions where you are asking two things instead of one
Example Questions:
- How have you been involved with…?
- Tell me about the experiences you have had with .…
- When I say …, what comes to mind?
- What impact did … have on …?
- How would you improve …?
- What recommendations would you make to …?
- How did you use …?
How do I analyze the data?
Typically you will have an audio recording of the sessions, which is transcribed and then analyzed along with moderator notes. When looking at the data, identify themes or similar topics, and focus on the frequency and extensiveness of those. Also consider context; comments may be specific to certain situations that were discussed, and it will be your job to infer and contextualize. This is also true for any non-verbal communication from the moderator notes. This analysis should link topics together into “Big Ideas.”
We hope you feel motivated by this overview to refresh your tools this fall! If you’d like additional resources, be sure to check out:
- The AAC&U Value Rubrics
- National Survey of Student Engagement (NSSE)
- Old Dominion Assessment Toolbox
- A Practical Approach to Using Focus Groups in Higher Education (Webinar)
- “Using Assessment Tools Effectively” (Webinar)
- Weave Academy Workshops on Rubrics, Surveys, and Focus Groups
