Evaluation Methods Overview
Once you have created a piece, it is important to objectively evaluate "how it was perceived" and "how it was experienced." This section introduces the two main evaluation methods used in this course.
Why Evaluate?
Subjective feelings like "I think it turned out well" or "it seems interesting" are valuable, but they are not enough on their own. Evaluation helps you:
- Check if your intent came through -- If you aimed for a "scary atmosphere" but people found it "fun," that is a clue for improvement
- Identify weaknesses -- Numbers and graphs make shortcomings visible
- Communicate with others -- Instead of "it's kind of good," you can explain with data
What We Measure
In this course, we primarily measure two things:
Impressions
We evaluate the impressions a work evokes using multiple pairs of adjectives. Each pair represents opposing qualities, such as "bright vs. dark" or "flashy vs. subdued."
Measured with the SD Method (Semantic Differential Method)
UX (User Experience Quality)
We evaluate the quality of the experience of using the work. UX has two major dimensions:
| Quality | Meaning | Example |
|---|---|---|
| Pragmatic Quality | Usability and clarity | "The controls were intuitive" / "I understood how to use it right away" |
| Hedonic Quality | Enjoyment, appeal, and novelty | "It was exciting" / "A completely new experience" |
Measured with the UEQ-S (User Experience Questionnaire - Short version)
Other Evaluation Methods (Reference)
This course primarily uses the SD method and UEQ-S, but the following methods are also commonly used in research:
| Method | What It Measures | Characteristics |
|---|---|---|
| SAM | Emotion (valence and arousal) | Respondents answer using illustrations for intuitive responses |
| AttrakDiff | UX (pragmatic and hedonic quality) | The precursor method to UEQ-S |
| POMS2 | Mood (negative aspects) | Widely used in medical and psychological research |
| Semi-structured interview | Experience details | Prepare questions in advance and follow up based on responses |
| Text mining | Trends in free-text responses | Analyzes word frequency and co-occurrence |
Going Deeper
Ways to deepen your evaluation:
- Compare variations -- Prepare different versions of your work (different sounds or effects) and compare results
- Compare media -- Headphones vs. speakers, monitor vs. projector
- Compare with existing works -- Compare against similar concept works or applications
- Factor analysis -- Statistically extract impression categories from SD method results