The article I selected for this critique is Benefits of emotional design in multimedia instruction by Richard E. Mayer, and Gabriel Estrella. In this study Mayer and Estrella (2014) attempt to determine whether or not emotional design in multimedia instruction has an effect on student learning. The research lists two hypothesis.

Hypothesis 1. The major prediction in this study is that students who receive multimedia lessons with graphics based on emotional design principles (enhanced group) should perform better on a learning outcome test (i.e., a comprehension test in which they write explanations) than students who receive the same lesson without emotional design (control group).

Hypothesis 2. A secondary prediction is that the enhanced group will report more effort and less difficulty than the control group, although the use of subjective self-report items to measure these variables provides only a limited and preliminary test.

Participants and Design: The design of the student included two experiments. In experiment 1, students received an 8-slide multimedia lesson on how viruses cause a cold. They were time limited to 5 minutes for viewing the lesson. The study uses a multiple group design with an enhanced group and a control group. The enhanced group received instruction using a lesson that has been enhanced with graphics that are designed according to emotional design principles. The control group, was given the same set of slides, with the same time limit, however, their graphics were not enhanced to include emotional design principles. In experiment 2, students received the experiment, with the same slides, and the same instruments, but they were not restricted by time.

Instruments: The study uses three distinct instruments including a participant questionnaire, a post test, and a post questionnaire. The participant questionnaire is designed to gather basic demographic information as well as set a baseline prior knowledge score for each participant. The post test is designed to determine what the student learned from the multimedia lesson. The post questionnaire was designed to gather information about the students’ perceived experience in participating in the lesson. The article includes very specific detail about each instrument including instrument design, exact questions, and scoring methods.

Results: Hypothesis 1 addressed the question of whether adding emotional design features in a lesson could improve learning gains after the lesson. This study shows that the enhanced group performed significantly better than the control group on the learning test in Experiment 1 and Experiment 2, but they did not perform significantly better than the control group on the transfer part of the test. In Experiment 1 and in Experiment 2, the groups did not differ significantly on their mean rating for appeal of the lesson, or for their desire for more similar lessons.

Hypothesis 2 addressed the questions of whether emotional design had an effect on learners’ effort and difficulty in completing the lesson. The results of the study show that neither of the two groups in Experiment 1 or Experiment 2 showed significant differences in the way they rated their experiences of affect, effort or difficulty.

Is the definition of the population to be sampled consistent with the level of generality claimed by the researcher?

There were two experiments in this study. Both experiments used randomly selected students from available research subject pools. All subjects were college students. Groups are fairly balanced as far as gender is concerned, although there are more women (0.76 & 0.28) than in the general population. There is no mention of cultural or social-economic diversity. Mayer and Gabriel suggest that additional research is needed to determine how emotional design might affect younger populations.

Is the method of sampling adequately described? How is it described?

The sample design consists of a participant questionnaire, a post-test and a post-questionnaire. The experimental design is

R O₁ X O₂ O₃

R O₁ O₂ O₃

In experiment group one, 64 college students were recruited from the Paid Subject Pool at University of California, Santa Barbara, who were paid $10 for their participation. The mean age was 19.5 years, the mean knowledge score based on the participant questionnaire was 3.9 out of 10, the average class standing was sophomore, and the proportion of women was 0.76. Thirty students served in the treatment group, and 34 served in the control group. Participants were randomly assigned to the treatment group.

In experiment group two, 47 college students were recruited from the Psychology Subject Pool at the University of California, Santa Barbara, who received course credit for their participation. The mean age was 18.6 years, the mean knowledge score based on the participant questionnaire was 4.9 out of 12, the average class standing was freshman, and the proportion of women was 0.68. Twenty-three students served in the treatment group, and 24 served in the control group. Participants were randomly assigned to the treatment group.

Up to 3 participants were tested in a session. Each participant was seated in an individual cubicle out of sight of other participants. Steps for experiment are as follows:

1. Experimenter described the study and participants read and signed an informed consent form.
2. Participants complete the participant questionnaire at their own pace.
3. Participants viewed the multimedia lesson corresponding to their treatment group. Five minute time limit for group 1, unlimited time for group 2. Study time was recorded for group 2.
4. Participants were given the retention sheet for 4 minutes.
5. Participants were given each of the five transfer sheets for 2.4 min. per sheet.
6. Participants completed the post-questionnaire at their own rate.
7. The entire session lasted about 30 minutes.

To what extent does the sampling design affect the generalizability of the findings? If a nonrandom sample is being used, is there an indication of how this is justified and how results are generalizable?

Gabriel and Mayer indicate the study may be criticized on the grounds of ecological validity. This is based on the fact it is a short-term study involving a 5-min (or shorter) lesson and an immediate 15-min post test conducted in a lab environment. This raises the question of setting. Their recommendation is to extend the research to a more authentic learning environment. Additional research is also suggested to determine how much time is necessary to process the presented material in a way that yields strong test performance, and to determine how emotional design might affect younger populations.

References

Kline, R. B. (2009). Becoming a Behavioral Science Researcher. New York, NY: Guilford.

Mayer, R. E., & Estrella, G. (2014). Benefits of emotional design in multimedia instructions. Learning and Instruction, 12-18.

Trochim, W. M. (2009). Research Methods Knowledge Base. Retrieved September 15, 2014, from socialresearchmethods.net: http://www.socialresearchmethods.net/kb/intmult.php