I wrote these articles for the course “Affective Computing” while pursuing a master’s degree in Human-Centered Computing at the University of Maryland, Baltimore County.
Recently, there have been many who have dived deeper into encapsulating emotions in technology. One such experiment is AffectAura which is a sensory array that continuously predicts the user’s affective state. The study was conducted to capture and assess emotional states over a period of time and review and reflect on them. The principle used to investigate such reflective properties of past experiences were triggers – or incomplete but sufficient details of events, such as labels and tags. The first user survey produced expected results such as users’ inability to remember events from the past week or month. There were also some additional triggers such as people, places, activities, weather, music and outdoor activities that were often used to recall emotional experiences. One of the most widely accepted notions of valence and arousal is stated in research for AffectAura (McDuff et al., 2012). Valence whether positive or negative affects decision making, and arousal has a significant impact on memory. AffectAura was developed to capture life logs tagged with a user’s affective states regarding valence, arousal, and engagement through their life experiences. This was basically to better understand people’s reactions to a retrospective memory through affective experiences.
There were various pieces of equipment such as a webcam to recognize facial expressions, a Kinnect to record and recognize posture, a microphone, an EDA sensor to record electrodermal activity, a GPS sensor and so on. Machine learning was used to recognize the user’s affective signals for valence, arousal and engagement factors. Visualization for these was represented as follows, arousal was depicted using shapes and engagement with opacity. This was chosen because there could be many iterations for size, shape, opacity and color. Icons were used to show the user’s location, e.g. home, office or an activity such as a meeting. This helped users to remember past events. The forms in the visualization consisted of detailed descriptions for the specific activities, it could show calendar events, people’s names, document names and so on. A summary of the predicted affective states was also included for display.
The activities of the six participants were recorded from Monday to Thursday during a working week. There were surveys at the end of each day, and a semi-structured interview each subsequent week for participants to report on their emotional experiences. The results of the study showed that people remembered recent interactions better compared to older people. The results also showed that people generally misremember tones as positive rather than negative in most interactions. There were noticeable pieces of evidence that people’s unpleasant memories fade faster than positive ones. From the collected observations, it can be said that people can suffer memory loss in emotional tones in as little as 24 hours. Through the analysis and interview period, the researchers realized that people are not interested in logging their daily or ordinary interactions, but rather they get happy from remembering an unusual event. The technology could be used to record interactions with people that trigger an atypical emotional response. Interviews also showed that people did not really remember past events with only effective cues, but reasoned about the affective information after reconstructing their past experiences. However, this also leads to the danger of creating false memories by rationalizing the information gathered.
In another research by D’Mello et al. (2007) investigated an affect-aware tutoring system to improve learning. AutoTutor was developed to help students learn Newtonian physics and computer science topics by engaging students in dialogue while answering questions. The first step towards building this was to develop a system that could correctly recognize a reaction to a student’s affective state. This led to the creation of an affective loop where the supervisor’s actions are maximized while affecting the student’s effect. The method of collecting this information was through a video of the participant’s faces, their posture patterns, and audio and video of the entire tutoring session.
When collecting information about the students’ different states when using AutoTutor, it was difficult for the judges to correctly identify “frustration” compared to when the students themselves used the emote-aloud technique to show their emotions. This may be due to the social pressure not to feel negative emotions freely compared to positive emotions. Students had a chance to see another student’s experience with AutoTutor and rate their session. The most common state was “neutral” followed by confusion, perhaps due to the content of the training session. Analysis of the results showed that students experienced more emotional episodes during the latter part of the session when they were unable to answer AutoTutor’s questions or took a long time to do so. They also experienced more emotion when receiving negative feedback.
Posture was also used to detect and measure affective states. It consisted of placing a thin film pressure pad on the seat and back of the student chairs. The relationship between attitude and affect confirmed the hypothesis that affective states are usually accompanied by some level of physiological changes. Through this it was seen that “boredom” was not associated with inactivity but rather with fidgeting. The accuracy of using posture to detect affect changes yielded the same result, i.e. 70 percent as detecting using dialogue. Using facial expressions to classify emotions was successful in detecting more animated emotions such as joy compared to those expressed subtly. In the future, AutoTutor may need to enable students to enter a circle of curiosity and confusion, away from boredom and frustration to encourage effective learning. It may also need to anticipate and proactively try to prevent the onset of negative emotions before they occur.
#technology #predict #emotions
