Affordance-HCI Research
#Machine Learning
#Physiological Sensors
#VR/AR
MIT Fulbright Scholar, Researchers
Guzden Varinlioglu, Han Tu
YH. LI
ZH.TIAN
DigitalFutures
CONTEXT + INTELLIGENCE
My Contribution:
concept development, research, 360 site context analysis, proofreading materials, build apk. file, questionnaire design, testing and interview, GSR set up, control experiment process
Tool:
Oculus Quest 2
Agisoft Metashape
Unity
Arduino GSR
Adobe Premiere
Collaborated with:
Timeline:
June 2023- Sept 2023
A Crowdsourcing Tool for Physiological and AI Data Collection in VR
We see how the virtual environment helped rebuild the Notre Dame de Paris, devastated by a fire, restoring the forms, shapes, materials, and textures. In the game, the architecture of buildings and cities shadows the real world and this time, surprisingly, even has an impact on the existing structures. The built environment in VR seems to have more meanings than illusions and veneers. People bring baggages from the real world — the physical structure of the city, the cultural implications and the perception of self.
How can we empower the context in virtual environments with our understanding of the real world?
Overview
Thesis
How does the conflict between environment and sound affects memory, concentration, judgment level, and emotional change?
Abstract
Individuals possess a strong innate ability to perceive environmental information. This means that when an object or a environment's functionality in line with people's sensory expectations, it results in a consistent perception of their surroundings. Conversely, if the visibility of an object or environment feature contradicts its actual purpose and is consistent with the way people perceive such things in real life, or if it alters their natural understanding of objects and environments. So how that might affect their emotional state? Our design approach focuses on dissecting people's instinctual responses to this situations and examining the depth of their concentration and judgment.
Key Word
affordance theory | spatial experience | sensory information
Methodology
What are Affordances?
An affordance is what a user can do with an object based on the user’s capabilities.
An affordance is, in essence, an action possibility in the relation between user and an object.
VR environment Design
We choose a city environment and a nature environment and it sound sources as our dependent variables, and MIT BAKER ROOM as independent variables.
Hypothesis
Exterior environment variables
Sound variables
Interior room variables
In interior spaces, affordance methodology examines how different sounds impact the perceived uses of that space. Does an open plan office afford focus work and conversations when phones are ringing and keyboards are clacking loudly? Background sounds communicate context and environment type to users before they even enter a space. The buzz of voices may afford a welcoming, lively social environment. The sound of vehicle engines and city noise outside affords a downtown, urban context. Familiar interior sounds can create continuity between the exterior and interior environment.
MIT BAKER ROOM
Experiment Design
Reading sections from Data Cities: How Satellites Are Transforming Architecture And Design by Davina Jackson
Data Cities explains how flows of data will inform our future behaviors in physical, virtual, and hybrid-reality situations, and architecture and cities are being reinvented as not merely static structures, but places that pulse.
Proofreading materials
We add proofreading materials in each VR environments. The purpose is to test how environment and sound affects memory, concentration, judgment level, and emotional change. Although the room variable are the same, but the setting inside the room are different
VR environment Setup with Meta Quest 2 + Unity Testing
Discontinuity between exterior and interior sounds can potentially cause perception dissonance that negatively impacts users. If outdoors sounds peaceful but indoors reverberates loudly, the mismatch might produce user stress and discomfort. Affordance methodology aims to choreograph complementary interior and exterior sounds.
Affordance methodology also considers the important relationship between sound and the interior and exterior environment. Sound is an integral aspect of how people perceive spaces and possibilities for action within those spaces.
GSR-skin sensor + Arudiono IDE Setup
In addition to the insights gained from the experiment regarding people's instinctual perception and emotional responses to the visibility of functionality in objects and environments, the inclusion of Galvanic Skin Response (GSR) sensor data in the VR experiment provides valuable quantitative information. The GSR sensor data captures changes in participants' skin conductance, which serves as a physiological indicator of emotional arousal.
Questionnaire
We conducted a research questionnaire that combine a mix of close-ended questions and open-ended questions, mostly about their feelings and experiences in VR.
Get Ready for the experiment
“Thank you for interesting and accepted for our research experiment！”
Step 1: You will need to sign a consent form and fill out the Google form about your basic information(name, age...)
Step 2: Choose one of the VR environment (4)
Step 3: Put on VR devices and GSR senor
Step 4: 2-3mins (exploring and get familiar with the vr environment), 2-3 mins complete the proofreading task
☺Adjust your headset until you see clearly
☺Be relax
☺A screen record will be on for your vr process
Step 5: take off the headset and GSR sensor
Step 6: fill out the Google form about your feeling
Step 7: user interview
you will watch your recording and tell about what you did, what you think and what you feel during the VR process.(not what you saw because I know how the interior and exterior look like, act as if you are alone, speaking to yourself as you solve the task.) you need to keep talking during you are watching
Step 8: get ready for next VR experiment!
If the process goes smoothly, probably will take you 15-20 mins for each VR environment. Thanks for you time!
Collection
&
Analysis
I interviewed 8 people, and they are…
During VR experiment…
Interviewing…
Collecting…
They said…
GSR Data Comparison
If the visibility of an object's functionality contradicts its actual functionality, the GSR data may show an increase in skin conductance, indicating heightened emotional arousal such as surprise, confusion, or frustration. Conversely, when the visibility aligns with the actual functionality, the GSR data may indicate lower levels of emotional arousal, suggesting a more stable and positive emotional state.
The analysis of GSR (Galvanic Skin Response) data in different VR environments reveals significant variations and the influence of external factors. In C2, where the environment is peaceful and calm, the flow of GSR data gradually decreases. This suggests that the participants' memory is not significantly affected since they find the sound pleasant. However, their attention is drawn to the outside view, resulting in an increase in GSR flow during the proofreading task. Towards the end, the flow returns to a calm state. Contrasting with C2, C1 presents a scenario where the sound and outside view are loud, leading to a greater impact on the participants. Surprisingly, despite the loud stimuli, the participants feel normal because the scenes match their expectations. This contradiction between the environment and sound may have adverse effects on memory,
In the case of C4, the scenes do not match the participants' expectations, resulting in a lack of calmness. Additionally, the loud sound further exacerbates the negative impact on memory test results. This suggests that when there is a conflict between the environment and sound, the participants' ability to concentrate, make judgments, and experience emotional changes may be significantly affected. It is important to consider the role of the VR environment and the GSR sensor in understanding how the conflict between the environment and sound affects various cognitive and emotional aspects. By analyzing the GSR data, we can gain insight into the relationship between external stimuli and their impact on memory, concentration, judgment level, and emotional states. This knowledge can contribute to a deeper understanding of human cognition and inform the design of more effective VR experiences.
In the context of the mentioned study, the VR environment and the accompanying sound can be seen as affordances. The peaceful and calm environment in C2 affords a sense of tranquility and relaxation, which may enhance the participants' ability to concentrate and retain information. On the other hand, the loud and mismatched environment in C4 may afford a sense of discomfort and distraction, negatively impacting memory and concentration.
Conclusion
In this study, when the environment and sound are not the same, it generate a heightened sense of curiosity in individuals. This research sheds light on the people's instinctual perception of environmental information. Our experiment recognizes the significant emotional changes that can occur when the visibility of an object or environment's functionality contradicts its actual functionality, thereby disrupting users' visual and auditory cognition or altering their inherent understanding of objects and environments.
Gibson, J. J. (1980). The ecological approach to visual Perception. The Journal of Aesthetics and Art Criticism, 39(2), 203. https://doi.org/10.2307/429816
Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., & Carey, T. (2003). Human-Computer interaction. In CRC Press eBooks. https://doi.org/10.1201/9780367804787
References
Reflections
Numbers of samples I will continue to interview more people for this experiment.
Model variables: I did 3D scanned another room model, however, the file was too big to bring into the VR devices, we may bring other variables add-on to interior environment.