MOMEX
Thesis
A slightly expanded version of the voiceover text for my video follows:
My name is Eric Dunstan Christopher Schubert, and I’m a poet, writer, creative technologist, and media artist. I’m a storyteller creating lyric and narrative immersive experiences in 3D and mixed-reality spaces, in addition to being an author, filmmaker, and bookmaker. My work as a designer concerns the expression of identity in digital space, both as it pertains to memory and storytelling—both individual and collective—and how digital spaces and places impact our sense of self and well-being.
When I began embarking on my thesis work, I was influenced by Walter Benjamin’s 1936 essay, “The Work of Art in the Age of Mechanical Reproduction.” In a nutshell, the essay argues that photography and film lack the aura of works of art made by hand. (The essay speaks more broadly about the cultural role of artists, the artefacts they make, and the role of religion and politics in art making. But the relationship between the aura and mechanical reproduction is most connected to my thesis project.) My response to Benjamin’s thesis is that the longer we’ve had to practice the techniques of photography and filmmaking, the better we’ve been able to imbue those works with aura. And the longer we’ve lived as connoisseurs of film and photography, the more soul we’ve found in them.
My critical position is that our digital tools, spaces, and places emphasize a distancing lack of dimensionality, driven by their development as tools of productivity and consumption and our relative inexperience creating with them. The prevailing industry bias is toward the reproduction of photorealistic high-definition imagery, and even our methods of crafting experiences in virtual and mixed reality depend mainly on the conversion of photorealistic 2D objects and scenes into a simulation of a 3D world. From Zoom conferences and metaverse interactions to first-person fighter games to colorful graphic delights made in augmented reality or with AI, the bulk of digital artefacts lack aura or soul. And we lack a sense of full human embodiment when we use these tools or experience digital places and spaces. (This is not to say that all digital creations lack aura. Some of the most soulful digital works of the current moment are made through creative coding, such as work by Zach Lieberman or Casey Reas. Perhaps the transfiguration of math and text into moving images is an alchemical process.)
I believe there is an opportunity to craft digital technologies, places and spaces that better enable embodied and soulful human experiences. In developing my thesis project, I have considered alternative ways to build immersive experiences that allow representations of the self featuring aspects of us typically occluded by digital technologies—including our emotions, thoughts, relations, memories, histories, and dreams. In doing so, I have been influenced by esoteric art practices such as spirit and aura photography as well as the holistic practices of the Bauhaus—particularly the way in which leaders of the school responded to the social fragmentation, alienation, and horrors of the Industrial Revolution and World War I with an insistence on a holistic approach to creation (the total work of art or Gesamtkunstwerk) and an emphasis on the union of mind, body, and spirit as the basis for making such work. My design work has also been influenced by my study of esoteric healing practices—specifically Traditional Chinese Medicine, herbalism, and homeopathy (which I studied in a two-year professional training program) and my research into theories of quantum physics, particularly the holographic principle of the universe. These practices and theories also underscore the importance of holistic approaches to living and making and emphasize the connection of mind, body, and spirit.
With MOMEX, I propose a speculative interface for storing, replaying, and sharing momes—moments and memories crafted into 3D augmented reality. The proposal illustrates possibilities for virtual, augmented, and mixed reality that engage us holistically—in emotional and psychological ways that impact our sense of self and well-being, whether for healing or forging a deeper connection with digital content or the people we interact with in digital places and spaces. Like the inspiration for its name—Vannevar Bush’s Memex, an early proposal for an electromechanical information retrieval device—MOMEX is intended to mirror the “associative trails” of human memory, demonstrating alignment between human mental processes and digitally created immersive experiences. While the user can call up specific momes, the MOMEX interface displays a stream of associated momes rather than a productivity-driven filing system. As with memories, an instructional mome that reminds one of a recipe might bleed into a mome of watching one’s grandmother make that particular recipe.
For this project, I have designed the MOMEX interface and a collection of momes—ranging from the instructional to the emotional to the narrative—that represents potential use cases, including:
The MOMEX interface is supported by apothecARies. In the classical definition, an apothecary was a medical practitioner who distributed medicine. AR apothecaries are designers who craft momes with the help of pervasive capture, generative AI, and narratives their clients provide.
To craft the momes you see in this demonstration, I created large, expansive 3D models of environments made with the help of drone photography, desktop 3D photogrammetry applications, and Lidar-based phone photogrammetry. The models are edited and refined in 3D modeling applications Blender and Cinema 4D. Momes have been deployed into AR applications such as Lens Studio and Spark AR or the Unity game engine and composited with supplemental video and sound. Layered and transparent effects have been created using AI tools, GPU shaders and volumetric capture (using the DepthKit application) to foreground the noisy, holistic, implicit qualities of our embodied reality that high-definition digital technologies tend to occlude.
MOMEX and the constructed momes are intended to speculate applications for health and well-being, such as memory augmentation to recall specific functional tasks or for meditative or psychological processing of life events. The project might also inform new methods for crafting immersive entertainment with greater emotional and narrative depth. Finally, MOMEX has applications for the training and representation of AI-generated digital twins of individuals or other beings, such as pets.
As I develop my design practice, I am interested in creating augmented and mixed-reality immersive experiences that impart some of the emotional and narrative depth we find in books, television, and film. On a parallel path, I hope to partner with specialists in neuroscience and psychology to inform proposals and prototypes for using augmented memories in healthcare applications. Finally, I hope to explore methods for using crafted memories and moments from our embodied, physical reality to advance artificial intelligence.
My name is Eric Dunstan Christopher Schubert, and I’m a poet, writer, creative technologist, and media artist. I’m a storyteller creating lyric and narrative immersive experiences in 3D and mixed-reality spaces, in addition to being an author, filmmaker, and bookmaker. My work as a designer concerns the expression of identity in digital space, both as it pertains to memory and storytelling—both individual and collective—and how digital spaces and places impact our sense of self and well-being.
When I began embarking on my thesis work, I was influenced by Walter Benjamin’s 1936 essay, “The Work of Art in the Age of Mechanical Reproduction.” In a nutshell, the essay argues that photography and film lack the aura of works of art made by hand. (The essay speaks more broadly about the cultural role of artists, the artefacts they make, and the role of religion and politics in art making. But the relationship between the aura and mechanical reproduction is most connected to my thesis project.) My response to Benjamin’s thesis is that the longer we’ve had to practice the techniques of photography and filmmaking, the better we’ve been able to imbue those works with aura. And the longer we’ve lived as connoisseurs of film and photography, the more soul we’ve found in them.
My critical position is that our digital tools, spaces, and places emphasize a distancing lack of dimensionality, driven by their development as tools of productivity and consumption and our relative inexperience creating with them. The prevailing industry bias is toward the reproduction of photorealistic high-definition imagery, and even our methods of crafting experiences in virtual and mixed reality depend mainly on the conversion of photorealistic 2D objects and scenes into a simulation of a 3D world. From Zoom conferences and metaverse interactions to first-person fighter games to colorful graphic delights made in augmented reality or with AI, the bulk of digital artefacts lack aura or soul. And we lack a sense of full human embodiment when we use these tools or experience digital places and spaces. (This is not to say that all digital creations lack aura. Some of the most soulful digital works of the current moment are made through creative coding, such as work by Zach Lieberman or Casey Reas. Perhaps the transfiguration of math and text into moving images is an alchemical process.)
I believe there is an opportunity to craft digital technologies, places and spaces that better enable embodied and soulful human experiences. In developing my thesis project, I have considered alternative ways to build immersive experiences that allow representations of the self featuring aspects of us typically occluded by digital technologies—including our emotions, thoughts, relations, memories, histories, and dreams. In doing so, I have been influenced by esoteric art practices such as spirit and aura photography as well as the holistic practices of the Bauhaus—particularly the way in which leaders of the school responded to the social fragmentation, alienation, and horrors of the Industrial Revolution and World War I with an insistence on a holistic approach to creation (the total work of art or Gesamtkunstwerk) and an emphasis on the union of mind, body, and spirit as the basis for making such work. My design work has also been influenced by my study of esoteric healing practices—specifically Traditional Chinese Medicine, herbalism, and homeopathy (which I studied in a two-year professional training program) and my research into theories of quantum physics, particularly the holographic principle of the universe. These practices and theories also underscore the importance of holistic approaches to living and making and emphasize the connection of mind, body, and spirit.
With MOMEX, I propose a speculative interface for storing, replaying, and sharing momes—moments and memories crafted into 3D augmented reality. The proposal illustrates possibilities for virtual, augmented, and mixed reality that engage us holistically—in emotional and psychological ways that impact our sense of self and well-being, whether for healing or forging a deeper connection with digital content or the people we interact with in digital places and spaces. Like the inspiration for its name—Vannevar Bush’s Memex, an early proposal for an electromechanical information retrieval device—MOMEX is intended to mirror the “associative trails” of human memory, demonstrating alignment between human mental processes and digitally created immersive experiences. While the user can call up specific momes, the MOMEX interface displays a stream of associated momes rather than a productivity-driven filing system. As with memories, an instructional mome that reminds one of a recipe might bleed into a mome of watching one’s grandmother make that particular recipe.
For this project, I have designed the MOMEX interface and a collection of momes—ranging from the instructional to the emotional to the narrative—that represents potential use cases, including:
- An instructional follow-along mome of a daily ritual of making tea
- A mome of one’s life partner and pets asleep in bed to replay while sleeping away from home on a business trip
- Momes of the self, whether of the self giving encouraging feedback during a diary session or momes of emotional moments infused into the present
- Momes of treasured or traumatic narrative events
The MOMEX interface is supported by apothecARies. In the classical definition, an apothecary was a medical practitioner who distributed medicine. AR apothecaries are designers who craft momes with the help of pervasive capture, generative AI, and narratives their clients provide.
To craft the momes you see in this demonstration, I created large, expansive 3D models of environments made with the help of drone photography, desktop 3D photogrammetry applications, and Lidar-based phone photogrammetry. The models are edited and refined in 3D modeling applications Blender and Cinema 4D. Momes have been deployed into AR applications such as Lens Studio and Spark AR or the Unity game engine and composited with supplemental video and sound. Layered and transparent effects have been created using AI tools, GPU shaders and volumetric capture (using the DepthKit application) to foreground the noisy, holistic, implicit qualities of our embodied reality that high-definition digital technologies tend to occlude.
MOMEX and the constructed momes are intended to speculate applications for health and well-being, such as memory augmentation to recall specific functional tasks or for meditative or psychological processing of life events. The project might also inform new methods for crafting immersive entertainment with greater emotional and narrative depth. Finally, MOMEX has applications for the training and representation of AI-generated digital twins of individuals or other beings, such as pets.
As I develop my design practice, I am interested in creating augmented and mixed-reality immersive experiences that impart some of the emotional and narrative depth we find in books, television, and film. On a parallel path, I hope to partner with specialists in neuroscience and psychology to inform proposals and prototypes for using augmented memories in healthcare applications. Finally, I hope to explore methods for using crafted memories and moments from our embodied, physical reality to advance artificial intelligence.