Initial thoughts on photo-narrative creation

In 2014, carrying maps marked with waypoints and a backpack laden with supplies, I embarked upon a long distance solo hike through rural Turkey. It did not go as planned, and I spent days grappling with questions of loneliness and purpose arisen from just under the surface of the everyday routine. After chancing upon new faces and walking away from my isolation, I was scrolling through photos on my camera’s LCD and jotting down observations and realizations, when I began to feel that there might be a structure to the experience that I had not seen while enveloped within it. The ensuing process of extracting a narrative from my images, frustrations and feelings, shaping something coherent from it, and sharing this story proved immensely cathartic. This catharsis was not short lived. It helped me understand that the photographs I’ve taken are more than a jumble of prints in a shoebox — they are a story captured in slices, and the composition of these fragments allows me to share that story with others.

Today, software universally presents photos in a kind of virtual shoebox. However, our oral reportage of events is not simply an unfiltered timeline of facts, but an evaluation of those facts1, and creating narratives from experiences is a critical part of personality building. 2 Building coherent narratives from life experiences, especially difficult or negative experiences, is strongly associated with greater life satisfaction and reduced risk of depression, 3 and it is now a common approach in psychotherapy to assist a patient in this process. It’s time that photo management tools were built for nurturing organic story telling, not sterile card-cataloging.

I intend to create emotionally aware data guided interfaces for assisting the transformation of personal media into narratives for self-reflection and sharing. From these prototypes I will study how technology designed to encourage narrative construction and sharing from life events might increase life satisfaction and well-being. The Media Lab’s pioneering work in playful and physical user interfaces, affective computing, and technology enabled storytelling make it an obvious place to pursue this research, and its integrative and demo-focused culture perfectly complements my hands-on background and multidisciplinary skill set.

I have broad experience implementing projects ranging from control systems to computational needlework. I am as comfortable writing code as essays and technical documentation, and have done significant amounts of all three during my academic career at MIT. My MEng thesis work with Professor Dennis Freeman at the Research Laboratory of Electronics involved a multi-year, self-managed effort to design, build, and test a new optical imaging apparatus capable of measuring motion within biological tissues on the scale of a single hydrogen atom. This research used a technique known as Doppler optical coherence microscopy, similar to laser vibrometry, and involved elements of optical design, mechanical fabrication, software development, and signal processing. Through this experience, I also developed the project management skills necessary to maintain a sustained research effort, including scheduling, real-time documentation, and budgeting.

After completing my MEng I sought work that could more directly impact on the way that people perceive and document the world. I found the opportunity to merge my photographic interest and engineering skills at the world’s largest photography company, Apple, Inc. In my year with the Camera Systems team, I have both been directly responsible for leading research on a successful computational photography feature, and have worked with a large and multi-disciplinary team to create and validate a second. These investigations have been motivated by product development but their exploratory nature has allowed me to probe beyond hardware and manufacturing into photogrammetry, neural networks, and statistical analysis — all skills which will enable me to begin prototyping my vision immediately. While my work at Apple has been extremely rewarding and involved research in many interesting directions, Apple’s intense secrecy limits discussion and I am not content to restricting my societal contribution to the development of the latest gadget. Furthermore, as a research endeavor my goals could be pursued as an open-source project conscientious of personal privacy, rather than as a commercial product that might lead to perverse data monetization incentives.

My specific interests in this endeavor connect three broad areas: data analysis of personal media for importance and interrelation, emotionally aware interfaces that encourage narrative creation from images, and prototyping and studying social systems for narrative sharing. The untenability of manual narrative creation from large quantities of unedited media has long been recognized. As smartphone camera use continues to increase and always-on wearable technology makes the jump from fantasy to product, this deluge of data will become even more overwhelming. While early projects such as Mathew Laibowitz’s SPINNER relied heavily on external sensors to interpret visual information, 4 diverse and detailed metadata is now available without external hardware from smart phones and fitness bands. Even simple metadata, such as time and location, can express surprising depth. Images as points in spacetime reveal places that the photographer has been, where they linger, and where they hurry through. The route the photographer has traveled can suggest a journey, the method of travel, a way of seeing. The availability of more complex data, including nearby friends, biometrics, and messaging activity, as well as the success of modern semantic image analysis, promises a richer diversity of perspective. While work has also shown the potential of analyzing images for aesthetics and style, this research has yet to be applied to a user interface. 5, 6 To assist the narrative creation process, analysis of media should draw on all of these pieces of information.

This image analysis is interdependent with the presentation of these conclusions to the user for creation and sharing. One possibility, a conversational interface, might ask the user directly about the experience to be organized. Analogous to ELIZA’s surprisingly deep simplicity, 7 this could allow more personal connection than might occur with a more traditional data entry interface. Hardware will also play an important role in the creation of playful interfaces and spatially distributed interactions,. For example, a digital newspaper that also asks about photos from last weekend’s hiking trip might make photo sorting less of a chore. In any implementation, algorithmic analysis of emotionally sensitive content carries the potential for severe mistakes and must be employed with care. Asking questions rather than applying labels allows the software to approach its assumptions with humility, creating opportunities for stories to grow.

I believe that social use of narrative-enabling technology can influence happiness and self-perception of one’s life trajectory. Social networks, through which most online media is shared, present photographs and stories against a curated highlight reel of hundreds of acquaintances’ lives. This carries a significant emotional toll. While many researchers have connected time spent on social networking with deleterious effects on well being, 8 9 10 11 others have found that specific use, particularly honest presentation and disclosure, has positive correlations. 12 13 A recent study analyzing the role of envy in social networks suggests specific interface design patterns that have particular psychological impact on users. 14 I plan to explore how tools designed for honest narrative creation and truly personal sharing can promote healthier and more meaningful use of social media.

My research goals, which span computer vision to psychology, and will require time and interdisciplinary collaboration to successfully realize. The experience I’ve gained from MIT and Apple leave me well positioned to immediately build real prototypes, and the uniquely protean nature of the Media Lab is the ideal incubation environment. Viewing photos and listening to stories, even our own, can be an intensely empathetic experience and I firmly believe technology has a role to play in multiplying and distributing this empathy. In Susan Sontag’s seminal work, On Photography, she laments that people now “seek to have their photographs taken — feel that they are images, and are made real by photographs.”15 This feeling may be here to stay. But perhaps, we can make the images reflect reality.


  1. Labov, W. & Waletzky, J. (1966). Narrative Analysis: Oral Narratives of Personal Experience. In J. Helm (Ed.), Essays on the Verbal and Visual Arts: Proceedings of the 1966 Annual Spring Meeting of the American Ethnological Society (pp. 12-44). Seattle, Washington: University of Washington Press. 

  2. McAdams, D. (2008). Personal Narratives and the Life Story. In O. Johns, R. Robins & L. Pervin (Eds.), Handbook of Personality, Theory and Research (pp. 242-262). New York, New York: Guilford Press. 

  3. Baerger, D. and McAdams, D. (1999). Life Story Coherence and its Relation to Psychological Well- Being. Narrative Inquiry, 9(1). doi:10.1075/ni.9.1.05bae 

  4. Laibowitz, M. (2010). Creating cohesive video with the narrative-informed use of ubiquitous wearable and imaging sensor networks (Doctoral thesis). Retrieved from DSpace@MIT, 1721.1/57695 

  5. Lu., X, et. al. (2014). RAPID: Rating Image Aesthetics using Deep Learning. In Proceedings of the 22nd ACM International Conference on Multimedia (pp 457-466). New York, New York: ACM. 

  6. Karayev, S., et. al. (2014). Recognizing Image Style. In M. Valstar et. al. (Eds.), Proceedings of the British Machine Vision Conference 2014. doi:10.5244/C.28.122 

  7. Weizenbaum, J. (1966). ELIZA — A Computer Program for the Study of Natural Language Communication Between Man and Machine. Compuational Linguistics, 9(1). pp. 36-45. 

  8. Kross, E. (2013). Facebook use predicts declines in subjective well-being in young adults. PLoS One, 8(8). doi:10.1371/journal.pone.0069841 

  9. Denti, L., et. al. (2012). Sweden’s Largest Facebook Study. Gothenburg Research Institute. Gothenburg, Sweden: University of Gothenburg. 

  10. Rosen, L., et. al. (2014) Is Facebook creating “iDisorders”? The link between clinical symptoms of psychiatric disorders and technology use, attitudes and anxiety. Computers in Human Behavior 29(3). pp. 1243-1254. 

  11. Wenninger, H. et. al. (2014). Activity matters: Investigating the influence of Facebook on life satisfaction of teenage users. In Proceedings of the 22nd European Conference on Information Systems. 

  12. Reinecke, L. & Trepte S.(2014). Authenticity and well-being on social network sites: A two-wave longitudinal study on the effects of online authenticity and the positivity bias in SNS communication. Computers in Human Behavior, 30. pp 95-102. 

  13. Kim, J., Chung, N., & Ahn, K. (2014). Why people use social networking services in Korea: The mediating role of self-disclosure on subjective well-being. Information Development 30(3). pp. 276-287. 

  14. Krasnova, H., et. al. (2015). Why Following Friends Can Hurt You: An Exploratory Investigation of the Effects of Envy on Social Networking Sites among College-Age Users. Information Systems Research, 26(3). doi:10.1287/isre.2015.0588 

  15. Sontag, S. (1977). On Photography. New York, New York: Picador. 

About Vignette

Vignette is an open source project investigating new ways of encouraging narrative creation from personal media. Vignette will be exploring data-driven methods for managing large quantities of photos and videos, visualizations that highlight the importance of place in narrative, and interfaces that engage users conversationally and emotionally.