Hugo Liu, Ph.D.
Chief Scientist,
Hunch.com
Research Affiliate,
MIT Media Lab
hugoathunchdotcom /
hugoatmediadotmitdotedu
selected writing
Social network profiles as taste performances
Hugo Liu (2007): Journal of Computer-Mediated Communication, 13(1), Blackwell Publishing.
This study examines how a social network profile's lists of interests—music, books, movies, television shows, etc.—can function as an expressive arena for taste performance. By composing interest tokens around a theme, profile users craft their "taste statements." First, socioeconomic and aesthetic influences on taste are considered, and the expressivity of interest tokens is analyzed using a semiotic framework. Then, a grounded theory approach is taken to identify four types of taste statements—those that convey prestige, differentiation, authenticity, and theatrical persona. The semantics of taste and taste statements are further investigated through a statistical analysis of 127,477 profiles collected from the MySpace social network site between November 2006 and January 2007. The major findings of the analysis include statistical evidence for prestige and differentiation taste statements and an interpretation of the taste semantics underlying the MySpace community—its motifs, paradigms, and demographic structures.
Small happiness: aesthetic strategies for witting consumersHugo Liu (2007): Trans. Wu Gang. Cultural Review, November, 2007: 32-39.re, Shanghai
Beyond being moral and intellectual, we believe in our heart of hearts that true happiness endures and surpasses all tests of time—a criterion that sometimes proves too challenging for those who stake happiness on but a handful of epic life events. Felicitously, robustness is the strong suit of the small when the small is practiced as a wise ecosystem. Finally, true happiness becomes substantial when it not only endures, but in fact, creates and flourishes. True love ripens like a fine wine, and children make their parents prouder by each year. Yet so too can small happiness flourish and multiply if one believes in the unlimited creative potential of aesthetic multiplicities. With the wisdom of a well-trained imagination, each passing day we will even recognize new beauties in that which is already before us. [English version]
From programming the unconscious to aesthetic technologiesHugo Liu, Paulo Urbano (2007): (interview), Nada 9, Portugal, nada.com.pt
My view of aesthetics is much closer to Freud’s. His insight was that the experience of art and beauty arises when everyday situations cause the unconscious mind to erupt with emotion. Using that revolutionary idea, Freud was able to interpret dreams and laughter—two central problems of aesthetics. Freud’s explanation emboldens computational inquiry into aesthetics because it implies that art and beauty’s sublime are psychological rather than divine; art and beauty are not inferior to the rational, but rather, are circumstances more contextually sophisticated than rationality. In Computational Aesthetics, we have applied many AI techniques to “implement” Freudian psychoanalysis in computer programs and machines. This affords new lines of inquiry into a great variety of aesthetic questions—for example, happiness, taste, imagination, humor, frustration, suffering, irony, myth, intimacy, memories, and morality.
Introduction to the semantics of people and culture (editorial preface)Hugo Liu & Pattie Maes (2007): International Journal on Semantic Web and Information Systems, Special Issue on Semantics of People and Culture (Eds. H. Liu & P. Maes), 3(1), Hersey,PA: Idea Publishing Group.
The human-grown semantic web has already proven its great potential. By researching semantic technologies to exploit real-world system usage, to cope with subjectivity, and to enhance interpretation using cultural context, we can create smarter systems to harness and enhance humans’ intrinsic semantic productivity.
Of men, women, and computers: data-driven gender modeling for improved user interfacesHugo Liu & Rada Mihalcea (2007): Proceedings of the International Conference on Weblogs and Social Media, Boulder, CO, USA.
In this paper, we tried to gain insights into how men and women perceive day-by-day events, and what they most value in their daily experiences, by looking at a very large number of diary entries extracted from the blogosphere. Our analysis of gender distinctions revealed that women's and men's sensibilities exhibited a particularity-generality dichotomy that swept all dimensions of gender space. Women focused on immediate time, nuanced colors, close-knit relationships, objects describable by size, the flavors of food, and were disposed to happiness and sadness. Men focused on months and years, primary colors, social hierarchies, abstract ideas, food as a tool for sating hunger, and were disposed to anger and arousal. These findings are in general agreement with previous research in gender psychology, but do articulate more specific preferences for factors directly relating to user interface design such as time, color, socialness, sizeability and concreteness, affect, and food (if one is designing a recipe interface, for example).
Superconsumer: a postmodern romanceHugo Liu (2006): (trans.), Nada 8, Portugal, nada.com.pt
But where is this in-between and how can a bricoleur find this space? Bhabha gives away the secret—it is located in “those moments or processes that are produced in the articulation of cultural differences.” The bricoleur is accustomed to undermining one culture’s teachings with the teachings of another, but if he were to focus on the difference between the two teachings and let that space of difference fill his imagination, a continuum of a thousands possible teachings would appear. Thus, to be in-between the space of difference affords even greater freedom than rebellion. [English version]
on computational aesthetics
Unraveling the taste fabric of social networks
Hugo Liu, Pattie Maes & Glorianna Davenport (2008): Social Networking Communities and E-Dating Services: Concepts and Implications (Eds: C. Romm-Livermore, K. Setzekorn), 18-43, Hershey,PA: Idea Group Inc. Originally published 2006 in International Journal on Semantic Web and Information Systems 2(1), 42-71.
Popular online social networks such as Friendster and MySpace do more than simply reveal the superficial structure of social connectedness—the rich meanings bottled within social network profiles themselves imply deeper patterns of culture and taste. If these latent semantic fabrics of taste could be harvested formally, the resultant resource would afford completely novel ways for representing and reasoning about web users and people in general. This paper narrates the theory and technique of such a feat—the natural language text of 100,000 social network profiles were captured, mapped into a diverse ontology of music, books, films, foods, etc., and machine learning was applied to infer a semantic fabric of taste. Taste fabrics bring us closer to improvisational manipulations of meaning, and afford us at least three semantic functions.
Computing Point-of-View: Modeling and Simulating Judgments of TasteHugo Liu (May, 2006): Ph.D. Dissertation, Program in Media Arts & Sciences, School of Architecture & Planning, MIT, Cambridge, MA, 163pp.
Our capacity for aesthetics and affectedness is one of the most celebrated bastions of humanity. Underlying our explicit knowledge and rationality is a faculty for judgment—the impulsion to prefer, to view the world through our individual lenses of taste. An interesting intellectual question is: can a computer model a person’s tastes, attitudes, and aesthetics richly enough to predict their judgments? This thesis explores one answer to the question. Our investigation flies under the banner of point-of-view for two reasons. Firstly, the term reflects an understanding that individual tastes are seated in, and articulated against a social and cultural fabric. Secondly, ‘point-of-view’ is developed to mean not isolated taste judgments, but rather, a coherent and systematic apparatus that engenders such judgments.
Rendering aesthetic impressions of text in color spaceHugo Liu & Pattie Maes (2006): International Journal on Artificial Intelligence Tools 15(4), 515-550, World Scientific Press.
What is an artwork and how could a machine become artist? This paper addresses the provocative question by theorizing a computational model of aesthetics and implementing the Aesthetiscope—a computer program that portrays aesthetic impressions of text and renders an abstract color grid artwork reminiscent of early twentieth century abstract expressionism. Following Dewey's psychological interpretation of "aesthetic" and Jung's ontology of fundamental psychological functions, we theorize that a viewer finds an artwork moving and satisfying because it seduces her into rich evocations of thoughts, sensations, intuitions, and feelings. The Aesthetiscope embodies this theory and aims to generate color grids paired with inspiration texts (a word, a poem, or song lyrics), which can be received as aesthetic and artistic by a viewer. The paper describes five Jungian aesthetic readers which are together capable of creative narrative understanding, and three color-logics that employ psycho-semantic principles to render the aesthetic readings in color space. Evaluations of the Aesthetiscope revealed that the program is best at portraying intuition and feeling, and that overall, the Aesthetiscope is capable of creating the aesthetic of art based on an inspiration text in a non-arbitrary way.
Taste fabrics and the beauty of homogeneityHugo Liu, Glorianna Davenport & Pattie Maes (2006): Association of Information Systems SIG SEMIS Bulletin, vol. 3.
Whereas ontology and metadata systems are bloating with semantic relation diversity, a semantic fabric is homogenous in its relation type (they are all affinity scores) and its method of reasoning (spreading activation). Because a semantic fabric represents just one semantic dimension, it can do it exhaustively and yield a very high-resolution resource for making semantic measurements along that dimension. Semantic fabrics re-introduce continuity into an increasingly symbolic and discretized semantic universe, begging us to heed that old adage, “do one thing, and do it well.”
Self-reflexive performance: dancing with the computed audience of cultureHugo Liu & Glorianna Davenport (2005): International Journal of Performance Arts and Digital Media 1(3), 237-247, Intellect Ltd.
Typically performance is a display for others, and is time-limited. But if we also regard everyday life as a performance, we see that it is a continuous improvisation—a multi-faceted dance with an audience that is our social and cultural milieu. In moments of self-reflection, we ourselves motivate this performance, seizing these occasions to explore and debate our relationship to culture and our reflexive situation within it. This article introduces a digitally mediated framework for real-time self-reflexive performance, called the Identity Mirror. Here, the audience is a computational model of culture himself—his moods complex and shifting constantly according to daily happenstance. The mirror shows the performer her dynamic and panoptic reflection against culture, which she can negotiate through dance. The article goes on to unravel the politics of self-reflexive performance—exploring the ideas of cultural persona, facets, and shadows, and gestating a future where these performances can be sustained as a daily dialogic, and co-performances can be had amongst friends.
Synesthetic Recipes: foraging for food with the family, in taste-spaceHugo Liu, Matthew Hockenberry & Ted Selker (2005): Proceedings of the 32nd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2005, Los Angeles.
This paper presents a new answer to one very old question, "What's for dinner?" Synesthetic Recipes is a graphical interface which allows a person to brainstorm dinner recipe ideas by describing how they imagine the recipe should taste (e.g. "hearty, mushy, moist, aromatic"); to keep mindful of the tastebuds of family members, on-screen avatars anticipating their reactions to recipes enrich the brainstorm with just-in-time family’s feedback. The forager clicks on a recipe suggestion, the recipe text is rendered with semantic highlighting such that the essence of their query is intelligible at-a-glance – e.g. you searched for spicy, and now, all the spicy ingredients are highlighted in this recipe view. Of course, deciding what to make for dinner cannot occur in a social vacuum, as the tastes of family members need to be.
The aesthetiscope: visualizing aesthetic readings of text in color spaceHugo Liu & Pattie Maes (2005b): Proceedings of the Eighteenth International Florida Artificial Intelligence Research Society Conference (AI in Music & Art Special Track), 74-79. Menlo Park, Calif.: AAAI Press.
Moorman & Ram’s revolt against the grain of the classical AI narrative understanding literature emboldens us in our task of aesthetic reading, which is the topic of this paper. Aesthetic reading is not reading purely for information. It is an emotionalized and personal reading, whereby the text’s primary purpose is to evoke aesthetic rumblings within the reader. Reading theorist Louise Rosenblatt states, “In aesthetic reading, the reader’s attention is centered directly on what he is living through during his relationship with that particular text” (Rosenblatt, 1978, p. 25); but this notion of “living through” can be quite a complex amalgamate of perceptions and sensations.
InterestMap: harvesting social network profiles for recommendationsHugo Liu & Pattie Maes (2005a): Proceedings of IUI Beyond Personalization 2005: A Workshop on the Next Stage of Recommender Systems Research, January 9, 2005, San Diego, CA, USA, 54-59.
Another confluence feature is a taste clique. Visible in Figure 2, for example, we can see that “Sonny Rollins,” is straddling two cliques with strong internal cohesion. While the identity descriptors are easy to articulate and can be expected to be given in the special interests category of the profile, tastes are often a fuzzy matter of aesthetics and may be harder to articulate using words. For example, a person in a Western European taste-echelon may fancy the band “Stereolab” and the philosopher “Jacques Derrida,” yet there may be no convenient keyword articulation to express this. However, when the InterestMap is learned, cliques of interests seemingly governed by nothing other than taste clearly emerge on the network.
Articulation, the letter, and the spirit in the aesthetics of narrativeHugo Liu (2004): Proceedings of the 2004 ACM Workshop on Story Representation, Mechanism, and Context (SRMC'04), New York.
We posed the aesthetics of narrative as problematics of articulation, the letter, and the spirit. We established the letter as the agency of social language, of the explicit, known, mundane, habituated, and thus, unaesthetic. In contrast, the spirit is the agency of the aesthetic; it is an amorphous, anomic space that is alive with meaning, fraught with creative tension, and home to the unarticulated, unarticulatable, mystified, sacred, and mythical. Whereas the letter is socially constructed and maintained, the spirit arises out of the personal and collective unconscious, its chief vehicle to the realm of the conscious being through the agency of intuition.
on sentiment analysis
A corpus-based approach to finding happiness
Rada Mihalcea & Hugo Liu (2006): Proceedings of Computational approaches for analysis of weblogs, AAAI Spring Symposium, March 2006, 6pp.
Recipe for Happiness: Go shop for something new – something cool, make sure that you love it. Then have lots of food, for dinner preferably, as the times of breakfast and lunch are to be avoided. Consider also including a new, hot taste, and one of your favorite drinks. Then go to an interesting place, it could be a movie, a concert, a party, or any other social place. Having fun, and optionally getting drunk, is also part of the recipe. Note that you should avoid any unnecessary actions, as they can occasionally trigger feelings of unhappiness. Ideally the recipe should be served on a Saturday, for maximum happiness effect. If all this happens on your birthday, even better. Bon appétit.
What would they think? a computational model of attitudesHugo Liu and Pattie Maes (2004): Proceedings of the ACM International Conference on Intelligent User Interfaces, IUI 2004, January 13–16, 2004, Madeira, Funchal, Portugal, 38-45, ACM Press.
A key to improving at any task is frequent feedback from people whose opinions we care about: our family, friends, mentors, and the experts. However, such input is not usually available from the right people at the time it is needed most, and attaining a deep understanding of someone else’s perspective requires immense effort. This paper introduces a technological solution. We present a novel method for automatically modeling a person’s attitudes and opinions, and a proactive interface called “What Would They Think?” which offers the just-in-time perspectives of people whose opinions we care about, based on whatever the user happens to be reading or writing. In the application, each person is represented by a “digital persona,” generated from an automated analysis of personal texts (e.g. weblogs and papers written by the person being modeled) using natural language processing and commonsense-based textual-affect sensing.
Visualizing the affective structure of a text documentHugo Liu, Ted Selker & Henry Lieberman (2003): Proceedings of the Conference on Human Factors in Computing Systems, CHI 2003, April 5-10, 2003, Ft. Lauderdale, FL, USA, 740-741, ACM Press.
This paper introduces an approach for graphically visualizing the affective structure of a text document. A document is first affectively analyzed using a unique textual affect sensing engine, which leverages commonsense knowledge to classify text more reliably and comprehensively than can be achieved with keyword spotting methods alone. Using this engine, sentences are annotated using six basic Ekman emotions. Colors used to represent each of these emotions are sequenced into a color bar, which represents the progression of affect through a text document. Smoothing techniques allow the user to vary the granularity of the affective structure being displayed on the color bar. The bar is hyperlinked in a way such that it can be used to easily navigate the document.color as the mode of representation.
A model of textual affect sensing using real-world knowledgeHugo Liu, Henry Lieberman & Ted Selker (2003): Proceedings of the ACM International Conference on Intelligent User Interfaces, IUI 2003, January 12-15, 2003, Miami, FL, USA, 125-132. ACM Press.
ACM IUI "Outstanding Paper Award" Recipient. This paper presents a novel way for assessing the affective qualities of natural language and a scenario for its use. Previous approaches to textual affect sensing have employed keyword spotting, lexical affinity, statistical methods, and handcrafted models. This paper demonstrates a new approach, using large-scale real-world knowledge about the inherent affective nature of everyday situations (such as “getting into a car accident”) to classify sentences into “basic” emotion categories.
on semantics of natural language
Saurus: an emotionally-weighted thesaurus
Jim Gouldstone, Hugo Liu, Henry Lieberman & Hiroshi Ishii (2006): Proceedings of the AAAI-06 Workshop on Computational Aesthetics, 107-110, AAAI Press. excerpt +
Saurus produces interesting results for larger bodies of text as well. In a speech made in Sept. 2005, U.S. Senator John Kerry criticized the government's response to Hurricane Katrina's devastation of New Orleans: "The incompetence of Katrina's response is not reserved to a hurricane. There's an enormous gap between American's daily expectations and government's daily performance." Had the speechwriter used Saurus to modify the tone with the guide phrase "violent hate", Senator Kerry might have spoken: "The incompetence of Katrina's reaction is not reserved to a hurricane. There's a heinous breach between American's daily prospects and management's daily execution."
NLP (natural language processing) for NLP (natural language programming)Rada Mihalcea, Hugo Liu & Henry Lieberman (2006): A Gelbukh (Ed.) Computational Linguistics and Intelligent Text Processing, LNCS 3878, 319-330, Springer.
Write a program to generate 10000 random numbers between 0 and 99 inclusive. You should count how many of times each number is generated and write these counts out to the screen.
Feasibility studies for programming in natural languageHenry Lieberman & Hugo Liu (2006): Lieberman, Paterno, Wulf (Eds.): End-User Development (Human-Computer Interaction Series Vol. 9), 459-474, Springer.
Pane and Myers conducted studies asking non-programming fifth-grade users to write descriptions of a Pac-Mac game (in another study, college students were given a spreadsheet programming task). The participants also drew sketches of the game so they could make deictic references. Pane and Myers then analyzed the descriptions to discover what underlying abstract programming models were implied by the users' natural language descriptions. They then used this analysis in the design of the HANDS programming language. HANDS uses a direct-manipulation, demonstrational interface. While still a formal programming language, it hopefully embodies a programming model that is closer to users' "natural" understanding of the programming process before they are "corrupted" by being taught a conventional programming language. They learned several important principles, such as that users rarely referred to loops explicitly, and preferred event-driven paradigms.
Langutils: a natural language toolkit for common lispIan Eslick & Hugo Liu (2005): Proceedings of the International Lisp Conference (ILC'2005), Stanford, CA, June 2005.
This paper describes the design and implementation of "langutils,” a high-performance natural language toolkit for Common Lisp. We introduce the techniques of real-world NLP and explore tradeoffs in the representation and implementation of tokenization, part-of-speech tagging, and parsing. The paper concludes with a discussion of the use of the toolkit in two natural language applications.language techniques to identify specific kinds of information within larger texts.
Programmatic semantics for natural language interfacesHugo Liu & Henry Lieberman (2005b): Proceedings of the ACM Conference on Human Factors in Computing Systems, CHI 2005, April 5-7, 2005, Portland, OR, USA, 1597-1600 ACM Press.
Perhaps one reason for the absence of explicit looping in natural language is that there already exists basic linguistic constructions that imply a class of procedure which reasons about sets using relational descriptions (e.g. “sweet drinks” as a subset of “drinks”); these set-theoretic constructions seem to supplant the need to narrate looping constructions explicitly. For example, consider the following utterance and the procedure it implies (expressed in Python): The bartender makes a random sweet drink from the menu. bartender.make(random.choice(filter(lambda drink: ‘sweet’ in drink.properties, menu.drinks)))
Metafor: visualizing stories as codeHugo Liu & Henry Lieberman (2005a): Proceedings of the ACM International Conference on Intelligent User Interfaces, IUI 2005, January 9-12, 2005, San Diego, CA, USA, 305-307, ACM Press.
As a person types a story into Metafor, the system continuously updates a side-by-side “visualization” of the person’s narrative as scaffolding code. This code may not be directly executable, but it is meant to help a person reify her thoughts. We believe that Metafor is a novel system which can accomplish at least two main goals: 1) assist novice programmers in developing intuitions about programming; and 2) facilitate intermediate programmers with a brainstorming and “outlining” tool, ahead of “writing.”
Toward a programmatic semantics of natural languageHugo Liu & Henry Lieberman (2004): Proceedings of VL/HCC'04: the 20th IEEE Symposium on Visual Languages and Human-Centric Computing, 281-282. September 26-29, 2004, Rome, IEEE Computer Society Press.
Natural language is also generic enough to use the same syntax to declare and compute variables, a manner similar to generic functions of the Common LISP Object System (e.g. “Pacman eats yellow dots” can depending on what’s known, declare that dots are yellow, or apply “eat” only to the subset of dots which are yellow). Unlike in most programming languages, the economic and goal-driven nature of story understanding causes evaluation of natural language expressions to almost always be lazy. For example, it may be sufficient to acknowledge that a procedure for generating the “shortest path” exists without actually specifying one.
Unpacking meaning from wordsHugo Liu (2003): Blackburn et al. (Eds.): Modeling and Using Context, LNCS 2680, 218-232, Springer.
No coherent meaning without simulation. In the Bubble Lexicon graph, different and possibly conflicting meanings can attach to each word-concept node; therefore, words hardly have any coherent meaning in the static view. We suggest that when human minds think about what a word or phrase means, meaning is always evaluated in some context. Similarly, a word only becomes coherently meaningful in a bubble lexicon as a result of simulation (graph traversal) via spreading activation (edges are weighted, though Fig. 1 does not show the weights) from the origin node, toward some destination. This helps to exclude meaning attachments which are irrelevant in the current context, to hammer down a more coherent meaning.
on common sense reasoning
ConceptNet: a practical commonsense reasoning toolkit
Hugo Liu & Push Singh (2004): BT Technology Journal 22(4), 211-226, Kluwer Academic Publishers.
ConceptNet is a freely available commonsense knowledge base and natural-language-processing tool-kit which supports many practical textual-reasoning tasks over real-world documents including topic-gisting, analogy-making, and other context oriented inferences. The knowledge base is a semantic network presently consisting of over 1.6 million assertions of commonsense knowledge encompassing the spatial, physical, social, temporal, and psychological aspects of everyday life. ConceptNet is generated automatically from the 700 000 sentences of the Open Mind Common Sense Project — a World Wide Web based collaboration with over 14 000 authors.
Commonsense reasoning in and over natural languageHugo Liu & Push Singh (2004): M Negoita, RJ Howlett, LC Jain (Eds.): Knowledge-Based Intelligent Information and Engineering Systems, LNCS 3215, 293-306, Springer.
For example, what is the precise color of a “red apple?” In logic, we might be able to formally represent the range in the color spectrum corresponding to a “red apple,” but in natural language, the word “red” is imprecise and has various interpretations. Consider the differing colors which map to “red apple” versus “red wine” versus “red hair.” WordNet has tried to address this issue of semantic leakage by imposing boundaries on word called word senses. In many cases, such boundaries are very clear, as in the case of homonyms (e.g. river bank versus financial bank), but in the case of more systematic polysemies (e.g. WordNet has different senses for a short sleep versus a long sleep), it is clear that such boundaries are artificial.
Beating common sense into interactive applicationsHenry Lieberman, Hugo Liu, Push Singh & Barbara Barry (2004): Artificial Intelligence Magazine 25(4), 63-76, AAAI Press.
Things fall down, not up. Weddings (sometimes) have a bride and a groom. If someone yells at you, they’re probably angry. One of the reasons that computers seem dumber than humans is that they don’t have common sense—a myriad of simple facts about everyday life and the ability to make use of that knowledge easily when appropriate. A long-standing dream of artificial intelligence has been to put that kind of knowledge into computers, but applications of commonsense knowledge have been slow in coming.
Teaching machines about everyday lifePush Singh, Barbara Barry & Hugo Liu (2004), BT Technology Journal 22(4), 227-240, Kluwer Academic Publishers.
We were interested in the question of whether it was possible to distribute the problem of building a commonsense knowledge base across thousands of people on the Web, and especially, people with little or no special training in computer science or artificial intelligence. We were interested in whether the ‘average person’ could participate in the process of building a commonsense knowledge base. After all, every ordinary person possesses the kind of commonsense we wish to give our machines!
Goose: a goal-oriented search engine with commonsenseHugo Liu, Henry Lieberman & Ted Selker (2002): De Bra, Brusilovsky, Conejo (Eds.): Adaptive Hypermedia and Adaptive Web-Based Systems, LNCS 2347, 253-263, Springer.
Asociación Española de Inteligencia Artificial "Best AI Paper Award" Recipient One novice user submitted the query: “I want to find other people who like movies,” and obtained many irrelevant and unwanted search results on the topic of movies. In contrast, a more experienced user formed the query: “ +‘my homepage’ +‘my interests’ +‘movies’ ” and was able to get many relevant results. The experienced user chose not only a keyword (“movies”) on the topic of the search, but also two keywords (“my homepage”, “my interests”) differentiating the context in which the topic keyword should appear. In choosing these keywords, the experienced user used her expertise to guide a series of inferences from the search goal.
Semantic Understanding and Commonsense Reasoning in an Adaptive Photo AgentHugo Liu (May, 2002): Master's Thesis, School of EECS, Massachusetts Institute of Technology, Cambridge, MA, 160pp.
We investigated three broad properties of intelligent software agents – communication through and understanding human language; exercising some commonsense to prevent obvious mistakes; and learning from past user interactions to improve future interactions. The ARIA Photo Agent provided an application platform from which approaches to these properties were tested, and what resulted was an intelligent software agent that can automatically annotate photos by understanding the user’s English text; robustly retrieve annotated photos by incorporating commonsense; and learn personal commonsense of the user and use that knowledge to improve future retrieval by knowledge specific to the user.
Makebelieve: using commonsense to generate storiesHugo Liu & Push Singh (2002): Proceedings of the Eighteenth National Conference on Artificial Intelligence, AAAI 2002, July 28 - August 1, 2002, Edmonton, Alberta, Canada, 957-958, AAAI Press.
Makebelieve, an interactive story generation agent that can generate short fictional texts of 5 to 20 lines when the user supplies the first line of the story. Our fail-soft approach to story generation represents a hybrid approach inheriting from both the structuralist and transformationalist traditions. It also incorporates a novel knowledge source, commonsense, which unlike other story knowledge bases, is not specifically purposed for story telling. Using a subset of knowledge in Open Mind, which describes causation, Makebelieve performs fuzzy and creative inference to generate casual chains, which become the basis for a storyline.
Robust photo retrieval using world semanticsHugo Liu & Henry Lieberman (2002): Proceedings of the LREC 2002 Workshop on Creating and Using Semantics for Information Retrieval and Filtering: State-of-the-art and Future Research, Las Palmas, Canary Islands, 15-20, LREC Press.
In our photo domain, we propose a mechanism for robust retrieval by expanding the concepts depicted in the photos, thus going beyond lexical-based expansion. Because photos often depict places, situations and events in everyday life, concepts depicted in photos such as place, event, and activity can be expanded based on our “common sense” notions of how concepts relate to each other in the real world. For example, given the concept “surfer” and our common sense knowledge that surfers can be found at the beach, we might provide the additional concepts: “beach”, “waves”, “ocean”, and “surfboard”.
Adaptive linking between text and photos using common sense reasoningHenry Lieberman & Hugo Liu (2002): De Bra, Brusilovsky, Conejo (Eds.): Adaptive Hypermedia and Adaptive Web-Based Systems, LNCS 2347, 2-11, Springer.
In user testing, we saw not only that ARIA adapts to the user, but that the user adapts to ARIA. Often a user's typing will bring up some photos relevant to the user's current text, but that also trigger the user's memory, encouraging him or her to explain related pictures in subsequent text, triggering new picture retrieval. This mutual adaptation is an important characteristic of adaptive systems, and our users particularly liked the continual interplay between their story and ARIA's suggestions.