Those who are programmers will tell you that the abstractions of computation are a tool chest of metaphors for understanding processes of all kinds. Unfortunately programming is not really taught until high school and college. The reasons for this are not good ones. The syntax and semantics of various programming languages tend to difficult to learn and to gain intuition about. But what if we could program in plain English? This is the dream of programming in natural language.

With Metafor, we are just beginning to scratch the surface of this dream. While full and complete understanding of language is a concern of "story understanding" and is often regarded as AI-complete, we are making the assumption that natural language can be constrained to simple constructions and inputs, without harming the intuitiveness of input in natural language. The major insight gained by Metafor is that natural language IS programmatic in nature. Thus, we simply need to transliterate this language into a traditional programming language.

While traditional programming languages are beginning just now to be dynamically typed, natural language was always dynamically typed, by part-of-speech, syntax, semantics, pragmatics, and common sense. In Metafor, if you declared that "there is a bin of apples," it is automatically understood that bins can hold things and is a container object like a list. English as a programming language is also much more concise than any traditional programming language. If i said, "Look in the bin and pick out just the red apples," that's the equivalent of programming: "map(Pick, filter(lambda apple: apple.color == red, bin.getApples()))."

Metafor doesn't construct whole programs. It builds just the scaffolding for a program. What visualizing scaffolding does is to help beginner programmers and non-programmers intuit programming. Perhaps young children wh play with Metafor can glean programmatic insights that will fill their tool chests with new metaphors for problem solving. That's our hope.

The Aesthetiscope is an interactive art installation whose wall of color reacts to portray the relationship between some idea (a word, a poem, a song) and a person (a realist, a dreamer, a neurotic) standing before it. Each idea, for example the word sunset, is rich in association for a person. Perhaps he remembers in his mind what a sunset looks like. Or a sunset mades him think of other ideas like warmth, fuzzy, beautiful, serenity, relaxation. Perhaps it reminds him of some past event in his life. The contextual sphere of these personal associations form the Aesthetic about the idea. And the experience of that aesthetic is called its pathos. I wanted to choose a medium through which pathos could be convincingly portrayed, and so I chose colors because they are a complete microconsciousness of pathos, like taste and smell.

The Aesthetic is hard to articulate because it is usually experienced it as an undeconstructed gestalt. Any analysis of Aesthetic needs to be sensitive to its complexity -- the multi-dimensional nature of connotation. The aesthetiscope analyzes each idea through a multi-perspectival linguistic analysis of connotation. The realms of analysis are "Think," "Culturalize," "See," "Intuit," and "Feel." Each of these realms brings to bear a different perspectival vocabulary to the pathos description of an idea. "Think" generates rational connotations and entailments of the idea. "Culturalize" looks at the cultural entailments of the idea through the lens of a particular culture. "See" takes the idea as a source of imagery, bringing to bear our collective visual memory of objects, places, and events. "Intuit" is an exercise in automatic free assocations with the idea as a cue. "Feel" takes a sentimental stance toward the idea, connecting it to a word of feelings. The results of these analyses are mapped to a world of colors through psycho-physiocological color surveys based on the work of Berlin & Kay, and Goethe, and naturalistic sampling of colors from photos.

With these different vocabularies of aesthetic, we can try to make sense of a "sunset." A sunset may be "Seen," revealing the dark purple swatches with splashes of warm hues that characterize the visual rememberance of a sunset. But there is also an inner sunset. A sunset "Felt" and "Intuited" recalls warmth, beauty, and serenity, and these will bring about brighter, warmer, and more intense colors than the outer sunset.

The aesthetiscope encourages us to experience and reflect on Aesthetic in a new way.

What if you could look in the mirror and see not just what you look like, but also who you are?

The Identity Mirror is an augmented evocative object. Looking into it, the viewer's face is painted over with identity keywords and interest keywords, sourced from a deep model of the viewer's identity. The identity model is computed automatically from a viewer's social network profile or webpage, using the InterestMap. For instance, the viewer specifies that he listens to "Kings of Convenience" and enjoys the fiction of Vladmir Nabakov, and using this, InterestMap situates the viewer within its multiple neighborhoods of taste. The keywords which paint over the viewer's face represent his context within taste-space.

By gazing into Identity Mirror, a viewer can glean his identity-situation. Is his hair out of place? Are one of his interests out of place? How do his facial features combine to compose a gestalt? How do his various interests come together to compose an identity or aesthetic gestalt?

Identity mirror reifies its metaphors in the workings of an ordinary mirror. When the viewer is distant from the object, a question mark is the only keyword painted over his face. As he approaches to a medium distance, larger font sized identity keywords such as "fitness buffs", "fashionistas", and "book lovers" identify him. Approaching further, his favorite book, film, and music genres are seen. Closer yet, his favorite authors, musicians, and auteurs are known, and finally, standing up close, the songs, movies, and book titles become visible.

In ongoing research, we're developing further two particular aspects. IdentityFixing and the Diderot Effect -- keywords are distributed between a hearth (keywords aesthetically co-consistent) and a periphery (outlier keywords seemingly out-of-place about a person); the hearth covers the face, the periphery covers the hair; the viewer can use his hands to adjust his hair -- he can dishevel those unwanted periperhal keywords, or accept them by packing them into his hair. A person with a strong degree of taste-coherence has ruly hair, whereas a postmodernist with scattered interests has unruly hair. The second aspect under development is temporality -- the image of the viewer will reflect the viewer in relation to the goings on of the world and of his life. Since the viewer has many facets, various facets can be teased out by biasing InterestMap with contemporaneous keywords of current worldly and lively goings-on.

"What's for dinner?"

In this work, we explore a technological answer to that famous question. Few of us know with great certitude the exact food we crave, but instead, we stew on the question and explore the nature of our craving through imaginative descriptions: "I feel like something light, fresh, sophisticated, not too mushy -- something influenced by thai or indian ingredients, something aromatic." Synaesthetic Recipes is a visual search program which allows such imaginative textual descriptions, and uses these to drive recipe recommendations. In the backend, a database of 100,000 recipes are automatically annotated with common sense about food. An artificial intelligence robotic reader reads each recipe and based on tastes of the ingredients and the types of cooking procedures, predicts how a food will look, taste, and smell. We are translating recipes into the rich descriptive vernacular of how people naturally conceptualize their cravings for food.

Over recent years, social network communities (e.g. inter alia, friendster, contact lists, weblog communities, newsgroups) have been steadily building up in the online world. There is now sufficient critical mass of such infrastructure to postulate things about identity and the Self, as reflected in the social fabric of the online world.

How might we sense identity from the online social fabric? Semiotician Jacques Lacan has argued that words and concepts carry meaning primarily by what they signify. Roland Barthes proposed that the particular mappings between signifier and signified originate in cultural systems of semiology. Such cultural systems of signs have grown in importance as mass consumption has replaced subjective culture as the dominant contemporary cultural paradigm. Because the signifying value of possessions associated with the Self serves an important social function in signalling identity, it is possible to view identity and the Self as a collection of consumption decisions (cf. Social Constructionist Theory of Identity).

InterestMap mines online social network communities to create a rich influence network of interests and subcultures. Some of the interests represented in the network are, inter alia, television and films, foods, geographies, music, sports, hobbies, activities, objects, and people. The strength of connections between interests are learned from the digestion of on the order of one hundred thousand user profiles from online social networks. For example, a person who likes X may also mention Y as an interest on her homepage. Identity and tastes emerge as patterns of intersection on InterestMap.

Currently, InterestMap contains 100,000 interest and subculture nodes, trained with over 100,000 thousand profiles. It is currently used as an interest engine, driving serendipitous recommendation and social introduction facilitation in the Ambient Semantics project, but it certainly has far broader implications.

We also see InterestMap as a novel kind of social recommendation mechanism. Whereas recommendation systems traditionally works within a single application domain of interest such as books or music, InterestMap represents users using a much larger vocabulary of interests (movies, music, television shows, foods, sports, hobbies, passions) and ways of describing people (identity labels e.g. "raver", geographical locations, etc). The result is a multi-dimensional model of a user that is portable across domains; it can be used by Amazon, or any other conceivable vendor. More importantly, it has the potential to instruct computers to understand and describe people as people do.

A key to improving at any task is frequent feedback from people whose opinions we care about: our family, friends, mentors, and the experts. However, such input is not usually available from the right people at the time it is needed most, and attaining a deep understanding of someone else’s perspective requires immense effort. This work introduces a technological solution. We have developed a novel method for automatically modeling a person’s attitudes and opinions, and a proactive interface called “What Would They Think?” which offers the just-in-time perspectives of people whose opinions we care about, based on whatever the user happens to be reading or writing. In the application, each person is represented by a “digital persona,” generated from an automated analysis of personal texts (e.g. weblogs and papers written by the person being modeled) using natural language processing and commonsense-based textual-affect sensing. In user studies, participants using our application were able to grasp the personalities and opinions of a panel of strangers more quickly and deeply than with either of two baseline methods. This research has exciting theoretical and pragmatic implications to intelligent user interfaces.

What is the essence of guacamole? The word's etyma tells us vaguely that it's an avocado (guaca) mixture (mole). But as with words, the practical meaning of guacamole is the sum of its myriad interpretations and uses. Out of a collection of 160,000 recipes collected from the web, many of which were clipped out of cookbooks and food magazines spanning the decades, we find 6200 distinct variants on a guacamole recipe. Applying machine learning and language processing to these recipes, the ingredients and procedures of guacamole are deconstructed into perceptual and functional descriptions. Avocado's creamy palate and richly umami profile lend it the quality of crab and egg. It also serves as a binding agent for herb-aromatic juicy bits like tomato, citrus zest, and onion. Based on the statistical variance across guacamole recipes, avocado's taste profile is almost indispensible to the essence of the dish, while its binding function is completely indispensible. Pea puree and chickpeas on rare occasion substitutes for binding function, but they don't fully replace avocados' rich umami profile. Herb-aromatic juicy bits accept the most variation--mango easily replaces tomato, scallion easily replaces onion, cilantro is discretionary. Garlic is least easily replacable. Out of these automatic deconstructions of guacamole, a machine can build a gestaltic conception of guacamole's essence.

Guacamole is but one example. Gulp fiction has modeled the essences of over 40,000 ingredients, dishes, cuisines, procedures, flavors, diets, and everything in-between. Gulp fiction remixes and recombines these influences into new experimental recipes-- all the user need supply is an evocative name for the new concoction. Gulp fiction is a dynamic cookbook for those whose sixth basic taste is imagination.

In the affective visualization I created for CHI'2003, I chose to represent the emotional slices of a story as a visually sequenced color bar. Left to right corresponds to story progression from beginning to end. The colors code for Paul Ekman's six basic emotions of happy (orange), sad (purple), angry (red), fearful (pink), disgusted (green), surprised (blue).

Stories have a thematic structure to them, and the text of stories are often well-annotated according to thematic structure, such as via chapters, a table of contents, et cetera. However, at a deeper level of meaning, stories can also be thought of as having many other structures. Affective structure may be a significant dimension of meaning because so much of interpretation is concerned with the engagement of our emotions. A plot does not climax without an emotional peak. Unlike thematic structure, affective structure is rarely made explicit in a story's organization. More likely than not, it emerges out of imagination and textual interpretation. Even though it is implicit, we still use it as a socially shared organizer and shared index in search and navigation, because we can expect that others will glean similar affective structures (for there is great culturally-driven commonality of human experience and emotion).

What Poseidon's Eye does is to allow the computer to read through stories for affective structure, and share this structure visually with people, who may find this preliminary and coarse reading of emotional subtext useful as an indexing and navigation tool. For example, perhaps you may want to read a book with the same kind of emotional rollercoaster as the book you've just finished reading. Or perhaps you want to skip to the climax of the book. Or perhaps you are a script writer and you want to compare the emotional developments of your characters against those in another script. One university English classroom is even using Poseidon's Eye to cultivate critical analysis. The professor says: "Poseidon's Eye says this chapter is angry. What did Tolstoy intend?"

Poseidon's Eye is backed by an emotion and sentiment analysis engine which uses a novel method for appraisal of events for a more comprehensive textual affect sensing machinery. Summarization of affective structure into increasingly larger chunks is accomplished with a bayesian classifier trained on conditional mutual information.