University Place

Welcome, everybody, to Edgewood College. I'm very happy to introduce Dr. Ferrine Spector, assistant professor in psychology here at Edgewood College. I'm Joan Schilling, chair of the psychology department, and I'm delighted to have Dr. Spector for my colleague. Tonight she is going to give our talk,

Multisensory Perception

The Brain's Original Hack. Dr. Spector earned her Bachelor of Science degree in psychology at Trinity College in Hartford, Connecticut, and her doctorate in psychology at McMaster University in Hamilton, Ontario, with a focus on multisensory perception in development. Here at Edgewood, Dr. Spector is busy exciting students about research in neuropsychology. Her students love her for her enthusiasm for her subject and her passion in teaching it to them. Please welcome Dr. Ferrine Spector. (audience applauds) Golly, hi everybody. So everybody was asking me if I was nervous today, but I see a lot of really friendly faces and I want to thank the honors students of Edgewood College for being here, and of course all of my friends and neighbors who are showing up for support. Thank you, Joan, for that introduction, and Wisconsin Public Television for coming to Edgewood College to film us. So hi, everybody. Joan actually asked what I meant by hack, and I thought that was interesting. We use that term so casually and I attempted to describe it appropriately by saying, "I think it's when you take something for which there is already an infrastructure, something that already exists, and you exploit it for your purposes. I think that's about right. And what I wanna start out, I want you to start out by thinking, and you can go ahead and draw, of course, I will give you more clear instructions as we go on, but you've got paper and crayons, draw away. But I wanna start out by thinking about something. I want you to think about the idea that every single piece of knowledge that you have ever had and you will ever have comes in through your sensory systems. Every single one. Think about that. And you're like, "Oh no, "there's stuff I knew when I was born." That's not knowledge. One might argue that's instinct. What is knowledge? Well, conveniently, definitions are easy to find. "Facts, informations, and skills acquired "by a person through experience," emphasis mine, "or education; "awareness or familiarity gained by experience." Where do you get that experience? We take our senses for granted, but some people argue that, and I might sometimes argue, that they're, in fact, more important than almost anything. And so where does this knowledge come from? And now I'm gonna show you a picture of a baby in the womb. This is how they show it to you when they're showing you pictures of babies, but actually it looks more like this. In the beginning it is dark. (audience laughing) But actually, (whooshing sounds) it's dark, but it's not quiet. And, well, if you live in my house, probably sounds a little bit more like (people talking) people talking in the background. Oh yes, of course. And so while you're not seeing anything, you are immersed, as a fetus, you are immersed in a rich auditory world. In addition, there's lots of tactile information that happens there too. And so a really important thing, and something that is a theme throughout the rest of this talk, is the idea that experience drives brain development. We are not passive receivers of information. We actively move through the world, and that active movement through the world actively shapes how our brain develops. So what am I showing you here? It's a highly-schematized graph that represents the difference between auditory development and visual development in the beginning of life through the first four years of life. And what you'll notice, I'm not supposed to move around very much, that's a little hard for me, but what you'll notice is that our auditory abilities are actually quite well developed when we're born, and our visual abilities are not. And you could probably start to guess why. Guess what? We're not using our eyes in the womb. I mean, we're opening them and we're closing them and we're getting random visual activity and things like that, but we're not seeing anything. There is no experience for us to be developing that system. And then we open our eyes, and all of a sudden, and that's the blue line, all of a sudden our visual sense gets really, really good and surpasses our auditory abilities, simply because we've seen. Well, now we're getting the experience, and that experience is driving our visual development. Our auditory system is going about its business in a sort of a standard, linear trajectory. So what do babies know? I mean, what is that knowledge we're born with? Because not all things that we're born with are necessarily instinctual. Guess what? Babies are born with a preference for their mother's native language. Babies are born with a preference for their mother's voice over the voice of any other woman. Babies are born with some kinds of auditory memories. That's knowledge, they have learned that. Your mother's native language was not programmed in your genes at conception. Okay, so experience drives brain development. And there are two different ways that we like to think about this. We have experience-expectant brain development and experience-dependent brain development. Experience-expectant development is the property of a system whereby, if given certain experiences that are typical, the system will develop optimally. So if you think about experiences that are typical for humans, you're talking about what? You're talking about language, you're talking about lights and movement and faces, by the way. Faces are the coolest things to babies, the coolest things. Ditch the mobiles, right? Ditch all the black and white things, just make faces at them, just talk to them. Have that, it's the most interesting thing they can imagine. Well, that's experience-expectant development. You don't have to do anything special to get the visual system to develop in an experience-expectant way. Or the language system, just immerse the child in an environment that is rich with language. Experience-expectant. Experience-dependent development, however, is the property of the system that there are variations, and what variations you experience, how your developmental outcomes may be more individual. So while every human can expect to be immersed in a linguistic environment, what language you learn is experience-dependent, how that drives your brain development. Okay, good, great. And so what does it look like, in the beginning of life, in the beginning, I am talking about the beginning, because that's the root, that's the root of everything. So what you're looking at here is this is a graph of the development of the number of synapses, synapses are the connections between neurons in your brain. Throughout we've got newborn, two months, six months, and all the way up to 70 years. On that vertical axis is synaptic density. And I want you to just see the pattern that I am seeing. What happens is, when a baby is born, all of a sudden he opens his eyes and there's all this stuff, and then there's all these connections, right? Connections and connections, they proliferate. And then they start to get pruned away. We don't need them all. But we might. And so, in other words, well, I'll make it, I'll make a scheme test for you. In other words, there's baby, right? Newborn baby, there's its brain, there's some connections. And then, very soon after birth, wow, you get so many. Golly, I mean, it's a cacophony. There's so much activity going back and forth that it's actually really hard to make sense of the world. Has anybody ever observed a newborn? They do a lot of, right? They do a lot of that. Well, you can understand why. Right? Now, eventually, a little bit later in the first year of life, as the baby gains experience with the world, the unnecessary connections get pruned. They're gone. You don't need them anymore. And you can start to see that not only do sensory areas become differentiated, but baby's behavior becomes differentiated. And they can respond very differently. An eight-month-old baby is very different from a newborn. But even then, and there, a little bit later in life, what we have done is we have... A brain, we didn't do that, actually, excuse me.

What your brain has done is

it has said, "But I didn't really know what it was gonna be like out there, "so I'm just gonna throw everything at it. "But now that I have moved through the world, "now that I have gained experience with the kinds "of things that I can expect to see, "I'm gonna get rid of the connections that I do not need anymore. I am going to maintain the connections that I do need because I want my brain to be an efficient machine. And this is a really important thing about brain development. The process of the developing brain is not more connections, but fewer. That's cool, in my opinion. And so how do we know this? How do we know that all these connections in infancy are functional? Well, lucky for you, there is a whole lot of evidence that suggests that. For example, if you tickle a baby's wrist, and they love that, when you tickle a baby's wrist you actually see activation in their brain not only where you would expect it, and where I would see it in all of your brains if I tickled your wrist, but also over the auditory cortex, which is kinda funny. And then, if you play a sound at the same time, do you know what happens? Their response to that wrist tickle is greater. It summates, it adds together. They're working together. But that's not gonna happen to you. If I tickle your wrist, and it would be fun, I mean, you can come to my office, I'll just give you a little wrist tickle. If I tickled your wrist and played sound at the same time, that's not gonna amplify the signal. Well, you're just gonna have some activation in your auditory cortex, and you're gonna have some activation where that gets processed, somatosensory cortex, it's called. It's not the wrist-touch cortex, that'd be kind of a cool thing. For babies, human speech activates auditory and visual areas of the brain. By the time you're three years old, it activates only auditory cortices. What is happening is the brain is becoming more differentiated, the senses are becoming more differentiated. Not only that, throughout development in general, and what you see, I know, you like to throw up some brain pictures and that makes everybody go, "Woo, brain pictures," but what we're actually seeing in blue is we're seeing the difference, we're seeing a reduction in brain activity over a number of years of development. And so a process of development is the process of diffuse brain activity, so activity over a large amount of the cortex, and it moves from diffuse to focal. So it starts with lots and lots of things being activated all the time, and then as you develop, as you gain active experience with the world, you start to have visual information stimulates the visual area, auditory information stimulates the auditory area, et cetera. But what that means, and see, here's funny, 'cause I'm sitting here, I'm telling you about differentiation of sensory areas, and I'm talking about multisensory perception. (whimpers) That's kind of a funny. But you see, those connections, the unnecessary ones got pruned. But don't you want some connections to remain? Don't you want to be able to seamlessly integrate sensory information? We've only gotten rid of the unnecessary ones, not all of them. And so as it turns out, the brain starts out multisensory. The brain starts out profoundly multisensory. In fact, the suggestion is that infants experience all sensory information in a completely undifferentiated way. They are not distinguishing between sights and sounds and touches, just as they're not really distinguishing between being hungry versus being scared versus being wet in the beginning of their lives. They're just like, "I am unhappy or I am fine." By the way. But if you think about this, the brain's default is to start out multisensory... And then gradually differentiate. Now, that's different than starting out differentiating and then having to actively integrate. There's a difference. And so this is really nice, because guess what? Not everybody's experience is expected. Not everybody's brain develops in a experience-expectant way. That's just a placeholder for me to make sure I talked about these things. For example, if you are born without one of your senses, what happens if you are born blind, for example? All those rich, strong connections connecting your visual areas to the other part of the brain, you don't want those to go away. Your visual cortex is not gonna get any information from your eyes. It's not gonna be able to process the information from your eyes. And guess what? People who are born blind or who go blind very early in their lives, the actual official term is early blind, what we're seeing here, what I've circled, up at the top we have blind adults, on the bottom we have sighted adults. What I have circled is I have circled visual cortical activity in response to braille reading. Oh. They're seeing with their hands, and that's no joke. That is how the visual world is being represented via touch. Now, imagine how difficult that would be if their brains started out unisensory. Differentiated sensory areas, and then they had to actively create those connections. Doesn't it make more sense to have them to begin with? And that's what the brain's doing. Again, got all those connections 'cause it's like, "I don't know, what's it gonna be like out there? "Am I gonna be raised in a cave? "Am I going to have hearing? "Or are my parents going to speak five languages? "Is my room gonna be blue?" I mean, that probably doesn't matter. This activation of the visual cortex, it correlates with tactile acuity. Have you ever felt braille? It's really hard to read it if you are untrained. And, of course, you could get trained, but if you already have more of your brain dedicated to touch, then you're gonna be better at it to begin with. So this activation, it's not only functional and measurable, but it actually is correlated to better ability, better tactile ability in general. Individuals who are born deaf, what we're seeing here, we have magnitude of activation in the auditory cortex, okay, so in the part of your brain that is dedicated to hearing. Now remember, a person who is born deaf is not going to have had hearing information feeding into that. So in white we have deaf individuals, in gray we have hearing individuals. When you present them, when you present deaf individuals with a, excuse me, if you present them with a visual stimulus, you see activation in their auditory cortex. And notice, over here on the right, when you present hearing individuals with an auditory stimulus, you see activation in the auditory cortex. That is what you would expect, right? So you're seeing the same patterns for people whose brain development is not experience-expectant. It is, in fact, experience-dependent. And they are missing some fundamental experiences and so their brains develop a little bit different. Yeah, if we didn't start out fundamentally multisensory, it would be very difficult to navigate through the world... With a deficit of this variety. But guess what? If I took any one of you sighted individuals and blindfolded you for five days, anybody want to volunteer? Blindfolded you for five days, what I would start to see, and that's what I've circled there, on day five I would start to see activation in your visual cortex when asking you to do something with your fingers, to feel something, to do a tactile task. And that activation, although not nearly as strong as if you had spent your entire life visually deprived, it does in fact correlate with increased tactile ability. So if I blindfolded you for five days and taught you how to read braille, you would get better at it, and that would be correlated with the amount of activity I'm seeing in your visual cortex. What does that mean about all those connections? They're not gone. They're not all gone. It's just the ones you don't need. Well, most of the time you don't need them until you do, and then you want them to be there so that you can continue to navigate through the world effectively. So what that means is experience-expectant and dependent brain development allows the baby to navigate in a world in which it can expect to experience integrated input. That's the experience-expectant part. But it also sets the stage for specific multisensory experience, experience-dependent multisensory experience. We think about things as being, "Okay, "well this is vision and this is hearing and this is touch." That is how we all learned about the senses. Now, some very extreme multisensory researchers will say there are no differentiated senses. They don't exist. I'm not sure I'm quite that extreme, but that's an idea that's out there. Because guess what? Every experience is inherently multisensory, and the world teaches us that. Did you know we all associate size to pitch? Larger objects make lower-pitched noises. We also associate size to loudness. Larger objects are in fact louder. We have learned that from maneuvering through our world, active experience. But it also helps that color indicates edibility in food. And not only that, but when you're looking at someone's face, you can understand what they're saying a little bit better. You ever noticed that? Every experience is inherently multisensory. I'd challenge anybody to think about a single, isolated sensory experience that doesn't involve any element of another one. We'll just see that guy again, he's so cute. Hello, hello. And that's why you have to turn your radio down when you're trying to see while you're driving. There's interference. So how does this influence our behavior and our experience? And that's what I get really interested in, and it's all very, very cool. And how does it influence our behavior? Well, we all know what a loud tie is. Right? And we all know what sharp cheddar is... Even though it doesn't actually cut us. We're really good at using multisensory metaphors. We don't even realize how good we are at using them. We're like, "Yeah, loud tie? Psht!" "Yeah, my uncle had one of those. "It was hideous." Well, you just used a multisensory metaphor to describe something that you have probably thought of as fundamentally visual. Sharp cheddar, you could probably all explain why that cheddar is sharp. I mean, we are from Wisconsin, right? And you could probably explain why that cheddar is sharp. "Well, it's something about the..." That one's a little harder than the loud tie, (laughing) to be perfectly honest. But the point is is that we really seamlessly integrate language and sensory information, and putting it all together without even realizing it. And that's what got me started when I started thinking about multisensory perception. I was like, "I feel like multisensory perception "is the default "and not the, what's the word, "exception," excuse me. I don't have a script in front of me. Okay, okay, fine. So this is how it influences our experience as we move through the world and we use language and we think about everyday things like loud ties and sharp cheeses. But what about in the lab? Okay, fair enough. In the lab, how does multisensory perception influence our experience? I've got some examples. Let's think about concurrent stimulation. So concurrent stimulation is when you've got stimulation of two senses at the same time. Now, I'm gonna give you a lot of examples from the audiovisual domain. This would be the point where I would ask you the question of why do you think that we have so many examples from the audiovisual domain? I'm not gonna ask you that question because we're filming. But it has to do with the dominance of our senses. Vision is, for sighted individuals, our dominant sense, and hearing does not fall too far behind. But moreover, if you think about how much audiovisual integration there is right now, you are listening to me and you are watching me talk. In fact, if you looked away, you would be less good at understanding my words, even though you think about speech as something that is fundamentally auditory. It's not. And so the audiovisual interactions are the ones that we're gonna see most often. There's this really neat one called the illusory flash effect. Someday I'd love to find an effect and give it a catchy little name like this, the illusory flash effect. What happens with the illusory flash effect is when a single flash is presented at the same time as two short beeps. What do you think I'm gonna say? You perceive multiple flashes, even though when it was presented with one beep you perceived one flash, when it was presented with no beeps you perceive one flash. When it is presented with two beeps, you perceive multiple flashes. Hmm. When we play noise and light together, concurrently, our perception of how intense the sound is, we actually perceive it as more intense even though the sound is actually the same intensity as it was when it was presented alone. But when we present it with a light, all of a sudden we perceive the noise as being more intense. Another really neat one. You ever notice sometimes you hear something, but you're not sure if you hear it? It's like, "Did I hear that?" That's usually something that's near threshold. You may have heard it, you may not have, sorta depends on all your other factors and your intentional factors and all of those things. But if you take a light that is just around your threshold, that means just where you are able to perceive it, and you take that light that you may or may not be able to perceive, and you pair it with a noise, then all of a sudden you are going to perceive that light far more than not. Usually if it's near threshold it's sort of maybe 50% of the time when you add the noise, and all of the sudden it's 75%-90% of the time that you are actually perceiving that light. Although, what is really cool too, and I love this. The illusory flash effect also counts for touches. So when a single flash is presented along with tactile stimulation in the form of taps, again, if you've got a single flash and two taps you're gonna perceive more than one flash. And so it's not just audiovisual, it's just that a lot of our examples are audiovisual. But this is clearly a vision to touch example. And there are many, many more, as I said here, but I wanna get to what I find most interesting, basic matching. You see me, I am gleefully rubbing my hands together because this is where I think it gets really fun. You know, it's all well and good to bring people in the lab and show them flashes and lights and tones and things like this, but I really like to just kinda ask people what they think, how they match things, I like them to draw pictures. I see lots of you drawing pictures, that's wonderful. And so this is a (laughing) we looked at sensory interference and sensory facilitation, but now we're just talking about, really, what sensory information do you pair together? Straightforward. And what I like about this, and why it fills me with glee, it's because the simplicity of it. How simple it is to just say, "Which one of these things "do you think matches?" Or the simplicity of saying, "Listen to this sound, "draw what you think it looks like." That's fun, and also I get to use crayons. And, it's win-win, right? So what do we learn from here? Okay. Which one of these balls goes pong and which one goes ping? You can think about it in your head, I'm not supposed to ask you questions. Well, one of them goes pong, and one of them goes ping. And I'm sorry, for those of you standing right there you can't really see. One of those balls is a dark gray and one of them is white. As it turns out, if we show people, toddlers and adults, who are both people, just at different times of their lifespan... Greater than can be expected due to chance, in fact over and well above what we would see as randomly, toddlers and adults will say that the gray ball goes pong, the darker ball says pong. That's also really fun, I get to say these things over and over again and it's all for science, so, I mean, it's really so good. And that light ball goes ping. We see this in adults, we see this in toddlers. Now, here's my question. Does not a white cat and a black cat meow at the same pitch? Yeah, does not a white mouse and a brown mouse squeak in the same frequency. We are not learning this the same way we are learning about our friend the Great Dane and our little buddy the Chihuahua. I mean, that's something you can say, "Well, yeah, you can learn that "through active experience with the world." How do you learn this through active experience in the world? Lighter-colored objects do not routinely make higher-pitched sounds. We didn't learn that. (moaning in frustration) It's a bit of a head scratcher, but that's why I find it really interesting. Here's another one for you. "Which one of these is a kiki and which one is a koko?" (audience laughing) Mmhmhm, you like that? (laughing) Well, (laughing) this is an interesting phenomenon. It's actually an incredibly robust phenomenon that has been studied for nearly 100 years, if you can believe it. And what we know is that words that have O, an O sound in them, are associated to rounded shapes. And words that have that E sound are associated to jagged shapes, and you can see lots of studies that suggest this. Now, what effect does the consonant sound on it? Not much. The consonant sound can facilitate it a little bit, but it's really all about the vowels. And actually, if we start to look at world languages, then we can start to see patterns whereby objects like fish that are rounded are often labeled with sounds that have vowels, rounded vowels in them. And objects that are angular, like lots of birds, are often labeled with sounds that have that E, E sound in them. And it's really neat because it's the idea that it's possible that some of these multisensory... Some of this multisensory integration, not only could it have possibly driven some of the evolution of language itself, that's kind of a big one, but it also helps bootstrap learning. Bootstrap learning in the individual child as that child learns language in a place where objects that have O's in them, like things that are round, or balls, okay, it's not all O's, right, rounded vowels, where those are reliably labeling objects that look just like they sound, versus spikes and points and teeny little things. And that's really cool. It's a good system. So we know information is consistently linked across the senses. Some of that information, some of those associations are learned, right? That Great Dane is barking louder, that Great Dane is barking lower than our little, wee Chihuahua. We've learned that, we can learn that. But some of these associations we can't explain by learning. And so I've coined a term, I don't know if I've coined it, it's not in the Oxford English Dictionary yet, but I do use it over and over again. It's called naturally-biased, naturally-biased. We can't say that it's innate. In order to say that something is innate we would have to see it in newborns, like minutes after they are born. I mean, I don't know if anybody's ever birthed a newborn, but you know nobody's walking in there in that moment and saying, "Oh, can I test your baby?" And people do that, and you know what? Bless their hearts, but it's very hard to say something is innate, because once, as we've said, as we saw with that visual development, once you gain experience with the world then all of a sudden, everything has changed. Experience drives brain development. And so we say that which we can't explain by learning is naturally-biased. Well, I think it's biased by our neural architecture, by the way in which our sensory systems developed together. But it also seems to be more common in audio, visual, tactile senses. And if you think about that, I mean, it makes sense. You can visually see rhythm the same way that you can bong, bong, bong, hear it, right? It's just coming down to frequency. But what about the other senses? And that's a place where I got my hands a little dirty against the better judgment of some of my academic mentors. The thing about vision and hearing and touch is they're frankly easier to measure. We know more about them. Do we know about them because they're more sophisticated systems, or do we know more about them because it's easier to figure out how to know about them? Smell's a tough one, it's a really tough one. But I'm up for the challenge. So what do we know? Okay, so we know that there is sensory interference and facilitation in the audio, visual, and the tactile domains. But there's also sensory interference and facilitation in the smell and taste domains. Those arrows are anomalous. Okay, we know that adding color, that adding color to a smell, right, just taking an odorant that is clear and then adding color to it, people will report that odorant as being more pleasant, if it's a pleasant smell, of course. They will rate it as being more intense, right? Remember it just like that light plus the noise, more intense when you add multisensory information. We know that the amount of color increases the perceived sweetness of a solution, even if the color doesn't make any sense. And if you take a lemon solution and you add red to it, people will still report it as being sweeter. This one's one of my favorites, if anybody has ever read a bottle of wine or heard somebody talk about wine and say things like, "Oh, this has got oak on the nose," or something like that. If you take white wine and you add red dye to it, then all of a sudden those wine experts who definitely have expertise in their area will actually start using words that they usually use to describe red wine, even though it's white wine with red dye to it. So they will say things like, I don't know, "rosehips and red floral tones." I have no idea, I'm really not a wine connoisseur, but the point is is that it can even fool an expert, somebody who has spent their entire career learning about the differences between wines. You simply manipulate one multisensory cue and then all of a sudden. It doesn't make them any less expert. I'm sorry if there's any wine experts in this room, but we'd fool you too, and then that's the point. (laughing) We also know that texture can influence perceived odor. So if you have a fabric and you add a pleasant odor, people will actually report that fabric as being softer. And if you add an unpleasant odor, they will report that fabric as being harsher, scratchier. And it was this series of things where I said, "I don't think it's all about audition and vision. Everybody thinks it's all about audition and vision, but what about smell, what about smell? Down here, it's smell's time down here, right now. And so, with glee, of course, I got a whole bunch of smells, and I got a whole bunch of people, not at the same time. And I had them smell the smells, and very simply, and this is what I love about my job, is I very simply just said, "What color do you think this smell is? "What texture do you think this smell is?" And sometimes I got some blank stares. "What do you mean what texture do I think this smell is?" "If this smell had a texture. "Just humor me." Sometimes I actually had to add that into my experimental instructions. "If this smell had a texture, what would it feel like? "If this smell had a color, what would it be?" And what's neat, and I'll get to this a little bit later, but what's neat is that some people were like, "Oh, yeah totally." And some people were like, "I don't, "I have no idea what you're talking about." And so there's a spectrum of ability to understand multisensory experience, which is really cool. And so what did we find? Okay, well as you can expect, odors that were easily identifiable, like cinnamon, were overwhelmingly reported as being the colors that you would expect. Right, cinnamon, red and brown, okay. Lemon, orange and yellow. Anise, which is like a licorice smell, black. Peppermint, yeah, okay. But those were the only, by the way, these were the only smells that were identified. None of the rest could be identified nearly by anybody, and we tested 80 people in this measure. Almond, red, white, and blue. That's actually purple on my screen. And dull, of a dull texture, right? Onion was brown, vanillin, vanilla was white and brown, and ginger was black. Well, I didn't learn that. I'm not putting black ginger on my kid's food. That's the ginger I throw away, right? And so what we're seeing, and I'm not gonna go through these in great detail, but what we're seeing is that, similar to the patterns that we've seen with audiovisual associations, some of the associations are clearly learned. Some of them are not. And we can't explain them all through learning. And if we can't explain them all through learning, then there's gotta be another explanation. Our idea is that these learned associations, they take over. We might have some naturally-biased associations. Here is our chemical, by the way, just in case you were wondering. Moth balls, although nobody was able to accurately identify moth balls, everybody said the smell was white. I know, isn't that interesting? Maybe somewhere, somebody was like, "Maybe somewhere they were identifying "the smell of moth balls." Fair enough, right? That's the point. Skepticism in science is the thing that makes science go forward. The environment, notice, environmental smells have these kinds of earth-toney type colors. That's pretty cool. Okay, well, now what? That's what was really funny about it, is I finished this study and my supervisors were like, "Okay, cool, now what?" I was like, "That's a great question." Because now what, or so what, is kind of the thing that drives a scientific discovery. You're like, "Oh, now what? "Okay, I think..." That there is a universal sensory code. I think that all of our senses can speak the same language. I don't know what it is. I can think about what it is, for those audio, visual, tactile domains. It's a little bit harder to think about what it is for the smell and the taste. But how do we crack this code? If only, if only there were a group of people somewhere in the world who had conscious access to multisensory experiences. If only. Do you guys know where I'm going with this? There are. There are people who have something called "synesthesia." Synesthesia officially means "syn," together, "aesthesis," sensation. So together sensation, as oopposed to anesthesia, which is without sensation. So people who have synesthesia, for an individual with synesthesia, external, typical stimulation of one sense such as the sound of a trumpet making a B flat, reliably elicits an experience in another sense. For example, the experience of red. That means that people who have synesthesia have a conscious experience of what we believe to be multisensory perceptual processes that are common to everybody. We're just not aware of them. I think I pressed a button. And so I use synesthesia as a tool for understanding typical perception because all evidence suggests that what is happening in their brains, in their perceptual systems, is not actually different than what's happening in our brains, in our perceptual systems, either at a different stage in development or at an earlier stage of processing. And why do I know that? Well, there are similar connections in synesthetes and typical infants. Right, remember all those connections? They may not all get pruned. We know that. And there is evidence that individuals with synesthesia have a little more connectivity between their brain areas. Some people have even gone out and said that infants, or neonates, newborns, are actually synesthetic because they experience all sensory information in an undifferentiated way. Now, an individual with synesthesia will say, "That's not the way it is, it's always going "to be between two senses, very specific." An individual with synesthesia does not experience chaotic sensory information like a baby does. They're not going all the time like babies are, okay, so just to be clear. It's typically specifically between a few sensory areas, but there is more connectivity. Maybe those connections, those connections just didn't pruned in these individuals. Not only that, when an individual with synesthesia who, for example, sees colored letters, so letters have color. When they're viewing a letter that is printed in black you actually see activation of their visual cortex which you would expect, but over areas of the visual cortex that deal with color, which you do not see in individuals who do not perceive letters as having color. But remember the pong, remember that? Remember how we associate large objects to loud sounds and teeny little objects to pointy things? Well, we see the similar associations in synesthetes. A person with synesthesia who has a visual experience to a sound, that visual experience is consistent with what other people are reporting. Higher pitches, somebody who has synesthesia, an audiovisual synesthete, when they hear higher pitches it reliably induces the experience of a lighter color. A smaller object. Lower pitches reliably induce the experience of a darker color, a larger object. We have every reason to believe that synesthesia and typical perception use common mechanisms. They're not different. They are at one end of a spectrum and the rest of us are somewhere in the middle. Some of us are very, very much at the bottom. Those are the ones that go, "Yeah, I really don't understand what you mean "when you're asking me to associate a color to a smell." You all have your crayons? Okay. So what we like to do is we like to, what I really want to do, in order to crack this multisensory code, and that is my life's work. I'm gonna crack that multisensory code, just like a coconut, just crack it right open. It's not gonna be that easy. Actually, cracking open a coconut is quite difficult. But we wanna crack the multisensory code, the universal sensory code, then we need to ask synesthetes and non-synesthetes what their experiences are. And one of the big projects that we're doing here right now with my students here at Edgewood College and with collaborators in other places, is we are collecting what I call audiovisualizations. We're playing sounds, we're having people draw pictures. And so, ready? I'm gonna play you a sound. (high-pitched twinkling music) I'm not collecting your data, don't worry. Here it was. Now, of course, in the actual experiment you can listen through it headphones and you can play it again, and draw your experience. I'm wondering, did anybody's picture look like those? These are visualizations drawn by non-synesthetes in response to this sound. These are visualizations drawn by synesthetes in response to this sound. Synesthetes for whom this is their actual, visual experience. I'm not asking them to imagine what the sound looks like, but in fact they are drawing what they see. They look kinda similar. Ready for the next one? Ready with your crayons? (low-pitched suspenseful orchestral music) I find that one very pleasing. I'll do it one more time. (low-pitched suspenseful orchestral music) Huh. Did it look something like that, maybe? These are the drawings for non-synesthetes. And these are the drawings for synesthetes. It's a little light, but it's yellow. You see some of the same elements, the jaggedness, the movement. (high-pitched violin music) Oh, you're not supposed to see this already. Oh well. And you can see that when the sound is represented in a more sweeping movement, that that is actually represented through the drawings of non-synesthetes. I like this one in particular. That one's very attractive. A lot of synesthetes happen to be artists, unrelated. All right, and the last one, which you'll notice differs very specifically from the previous one. (high-pitched staccato music) The difference is the style of the musical note. And so what do we see differently? That's the goal here, is how are these audiovisualizations represented differently across these different characteristics of the sound? Well, the drawings look very staccato, as do the drawings of non-synesthetes. And so what we're doing right now through some tireless efforts of my students here, is we are actually combing through all of this data, non-synesthetes and synesthete data, and we are looking across individuals within a certain musical clip, and we are trying to figure out, what are the characteristics of the sound that reliably induce the visualization? Is it the pitch or the timbre of the sound? Is it the rhythm, is it the tone? One thing we are seeing over and over again is something called form constants. And these form constants were proposed a long time ago when somebody was studying hallucinations. And that individual said, "For some reason, "in all simple hallucinations," we often think about hallucinations as being complex, like horses running through a field, that's not usually the way it is. Most hallucinations are actually quite simple. They involve very basic sensory input. And most of the time when you're having a simple hallucination you are having a simple hallucination that involves what are called form constants. Some variation of this very basic set of shapes. Did you see those in the drawings I showed you? Because they were there too. There's something real special about them. And just in case you were wondering, when we show people two options, a visualization that was drawn in response to the music, and a visualization that was drawn in response to different music, they are able to select... Reliably which image fits with which picture. Which image, excuse me, fits with which song. But why? (laughing) Isn't that the question? But why? The world is mapped out on your brain. The world is mapped out on your brain everywhere. The visual world is mapped out on your retina and it's mapped out in your visual cortex. The auditory world is mapped out in your cochlea, it's mapped out on your auditory cortex. Why shouldn't all elements of the world be mapped out on your brain? We know that there are specific parts of the visual cortex that specifically process basic shapes or angular versus curved objects. There are similar components in auditory neurons. This is our universal sensory code. It makes me very excited. So, we have processes that allow us to seamlessly integrate information that we can expect to encounter so that we can move through the world without being like, "Wait, there's a loud sound "and there's a large object. "What am I supposed to do here?" We've seamlessly integrated the information so that we can seamlessly react to it, just like everything else that our brain does. As a consequence, we have consistent associations that we can't explain by learning. Well, that's what I like to have. How can I measure those? Because it's very easy to be like, "Oh, well you just learned that." Well, but what about the ones that I didn't learn? What does that mean? The ones that I didn't learn, how are elements reliably associated across sensory areas? That's the universal sensory code. And finally, I believe that we all exist on a multisensory spectrum. And on one end, what you see here is that is a visualization that was drawn in response to one of those sounds. It's the drawing of a piano. That is a non-abstract image, totally appropriate. We never said that the images had to be abstract. Most people choose abstract images, most people draw abstract images, some people drew non-abstract images. I think that those are the people that live on this side of the spectrum. Nothing wrong with that. Those are the ones that go, "What do you mean, "what color is this smell?" They look at me like I've got lobsters crawling out of my ears. It's kind of funny. And then somewhere in the middle is most of us, who say, "Oh yeah, I drew something "that looked kind of like that." And then all the way up here are people who have synesthesia, who actually have conscious and reliable and involuntary multisensory percepts. And that is how we're gonna start to crack our universal sensory code. But of course, I couldn't even dream of cracking our universal sensory code without extraordinary help from my wonderful students here at Edgewood College, and all of my collaborators. To them, and to you, I say thank you. I'm done. (audience applauds)