What Memory Is Made Of

The relationship between synesthesia, vivid mental imagery, and memory may offer a view into our unique inner theaters.

By Mark Kraus

Updated July, 11, 2024.

“The accident was in April of 2018. There was a handyman on the roof, a couple of stories up. He dropped a pair of shears, a heavy-duty pair of tin snips...I was hit with the rubberized handle.” Callie, a playwright and educator, recalled the accident that led to a severe concussion, “I didn’t want to sit down because I thought if I sat down, I would pass out, and I didn’t want to pass out.” She had relied on her quick memory and vivid imagination to build storylines and dream up characters. She described her mental experience of writing before the incident, “I saw things like a movie in my head. I would see the location...I’d sort of see arcs of movement and three-dimensional sculptures of lines.” The visualizations used to reference plot points are common in people with certain types of synesthesia — a blending of sensory experiences — and those with highly visual imaginations. Due to her accident, her ability to think visually had vanished — she could no longer create or manipulate images in her mind. She had also lost most of her ability to remember basic information like numbers, directions, or names.

Research into memory gives insight into its connection with this type of episodic visualization. People who lack the capibility to make mental images — a condition known as aphantasia — have different types of memories than those with vivid visual mental experiences — such as people with synesthesia. According to a few studies, recall, mental imagery, and episodic memory in those with synesthesia and highly visual imaginations are markedly higher than those without the condition. The findings may offer a clue into how we use mental imagery to create imagined visual stories and better understanding of those who don’t.

“There is a continuum, from people who have no visual imagery to people who have very strong and vivid visual imagery,” explained Dr. Beat Meier from the University of Bern, whose research first showed memory advantages in people with synesthesia. “Synesthesia is just one case where we know that people have more vivid imagery,” he said. His research focuses on the brain’s ability to encode information and his interest in synesthesia is due to its unique advantage in that area. He suspects that dual-coding, meaning using more than one reference to encode a memory, such as color, smell, or sound, can increase retention, even when associations are automatic.

People with synesthesia experience a blending of two cognitive experiences. It can take different forms, some people see colors when they read or hear sound, while others can smell shapes or taste music. This might seem bizarre to people without the condition, but the experience often goes unnoticed by synesthes for a portion of their lives. People with the condition can also experience more than one type. Spatial-sequence synesthesia can manifest as three-dimensional shapes that relate to numbers, time, or even plot lines — providing the ability for synesthes to jump between “scenes,” like scrolling through a highlight real. Associative synesthetes experience everything within the mind’s eye, while projector synesthetes experience them outside of their body, as if they’re wearing a pair of augmented reality goggles.

The condition may arise due to brain cell connections and the proximity of our neurons. “For grapheme-color synesthesia, people have activations in their temporal lobe...in the area in the letter area and the color area; they are quite close together,” explained Meier. As our brains grow normally there may be a pruning among the connections. In synesthesia those connections may be somehow inhibited. “It is possible that kind of an obsessive exposure to these connections is necessary.”

Callie described the contrast between her pre- and post-concussion experience, “I didn’t know there was any other way to experience the world. I was always seeing stories. I’d drive by a house, and I would see a whole movie in that house. I didn’t even know the people living there, but there just be a story. I just had a very, very vivid imagination.” After the head injury, Callie lost that normally rich ability to make images. “There was nothing. It was like this black void. Everything was just flat.” She struggled to navigate ordinarily simple tasks due to her impaired memory, “I had to use GPS to go to the grocery store because I couldn’t remember where I was going.”

“

I didn’t know there was any other way to experience the world. I was always seeing stories. I’d drive by a house, and I would see a whole movie in that house.”

Memory and visualization have long been associated. Ancient Greek senators used a technique called “a memory palace” to give speeches. It works like this: items representing words or phrases are placed in a familiar space, such as a person’s home, within the mind’s eye. As the observer “walks” through its hallways and rooms, they are able to convert the items back into words and sentences, allowing them to reproduce long orations from memory. In an example of the hyper visualizations of eidetic memory, the actress Marilu Henner described her ability to remember events and details in perfect detail on an episode of 60 Minutes. “It’s like putting in a DVD and it cues up to a certain place. I’m there again. So, I’m looking out from my eyes and seeing things visually as I would have that day,” she said.

Meier shared the story of the artist Stephen Wiltshire, who has an extraordinary gift. “He is able to, when he flies over a city...kind of imprint the structure of a city. Then he goes and draws the city astonishingly well.” His drawings, done from memory, are rendered in black and white in perfect detail. “There is evidence that there are really people who have excellent visual memory,” he said. Notably, Wiltshire is autistic, a condition that typically is associated with poor visual and episodic memory. His process is described as an obsession with buildings, cars, and cityscapes, with little concern for the context or narrative detail that would normally support a memory.

There is a recent study that explores the relationship between aphantasia — or the inability to make mentals images — synesthesia, and autism out of the University of Sussex. It found that synesthesia co-occurred across all participants equally in all three conditions. This may mean that synesthesia is not necessary to drive how vivid our mental landscapes are. More likely, the condition is a supplemental tool to enhance memory in already visual people. When co-occurring with aphantasia the subjects scored higher on memory tests than others with the condition, showing an advantage even when mental images aren’t present.

Meier’s research found that memory advantages in synesthetes seem to be “type-specific.” Color-graphene synesthetes, those would see colors associated within the letters of the alphabet, tend to score highest on memory tests. He’s currently studying chromesthesia, or those who see color associated with music, and how it relates to perfect pitch. “Musicians who have absolute pitch, the prevalence of synesthesia is higher in these people than in the normal population,” he said. Having a second reference point to musical scale may allow the brain to reference notes visually — a huge advantage when identifying pitch out of context.

Synesthesia may arise due to brain cell connections and the proximity of our neurons. “For grapheme-color synesthesia, people have activations in their temporal lobe...in the area in the letter area and the color area; they are quite close together,” explained Meier. As our brains grow normally there may be a pruning among the connections, or those connections are somehow inhibited. “It is possible that kind of an obsessive exposure to these connections, is necessary,” he said.

As Callie’s recieved treatment and her concussion healed, her memory and visual forms returned. “That ability to imagine came back,” she said. “I came up with the entire timeline, characters, character names, exactly how many scenes there were, the main events...like the resonance objects, the pivotal moments, all of it...I was like, ‘oh, I can imagine!’” Her experience sheds light on the mind’s visual projector and how it relates to our episodic imagination and memory. The pathway of a thought and a visualization may cross brain regions that are usually separated. In some people, these regions are intertwined, leading to extraordinary experiences of perception and thought: three-dimensional sculptures of lines or color enhanced experience — and for others, near perfect memory. It may be indicative of a mind that stages and poses, conducts, and illuminates vs. one that tells its stories in the dark.

1.

Meier, Beat. Synesthesia. (2021). 10.1016/B978-0-12-819641-0.00134-1.

2.

Dr. Beat Meier, Institute of Psychology, University of Bern. Teams interview, 07/21/23. beat.meier at unibe.ch

3.

Callie Kimball, 07/24/23. Phone interview. calliekimball at gmail.com

4.

Rothen N, Meier B, Ward J. Enhanced memory ability: Insights from synaesthesia. Neurosci Biobehav Rev. (2012) https://pubmed.ncbi.nlm.nih.gov/22634573/

5.

Lunke K, Meier B. New insights into mechanisms of enhanced synaesthetic memory: Benefits are synaesthesia-type-specific. PLoS One. (2018) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6124748/

6.

C.J. Dance, M. Jaquiery, D.M. Eagleman, D. Porteous, A. Zeman, J. Simner, What is the relationship between Aphantasia, Synaesthesia and Autism?, Consciousness and Cognition,Volume 89. (2021). https://doi.org/10.1016/j.concog.2021.103087. https://www.sciencedirect.com/science/article/pii/S1053810021000131

7.

Oliver Sacks. Prodigies. The New Yorker (1995) https://www.newyorker.com/magazine/1995/01/09/prodigies

8.

Lesley Stahl. 60 Minutes, The Gift of Endless Memory. CBS (2010) https://www.cbsnews.com/news/the-gift-of-endless-memory/

9.

Simon Leipold, Carina Klein, Lutz Jäncke. Musical Expertise Shapes Functional and Structural Brain Networks Independent of Absolute Pitch Ability. Journal of Neuroscience (2021). https://www.jneurosci.org/content/41/11/2496

10.

Foer, Joshua. Moonwalking with Einstein: The Art and Science of Remembering Everything. (2011) New York: Penguin Press. ISBN 978-1-59420-229-2.