What Memory Is Made Of

The relationship between synesthesia, vivid mental imagery, and memory may offer a view into our unique inner theatres.

By Mark Kraus

Published February, 19, 2024.

“The accident was in April of 2018. There was a handyman on the roof, a couple of stories up. He dropped a pair of shears, a heavy-duty pair of tin snips...I was hit with the rubberized handle,” Callie recalled the incident that led to her lossing both her memory and the ability to make mental images after suffering a severe concussion. “I didn’t want to sit down because I thought if I sat down, I would pass out, and I didn’t want to pass out.” A playwright and educator, she relied on her quick memory and vivid imagination to build storylines and dream up characters. She described her mental experience of writing before the incident, “I saw things like a movie in my head. I would see the location... I’d sort of see arcs of movement and three-dimensional sculptures of lines.” The forms used to reference plot points are common in people with a certain type of synesthesia and those with highly visual imaginations. Due to her accident, her ability to think visually had vanished; she could no longer create or manipulate images in her mind. She had also lost some of her working memory, or the ability to remember basic information, like numbers, directions, or names.

Research into synesthesia — a blending of sensory experiences — and memory gives insight into its connection with this type of episodic visualization. Studies have uncovered a gradient between people who lack mental images on one end, as seen in aphantasia and some types of autism, and those with vivid visual imagery on the other, such as people with synesthesia or highly visual thinkers. According to studies, recall, mental imagery, and episodic memory in those with synesthesia are markedly higher than those who lack the ability. The findings may offer a clue into how we use mental imagery to create imagined visual stories and better understanding of those who don’t.

“There is a continuum, from people who have no visual imagery to people who have very strong and vivid visual imagery,” explained Dr. Beat Meier from the University of Bern, whose research first showed memory advantages in people with synesthesia. “Synesthesia is just one case where we know that people have more vivid imagery,” he said. His research focuses on the brain’s ability to encode information and his interest in synesthesia is due to its unique advantage in that area. He suspects that dual-coding — meaning using more than one reference to encode a memory, such as color, smell, or sound — can increase retention, even when the associations are automatic.

“

Synesthesia is just one case where we know that people have more vivid imagery.”

Synesthesia can take different forms, some people see colors when they read or hear sound, while others can smell shapes or even taste music. This might seem bizarre to people without the condition, but the experience often goes unnoticed by synesthes for a portion of their lives. People with the condition can also experience more than one type. Spatial-sequence synesthesia can manifest as three-dimensional shapes that relate to numbers, time, or even plot lines — providing the ability for synesthes to jump between “scenes,” like scrolling through a highlight real. Associative synesthetes experience everything within the mind’s eye, while projector synesthetes experience them outside of their body, as if they’re wearing a pair of augmented reality goggles.

Memory and visualization have long been associated. Ancient Greek senators used a technique called “the memory palace” to give speeches. They'd use items representing words or phrases are placed in a familiar space, such as a person’s home, within the mind’s eye. As the observer “walks” through its hallways and rooms, they are able to convert the items back into words and sentences, allowing them to reproduce long orations from memory.

Callie, whose accident left her without mental visualizations for a period, described the contrast between her pre- and post-concussion experience, “I didn’t know there was any other way to experience the world. I was always seeing stories. I’d drive by a house, and I would see a whole movie and in that house. I didn’t even know the people living there, but there just be a story. I would imagine it all, I just had a very, very vivid imagination.” After the head injury, Callie lost that normally rich ability to make images. “There was nothing. It was like this black void. Everything was just flat.” She struggled to navigate ordinarily simple tasks due to her impaired memory, “I had to use GPS to go to the grocery store because I couldn’t remember where I was going.”

While speaking about extrodinary memory, Dr. Meier shared the story of the artist Stephen Wiltshire, who has an extraordinary gift. “He is able to, when he flies over a city...kind of imprint the structure of a city. Then he goes and draws the city astonishingly well.” His drawings, done from memory, are rendered in black and white in perfect detail. “There is evidence that there are really people who have excellent visual memory,” he said. Notably, Wiltshire is autistic, a condition that typically is associated with poor visual and episodic memory. His process is described as an obsession with buildings, cars, and cityscapes, with little concern for the context or narrative detail that would normally support a memory.

There is a recent study that explores the relationship between aphantasia — or the inability to make mentals images — synesthesia, and autism out of the University of Sussex. It found that synesthesia co-occurred across all participants equally in all three conditions. This may mean that synesthesia is not necessary to drive how vivid our mental landscapes are. More likely, the condition is a supplemental tool to enhance memory in already visual people. When co-occurring with aphantasia the subjects scored higher on memory tests than others with the condition, showing an advantage even when mental images aren’t present.

Meier’s research found that memory advantages seem to be “type-specific.” Color-graphene synesthetes, those would see colors associated within the letters of the alphabet, tend to score highest on memory tests. He’s currently studying chromesthesia, or those who see color associated with music, and how it relates to perfect pitch. “Musicians who have absolute pitch, the prevalence of synesthesia is higher in these people than in the normal population,” he said. Having a second reference point to musical scale may allow the brain to reference notes visually — a huge advantage when identifying pitch out of context.

Synesthesia may arise due to brain cell connections and the proximity of our neurons. “For grapheme-color synesthesia, people have activations in their temporal lobe...in the area in the letter area and the color area; they are quite close together,” explained Meier. As our brains grow normally there may be a pruning among the connections. In synesthesia those connections may be somehow inhibited. “It is possible that kind of an obsessive exposure to these connections is necessary.”

As Callie’s recieved treatment and her concussion healed, her memory and “sculptures of lines” returned. “That ability to imagine came back,” she said. “I came up with the entire timeline, characters, character names, exactly how many scenes there were, the main events...like the resonance objects, the pivotal moments, all of it...I was like, ‘oh, I can imagine!’” Her experience sheds light on the mind’s visual projector and how it relates to our episodic imagination and memory. The pathway of a thought and a visualization may cross brain regions that are usually separated. In some people, these regions are intertwined, leading to extraordinary experiences of perception and thought: three-dimensional sculptures of lines or color enhanced experience — and for others, near perfect memory. It may be indicative of a mind that stages and poses, conducts, and illuminates vs. one that tells its stories in the dark.

1.

Meier, Beat. Synesthesia. (2021). 10.1016/B978-0-12-819641-0.00134-1.

2.

Dr. Beat Meier, Institute of Psychology, University of Bern. Teams interview, 07/21/23. beat.meier at unibe.ch

3.

Callie Kimball, 07/24/23. Phone interview. calliekimball at gmail.com

4.

Rothen N, Meier B, Ward J. Enhanced memory ability: Insights from synaesthesia. Neurosci Biobehav Rev. (2012) https://pubmed.ncbi.nlm.nih.gov/22634573/

5.

Lunke K, Meier B. New insights into mechanisms of enhanced synaesthetic memory: Benefits are synaesthesia-type-specific. PLoS One. (2018) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6124748/

6.

C.J. Dance, M. Jaquiery, D.M. Eagleman, D. Porteous, A. Zeman, J. Simner, What is the relationship between Aphantasia, Synaesthesia and Autism?, Consciousness and Cognition,Volume 89. (2021). https://doi.org/10.1016/j.concog.2021.103087. https://www.sciencedirect.com/science/article/pii/S1053810021000131

7.

Oliver Sacks. Prodigies. The New Yorker (1995) https://www.newyorker.com/magazine/1995/01/09/prodigies

8.

Simon Leipold, Carina Klein, Lutz Jäncke. Musical Expertise Shapes Functional and Structural Brain Networks Independent of Absolute Pitch Ability. Journal of Neuroscience (2021). https://www.jneurosci.org/content/41/11/2496

9.

Foer, Joshua. Moonwalking with Einstein: The Art and Science of Remembering Everything. (2011) New York: Penguin Press. ISBN 978-1-59420-229-2.