"Perception is never purely in the present–it has to draw on experience of the past.” O. Sacks illustrates a grounding principle in cognitive neuroscience: Eyes are not vision. Instead, seeing is an inferential process, with the brain integrating incoming with past sensory information. This idea, that perception and memory are deeply intertwined, has shaped many theories of neural computation. Yet research on these building blocks of cognition has largely evolved in separate fields. I aim to overcome this conceptual divide and understand the mechanistic principles by which memory and perception shape each other. One phenomenon, which provides a window into this frontier, is a perceptual memory known as serial dependence: It comes in multiple flavors (attractive & repulsive effects) and occurs at multiple timescales. A simple, untested hypothesis may account for this diversity: I propose that there is a hierarchy of perceptual memories, with different integration times and tuning features, all acting jointly to shape perception. I will test this framework with behavioral studies, computational modeling, and non-invasive and invasive electrophysiology.