Humans and other primates have a remarkable ability to remember the images that they have seen, even after seeing thousands, each only once and only for a few seconds. In this talk, I will describe recent work from our group focused on the mechanisms that support visual familiarity memory in the primate brain. In the first part of the talk, I will describe the correlates of the natural variation with which some images are inherently more memorable than others, both the brain as well as deep neural networks trained to categorize objects. In the second part of the talk, I will focus on how information about visual familiarity is signaled and then decoded to produce visual familiarity behavior.