This article puts research on socially intelligent agents (SIA) in the broader context of how humans (and other primates) perceive and interact with the social world. Phylogenetic (evolutionary) and ontogenetic (developmental) issues are discussed with respect the social origin of primate and human intelligence and human culture. Implications for designing artifacts and for the evolvability of human societies are outlined. A theory of empathy is presented that is based on current research on the primate social brain. Research projects that investigate some of these issues are reviewed. I argue that Socially Intelligent Agents (SIA) research, although strongly linked software and robotic engineering, goes beyond a software engineering paradigm: it can potentially serve as a paradigm for a science of social minds. A systematic and experimental investigation of human social minds and the way humans perceive the social world can result in truly social artifacts, that are integrated in human society.