Behaviorally-relevant features of observed actions dominate cortical representational geometry in natural vision.

We effortlessly extract behaviorally relevant information from dynamic visual input to understand the actions of others. Here, we compare behavioral judgments of action meaning against other models capturing semantic and visual features. We measured brain activity using fMRI while participants viewed 90 video clips depicting social and nonsocial actions in real-world contexts. Using arrangement tasks, we developed two behavioral models capturing judgments of action meaning (transitive purpose, social content), as well as three control models capturing judgments of visual content (people, objects, scene). We compared these models with semantic models based on word embeddings and visual models based on gaze and motion energy. Behavioral models of action meaning outperformed models of visual content, semantics, and low-level visual features across much of cortex. Transitivity and sociality captured large portions of unique variance throughout the action observation network, extending into regions not typically associated with action perception, such as ventral temporal cortex.