Social learning helps humans and animals rapidly adapt to new circumstances, and drives the emergence of complex learned behaviors. This talk focuses on how Reinforcement Learning (RL) agents can benefit from social learning in naturalistic multi-agent environments. In this setting, which is analogous to that of autonomous driving, there are other agents that may have relevant knowledge, but they are not explicitly interested in teaching the RL agent. We first show that traditional model-free RL algorithms do not benefit from social learning in such contexts, and cannot discover the optimal policy even when nearby agents are visibly following it. However, by learning an unsupervised model that predicts the next state, agents implicitly model the behavior of other agents and can leverage social learning to improve their performance. Further, agents that engage in social learning can generalize better to new environments, by following a strategy of using social learning to obtain information about how to perform well on the new task. We then introduce an improved method for social learning, PsiPhi-learning, which leverages the power of successor features to improve RL through modeling other agents, and improve modeling other agents through collecting individual experience via RL. PsiPhi-learning improves over both traditional RL techniques and recent imitation learning techniques, flexibly benefitting from learning from other agents when it is relevant to the task at hand.