In the study of prosopagnosia, several issues (such as the specific or non-specific manifestations of prosopagnosia, the unitary or non-unitary nature of this syndrome and the mechanisms underlying face recognition disorders) are still controversial. Two main sources of variance partially accounting for these controversies could be the qualitative differences between the face recognition disorders observed in patients with prevalent lesions of the right or left hemisphere and in those with lesions encroaching upon the temporo-occipital (TO) or the (right) anterior temporal cortex. Results of our review seem to confirm these suggestions. Indeed, they show that (a) the most specific forms of prosopagnosia are due to lesions of a right posterior network including the occipital face area and the fusiform face area, whereas (b) the face identification defects observed in patients with left TO lesions seem due to a semantic defect impeding access to person-specific semantic information from the visual modality. Furthermore, face recognition defects resulting from right anterior temporal lesions can usually be considered as part of a multimodal people recognition disorder. The implications of our review are, therefore, the following: (1) to consider the components of visual agnosia often observed in prosopagnosic patients with bilateral TO lesions as part of a semantic defect, resulting from left-sided lesions (and not from prosopagnosia proper); (2) to systematically investigate voice recognition disorders in patients with right anterior temporal lesions to determine whether the face recognition defect should be considered a form of "associative prosopagnosia" or a form of the "multimodal people recognition disorder."