CD47 is a widely distributed cell-surface protein that acts a marker of self through interactions of myeloid and neural cells. We describe the high-resolution X-ray crystallographic structures of the immunoglobulin superfamily domain of CD47 alone and in complex with the N-terminal ligand-binding domain of signal regulatory protein alpha (SIRPα). The unusual and convoluted interacting face of CD47, comprising the N terminus and loops at the end of the domain, intercalates with the corresponding regions in SIRPα. We have also determined structures of the N-terminal domains of SIRPβ, SIRPβ(2), and SIRPγ; proteins that are closely related to SIRPα but bind CD47 with negligible or reduced affinity. These results explain the specificity of CD47 for the SIRP family of paired receptors in atomic detail. Analysis of SIRPα polymorphisms suggests that these, as well as the activating SIRPs, may have evolved to counteract pathogen binding to the inhibitory SIRPα receptor.