We describe a model of invariant visual object recognition in the brain that incorporates feedback biasing effects of top–down attentional mechanisms on a hierarchically organized set of visual cortical areas with convergent forward connectivity, reciprocal feedback connections, and local intra-area competition. The model displays space-based and object-based covert visual search by using attentional top–down feedback from either the posterior parietal or the inferior temporal cortex (IT) modules, and interactions between the two processing streams occurring in V1 and V2. The model explains the gradually increasing magnitude of the attentional modulation that is found in fMRI experiments from earlier visual areas (V1, V2) to higher ventral stream visual areas (V4, IT); how the effective size of the receptive fields of IT neurons becomes smaller in natural cluttered scenes; and makes predictions about interactions between stimuli in their receptive fields.