Cortical network structure has been extensively characterized at the level of local circuits and in terms of long-range connectivity, but seldom in a manner that integrates both of these scales. Furthermore, while the connectivity of cortex is known to be related to its architecture, this knowledge has not been used to derive a comprehensive cortical connectivity map. In this study, we integrate data on cortical architecture and axonal tracing data into a consistent multi-scale framework of the structure of one hemisphere of macaque vision-related cortex. The connectivity model predicts the connection probability between any two neurons based on their types and locations within areas and layers. Our analysis reveals regularities of cortical structure. We confirm that cortical thickness decays with cell density. A gradual reduction in neuron density together with the relative constancy of the volume density of synapses across cortical areas yields denser connectivity in visual areas more remote from sensory inputs and of lower structural differentiation. Further, we find a systematic relation between laminar patterns on source and target sides of cortical projections, extending previous findings from combined anterograde and retrograde tracing experiments. Going beyond the classical schemes, we statistically assign synapses to target neurons based on anatomical reconstructions, which suggests that layer 4 neurons receive substantial feedback input. Our derived connectivity exhibits a community structure that corresponds more closely with known functional groupings than previous connectivity maps and identifies layer-specific directional differences in cortico-cortical pathways. The resulting network can form the basis for studies relating structure to neural dynamics in mammalian cortex at multiple scales.