Purpose To investigate factors associated with expectations of delayed return to work (RTW) in patients with prostate cancer recently admitted to a cancer rehabilitation program. Methods In this multicenter study, data about expected time until RTW and potential correlates (personal, medical, psychosocial and work-related factors) were obtained from 822 employed cancer rehabilitation participants at the beginning of the program. Participants expecting early RTW (≤ 3 months) and delayed RTW (> 3 months) were compared. Hierarchical multivariate logistic regression was applied to study which factors are associated with expecting delayed RTW. Results In total, 171 cancer rehabilitation participants (21%) expected delayed RTW. Group comparison showed education, type of occupation, income, number of comorbid conditions, tumor stage according to the staging system of the Union for International Cancer Control (UICC), anxiety and depression, quality-of-life functioning scales, urinary and treatment-related symptoms, duration of sick leave, subjective work ability, perceived ability to return to the former job, intention to apply for a disability pension, effort-reward-imbalance and occupational stress to be associated in bivariate analysis with participants' expectations. Multivariate analysis revealed UICC tumor stage III (compared to stages I/II, OR 2.36), lower subjective work ability (OR 0.82), perceived inability to return to the former job (OR 1.88) and intention to apply for a disability pension (OR 1.94) to increase the likelihood of expecting delayed RTW. Conclusions Negative or non-beneficial RTW expectations, which are related to self-perception and behavioral intention, seem to be key factors for expecting delayed RTW. Interventions to early identify and adjust such expectations might empower cancer rehabilitation participants to develop appropriate expectations for work recovery.