Background The Doloplus-2 is used for behavioural pain assessment in cognitively impaired patients. Little data exists on the psychometric properties of the Doloplus-2. Our objectives were to test the criterion validity and inter-rater reliability of the Doloplus-2, and to explore a design for validations of behavioural pain assessment tools. Methods Fifty-one nursing home patients and 22 patients admitted to a geriatric hospital ward were included. All were cognitively impaired and unable to self-report pain. Each patient was examined by an expert in pain evaluation and treatment, who rated the pain on a numerical rating scale. The ratings were based on information from the medical record, reports from nurses and patients (if possible) about pain during the past 24 hours, and a clinical examination. These ratings were used as pain criterion. The Doloplus-2 was administered by the attending nurse. Regression analyses were used to estimate the ability of the Doloplus-2 to explain the expert's ratings. The inter-rater reliability of the Doloplus-2 was evaluated in 16 patients by comparing the ratings of two nurses administrating the Doloplus-2. Results There was no association between the Doloplus-2 and the expert's pain ratings (R2 = 0.02). There was an association (R2 = 0.54) between the expert's ratings and the Doloplus-2 scores in a subgroup of 16 patients assessed by a geriatric expert nurse (the most experienced Doloplus-2 administrator). The inter-rater reliability between the Doloplus-2 administrators assessed by the intra-class coefficient was 0.77. The pain expert's ratings were compared with ratings of two independent geriatricians in a sub sample of 15, and were found satisfactory (intra-class correlation 0.74). Conclusion It was challenging to conduct such a study in patients with cognitive impairment and the study has several limitations. The results do not support the validity of the Doloplus-2 in its present version and they indicate that it demands specific administration skills.