Measuring Programming Skill - Construction and Validation of an Instrument for Evaluating Java Developers

Bergersen, Gunnar Rye

Doctoral thesis

View/Open

PhD-Bergersen.pdf (4.626Mb)

Year

2015

Abstract

Skilled developers are important to the software industry. In empirical studies in software engineering, knowing the skill level of the participants is also important for correct interpretation of results. The current practice in industry and research for assessing programming skills is mostly to use proxy variables of skill, such as education, experience, and multiple-choice knowledge tests. There is as yet no valid and efficient way to measure programming skill. Consequently, this thesis aimed to construct a valid instrument for measuring programming skill, where skill is inferred from performance on programming tasks.

The Rasch measurement model was used to construct the instrument. Sixty-five professional developers from eight countries participated in validating the instrument, solving 19 Java programming tasks over two days. The validity of the instrument was theoretically investigated through commercial and research-based tests. Programming skill, as measured by the instrument, was also investigated in terms of experience and other background variables.

The instrument was found to have desirable psychometric properties, and the overall results appear well aligned with theoretical expectations. This work has shown that acceptable measures of programming skill may be obtained with less than one day of testing. Further work should be directed at reducing the time needed to measure programming skill without affecting the validity of the instrument. The results of the research have already been transferred to the industry through a commercial prototype.

List of papers

(I) Inferring skill from tests of programming performance: combining time and quality. Gunnar R. Bergersen, Jo E. Hannay, Dag I. K. Sjøberg, Tore Dybå, and Amela Karahasanovic. In Proceedings of the 5th International Symposium on Empirical Software Engineering and Measurement, 2011, pp. 305–314. Author version. © 2011 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The published version is available at: https://doi.org/10.1109/ESEM.2011.39
(II) Construction and validation of an instrument for measuring programming skill. Gunnar R. Bergersen, Dag I. K. Sjøberg, and Tore Dybå. In IEEE Transactions on Software Engineering, Vol. 40, No. 12, pp. 1163–1184, 2014. Author version. © 2014 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The published version is available at: https://doi.org/10.1109/TSE.2014.2348997
(III) Programming skill, knowledge, and working memory among professional developers from an investment theory perspective. Gunnar R. Bergersen and Jan-Eric Gustafsson. In Journal of Individual Differences, Vol. 32, No. 4, pp. 201–209, 2011. Author version. This article does not exactly replicate the final version published in the journal Journal of Individual Differences. It is not a copy of the original published article and is not suitable for citation. The published version is available at: https://doi.org/10.1027/1614-0001/a000052
(IV) Evaluating methods and technologies in software engineering with respect to developers’ skill level. Gunnar R. Bergersen and Dag I. K. Sjøberg. In Proceedings of the 16th International Symposium on on Evaluation and Assessment in Software Engineering, 2012, pp. 101–110. Author version. The published version is available at: https://doi.org/10.1049/ic.2012.0013