So I’m pretty happy about this paper. Neil Borle and Meysam Feghni worked very hard in my grad class on this project. Their work was to investigate if TDD (test driven development, or more specifically test-first) existed with Github Java projects and if it could be shown statistically to improve software quality over projects who were not Test-First followers of TDD.

They worked hard and they came up with a systematic definition of how to identify TDD test-first projects with a threshold for how “test-first” it was. They showed that for a variety of thresholds that TDD on Github Java projects was rare and that there was not a clear signal of quality improvement.

So is TDD more a suggestion than a process actually followed? I know personally from my experience with undergraduates that testing is hard, and testing first is even harder.

Anyways onto the paper and see for yourself:

Paper Title: Analyzing The Effects of Test Driven Development In GitHub

Journal: EMSE

Journal-ID: EMSE-D-17-00057R2

Authors: Neil C. Borle, Meysam Feghni, Eleni Stroulia, Russell Greiner, Abram Hindle from Department of Computing Science, University of Alberta, Edmonton, Canada

EMSE URL: http://rdcu.be/zqhX

Self-archived PDF URL/Pointer: http://softwareprocess.es/pubs/borle2017EMSE-TDD.pdf

Here’s our abstract:

Testing is an integral part of the software development lifecycle, approached with varying degrees of rigor by different process models. Agile process models recommend Test Driven Development (TDD) as a key practice for reducing costs and improving code quality. The objective of this work is to perform a cost-benefit analysis of this practice. To that end, we have conducted a comparative analysis of GitHub repositories that adopts TDD to a lesser or greater extent, in order to determine how TDD affects software development productivity and software quality. We classified GitHub repositories archived in 2015 in terms of how rigorously they practiced TDD, thus creating a TDD spectrum. We then matched and compared various subsets of these repositories on this TDD spectrum with control sets of equal size. The control sets were samples from all github repositories that matched certain characteristics, and that contained at least one test file. We compared how the TDD sets differed from the control sets on the following characteristics: number of test files, average commit velocity, number of bug-referencing commits, number of issues recorded, usage of continuous integration, number of pull requests, and distribution of commits per author. We found that Java TDD projects were relatively rare. In addition, there were very few significant differences in any of the metrics we used to compare TDD-like and non-TDD projects; therefore, {our results do} not reveal any observable benefits from using TDD.

Here’s the data and scripts if you need them: https://archive.org/details/tdd_project

@Article{Borle2017,
    author="Borle, Neil C.
        and Feghhi, Meysam
        and Stroulia, Eleni
        and Greiner, Russell
        and Hindle, Abram",
    title="Analyzing the effects of test driven development in GitHub",
    journal="Empirical Software Engineering",
    year="2017",
    month="Nov",
    day="25",
    abstract="Testing is an integral part of the software development lifecycle, approached with varying degrees of rigor by different process models. Agile process models recommend Test Driven Development (TDD) as a key practice for reducing costs and improving code quality. The objective of this work is to perform a cost-benefit analysis of this practice. To that end, we have conducted a comparative analysis of GitHub repositories that adopts TDD to a lesser or greater extent, in order to determine how TDD affects software development productivity and software quality. We classified GitHub repositories archived in 2015 in terms of how rigorously they practiced TDD, thus creating a TDD spectrum. We then matched and compared various subsets of these repositories on this TDD spectrum with control sets of equal size. The control sets were samples from all GitHub repositories that matched certain characteristics, and that contained at least one test file. We compared how the TDD sets differed from the control sets on the following characteristics: number of test files, average commit velocity, number of bug-referencing commits, number of issues recorded, usage of continuous integration, number of pull requests, and distribution of commits per author. We found that Java TDD projects were relatively rare. In addition, there were very few significant differences in any of the metrics we used to compare TDD-like and non-TDD projects; therefore, our results do not reveal any observable benefits from using TDD.",
    issn="1573-7616",
    doi="10.1007/s10664-017-9576-3",
    url="https://doi.org/10.1007/s10664-017-9576-3"
}