Predicting Defective Visual Code Changes in a Multi-Language AAA Video Game Project

Kalvin Eng and Abram Hindle and Alexander Senchenko

2023/08/10

Predicting Defective Visual Code Changes in a Multi-Language AAA Video Game Project

Authors

Kalvin Eng and Abram Hindle and Alexander Senchenko

Venue

Abstract

Video game development increasingly relies on using visual programming languages as the primary way to build video game features. The aim of using visual programming is to move game logic into the hands of game designers, who may not be as well versed in textual coding. In this paper, we empirically observe that there are more defect-inducing commits containing visual code than textual code in a AAA video game project codebase. This indicates that the existing textual code Just-in-Time (JIT) defect prediction models under evaluation by Electronic Arts (EA) may be ineffective as they do not account for changes in visual code. Thus, we focus our research on constructing visual code defect prediction models that encompass visual code metrics and evaluate the models against defect prediction models that use language agnostic features, and textual code metrics. We test our models using features extracted from the historical codebase of a AAA video game project, as well as the historical codebases of 70 open source projects that use textual and visual code. We find that defect prediction models have better performance overall in terms of the area under the ROC curve (AUC), and Mathews Correlation Coefficient (MCC) when incorporating visual code features for projects that contain more commits with visual code than textual code.

Bibtex

@inproceedings{eng2023ICSME-defect-visual-code,
 abstract = {Video game development increasingly relies on using visual programming languages as the primary way to build video game features. The aim of using visual programming is to move game logic into the hands of game designers, who may not be as well versed in textual coding. In this paper, we empirically observe that there are more defect-inducing commits containing visual code than textual code in a AAA video game project codebase. This indicates that the existing textual code Just-in-Time (JIT) defect prediction models under evaluation by Electronic Arts (EA) may be ineffective as they do not account for changes in visual code. Thus, we focus our research on constructing visual code defect prediction models that encompass visual code metrics and evaluate the models against defect prediction models that use language agnostic features, and textual code metrics. We test our models using features extracted from the historical codebase of a AAA video game project, as well as the historical codebases of 70 open source projects that use textual and visual code. We find that defect prediction models have better performance overall in terms of the area under the ROC curve (AUC), and Mathews Correlation Coefficient (MCC) when incorporating visual code features for projects that contain more commits with visual code than textual code.},
 accepted = {2023-08-10},
 author = {Kalvin Eng and Abram Hindle and Alexander Senchenko},
 authors = {Kalvin Eng and Abram Hindle and Alexander Senchenko},
 booktitle = {2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) Industry Track},
 code = {eng2023ICSME-defect-visual-code},
 date = {2023-10-01},
 doi = {10.1109/ICSME58846.2023.00062},
 funding = {NSERC Discovery},
 location = {Bogotá, Colombia},
 pagerange = {485-494},
 pages = {485-494},
 rate = {},
 role = {Co-Author},
 title = {Predicting Defective Visual Code Changes in a Multi-Language AAA Video Game Project},
 type = {inproceedings},
 url = {http://softwareprocess.ca/pubs/eng2023ICSME-defect-visual-code.pdf},
 venue = {2023 IEEE International Conference on Software Maintenance and Evolution (ICSME) Industry Track},
 year = {2023}
}