This is the online replication package for the ICPC 2018 Technical Research paper:
Sarah Fakhoury; Yuzhan Ma; Venera Arnaoudova; Olusola Adesope The Effect of Poor Source Code Lexicon on Developers’ Cognitive Load (Inproceeding) Proceedings of the International Conference on Program Comprehension (ICPC), pp. 286–296, 2018.
and the Empirical Software Engineering (EMSE) special issue extension of the ICPC 2018 paper.
Sarah Fakhoury; Devjeet Roy; Yuzhan Ma; Venera Arnaoudova; Olusola Adesope TBD
This replication package includes the code snippets used for control group and three treatment groups (LA, Structural, LA & Structural).
The main contributions with respect to the previous work are as follows:
- Experiment conducted with a larger set of participants, we expand our participant pool from 15 to 25 participants, adding 10 new subjects. We answer the original research questions (RQ1 - RQ4) on the 25 participants. 2.Expanded set of research questions, we answer four new research questions for a total of 8 research questions. The contributions of the four new research questions (RQ5 - RQ8) are outlined next.
- RQ5: A quantitative and qualitative analysis of identifiers from control and lexical snippets that contain the highest average oxy. We compare the normalized Oxy distributions over identifiers from control snippets and those from linguistic treatment snippets. We also rank identifiers in control snippets and lexical snippets by the average normalized Oxy to determine which identifiers have high cognitive load in both treatments.
- RQ6: An assessment of participant performance in terms of success rate and task duration. Difficult tasks are expected to cause an increase in cognitive load of developers but also they could have a lower success rate and higher task duration. We want to determine how structural and lexical inconsistencies affect these performance metrics. 5.RQ7: A quantitative and qualitative analysis of identifiers from control and LA snippets that contain the highest fixation durations. Fixation duration has been shown to be positively correlated with cognitive effort. In this research question we want to determine if fixation duration increases over identifiers that contain linguistic antipatterns.
- RQ8: An assessment of the consistency of self-reported measures with cognitive load and eye tracking data. We use answers from the post analysis survey to determine the self-reported difficulty rating of each task. We also calculate the average Oxy and fixation duration per participant task, and perform a pairwise test for correlation between the three metrics.