April 9, 2019
Every single day we receive tens of thousands of students submissions across hundreds of programming projects on Mimir Classroom. Since we can automatically evaluate all of these submissions, instructors can choose to allow unlimited submissions on projects without adding more work to their plate; however, limiting submissions is also an option and can alter how students tackle projects.
We decided to take a look at projects that limit submissions to better understand the impact.
Instructors can set the submission limit to a number greater than 0 or select unlimited submissions. Pulling data from 2014 to early 2019, we found that we had a pretty close split between projects that allowed for unlimited submissions (44%) and those that had a maximum limit set (56%).
The graph below shows the distribution of projects by submission limit. About 30% of projects allowed for less than 10 submissions and we found no correlation between the level of the course and the submission limit.
Looking at student submission patterns based on the limits imposed on the project resulted that limiting submissions to 20 or more was equivalent to allowing unlimited submissions. More interestingly, when students have fewer than 10 submissions, they are more likely to get a passing score in their initial submission.
We spoke to instructors who offer fewer than ten submissions and those that offer unlimited and learned the following:
If you use automated grading in your computer science courses and would like to share you experience with us, reach out at email@example.com.