Every single day we receive tens of thousands of students submissions across hundreds of programming projects on Mimir Classroom. Since we can automatically evaluate all of these submissions, instructors can choose to allow unlimited submissions on projects without adding more work to their plate; however, limiting submissions is also an option and can alter how students tackle projects.
We decided to take a look at projects that limit submissions to better understand the impact.
Instructors can set the submission limit to a number greater than 0 or select unlimited submissions. Pulling data from 2014 to early 2019, we found that we had a pretty close split between projects that allowed for unlimited submissions (44%) and those that had a maximum limit set (56%).
The graph below shows the distribution of projects by submission limit. About 30% of projects allowed for less than 10 submissions and we found no correlation between the level of the course and the submission limit.
Looking at student submission patterns based on the limits imposed on the project resulted that limiting submissions to 20 or more was equivalent to allowing unlimited submissions. More interestingly, when students have fewer than 10 submissions, they are more likely to get a passing score in their initial submission.
We spoke to instructors who offer fewer than ten submissions and those that offer unlimited and learned the following:
- Instructors that offer unlimited submissions often hide the results of a few of their automated test case until after the due date (using one of Mimir Classrooms features). This way students still have to debug on their own to make sure their code covers all the requirements.
- Instructors that offer fewer than ten submissions usually do not hide any of their automated tests and ask students to debug on their own before submitting and treat every submission like a final one.
If you use automated grading in your computer science courses and would like to share you experience with us, reach out at hello@mimirhq.com.