Make the cheat guard more resilient to students' tricks
Closed this issue · 9 comments
I've been using this plugin for a while and I noticed that students always find a way to cheat my rules ;)
First they found that doing repetitive small actions was increasing their level (like login in). Then I zeroed crud default rules from code.
Now I only give points when they submit tasks and questions, and they found another trick: resubmitting tasks and quizzes when they can (I can restrict the number of attempts in quizzes but not in tasks).
I don't know if it's possible to extract the number of submissions when the plugin processes a "submit task" event but it would be nice to have a rule that only triggers the first time a user sends a task, to avoid cheaters.
Maybe it could be done generalizing for all activity events: a meta-event generated by xp plugin associated to the first time an event is triggered for a specific activity and user?
I wish I could do it myself but I'm still in the process of learning more about Moodle development and PHP programming!
Temporarily I did this: I created a role based on Student that didn't have the "block/xp:earnxp" capability, and I'm applying the role to the students that I detected where cheating.
Hi,
Nice workaround, thanks for sharing.
It is possible to develop a stronger cheat guard, the current one was the best compromise for having something that works decently while not impacting performance too much.
I would like the option of a lightweight cheat guard to remain, but adding a more robust one which is known for being more demanding on the database. The way I would do it is by recording the events in the database and in the session, and repopulate the list of events on login. However, this will still allow a student to open two sessions simultaneously to trick the system. The even more robust solution is to constantly query the database before attributing them points, that's also the most costly.
Hi Fred,
the current cheat guard also has another problem.
I think you designed it to address the refresh page cheat and it works great for that. But sometimes a user maybe doesn't get legit points due to the time restrictions, for example if a user uploads two different tasks in a small period of time. That's a problem, because from a motivational point of view the cheat guard should not cause false positives (a user should always get their points when they deserve it) and secondary pursue the cheaters if it can. I've had complaints from some students about this so I disabled the cheat guard to be sure (zero time values in config).
If I've got time and mental energy I will try to do a cheat guard for my special case: checking that the event hasn't been pointed before to avoid the multiple submission cheat.
Talking about implementation, I understand the performance problem that this and others checks could cause. I've read the code a little and I understand that the event observer can't do too much job (or I am wrong?), but it could queue events for future background processing (API task?), and then it could do a more robust check in the future... What do you think?
I wouldn't delay the events to future processing because students need to have immediate feedback. Getting XP the next time they login will not be a great motivator, I don't think. My suggestion is to keep the cheat guard log in the session but save it to the database every now and then, to restore it upon login. This should cope with simultaneous sessions though.
The alternative is to use add more information to the logging table and use that as a determining factor, however we may end-up storing too much information there.
I didn't thought about that, you're right... Immediate feedback is at least as important as right feedback.
Now that I inspected your code I have a little more understanding about the session guard log, but I still need to figure out what's inside "yui" folder... I need to catch up!
Talking about not cheating, I think default CRUD rules are great for starting with the plugin but have two major flaws:
- They encourage cheating (or at least discovering how to cheat).
- They are not related to doing real learning/effort in the platform.
A more interesting default rule set could include giving points for event task submission/quiz submission. And if the points given could be the maximum total points of the activity related to the event that would be orgasmic! :)
That would mean adding more data to the event, at least for having a reference to the activity. And that could kill 2 birds with one stone, because now we could do a cheat guard that cared about multiple submissions!
Well, maybe the event log table can't be used now because you purge it every day, but a similar and permanent table with more event info could be added. You would only add rules there that didn't cheat. And then it could be used also to recalculate total points if rules changed! I sense this will be banned for performance reasons ;)
Another idea is that each rule could have associated a different cheat guard. For example, the current cheat guard is appropiate for CRUD events (login), but not for submission events. Ideally I would like to choose which cheat guard to associate with each rule, or leave defaults.
Another flaw in general is that a student doesn't now the rules to increase their points. It's difficult to play a game when you don't now the rules!
So today, with my current understanding, my wish list would be:
- being able to remove default CRUD rules from GUI.
- showing the rules to the students.
- add a default rule associated with event task/quiz submissions.
- Create a rule type that could get points from the linked activity.
- Create a cheat guard that checks multiple submissions.
- Being able to associate different cheat guards for each rule.
Tell me your thoughts! So this was only to talk about cheat guard but everything is so connected that I could not restrain ;)
Hi Canx,
Thanks for all those suggestions, it's great!
While I agree that all of these suggestions could be great for many, I believe that a key feature of Level Up is the easy at which you can get it working. Moodle has so many features that it imposes a very steep learning curve and is hard to get around when you're not familiar with it. As much as possible, I would like Level Up to remain as easy to get around as possible.
A few others have expressed their will to change the default rules, or remove them. It's interesting because to ignore them completely you can just add a rule which captures everything and gives 0 points. I understand that it's not clear to users, so I'm all for improving this aspect. However, for the reasons mentioned above, I would prefer if the default rules remained a default just to make the block a drop it in and it works plugin. Someone (wasn't it you?) also suggested site-wide default rules, and import/export, that could be useful too.
Yes, default rules definitely have their flaws, I made them and I know close to nothing about pedagogy. Though, they are a good way to start, and their advantage (as default rules) is that students get instant gratifications for doing stuff. It might not even be needed for them to be told how much they get for doing something if they get additional XP relatively often. I understand that some students will try to trick the system, some will not, and some won't care about the XP altogether.
Let's also not forget that different tutors will want to set things up differently. Perhaps you have a course that does not have any assessment, though you still want to motivate your students for reading through the content. Perhaps you only want to reward them when they submit an assignment. Perhaps your students are very competitive, perhaps they aren't.
I believe that what matters is finding the right balance between being specific, or too customisable, and being too strict or against most common use cases. There will be times where custom development is more appropriate than introducing new features in the plugin.
- being able to remove default CRUD rules from GUI.
** I agree that work needs to be done about them. Rather than trying to find the solution, how would you describe the problem that you're facing? #26? - showing the rules to the students.
** Sometimes the description of a rule can be too cryptic for a student. But, ultimately I don't see why we couldn't make the rules visible to students. - add a default rule associated with event task/quiz submissions.
** I don't mind changing the default rules, though if possible I'd like to have evidence backing up the changes to be made. As said, it's very important that the block works without the need for pages of documentation. - Create a rule type that could get points from the linked activity.
** Again, simplicity over complexity. Why not though, if we find a sensible way to make it work. - Create a cheat guard that checks multiple submissions.
** I'm not sure what you mean by that. - Being able to associate different cheat guards for each rule.
** I suspect that this will only come useful to a very low percentage of users, which makes me think that perhaps it should not be part of the plugin.
I want to make it clear that I really appreciate your time and research and am not trying to discard your comments to keep the plugin "the way I want it". My point is really to add features which are wanted, not features which could be used by a very low percentage, maybe. This is simply to keep the plugin as enjoyable as possible, and as friendly as possible for the large majority of users.
Few things to clarify about the cheat guard. It's meant to catch actions within a relatively short time frame, if an event should only be captured once, then this is more to do with a rule than the cheat guard. We're trying to prevent cheating, not to control what a student can earn.
The logs are kept 3 days by default, but can extend to "forever". Storing more information in the log table ends up duplicating what is already meant to be stored in the "log stores" (a Moodle-wide stuff). We could use the log stores for a lot of things, but we cannot guarantee what they contain, that's the problem. We shouldn't end-up storing everything that happened.
There was one more thing I wanted to comment about but I forgot what it was.
Thanks a lot for your input!
Fred
PS: Shameless ad, I can do custom development ;-)
Hi Fred,
yeah, I agree that is difficult to keep balance between simplicity and adding more features, and it's also difficult prioritize features. I admit I'm selfish asking for my use case and not thinking in the common benefit ;)
Having said that I'm with you that an easy starting with the plugin is good. I'm not against CRUD rules, they are a great for beginning... As you said, I would only add a way of disabling them more easily than overriding them or touching code.
I just read #53 where he also comments on that and suggests publishing the rules to the students. I'm with you that the standard student wouldn't understand what CRUD rules are... but he/she can understand that submitting a task will earn him 200 points more easily.
You could create manually a rule for each task to earn 200 points but that's too much work and you can forget when you create a new task. A general catch all "submitted task" event would be great but has the problem of not knowing which task was submitted and which one was not. And that's why I would like a "cheat guard that checks multiple submissions". I don't know what percentage of users would need this but we can ask...
Custom development! I wish I had Trump's money... My school has limited funding (Spain has suffered a lot of public funding cuts) and I would need first to evangelize and convince my department to fund on this. It's only an idea but if you had some bountysource or similar add in the README maybe we could guess what really people wants and having time=money to implement features. It's not ideal but better than nothing, I would give my two cents (hope not literally :)
We keep talking!
** I agree that work needs to be done about them. Rather than trying to find the solution, how would you describe the problem that you're facing? #26?
I forgot about this one! yes, this guy that wrote that issue two years ago pointed to the problem ;)
This resilient feature is now available through the add-on Level up! Plus.