ll, I actually figured out how to do it via setting up a sub OU, and setting email compliance rule to only allow send/receive for teacher/staff domain, and moved that student into that OU. Tested and it seems to work.
If the Penalized student account emails another student, the email seems to never get sent, (not even a trace of it in the Vault), and despite there being a reject message, the student does not get notified that their email did not go through. This does achieve the end goal (so it seems with my small testing) but it would be easier if could set up that same scenario with a group, as opposed to having to move the student into that penalty box.