Groups can indeed only be used to turn services ON not OFF. One possibility would be to use Context Aware Access, add the CAA rule to a group and the Gmail service and add the user to the group. One option could be to use location-based and set them to only allow access if they are in some random country they aren't in for sure. Or a specific IP Address or anything that prevents them from accessing it.
It's not the nicest solution but it works for now. I am sure there is a better solution to it but this was my first thought process.