I think this is better than a direct user-queue Many2Many, at least for large organizations where tens of people may have the exact same configuration. I also agree that it is better to have a separate entity to store a group of queue. In that case, shouldn't the priority set per queue / per group ? i.e. a given queue could have different priorities depending on the group they are in.
The use case would be two "teams" with the same qualifications, but different priorities. For instance, they both are able to manage invoices and payments, but one will prioritize invoices while the other will first work on payments.