I am well into making an implementation of this feature and had programmed in that constraint upon your suggestion. However, in my use case (e-commerce web app) all of the created sales followed those constraints already so it wouldn't really be a factor.
I added a boolean field to the sale called "prepaid" to indicate a sale is intended to be paid before it is invoiced, and when that value is true it performs extra processing. On payments that reference a sale I do a check against the total amount of the sale less the total amounts of any previously created payments that aren't in failed state (including pending payments--those that have not yet gotten past the processing state):
sale.prepaid_remaining = sale.total_amount - sum(p.amount for p in successful_payments)
sale.prepaid_pending = sum(p.amount for p in pending_payments)
max_amount = sale.prepaid_remaining - sale.prepaid_pending
This way we can remind the user of an over-payment before the creation and posting of an invoice and do some automatic stuff like automatically create new payments with the amount set to max_amount and so on. It is still possible to force over-payment.
Anyways once our sale_payment module is functioning reasonably well (hopefully by the end of the month) I will be submitting it for proposed inclusion in Tryton. If you have already got something in the works let me know and perhaps we can combine our efforts