I deleted the "answer" that led to the clarification above... I think the clarification needs to stay, but it covers enough that my pseudo-answer, which could've been a comment, can be deleted entirely.
This answer + the GPM answer by Brian answers my question, but I can only select 1 answer.
Note that GPM will probably not interact comfortably with X active on the same terminal... read the end of the man page "BUGS" section carefully. Even if it looks workable, I'd go to the X server for both kinds of event.