Is there a good library to use for gathering user input in Linux from the mouse/keyboard/joystick that doesn't force you to create a visible window to do so? SDL lets you get user input in a reasonable way, but seems to force you to create a window, which is troublesome if you have abstracted control so the control machine doesn't have to be the same as the render machine. However, if the control and render machines are the same, this results in an ugly little SDL window on top of your display.
Edit To Clarify:
The renderer has an output window, in its normal use case, that window is full screen, except when they are both running on the same computer, just so it is possible to give the controller focus. There can actually be multiple renderers displaying a different view of the same data on different computers all controlled by the same controller, hence the total decoupling of the input from the output (Making taking advantage of the built in X11 client/server stuff for display less useable) Also, multiple controller applications for one renderer is also possible. Communication between the controllers and renderers is via sockets.
I deleted the "answer" that led to the clarification above... I think the clarification needs to stay, but it covers enough that my pseudo-answer, which could've been a comment, can be deleted entirely.
This answer + the GPM answer by Brian answers my question, but I can only select 1 answer.
Note that GPM will probably not interact comfortably with X active on the same terminal... read the end of the man page "BUGS" section carefully. Even if it looks workable, I'd go to the X server for both kinds of event.