Someone working with a modest interface will probably find a modest monitor controller entirely adequate -- as you suggested yourself earlier. But someone working with more upmarket outboard converters will soon discover the limitations... as narcoman discovered.
As I said before, there are ways of minimising the potential problems, but the fact remains that cheap equipment is always cheap for a reason, and most good expensive equipment is expensive because doing it right is inherently more expensive.
Going back to the passive controller idea, within certain physical constraints involving cable lengths and impedances -- which are rarely a problem for most -- there is no quality bottle neck to worry about.
...but you might not always get all the bells and whistles of other active systems.
Technical Editor, Sound On Sound