But even if we completely remove malicious intent, open sourcing the engine and allowing people to connect with non-standard assets or non-standard binaries basically throws all integrity out the window. What I'm saying is it would make zero sense to have two people playing in the same game with different assets and different rules. You would really need a server to arbitrate the game (and the server would have to be proprietary or at least unmodifiable to the participants in the game, which just shifts the open source question to the server). The server would have to decide if two players can play in the same game compatibly under the same rules. Even if both players don't WANT to cheat, it's still possible that their mod would make the game unfair whether intentional or not.
As far as sending an enemy's position only when they're supposed to be "in sight": a few games have tried this, but it's really unreliable and degrades the quality of the game. Anyone with more than 50 ms ping or so is going to be at an enormous disadvantage: they won't see enemies at all until after they've already been in view for a fraction of a second. By the time they have a chance to react, they're dead. For twitch games, there are some things that you just can't take away from the client. You can't make it a VNC client. The latency is too high.
Maybe for a turn based strategy game you could go the VNC route and ensure absolute security. And you could open source the server and let people run whatever mods they want on the server. That could work.
FPSes present real engineering challenges though because you have to balance playability and hackability, and unfortunately FPSes are the most likely genre to get hacked in the first place. Start introducing open source mods into this mix and it's just ugly.
Something like a spectrum of "open source suitability / feasibility" for games would be :
Single player = Very suitable
Multiplayer turn based = Somewhat suitable
Multiplayer real time = Not very suitable
Multiplayer FPS (or other twitch game) = Not suitable at all