I guess it is a little harsh, but considering the number of games I've played with coop that don't have this problem I still think it's fair enough. Games that don't have these problems as follows:
COD
ESO
Gears of War
Destiny
...
COD
ESO
Gears of War
Destiny
...