The Brutes were part of the Covenant. Unless you are referring to the Covenant as pre-Schism and the Brutes as post-Schism. In that case, I'll pick the Covenant. I enjoyed the variety in enemy types between the Elites, Brutes, Grunts, Jackals, and Drones. With the post-Schism Covenant, the Elites were no longer an enemy, and they were just so much fun and challenging to fight in CE, H2, and Reach. I especially enjoyed taking them out first and watching their underlings scatter without a leader.
As for the Forerunners, I'm assuming you are referring to the Prometheans... and fighting those just kinda becomes a grind. In H4 the combat loop was completely thrown off because the Watchers were too overpowered. They could shield their allies, block grenade attacks, and resurrect their allies. So you HAD to take them out first. There was far less variety in fighting Prometheans because you always had to take Watchers out first. Then the Knights were just too overpowered as well and could teleport at will with a devastating melee attack, meaning you had to take them out from range and couldn't get in close. Then H5 fixed some issues by adding the Armigers and increasing enemy variety... but then H5 continued the grind fest by loading the entire battle field with way more enemies during each encounter. This appears to have been done because you had yourself and three friends to fight alongside. But when you play by yourself, the number of enemies just gets to be too much and turns almost every encounter into an over-saturated shooting gallery.