Skip to main content

Forums / Community / Matchmaking Feedback & Discussion

[Locked] MATCHMAKING FEEDBACK UPDATE – FEB 20

OP ZaedynFel

  1. 1
  2. ...
  3. 2
  4. 3
  5. 4
  6. ...
  7. 5
ZaedynFel wrote:
If you mess with Elo’s math to force Onyx to be smaller it’ll mess everything upI don’t need to touch the math, just the boundaries on the raw numbers that are used to decide your ranks. For example, right now, 1500 is the boundary for Onyx. Maybe that needs to be 1700. But your raw CSR and the math behind it would stay the same.

As a side note, I reserve the right to mess with Elo’s math as much as I want since I've published academic papers in that type of math(s).
Fair points. Any chance you could point us to some of those academic papers, or other recommended reading in this field? I'm genuinely interested by this topic and would love to read more about recent innovations and research. I've got a masters in applied math (ahem... nerd), but my job is much more focused on statistics than anything specific to ranking systems.

ZaedynFel wrote:
Not Using the Average Skill in PartyI’ve heard people suggest that we should use the max skill of a party instead of its average in order to discourage partying up with smurfs. I’m not opposed to that and may try it in the future, but:
  • I want to first to see how well the skill improvements alone can mitigate this problem.
  • If still needed I may do it for only Team Arena
  • A softer option is to force your CSR to move towards your fireteam’s average after every match rather than your personal MMR. For example, in the case of an Onyx player playing with Bronze, both will see their CSR drift towards Gold over time. This will have the result of discouraging smurfing since it will drag the better player’s CSR down. In cases where your friends really are bad, it also will still give the more balanced matches for everyone that we have today. It does have the downside that when you play with bad friends, your CSR goes down. But so does matchmaking on the max. I think this option is better overall.
Thanks. I really appreciate that you're looking into doing something to discourage smurfing (which has become a really annoying problem in multiple playlists - I match smurfs once every ~5 games) - your suggestion of forcing CSRs towards the fireteam average after every match is reasonable, since halo is SUCH a team-oriented game that it seems near impossible to accurately determine the relative contributions of two players in a game when they have dramatically different skill levels. So, slowly forcing both players' CSRs towards the team average emphasizes the team aspect of Halo. I think the "downside" you mention is tolerable if it decreases smurfing... If you're legit playing with worse players but not boosting, you probably don't care much about your rank anyway! Plus you're playing against lesser skilled opponents so it's not a fair assessment of your supposed champ-level skill to run train on plats. ;-)

ZaedynFel wrote:
Team Rankings vs. PersonalYes, I like the idea of having a Personal Ranking that moves based on individual in-game performance, and one that is based on winning. I would want these both separate and want to make it clear that the winning version is more important.

Keep in mind we don’t use Arpad Elo’s method for Ranks. We use something more advanced. We can integrate Kills, Deaths, Assists, Headshots, Objectives, etc., into your skills. We can also create a win-based one CSR separate from your individual performance-based one. I’m constantly evaluating which would be best for each part of the game.
So are you saying that our current MMR/CSR ranks are based both on wins and on individual performance stats, but a future idea is to separate into two CSR numbers, one based solely on wins (Elo-like), the other based solely on performance stats? I would strongly support that.

I'm a huge skeptic about incorporating individual performance stats into primary MMR skill ranking (and therefore influencing CSR) in a team game like Halo... I know you currently do it that way, and this is an age-old debate. But why do you feel so strongly that incorporating performance stats is necessary, when Halo skill is SO situational and performance stats never tell the whole story? I've watched my games in theater mode thinking "I went HAM that game, how the heck did we lose?", and discover that my higher-ranked teammate, who had great stats, cost us the game by playing selfishly/conservatively. In one case, he even had awesome objective stats (lots of stronghold defense/secured - although turns out none of them were at critical moments), but in watching his gameplay I discovered that he kept letting us die and preserving his own life by not getting in the strongholds to stall or maintain progress at critical moments-- we lost 100-95, largely because of him.

Often, the "best" player on the team is taking on more risk, aggressively challenging the key map positions / weapons / powerups, and may have worse stats! I see this in practice all the time - I'm a Diamond level player but often end up with better post-game stats than onyx teammates when solo-queuing, because they're challenging for map control and trying to make my life easier.

ZaedynFel wrote:
Can I show the bell population of the ranks?I won't at the moment, but since enabling demotions, it’s smoothed out quite a bit. The amount of players within each rank among those who have completed placement is at the intended distribution, except for perhaps Onyx being larger than intended, which I spoke on last week.
I wrote Python software to scrape the Halo 5 API game histories and player ranks, to generate these bell curves and "percentiles" to go along with them. It took a lot of time to scrape the API, so I stopped doing it after a few months last winter, but the response on Reddit was really positive! I think a lot of folks would love to get a sense what percentile they fall into, so I'd love to see official data on this matter if you are able to publish it at some point. For context:
Thanks for taking the time to post so many of your thoughts for community discussion. Any specifics on when the changes will go live?
1 thing I would like to see addressed in MM is the how the players are matched up. I have solo queued my way to diamond playing with the odd good player but majority of the time many teammates were struggling to keep a positive KD going 2-10 or 5-13. Granted they may not be the best of players but still I would like to have a good team when I play Ranked rather than be matched up with players below my level. Due to this I have deranked from D5 to D1 because of trying to solo queue and being match with bad players.
In my opinion all free for all game modes should give a win on record for the top 3 players at the end of a game. This includes ranked ffa but also modes like Infection
Guys, we get it. You don't like the new Breakout. Use the feedback thread for that at this point instead of co-opting every new thread for your soapbox.

Thanks!
I will gladly do that right after I point out that Shotguns and Breakout were brought up in the one dudes update. If you do not want people to respond to something do not bring it up. Now I will go back to my soapbox and dream of a world where I can play the original Breakout.
What's a soapbox?
We should all know that these changes are not coming anytime soon or even never. I believe this so called "updates" ZaedynFel are giving us are just an explanation of issues that the community are complaining about and he just talks about it with his own opinion and giving possible "solutions" that hes supposedly "discussing" with the "343'' team.

Some or probably most of the issues he talks about are not possible to fix as he even said it. I really don't see the point of posting this threat of "updates" that will never be fixed. i understand that they might be collecting information and data from the community to try to solve these problems but like i said, is it really necessary to make a separate treat to discuss this so called "updates"? we have a complete rant chaos on individual player treats of people complaining about matchmaking. go ahead and take a look 343 and tell me that you don't see a single complaint below. its honestly congested of complaint after complaint.

Anyways I'm just sharing my opinion with logic and evidence so read carefully before doping a negative storm on me.
Jynx Fatal wrote:
We should all know that these changes are not coming anytime soon or even never. I believe this so called "updates" ZaedynFel are giving us are just an explanation of issues that the community are complaining about and he just talks about it with his own opinion and giving possible "solutions" that hes supposedly "discussing" with the "343'' team.
ZaedynFel, could you possibly clarify this point for us? In any case, appreciate your taking the time to post here.
VETOED wrote:
Glad to see a lot of these issues being talked about. I can tell this will result in massive improvements in the ranked matchmaking experience over time if those changes come to fruition. However
ZaedynFel wrote:
Aren’t people quitting because they don’t like the Map and / or Mode?For most playlists, there’s no pattern on which maps and / or modes people quit the most. It’s equal. This is the case in team arena. This means removing a map-mode combination wouldn’t make a difference in quit rates. Neither would vetoes because players are just as likely to quit the alternative as they are the vetoed map.
This part, I don't think is 100% accurate. Sure, DNFs tend to be equal across the board, and people tend to just leave games when getting stomped or matching a pro team that makes them scared, but there's a significant amount of map dodging in certain hoppers, just based on maps alone. Unfortunately, in things like FFA where the population is low at high ranks, dodging means a significant increase in search times. I'd recommend looking into this further for hoppers where the experience hasn't been as refined as Team Arena, such as FFA, Doubles, Snipers, etc.
I concur. Do your stats about quitting games include when people dashboard and quit that way as opposed to quitting once a game has already started? There are definitely maps where you see people get queued with you on say, Orion 2, then "mysteriously disappear" before the game starts so the matchmaker has to find a new player again. On some maps this happens repeatedly on a regular basis.

Quote:
MMR moves much faster in FFA than CSR. This means the system knows much sooner where to match you than is visibly represented in CSR.
Why is that? I know this example is specific to FFA, but if this issue happens in one playlist, it must happen in others even if its to a lesser extent? Why can't the system be designed so that the skill metric being used to create matches is represented exactly in the visual skill indicator? If my MMR is higher or lower than my CSR, it will present as if im getting unfairly matched against players even if that's not the case.
Jynx Fatal wrote:
We should all know that these changes are not coming anytime soon or even never. I believe this so called "updates" ZaedynFel are giving us are just an explanation of issues that the community are complaining about and he just talks about it with his own opinion and giving possible "solutions" that hes supposedly "discussing" with the "343'' team.

Some or probably most of the issues he talks about are not possible to fix as he even said it. I really don't see the point of posting this threat of "updates" that will never be fixed. i understand that they might be collecting information and data from the community to try to solve these problems but like i said, is it really necessary to make a separate treat to discuss this so called "updates"? we have a complete rant chaos on individual player treats of people complaining about matchmaking. go ahead and take a look 343 and tell me that you don't see a single complaint below. its honestly congested of complaint after complaint.

Anyways I'm just sharing my opinion with logic and evidence so read carefully before doping a negative storm on me.
Makes negative, tell people to read before dropping a negative post on him.

You sir generate enough negativity, I'm sure you will be fine.
Jynx Fatal wrote:
We should all know that these changes are not coming anytime soon or even never. I believe this so called "updates" ZaedynFel are giving us are just an explanation of issues that the community are complaining about and he just talks about it with his own opinion and giving possible "solutions" that hes supposedly "discussing" with the "343'' team.

Some or probably most of the issues he talks about are not possible to fix as he even said it. I really don't see the point of posting this threat of "updates" that will never be fixed. i understand that they might be collecting information and data from the community to try to solve these problems but like i said, is it really necessary to make a separate treat to discuss this so called "updates"? we have a complete rant chaos on individual player treats of people complaining about matchmaking. go ahead and take a look 343 and tell me that you don't see a single complaint below. its honestly congested of complaint after complaint.

Anyways I'm just sharing my opinion with logic and evidence so read carefully before doping a negative storm on me.
It's pretty clear from this post that you have no idea who Josh Menke is....
This post has been edited by a moderator. Please refrain from making non-constructive posts that do not contribute to the topic at hand.

*Original post. Click at your own discretion.
Spoiler:
Show
RzR J3ST3R wrote:
Jynx Fatal wrote:
We should all know that these changes are not coming anytime soon or even never. I believe this so called "updates" ZaedynFel are giving us are just an explanation of issues that the community are complaining about and he just talks about it with his own opinion and giving possible "solutions" that hes supposedly "discussing" with the "343'' team.
ZaedynFel, could you possibly clarify this point for us? In any case, appreciate your taking the time to post here.
The updates are exactly what they appear to be. In fact, at least one improvement went live within 2 days of me discussing it (high onyx scoring).

Any feature that I didn't say, "we can't do this for Halo 5" is probably doable, and in most cases I have gone and done the research on exactly how and when we could do it. Some of these features are already in flight, though not far enough a long that I would get your hopes up about seeing them soon. One thing you learn as a designer is that while some improvements can be made quickly, most can not and take time and patience.

Stuff always takes longer than you'd hoped, but gets done sooner than you fear, if that makes sense. Sometimes just when you give up, bam, it's done and in the game.

What I am trying to do with these posts is involve you in the design process. I want to take the same issues you bring to us, tell you why we did things a certain way, and let you give suggestions I can sift through. Ideally, your suggestions need to be defended and supported in a reasonable and mature manner, like any other designer. I tend to focus more on feedback given the same way a designer would.

Once we, working together, get a well-baked suggestion that is doable, it will absolutely get into the game.
RzR J3ST3R wrote:
Fair points. Any chance you could point us to some of those academic papers, or other recommended reading in this field? I'm genuinely interested by this topic and would love to read more about recent innovations and research. I've got a masters in applied math (ahem... nerd), but my job is much more focused on statistics than anything specific to ranking systems.
A good starting point is "Online Bayesian Ranking" by Weng and Lin. They reference other papers (and mine) that are helpful if you want to keep going.

RzR J3ST3R wrote:
So are you saying that our current MMR/CSR ranks are based both on wins and on individual performance stats, but a future idea is to separate into two CSR numbers, one based solely on wins (Elo-like), the other based solely on performance stats? I would strongly support that.
Right now it's based solely on wins. My point is it doesn't have to be. We can add in other bits, and at the same time infer how important each bit is. So, if like you say, the better player takes on more risk, gets lower kills, but pulls off the win, it will learn that kills aren't important in that case, and wins will still dominate. So I think we can handle that concern you have nicely because we still hold up winning as the main requirement.

This also lets us do either of the above. We could have a single score that's fair in situations where you carry and lose, but still focuses on the win. Or, we could have two numbers. Still evaluating that.

RzR J3ST3R wrote:
I wrote Python software to scrape the Halo 5 API game histories and player ranks, to generate these bell curves and "percentiles" to go along with them. It took a lot of time to scrape the API, so I stopped doing it after a few months last winter, but the response on Reddit was really positive! I think a lot of folks would love to get a sense what percentile they fall into, so I'd love to see official data on this matter if you are able to publish it at some point.
The graphs are very cool. We do something similar internally. The proportions are roughly:
Bronze (2-3%)
Silver(~13%)
Gold/Plat (~30% each)
Onyx(~10%)

This roughly matches the original plan, except Gold/Plat are each about 4% smaller than planned, and Onyx about 8% bigger. I may look at what I would need to do to change that. Or, at least drastically shrink Onyx.

Side note: The goal isn't to have them be equal in size, but to instead have the skill gaps between them be equal. So Silver beats Bronze as much as Gold beats Silver. So everyone knows what it feels like to play someone above / below them. If you do equally-sized buckets, every gap is different.

RzR J3ST3R wrote:
Thanks for taking the time to post so many of your thoughts for community discussion. Any specifics on when the changes will go live?
Unfortunately, no. Some of this is in progress, but there's quite a bit left to do. No ETA.
Apoll0 wrote:
I concur. Do your stats about quitting games include when people dashboard and quit that way as opposed to quitting once a game has already started? There are definitely maps where you see people get queued with you on say, Orion 2, then "mysteriously disappear" before the game starts so the matchmaker has to find a new player again. On some maps this happens repeatedly on a regular basis.
We have both stats, and we look at all modes / playlists. So we do notice when maps aren't loved. If there are any in there now, we just haven't moved on it yet.

Apoll0 wrote:
Why is that? I know this example is specific to FFA, but if this issue happens in one playlist, it must happen in others even if its to a lesser extent? Why can't the system be designed so that the skill metric being used to create matches is represented exactly in the visual skill indicator? If my MMR is higher or lower than my CSR, it will present as if im getting unfairly matched against players even if that's not the case.
An FFA matches gives roughly 120 times more information about the players in the match than a 4v4 match. This comes from having 15 player to player comparisons from the ordered ranks compared to 4v4 which needs to share 1 comparison over 8 players = 1/8. 15 / ( 1/8) = 120.

So the extent to which this happens in other playlists is tiny in comparison.

MMR serves too different of a purpose for it to be shown visibly. It's not always a fair indicator of what your rank should be, but instead where you should be matchmade at that moment to the best of our knowledge. At this point, it would only be confusing to show it raw. CSR eventually catchces up to MMR, and exactly hits it on season boundaries, enough that I'm not worried about showing more.

In the future, we may have representations of MMR that are helpful rather than confusing, but that's too far off to worry about.
ZaedynFel wrote:
Apoll0 wrote:
I concur. Do your stats about quitting games include when people dashboard and quit that way as opposed to quitting once a game has already started? There are definitely maps where you see people get queued with you on say, Orion 2, then "mysteriously disappear" before the game starts so the matchmaker has to find a new player again. On some maps this happens repeatedly on a regular basis.
We have both stats, and we look at all modes / playlists. So we do notice when maps aren't loved. If there are any in there now, we just haven't moved on it yet.

Apoll0 wrote:
Why is that? I know this example is specific to FFA, but if this issue happens in one playlist, it must happen in others even if its to a lesser extent? Why can't the system be designed so that the skill metric being used to create matches is represented exactly in the visual skill indicator? If my MMR is higher or lower than my CSR, it will present as if im getting unfairly matched against players even if that's not the case.
An FFA matches gives roughly 120 times more information about the players in the match than a 4v4 match. This comes from having 15 player to player comparisons from the ordered ranks compared to 4v4 which needs to share 1 comparison over 8 players = 1/8. 15 / ( 1/8) = 120.

So the extent to which this happens in other playlists is tiny in comparison.

MMR serves too different of a purpose for it to be shown visibly. It's not always a fair indicator of what your rank should be, but instead where you should be matchmade at that moment to the best of our knowledge. At this point, it would only be confusing to show it raw. CSR eventually catchces up to MMR, and exactly hits it on season boundaries, enough that I'm not worried about showing more.

In the future, we may have representations of MMR that are helpful rather than confusing, but that's too far off to worry about.
Ah ok. Would making seasons longer help mitigate this since "CSR eventually catches up to MMR"? It seems like compared to other games with good rank systems, the seasons for Halo are very short. I used to like the short seasons, but see them as more of a detriment now because i feel like im always playing "catch up" with my rank.
Apoll0 wrote:
Ah ok. Would making seasons longer help mitigate this since "CSR eventually catches up to MMR"? It seems like compared to other games with good rank systems, the seasons for Halo are very short. I used to like the short seasons, but see them as more of a detriment now because i feel like im always playing "catch up" with my rank.
Making seasons longer might be worth considering for that reason, good though to consider at this point.
I would love to see the boundary for onyx moved up to 1700.
It's most likely the reason why 343 doesn't wants us play a fair game or they are unable to figure out way to make it fair. Lets be honest they are not Bungie(partner with Activation) and they will never be. They will never be able to create something like Halo3. They don't have the talent. So the results are followed- they take away population counter, good player playing in team and pup stomping (great feeling and easy to rank up) and my favorite force to play in sever that lags. Creating halo war and halo FPS are two different thing. FPS game take knowledge to develop and it is hard game to create. 343 industry needs to really look into how people feels about Halo. There is tons of video posted youtube(google product) with feedback. I have played 10 games on social today and all 10 of them I had a 1 or 2 teammate short while I was paired against a full team. Gamer tag Ronni416 and date of game play Feb 27.
Seeehawks wrote:
ZaedynFel wrote:
Why is there a wider variety of skills in my FFA matches?There are two main reasons, I believe, for this:
  • MMR moves much faster in FFA than CSR. This means the system knows much sooner where to match you than is visibly represented in CSR. So you may have what appears to be a Gold player in your match, but the matchmaker already knows he’s really Bronze. In these cases, rest assured that the matchmaking itself will be fair despite the ranking appearance. The matchmaker will not allow more a GAP of roughly 1 whole CSR rank. (e.g. Gold to Silver) in terms of performance. You’ll see larger gaps when a Bronze player still looks Gold before he’s been demoted, etc.
  • Higher-skilled players in FFA are more likely to wait longer for a match, and more likely to give up on trying to find close matches. Because of this, you’ll see them more in your games. Players that are near the top and bottom will also get matches that are a bit too hard / easy because we won’t have enough players at the ends to match. Note: this isn’t a population size issue, no matter how large your population, you will have sparseness at the ends.
Ranks and Win %Every now and then I hear comments like, “Why do they outrank me if I have a higher win %?” The whole point of a CSR-type system is that because of matchmaking, win percentage is just wrong. It’s not a good measure except for maybe the top 10 or so players in the whole world. If you have a higher win% than someone with a higher CSR than you, it means by definition that they have played harder matches. The harder the match, the more CSR you get for winning, and then less you lose for losing. So if you dodge or play with noobs and get a lot wins vs. other noobs, you’ll have a high win%, but a lower CSR.

Can I show the bell population of the ranks?I won't at the moment, but since enabling demotions, it’s smoothed out quite a bit. The amount of players within each rank among those who have completed placement is at the intended distribution, except for perhaps Onyx being larger than intended, which I spoke on last week.
Can I tell how popular the playlists are?Maybe in a future post, but this can lead to a self-fulfilling prophecy if we’re not careful. I’ll think about it. I can say that we monitor playlist popularity weekly if not daily and it does inform our decisions. It’s not the only thing we look at, of course. As designers we also have more creative reasons for how we set up the playlists.
Why so few players Ranked vs. Placed? Where did they go?I hear this concern now and then from people who see get a glimpse of the stats and notice how many people don’t finish their placement matches. This is totally normal. All games look like this, no matter how new or popular the game and regardless of the ranking system. This is because of casuals who just try a game here or there each season. There are almost always more players who play 1 game than who play 2, more who play 2 than 3, etc. In fact, there are way MORE players who play “0” games if you include everyone in the world. That’s an extreme example, but that’s the main idea on why this happens. A lot of players will try something just once a season, it’s the more dedicated players who play their 10 games, and even more so who play a full season. Remember, the casual audience dwarves the rest by far, and they do come and play a game here or there.

Breakout ShotgunI’ve read a few concerns about starting with shotguns in breakout. First, keep in mind my primary design role isn’t weapon and mode balancing, so I’ll listen to your comments but I won’t personally argue the merits either way. My understanding of why we have shotguns is to increase lethality now that we have shields. So like the original breakout, a player still has a chance of winning 1v3, etc. whereas that would be much harder with the standard loadout.

Fun Facts
  • Many of you have referred to the “ELO System”
  • There is no such thing.
  • ELO is Electric Light Orchestra, a rock band.
  • Arpad Elo was a Hungarian American and a physics professor
  • Arpad created what we call the Elo system, named after him.
  • CSR uses a form of Elo’s system that I have adapted, but it is anchored in the MMR system which is more efficient and accurate at tracking skill.
  • ELO the rock band, will be playing at Wembley Stadium June 24, 2017
  • Wembley Stadium is where the Halo World Champion 2017 EU Qualifiers just took place over this past weekend.
  • So, in some weird way, Elo and ELO have become related
( 2 / 2)
If u want people to able to survive a 3v1 in breakout then maybe take shields back off and go back to standard loadout rather than adding a power weapon start out to a ranked playlist. Rockets could do the trick too....... don't fix it if it's not broken
You took the words right outta my mouth. Breakout used to be one of my favorite playlists, but I can't force myself to play even the 10 placement matches with the current settings. I'd definitely go back to it if they'd revert it to the original.
Thanks for the updates, especially for explaining more on how the MM system works. Really clarified a lot of things for me.
ZaedynFel wrote:
Apoll0 wrote:
Ah ok. Would making seasons longer help mitigate this since "CSR eventually catches up to MMR"? It seems like compared to other games with good rank systems, the seasons for Halo are very short. I used to like the short seasons, but see them as more of a detriment now because i feel like im always playing "catch up" with my rank.
Making seasons longer might be worth considering for that reason, good though to consider at this point.
Making seasons a bit longer like 3 months (so they match roughly the reason seasons :) could help stretch out the limited pool of unique rewards that can be used for completing seasons within certain rank tiers.
  1. 1
  2. ...
  3. 2
  4. 3
  5. 4
  6. ...
  7. 5