Closer Look: Halo Infinite’s Online Experience
Hey, I'm Richard Watson, Lead Engineer of the Sandbox team here at 343. There has been a lot of discussion and feedback given related to the online experience in Halo Infinite’s Multiplayer including examples of various issues and speculation around potential causes, and solutions, to these problems.
The 343 team and myself want to make sure that we can maintain an open discussion about the state of networked play in Halo Infinite, so with that in mind I want to:
- Go over some of the common network-related issues that people are experiencing and explain what's happening
- Discuss our plans for mitigating these issues
- Go over some commonly asked questions and clear up some misunderstandings about the way things work
If you haven’t already, I encourage reading our post on the Waypoint forums covering “geofiltering” that released today as well.
Multiplayer Networking Community Feedback Themes
BEING SHOT / SHOOTING AROUND CORNERS
This seems to be the issue that gets the most vocal response online, so let's explain what's happening. Fundamentally, this has to do with latency compensation. Latency compensation is how we handle the fact that there is a delay between actions on your machine, actions on the server, and the results of those actions being communicated to other players.
Ultimately, we need to ensure that all players are getting the best experience possible. To do this, the server has to manage the actions from various machines that are all running at different latencies. Imagine the server as the arbiter of multiple different timelines and having to reconcile everything that happened in those timelines into something that makes sense for all players.
There are multiple ways that you can solve the problem of latency compensation, but in Halo we choose to favor the shooter. Practically, this means that whatever happened on the shooter’s screen the server endeavors to honor. That means whenever a player sees themselves hitting a target, our system does its best to give them the hit on the server. So, how does this work?
When the shooter fires, we first simulate that locally. The shooter's client creates the projectile and fires it in the direction the shooter was aiming. Locally, the shooter should see all effects that happened based on that shot: the projectile should fire, the shooter will see the tracer, and ultimately the impact that occurred as a result of it. On the same frame, that the shooter shot they also send a message to the server to let it know that they shot.
When the server receives the message that the shot occurred, it will also execute the shot. If the server just fired the projectile the same way that the shooter fired it, there would be a big problem. Since it took time for the message to reach the server, the target that the shooter was shooting at won't be in the same position they were in when they shot on their machine. This would be bad because it would result in a game where the shooter had to constantly lead their target based on their latency to the server.
To ensure that the shooter doesn't need to lead their target, the server rewinds time and places all objects that they might be shooting towards at the position / orientation they were at when the shooter took the shot. This ensures that whatever the shooter saw on their screen happens on the server. The server then resolves the result of the shot and sends that result out to all clients (player’s PC or Xbox).
If the shot was a hit, the shooter should see hit ticks on their screen and the target should take damage or die. It will take time for that message to arrive on the shooter’s machine and the target’s machine, resulting in delay on the hit ticks or damage being taken.
It’s this delay on damage being taken that leads to cases where you could be shot around a corner. If the person shooting you has high latency, then the server will have to rewind time further as it will take longer for the message that the player shot to reach the server. If you have high latency, then it will take longer for the message that you were shot to reach you and you will be a further distance away from where you were shot. If you both have high latency, then the problem becomes even worse.
Let's look at some examples. In Halo Infinite, our players can move very fast – at full sprint, a Spartan moves at approximately 8.5 meters per second, when grappling this is increased to a massive 24 meters per second. The latency that you will experience from a shot is a product of the following calculation (from the moment the shooter sends the signal they fired):
Shooter’s Latency to Server + Shooter’s Command Buffer + Server Processing Time (16 - 33ms depending on tick rate) + Target’s Latency from Server.
The first three parts of this equation (Shooter’s Latency to Server + Shooter’s Command Buffer + Server Processing Time) are the amount of latency compensation we need to perform or how long it takes the server to perform your input, the last part is how long it then takes the server to tell the target what happened.
The Server Processing Time mentioned above is a product of Tick Rate, the number of times the server updates per second, and when the message arrives on the server. Ideally, the message would arrive exactly as the server is ready to process it, but most of the time that won’t be the case. The worst case is that the message arrives right after the server has begun processing messages from everyone, so we have to wait that entire frame (server tick) and then the time it takes for the server to process the next frame before we see a result.
The Shooter’s Command Buffer is a tricky topic. The server wants to execute a consistent stream of inputs from the client and as such assumes that it receives updates at a consistent rate. However, if you have some variability in your connection, inputs might arrive earlier or later instead of at a consistent rate. All connections have some kind of variability out of the box, but this can be exacerbated by peak times or other users on your network using the internet. To account for this variability, we buffer inputs on the server. For a good connection, this should be either 0 or 1 frame of buffering (so an extra 16ms worst case), but for a bad connection this could be multiple frames as we aren't reliably getting data from them.
In these examples, let's assume 0 command buffer and that messages arrive in perfect time for the server to execute them and that RTT (Round Trip Time, the time it takes for data from the client to reach the server and the response be returned to the client) is symmetrical (it takes the same time for data to get from server → client as it does client → server). If you’ve enabled the Network Statistics option in-game, RTT is what you’ll see displayed on screen.
The images below show how far a client would be around a corner by the time a shot resolves at various levels of end-to-end latencies.
Target's View (0ms)
With two clients with 33ms RTT we would see a latency of
16.66 (1/2 target RTT) + 16.66 (Server Processing Time) + 16.66 (1/2 Target RTT), so 50ms.
If you have 33ms RTT but you are shot by a player with 100ms RTT, you are looking at a delay of
50 (1/2 Shooter RTT) + 16.66 (Server Processing Time) + 16.66 (1/2 Target RTT) = 83.33ms.
If you have 33ms RTT but you are shot by a player with 150ms RTT, you are looking at a delay of
75 (1/2 Shooter RTT) + 16.66 (Server Processing Time) + 16.66 (1/2 Target RTT) = 108.33ms.
As you can see, even with both clients with a good connection to the server, you get some degree of being shot around the corner and playing with people with higher latencies makes this worse. This is the case with all online games but is especially noticeable in first and third-person shooters. In short, the higher the value of latency for the shooter and the target combined is, the longer it will take the target to be notified that they were killed. This effect is also the same if the connections are reversed, so if you have a slow connection and the shooter has a good connection you will still feel like you were shot around a corner.
Note: If you’re the shooter and have a good connection, you should still be told that you killed the target in good time. The delay will only be felt on the target’s machine.
So, what can we do to make this [getting shot around corners] better? From the data above, it's clear that we need to reduce players’ latencies to the game datacenter. In other words, we need to give players games on datacenters that they are closer to decrease latency and improve the online experience. This is something that the Halo Infinite matchmaking system was designed to do, but it hasn’t been performing as well as we’d planned, particularly for players in certain low population regions.
Today, we released multiple changes that should reduce players’ latencies, especially for those in lower population regions. Though these may lead to somewhat longer matchmaking wait times, we're eager to see its impact on matchmaking and the in-game experience.
As outlined in this post on Waypoint, the first steps that we took today include:
- As of today, we’ve increased the priority of low pings to our servers in the matchmaking process. This step should help you match with local players, and therefore have a better connection, more often. As you get more of these local matches, instances of playing on further data centers will also decline.
- Separate from the bullet point above, our Services team has taken additional steps to help ensure you can have quality matches with players from your respective regions, more often. We expect this to have a noticeable impact when making matches for players in less populated regions around the world including - but not limited to - Australia and Asia. This update, which went live yesterday, has already shown promising results and should take further advantage of this morning’s changes. Please keep an eye out and let us know how your games start to feel.
- This morning's update should also prevent players from “geofiltering” their online matches. Since this was impacting matches after they were put together via our matchmaker, it was causing unstable connections for the other players in the lobby. To learn more about this and why it’s being prevented, please read this post on Waypoint.
In combination with the changes above and additional tuning to our matchmaking system, we believe that we can create better and better matches over time. If we ever feel like the matches we are making do not meet player expectations or our quality bar, we will evaluate additional means of improving the online experience - including considering features like a server selection similar to MCC or search preferences similar to Halo 5 - further down the road.
We can also do a better job of communicating to the player the network quality of other players in the game. Since it is both your connection quality and other players’ connection quality that affects how long it takes to get feedback when you are shot, we want to give you a way to know how good the other player’s connection is. Our team is working on adding round trip time to the scoreboard so you can tell if you are in a game with someone who doesn’t have an optimal connection.
We are also making sure that if your connection is poor, the game will notify you. We have diagnostics that communicate the state of your connection, but they are only turned on when you choose to have the network statistics enabled. We are changing this so that if the network conditions in the game are not optimal, you will see it on screen.
One area that we do have direct control over is ensuring that we don’t have any issues on our side that may be artificially increasing the Shooter’s Command Buffer or causing unnecessary spikes in latency. We are also about to deploy work in our February update that gives us a better view of the shot latencies that people are reporting, so we should be able to dive into the data and get an even clearer picture as to how the game is playing. With that data we will be able to ensure that our systems are working as expected and respond if we detect any issues.
MELEE AND SHOT REGISTRATION ISSUES
There is nothing more frustrating than when a shot / melee that appeared to land on your screen does not do damage. As explained above: when someone shoots, a message is sent to the server. The server attempts to get its view of the world into the same state that the shooter’s world was in when the shot was fired. If there is a discrepancy between these two views, then the shot may not land on its intended target. The question then becomes, why would the server see the game in a different state to the client? There are three main causes for this.
The first cause is rooted in the fact that the simulation needs to be deterministic. This means that when the client performs an action, the result of that action is the same when the server performs it. If the server does something different than the client, the two machines will diverge. Sometimes nondeterminism is unavoidable, something that the client could not predict happens (e.g. a Warthog hitting you) and the state of the world on the client’s machine is now different to the server’s state of the world. But in most circumstances, especially in Arena environments, the client and the server should be deterministic, but we believe there are some bugs we can address to improve this.
The second cause is due to network bandwidth. In order to replicate the state of the game to the client, we have to send a significant amount of data from the server to the client. Even in a 4v4 arena match there is too much data to send the entire state of the game world every server tick, so we only send the information we deem most relevant to the player based on a number of different heuristics. Players you are shooting at, and those who are shooting you, as well as data about your own player are the highest priority in this system, but if bandwidth is constrained or there is a lot going on around you, we may not be able to send all that data every frame.
The third is due to the fact that there is latency in the simulation and it occurs in situations where players would expect to trade or win a close battle but actually die. The problem lies in the fact that it takes time for you to be told that you died. You will die on the server before you are told about it on your client. In that window, any shots you make will be ignored when they reach the server, as you are already dead. For melees, however, we do have a special code path that should allow the melee to go through even if you died on the server, but we do not do this for shots.
On our side, we have some initial improvements that should be landing in a future update that should both improve determinism and bandwidth usage, and our test teams are constantly looking for scenarios where our determinism systems break.
Player reports are very useful here to help us uncover bugs. I’d encourage you to report issues through the Halo Support site when you find them with client-side clips. We can analyze these locally and hopefully track down what failed and caused the client and server simulations to diverge. Your reports are incredibly helpful in helping us track down edge cases in our systems and I’d like to thank everyone who reports problems.
I’d also advise all PC players to set Simulation Quality to Ultra in the settings. This setting drives a few of our game systems but, most importantly for this discussion, it controls the amount of time we give to our system that corrects determinism errors on the client. Setting this to Ultra should result in better determinism between client and server. The CPU cost of this is also quite low, so it should be manageable on most PCs.
For posthumous shooting, we have a work item on our backlog to look at shots post death to better represent what players see on their screen, keep an eye on the patch notes to see when this lands.
This is one of the top priority issues for my team because if you can’t trust your shots or melees to land, your whole Halo experience begins to suffer. We know we still have a few bugs here and are actively working to make sure we fix them as fast as we can.
MISSING COLLISION AND MELEE PHASING
Players have reported cases, especially in close melee encounters, where they can appear to phase through other players or that player collision appears to be disabled. The first thing I would like to stress is that player collision is enabled against enemies. That said, cases where you phase through enemies are not intentional, so let’s explore what’s happening there.
Due to latency, movements that you make on your machine will take time to arrive on the server and therefore the state of the world when the movement is applied on the server will be different. On your machine where you wanted to go may have been clear, but by the time the server processes the move there may be a player obstructing you. If we didn’t do any latency compensation here this would result in rubberbanding when near other players, as you were not able to proceed forward on the server but did on the client.
In some shooters this may be acceptable as close quarters combat may not be that frequent, but in Halo the melee dance is an important part of our combat, so we have systems that attempt to deal with these edge cases by performing similar latency compensation to what happens when a shot is taken.
The melee lunge is a particularly challenging case, as you have two players moving towards locations that are actually where the player they are targeting was in the past. Imagine two players strafing in opposite directions to each other. In the images below, imagine the purple player has been strafing to the right and the green player has been strafing to the left.
Due to latency, each player sees the other directly in front of them (the colored circles).
When each player lunges on the server, they take the path indicated by the dashed line.
But on their machines, it looks like the following happened (the green player’s perspective is shown below, the purple player sees the same in reverse).
To the player, it would appear that the other player phased through you ignoring your hit.
This is also the reason that you sometimes see double back smacks. As the server performs latency compensation for each player in a melee encounter, it is possible that each player did see a back smack on their machine, leading the server to apply what each client saw resulting in a double back smack.
We are working on improvements to our player collision systems, especially during melee. Given how important melee is to our combat dance, this is also a very high priority work item for my team. We will share more in patch notes as we resolve issues.
Any improvements that we make to latency all-up should also help reduce these issues. The closer the clients’ simulations are to each other and to the server, the better the experience for all players will be.
SERVER -> CLIENT DESYNCHRONIZATION
Players have found cases where you can permanently desync the client from the server. This thread on the Halo subreddit is a good example of these cases. Usually, the moment we detect a desync, no matter how small, the client should begin correcting to the server’s position. In bad cases this results in rubberbanding, but in normal scenarios these changes should be unnoticeable. The cases where you permanently desync occur when our systems fail to detect and respond to desyncs.
We have identified the issue linked above, along with some other cases where you can permanently desync, and we should have the fix deployed in upcoming builds. Keep an eye on the patch notes for more info.
I want to wrap up by saying we hear your frustrations and understand that it feels terrible when you experience any issues relating to the game’s networking architecture. We are continuously working to get this to the best state it can possibly be and make sure these issues are as rare as they can possibly be.
As mentioned above, I hope today’s matchmaking updates to prioritize latency help you get better matches. Beyond that, look for improvements to shot and melee registration in future game updates as well.
I’d also note that these issues are not the only things we are looking at, they are just the items that will have the biggest impact when addressed. Hopefully, starting with this post, we can foster a strong dialog between 343 and the community so that together we can help identify and resolve these issues, and that you as a community know that our goal is to make the game experience the best it can possibly be.
Thank you for reading and reporting all the issues you have so far.
Q: What's the server tick rate?
A: Our 4v4 matches run at a 60hz tick rate, and Big Team Battle games run at 30hz. In the future, online campaign coop will run at a 30hz tick rate as well.
Q: Why do my shots appear to miss when I watch back in Theater films / Observer mode?
A: This is due to the latency compensation the server does to ensure shots land correctly. Remember that when the server receives a shot it rewinds time for that shot so that the shot is performed in the same way as the client fired it, this means that shots in Observer / Theater mode will appear to miss as we don't rewind time in these modes.
We have future work planned that will perform the latency compensation when viewing a player in first person, but unfortunately due to the way the system works, third person in theater / observer will always be slightly off. Again, the higher the latency of the shooter, the more significant the error will appear.