MEASURING GAME PERFORMANCE
Our team has already put in the hard work developing the tools to measure and collect information about nearly every scenario imaginable, such as game crashes or matchmaking failures, so technical previews will give us a unique opportunity to compare our internal data to what we see show up in the wild. Measuring the game’s stability and performance, especially at scale, is an integral part of a game’s journey to launch.
Now, let’s hear from members of the Halo Infinite team to see what they’re monitoring and how they’re measuring it during our upcoming technical previews.
What are you/your team monitoring during flights?
Sam Hanshaw – Producer, Live Team: We’re always looking at the rates of matchmaking success and the availability of our servers, after all it’s hard for people to playtest the game if they’re unable to play matches. We also keep an eye on how many matches are getting played to completion, and we receive data about how many people crash while playing.
Brian Dunn – Multiplayer Test Lead & Brian Hughes – Live Test Lead: The test team monitors a large number of specific areas over the course of a flight. First, we are watching for overall crashes and stability. We analyze this data on a per-platform basis and identify specific things like most frequent crashes and differentiate between known and new crash instances. We use a metrics such as “MTTF” (Mean Time to Fail) or “crashes per 1,000 hours of gameplay” as a way of quantifying overall stability.
Next, we are also closely monitoring the overall quality of matchmaking which includes reliability, load times, and skill matching. In addition, the performance of our dedicated servers, performance of the game itself (particularly across different PC configurations) and our overall netcode quality are all under scrutiny. And with Halo Infinite's greater emphasis on player customization, we’re going to be paying close attention to progression, challenges, and which customization items people are unlocking and using.
Lastly, we partner with the Halo Support team to review tickets and bug reports to cross-reference against our telemetry and to help understand real-world player impact and scope of an issue. We also lurk in the Halo Waypoint forums, Reddit, Twitch, and social to keep an eye out for anecdotal reports.
Nate Jones – Engineering Lead, Services Lifecycle Team: If everything’s going well, I’m able to play some matches during a flight and I’m not glued to logs/monitoring. I keep the lobby service dashboards up to the side during the flight (watching backend service performance health, dedicated server usage, number of players connected, matchmaking ticket success rates, etc), and I’ll occasionally get pulled into a mid-flight investigation and have to dig into the logs. Most of my (and my team’s) flight monitoring takes place after the flight ends. Our normal post-flight plan is to thoroughly look over the lobby and skill service logs for errors, warnings, or ‘weird stuff’; that’s on top of any deeper dives into specific experiments we were running during a flight.
Jeff Guy – Producer, PC Team: I’m going to be monitoring Brian Dunn for cheating because he wipes the floor with me every time I play against him in a match. I’m only half joking. :P
We've built a TON of different player-controlled settings into Halo Infinite, as well as support for a vast array of player hardware and peripherals. The PC team will be monitoring how the game holds up across that matrix of player choices. We’ll be looking for things like drops in framerate and crashes that happen with specific hardware configurations. We’ll be combining that information with their graphics settings and peripherals – how many people are rocking ultrawide setups, who is pushing 4K or 8K resolution, who is playing with unlocked framerate versus locked at 144, what custom key bindings are players choosing, etc? Ensuring our PC players can play Infinite the way they want is everything to us, and we’ll be closely watching for any problems when they do.
How do you/your team measure that information?
Sam Hanshaw : We have a general release health dashboard that displays information on matchmaking errors, and any crashes that are happening. There is also a wealth of information in the server logs that the team can dig into during investigations to figure out what broke and when. We also have a great support team who are capturing bug reports and support ticket submissions, these are what we’ll dive into after each flight is completed to get a picture of the experience through player observations of bugs. The information we get from those tickets is valuable for helping us track down the issue, and also for knowing how many individual people an issue is bothering, and how often it’s bothering each of them.
Brian Dunn & Brian Hughes: We have a robust suite of tools and systems that feeds critical information into our team for real time and reactive monitoring. Any time a player crashes, a system called “Watson” uploads a detailed crash report which feeds into our internal crash reporting website, “Ticket Track.” Like Nate and the Services Team, we also use Kusto which helps us understand the data we’re receiving and converts it into something useful for reporting purposes. Speaking of reports, we use Power BI reports for any of the data we need to visualize and then all of this feeds into our Azure DevOps database for bug tracking. Our team uses data from all of these sources to get an understanding of the quality of the flight, and ensure we're making the right improvements for current and future flights.
Nate Jones: We use Kusto (Azure Data Explorer) for most of our ad-hoc data analysis; I spent a lot of time writing queries that join service, dedicated server, and client telemetry events together to figure stuff out. We have a bunch of pre-built graphs and dashboards using an internal Microsoft system called Geneva. We’ve also got a few internal tools (bespoke websites). One’s called “lobby logs”; it’s like a mini search engine for the Halo lobby service that can trace a player’s parties/matches and correlate their sessions with the other players/parties involved in the matchmaking process or eventual game session.
Jeff Guy: We have a lot of tools and telemetry already set up to capture a lot of this data, even at scale. When someone crashes, for example, the game will automatically upload a crash report to us. Our team can then look at the details of each specific crash or use that information to spot a pattern that could indicate a larger problem. In terms of players’ settings, we’ll get telemetry data when players save their key binds and video settings, since a good portion of that information is saved in our online services. We can even check to see if a player’s framerate drops during a session. Gathering the full picture of all the different settings and variables enables us to quickly narrow down our investigations and fix issues fast.
What do you/your team consider a success?
Sam Hanshaw: We want players to be able to consistently play matches of Halo Infinite. The more matches they can play, the more issues they can find for us to deal with. Every new issue players are able to find that we were not is a success for me. Anything that gets found during flighting is something that’s found before the game launches, resulting in a better game for everyone when we release.
I know I’ve talked about problems a lot, but overall, we also consider a flight a success when we see people having fun playing the game. In the studio we’ll be in the flight with you, and while we’ve got a ton of information we need to gather to make this game the best it can be at launch, we’re also looking forward to playing alongside you and opening a dialogue about our road to launch. To everyone who is going to participate in our upcoming flights I want to say thank you for helping us make this game better!
Brian Dunn & Brian Hughes: Success for us is when we're able to meet or exceed our internal targets, while not having too many big surprises. You always learn something new when so many people play your game for the first time, but hopefully there's nothing completely unexpected. We’ll be looking specifically at our matchmaking quality real world results vs. our targets as well as our crash counts and rates actuals vs. what we anticipated. We also hope to see player feedback and data coming in that aligns with our own internal expectations for overall quality at this stage of the project.
Nate Jones: Selfishly, one hallmark of success is when nobody needed to activate my team’s on-call process and phone someone for mid-flight engineering support 😊.
At a high level, success is getting usable data from the flight. Good news is great, but bad news is still valuable. We’re always looking at general health (CPU/mem/bandwidth, matchmaking speed and quality, etc) and we’ll have general quality/performance targets, but any given flight also has specific experiments we’re running. For example, during an internal flight in April, we did some explicit tests around how our pool of dedicated servers would fallback/failover to secondary pool(s) if we run low on machines. That April test was technically a failure (we didn’t see the expected players using the secondary server pool), but we narrowed the issue down to a bug in an external team’s system, and they were able to find/fix the problem from our flight data.
Jeff Guy: Creating a first-class PC game is all about embracing player identity and all the awesome diversity that comes with it. PC players build their gaming rigs to reflect the way they want to play, and we are making sure Halo Infinite honors that. The challenge of course is that supporting such a diverse array of hardware and software creates a lot of ways for things to break.
Our team considers flights a success if we're able to find issues that we might not have seen if we didn’t put the game in the hands of our players early. This is critical to ensuring every PC player within our min-spec has an awesome experience on day one. Also, we’d love to see which custom key bindings players use...or better yet, see players choose to use the default key bindings because that would mean we chose them well. :)
We appreciate you taking the time to share your insights and expertise with our community today. It’s reassuring to know that the issues they may encounter during flighting are usually seen by the team, and there’s rarely a need to wonder, “Is 343 seeing this?!” since you have built the tools to do exactly that. We'll let you get back to preparing for the technical preview now, thank you again!