Games

Halo Infinite

343 INDUSTRIES

Fracture: Entrenched Event (Updated)


<p>We’ve got two changes coming to <em>Halo Infinite</em>’s playlist line up next Tuesday, September 20: <strong>The Yappening BTB shenanigans will continue, and we’ll be rotating out the dedicated Land Grab playlist</strong>.</p>



<p>Before we jump into those specifics, we wanted to share some insights around how the current playlists are performing, as well as the team’s philosophy and priorities for ongoing playlist management, as these are questions we hear often.</p>



<hr class="wp-block-separator"/>



<h2><strong>Playlist Fundamentals</strong></h2>



<p>Ever since <em>Halo 2</em>, matchmaking playlists have been at the heart of <em>Halo</em>’s online multiplayer and that remains true today in both <em>MCC</em> and <em>Halo Infinite</em>. Likewise, for as long as there have been <em>Halo </em>playlists, there have been rotational plans that see various modes coming and going at regular intervals. With <em>Infinite</em>, we haven’t seen a lot of rotations thus far, but as the team continues to amass a backlog of tried-and-true modes—along with brand new offerings—this will occur more regularly.</p>



<p>When it comes to playlist management, the team’s priorities are offering variety and breadth of experiences, focusing on the things that are most popular to players (while improving and evolving those that are less popular), and maintaining the best player experience possible.</p>



<p>In general, when new playlists are added, there’s a likelihood that overall match quality declines as the player population is spread across more options (unless, of course, a suitable influx of incremental new players is seen). As a result, some playlists become “unhealthy,” meaning they will experience longer than desired wait times, degraded online performance due to smaller populations around specific data centers, and/or larger than desired variances in player skill levels. Each of these factors can contribute negatively to the overall experience, with the worst case being some players have such a bad time that they move on entirely.</p>



<p>Most recently, Last Spartan Standing was removed as a dedicated playlist due to it no longer sustaining a suitable level of engagement to hit the match quality and player experience bars the team has set. For context, LSS accounted for less than 1% of all player activity (despite it also being some players’ personal favorite mode).</p>



<p>The team has to weigh all of these considerations any time a new playlist is added and/or the population changes. Meanwhile, rotational playlists will continue to play a part in matchmaking and should become more commonplace as the game continues to evolve.</p>



<p>Overall, playlist management is a dynamic process and our plans will continue to evolve based on data and player feedback.</p>



<hr class="wp-block-separator"/>



<h2><strong>PLAYLIST RANKINGS</strong></h2>



<figure class="wp-block-image size-full"><img src="https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_06_Watermarked_1920x1080.jpg" alt="A multiplayer Spartan rocking the citron white coating which is part of the Yappening Event Pass." class="wp-image-45501" srcset="https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_06_Watermarked_1920x1080.jpg 1920w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_06_Watermarked_1920x1080-300x169.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_06_Watermarked_1920x1080-1024x576.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_06_Watermarked_1920x1080-768x432.jpg 768w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_06_Watermarked_1920x1080-1536x864.jpg 1536w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_06_Watermarked_1920x1080-1568x882.jpg 1568w" sizes="(max-width: 1920px) 100vw, 1920px" /></figure>



<p>Quite a few folks were surprised to hear that Last Spartan Standing was hovering around the bottom of the list given how much they personally enjoy it. Naturally, this raises the questions of <em>what are the most popular playlists</em> and <em>how do my favorite playlists currently fare</em>?</p>



<p>Here’s a stack ranked list of <em>Halo Infinite</em>’s playlists over the past month to give you an idea of what’s most popular and where players are spending their time:</p>



<ol><li>1) Quick Play</li><li>2) Ranked Arena</li><li>3) Fiesta</li><li>4) Team Slayer</li><li>5) Big Team Battle</li><li>6) Tactical Slayer</li><li>7) Land Grab</li><li>8) Ranked Doubles</li><li>9) Team Doubles</li><li>10) Team Snipers</li><li>11) Rumble Pit</li><li>12) Bot Bootcamp</li></ol>



<p>Anytime we have an active event, we see significant impact and shifts across matchmaking. Since it debuted, the Yappening playlist has been the overwhelming favorite playlist (nearly 4x as popular as the #2 playlist).</p>



<hr class="wp-block-separator"/>



<h2><strong>THE YAPPENING KEEPS ON HAPPENING</strong></h2>



<figure class="wp-block-image size-full"><img src="https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_04_Watermarked_1920x1080.jpg" alt="A multiplayer Spartan rocking the Quilhawk cosmetic item which is part of the Yappening Event Pass." class="wp-image-45473" srcset="https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_04_Watermarked_1920x1080.jpg 1920w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_04_Watermarked_1920x1080-300x169.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_04_Watermarked_1920x1080-1024x576.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_04_Watermarked_1920x1080-768x432.jpg 768w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_04_Watermarked_1920x1080-1536x864.jpg 1536w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_2022_Yappening_04_Watermarked_1920x1080-1568x882.jpg 1568w" sizes="(max-width: 1920px) 100vw, 1920px" /></figure>



<p>It’s been great to see so many people enjoying BTB antics as part of the Yappening event (which we’ve also been having a blast playing!). Our pal Flipflop the Grunt has thrown one heckuva party and, with his permission, we’re going to keep the good times rolling with a minor change in branding.</p>



<p>When the Yappening officially ends on Tuesday, September 20, the new “<strong>BTB Social</strong>” playlist will emerge as the new home for the shenanigans you’ve come to know and love in the event playlist.</p>



<p>The Yappening has been a huge success in terms of popularity and engagement, but we’ve also seen the core BTB playlist activity significantly decline in its shadow. The team will be monitoring closely to see how both playlists hold up together and what the overall player experience is to determine what future steps may be needed.</p>



<p>In order to maintain a positive player experience and optimal playlist health, we may need to consolidate and/or rotate things around while we also apply the insights gathered during the Yappening to further improve BTB as a whole further down the road. We’re eager to see how things go and as always, look forward to your feedback.</p>



<hr class="wp-block-separator"/>



<h2><strong>Land Grab</strong></h2>



<p>On Tuesday, Sept. 20, the dedicated Land Grab playlist will be rotated out of matchmaking. While we know many players have enjoyed this new mode introduced in Season 2, at this stage it’s been steadily declining and starting to fall short of the quality bar for a positive player experience.</p>



<p>Land Grab will be back as part of the Entrenched event with Week 5 kicking off Sept. 27, and the final week slated to begin on Oct. 25. Looking further ahead, Land Grab will roll into Quick Play as part of the Winter Update coming Nov. 8.</p>



<hr class="wp-block-separator"/>



<h2><strong>LOOKING AHEAD</strong></h2>



<p>As <em>Halo Infinite</em> continues to evolve and we see new modes brought online we expect to see more regular rotations to offer experiences players want while also maintaining that high quality bar we have set for ourselves. We can’t always offer everything that every individual player wants today, but we’ll continue doing our best to focus on the breadth <em>Infinite </em>has to offer, the modes that are most popular to players, and keep addressing areas that negatively impact your online experience.</p>



<p>We’ll see you online, and thank you for your continued support!</p>


Halo Infinite – September Playlist Update


<p>With the end of Season 2 drawing ever closer, we want to provide you with an update on the future of all things <em>Halo Infinite</em>.</p>



<p>Joseph Staten, Sean Baron, and Brian Jarrard sat down for a roundtable discussion about the current state of the game, priorities, and player experience goals. Also covered is the upcoming “Winter Update,” Season 3, and other features (such as the Forge Beta and Campaign Network Co-Op) which you can check out in the video below.</p>



<figure class="wp-block-embed-youtube wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
<iframe title="Halo Infinite |  Update – September 2022" width="640" height="360" src="https://www.youtube.com/embed/Ramqlt7vxJ8?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div></figure>



<p>In addition, we have also updated the roadmap for what’s coming in the Winter Update and Season 3, featuring new maps, modes, weapons, armor, and much more.</p>



<figure class="wp-block-image size-full"><img src="https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_Roadmap_Sep2022.png" alt="Halo Infinite roadmap update for September 2022" class="wp-image-45372" srcset="https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_Roadmap_Sep2022.png 1920w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_Roadmap_Sep2022-300x249.png 300w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_Roadmap_Sep2022-1024x850.png 1024w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_Roadmap_Sep2022-768x638.png 768w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_Roadmap_Sep2022-1536x1275.png 1536w, https://wpassets.halowaypoint.com/wp-content/2022/09/HaloInfinite_Roadmap_Sep2022-1568x1302.png 1568w" sizes="(max-width: 1920px) 100vw, 1920px" /></figure>



<p>We continue to be grateful for the community’s passion and patience as we look to what’s coming on the horizon. Nothing excites us more than being able to deliver the experiences of everything that’s coming on the roadmap (and beyond!), and for your creativity to soar when Forge is in your hands, knowing that our best days are yet ahead of us!</p>


Halo Infinite Update – September 2022


<p>Introducing Halo Gear Rewards from the Xbox Gear Shop! Halo Gear Rewards are exclusive physical products players can earn access to after completing specific in-game accomplishments.</p>



<p>And with that, we are excited to announce our first Halo Gear Rewards item, the Fracture: Entrenched Tanker Jacket. This high-quality jacket features embroidered emblem patches, beautifully screen-printed <em>Halo </em>art, and a personalized removable Gamertag patch. And as a bonus, it will also come with an additional UNSC patch for a stealthier option.</p>



<figure class="wp-block-image size-large"><img src="https://wpassets.halowaypoint.com/wp-content/2022/07/Halo_Entrenched_Jacket_Inside_03-62e00fe293c68-1024x576.png" alt="Inside of the new Halo Fracture: Entrenched Jacket" class="wp-image-43388" srcset="https://wpassets.halowaypoint.com/wp-content/2022/07/Halo_Entrenched_Jacket_Inside_03-62e00fe293c68-1024x576.png 1024w, https://wpassets.halowaypoint.com/wp-content/2022/07/Halo_Entrenched_Jacket_Inside_03-62e00fe293c68-300x169.png 300w, https://wpassets.halowaypoint.com/wp-content/2022/07/Halo_Entrenched_Jacket_Inside_03-62e00fe293c68-768x432.png 768w, https://wpassets.halowaypoint.com/wp-content/2022/07/Halo_Entrenched_Jacket_Inside_03-62e00fe293c68-1536x864.png 1536w, https://wpassets.halowaypoint.com/wp-content/2022/07/Halo_Entrenched_Jacket_Inside_03-62e00fe293c68-1568x882.png 1568w, https://wpassets.halowaypoint.com/wp-content/2022/07/Halo_Entrenched_Jacket_Inside_03-62e00fe293c68.png 1920w" sizes="(max-width: 1024px) 100vw, 1024px" /></figure>



<p>So, how does it work? </p>



<p>Players who have completed the Fracture: Entrenched Event Pass will need to log into their Halo Waypoint profile via web or mobile. Once logged in they will automatically receive a unique code via the Waypoint Notification system. Please note that it can take up to 24 hours for Halo Waypoint to recognize completion of the event pass and to deliver the code. There is a limit of one code per account, and codes will begin rolling out to qualifying players on July 26, 2022.</p>



<p>Once you have received your code, just head over to the <a href="https://gear.xbox.com/products/halo-entrenched-personalized-jacket"><strong>Halo Gear Shop</strong></a> to select your jacket size and personalization details on the product page and complete your purchase. Follow the code redemption instructions listed on the product page.</p>



<p>To qualify for this Halo Gear Rewards item, players must complete the <a href="https://www.halowaypoint.com/news/fracture-entrenched-event">Fracture: Entrenched Event Pass</a> by November 1, 2022.</p>



<p>The Fracture: Entrenched Tanker Jacket will be available for purchase until November 15, 2022. Jackets will ship in Q1 2023.</p>


Introducing Halo Gear Rewards

Season 2 Lone Wolves Launch

Halo: The Master Chief Collection


<p>“So, who spilled coffee on the MCC servers on Friday 04/01?”</p>



<p>My name is Dana and I’m one of the engineers on the <em>Halo </em>Publishing Team who responded to the recent incident which affected matchmaking and custom games browser in <em>Halo: The Master Chief Collection</em>.</p>



<hr class="wp-block-separator"/>



<h2><strong>MCC’s Dedicated Server Model</strong></h2>



<p>In order to contextualize the details shared in this blog, here are some basic details on MCC’s server model. MCC uses dedicated servers for matchmade games and custom game browser sessions. Other game modes in MCC use peer-hosted session (where one player within the game is designated the game host).</p>



<p>These servers are virtual machines hosted by another team at Microsoft which serves first- and third-party Xbox studios and that team is in turn served by Azure technologies. These servers run an application which we call the Unified Dedicated Server (UDS). The UDS is effectively the server counterpart to the MCC game client and contains a wrapper which can configure and launch server versions of any of the games within MCC.</p>



<p>The UDS is a “trusted” authority meaning that it can’t be tampered with like MCC game clients potentially can be on PC. As such, in addition to creating a fair and reliable game experience (where all players in a game have comparable ping, the host has a stable connection, and no one has “host advantage” or “host disadvantage”), it is also a trusted source for tracking match statistics. These stats and other trusted events are used for things like calculating competitive skill ranks and progressing challenges.</p>



<p>From an MCC client perspective (i.e. the version of MCC that runs on your local PC or Xbox), we look at the setup of the UDS in two phases: allocation and configuration.</p>



<p>To allocate a dedicated server, the MCC game client makes a request to our microservice which handles some other details of the allocation on the client’s behalf. A virtual machine is allocated in the optimal region (based on ping—or manual selection in the case of the custom game browser) with sufficient machines standing by and information about the allocated machine is communicated back to the game client. From there, we enter the configuration stage where a connection between the client is established and the client begins sending configuration data for the match (which <em>Halo </em>game to launch, what map and game variant to load, etc.)</p>



<hr class="wp-block-separator"/>



<h2><strong>The Incident</strong></h2>



<p>On the afternoon of 04/01, our internal QA team escalated a potential matchmaking issue they were encountering on our release candidate (RC) build (the build expected to ultimately release to our players) for our upcoming MCC game update. Ensuring that this update is in good shape and able to ship by our target date has been my top priority in recent weeks, so I began investigating immediately.</p>



<p>After some initial investigation, the symptom became apparent. A server was being successfully allocated, but configuration was failing. It seemed surprising that this issue wouldn’t have been observed earlier in our RC testing, but as this was observed on a relatively new MCC build, the investigation was directed towards identifying what code changes since our last released game update (roughly a sixth month delta during which time numerous features were added and hundreds of bugs were fixed) could have introduced the issue.</p>



<p>However, it was around this time that reports started rolling in from MCC players that they were experiencing their own matchmaking issues in our production environment (the “real” environment where you play MCC as opposed to the “artificial” internal environment where we test iterative development changes for MCC). The issue we were observing in our internal environment was logically inferred to be caused by code churn from the past six months of development, so surely the production issue was not related. An issue actively impacting our players in the production environment takes priority of an internal issue, though, so our investigation retargeted that issue immediately.</p>



<p>It didn’t take long to confirm that the production issue was, in fact, the very same one impacting our internal RC build!</p>



<p>On one hand, it was a relief to confirm that the issue wasn’t something unique to our RC that would jeopardize our upcoming game update. But the fact that both builds (which were separated by six months of iterative development) were abruptly impacted in exactly the same way had concerning implications of its own. We had made two backend changes for MCC in the preceding days (in preparation for our upcoming game update), but these were logically incapable of introducing the behavior we were observing. Still, we rolled both changes back to entirely rule them out as potential causes.</p>



<p>Shortly after confirming this, we rolled out our first mitigation step for the issue which was to enable peer-to-peer fallback for MCC matchmaking. This allows matches where server configuration fails to fall back to a peer host (like custom games use). The peer-to-peer experience is not the ideal or intended experience and, with the lack of a trusted host we described earlier, doesn’t allow certain challenges to be progressed or skill ranks to be calculated. Still, it would at least increase matchmaking success rates and allow folks to get into matches more reliably for their Friday evening <em>Halo</em>-ing.</p>



<p>It was approaching midnight on Friday. Here’s what we knew so far…</p>



<p>Players were frequently failing to configure dedicated servers. From looking through logs corresponding to failures on the dedicated servers themselves, the primary configuration data was never being received from clients in the first place. Two wildly different MCC builds (one of which had been working as expected in the production environment for the past several months) were affected by the issue in exactly the same way. Recent service changes we’d made were not causal (but had been reverted just in case anyway). The primary mitigation steps we’d taken so far were enabling peer-to-peer fallback and disabling bans. Unfortunately, as we learned further into the weekend, this ban change failed to immediately take effect due to a completely unrelated issue.</p>



<p>On Saturday morning, now that we’d ruled out any MCC-specific changes being causal, we began engaging partner teams across Microsoft. We cast a gradually wider net as each team owning services along the path between client and server conducted their own investigation, slowly narrowing things down. On Saturday we also built a more complete understanding of the issue from a combination of data aggregation and manual reproduction of the issue aided by network traffic capture tools.</p>



<hr class="wp-block-separator"/>



<h2><strong>On Thin ICE</strong></h2>



<p>The server configuration process is conducted by way of a series of packets sent between the client and server. The client-server connection itself is negotiated via <a rel="noreferrer noopener" href="https://datatracker.ietf.org/doc/html/rfc8445" target="_blank">Internet Connectivity Establishment (ICE)</a> in order to send STUN packets between them. STUN is a protocol for <a rel="noreferrer noopener" href="https://datatracker.ietf.org/doc/html/rfc8489" target="_blank">Session Traversal Utilities for Network Address Translator (NAT)</a>. In our case, these STUN packets are used for managing <a rel="noreferrer noopener" href="https://www.ietf.org/rfc/rfc768.txt" target="_blank">User Datagram Protocol (UDP)</a> flow between the client and server. The connection is negotiated via binding requests which allow the client to identify the address of the server. The client will only accept solicited traffic from the server (traffic sent along an established connection), but the server can accept unsolicited traffic (allowing the client to initially establish the connection).</p>



<p>This process looks like an initial establishment followed by a series of configuration messages telling the server which <em>Halo</em> game library to launch and optionally providing map and game variant data. What we observed was that the client would send the expected initial packets, the server would receive them and send the expected packets in response, but the client never received them!</p>



<p>Without receiving the initial acknowledgement, the client could never move on to sending the configuration data itself. After ten seconds of not receiving acknowledgment, the client would time out which would manifest as a failure in matchmaking or the custom game browser. 30 seconds after allocation with no configuration, the UDS would shut down (putting the virtual machine back into standby, ready to be allocated again).</p>



<figure class="wp-block-image size-full"><img src="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_01.jpg" alt="" class="wp-image-33806" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_01.jpg 1100w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_01-300x216.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_01-1024x736.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_01-768x552.jpg 768w" sizes="(max-width: 1100px) 100vw, 1100px" /></figure>



<p><em>Parallel Wireshark captures from the client (left) and server (right) when reproducing the issue with trace view (top) and UDP stream view (bottom). IP addresses have been anonymized. The server responses (blue in the UPP stream view) are not received by the client.</em></p>



<p>Interestingly, in the cases where the client did receive the response from the server and successfully configure, it would continue to succeed any subsequent attempts to configure that virtual machine for a period of about 15 minutes. Failures were also inconsistent; a client could fail to configure a given server and then succeed on its next attempt. In the retail environment, it is highly unlikely that a client would ever make two contiguous configuration requests to the same server, but this provided some further details for our investigation.</p>



<p>The number of virtual machines allocated rose to higher-than-normal levels because, in most cases, they would fail configuration and clients would move onto allocate another machine and repeat the process. Meanwhile, the amount of actual UDP traffic between clients and servers fell well below normal.</p>



<figure class="wp-block-image size-full"><img src="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_02.jpg" alt="" class="wp-image-33820" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_02.jpg 1100w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_02-300x108.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_02-1024x368.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_02-768x276.jpg 768w" sizes="(max-width: 1100px) 100vw, 1100px" /></figure>



<p> <em>MCC dedicated server machine allocations from 03/28 through 04/03</em></p>



<figure class="wp-block-image size-full"><img src="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_03.jpg" alt="" class="wp-image-33834" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_03.jpg 1100w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_03-300x99.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_03-1024x338.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_03-768x253.jpg 768w" sizes="(max-width: 1100px) 100vw, 1100px" /></figure>



<p> <em>MCC dedicated server outgoing User Datagram Protocol (UDP) bytes from 03/28 through 04/03</em></p>



<figure class="wp-block-image size-full"><img src="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_04.jpg" alt="" class="wp-image-33848" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_04.jpg 1100w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_04-300x97.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_04-1024x330.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_04-768x248.jpg 768w" sizes="(max-width: 1100px) 100vw, 1100px" /></figure>



<p><em>MCC dedicated server outgoing User Datagram Protocol (UDP) packets from 03/28 through 04/03</em></p>



<hr class="wp-block-separator"/>



<h2><strong>Mitigation</strong></h2>



<p>Thankfully, we were able to reliably reproduce the issue ourselves which allowed us to set up and capture various network diagnostics in real-time from the client and server simultaneously. This allowed us to narrow down the issue somewhat as we tracked the returning packets on their way back to the client. However, the network topology between client and server is somewhat complex and some components of it are stateful—meaning that the same inputs can result in different outputs based on surrounding factors. As a result, identifying exactly where packets were getting lost along the way was an unfortunately time-consuming process.</p>



<p>In parallel to this investigation, we sought to simplify our test case. MCC and the UDS are each complex applications which can introduce noise or red herrings into this kind of investigation. To combat this, we built a simple test application to replicate our traffic scenario without any other implementation details getting in the way. We found that we could consistently reproduce the issue with our simple test app if the server sent an unsolicited packet to the client (before the client established a connection). This behavior which was consistently resulting in failure would have resulted in success prior to 04/01.</p>



<p>Armed with this information, we were able to make a targeted change to the traffic pattern on our UDS (i.e. a change which would not require a game update for MCC). We rolled this change out in parallel to two of our development branches and began testing the fix against our live MCC build and our upcoming release candidate at the same time. With this change, we observed our server configuration success rate go from something like 25% to 100%. This testing took place on Tuesday afternoon into Wednesday morning, and we deployed our new UDS build containing the change at scale early Wednesday afternoon.</p>



<p>Unfortunately, players currently need to relaunch MCC to receive the new server pairing information (this is something we’re looking to improve in the future). As a result, players saw a lot of false negatives throughout Wednesday afternoon where, even if they had relaunched, someone they matched with hadn’t and they attempted to configure a server from the old deployment (without the fix). Error rates declined as more and more of the population moved up to the new server deployment.</p>



<figure class="wp-block-image size-full"><img src="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_05.jpg" alt="" class="wp-image-33862" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_05.jpg 1100w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_05-300x147.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_05-1024x501.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_05-768x376.jpg 768w" sizes="(max-width: 1100px) 100vw, 1100px" /></figure>



<p><em>MCC dedicated server configuration and reservation failures per hour from 03/31 through 04/08</em></p>



<hr class="wp-block-separator"/>



<h2><strong>STUNning Conclusion</strong></h2>



<p>We finally had our fix, but what had caused this established traffic pattern (which adhered to the Internet Engineering Task Force (IETF) standard and had served us well for the past few years) to abruptly stop working? Investigation continued. Finally, it was identified that there had been some updates behind the scenes to the servers we use to relay STUN traffic as part of the ICE process which had resulted in misconfigurations.</p>



<p>We took other steps along the way such as adding more logging and other functionality to our UDS builds to make potential issues like this easier to diagnose in the future. We also implemented the traffic pattern change via a configurable switch (so we can reenable the old behavior at any time if needed). We were ultimately able to deploy a fix very quickly (under 24 hours from implementation to release) and without prompting a client-side MCC update (which requires certification and generally longer lead time), but the nature of the investigation to identify the issue in the first place meant that the whole process took much longer than we would’ve hoped.</p>



<p>Modern multiplayer games are complex and rely on a variety of technologies spanning multiple devices and networks. We’re always seeking to evolve and improve our own tech stack and how it interacts with dependent services. Thank you for joining us for this peek behind the curtain, and for your patience while we resolved the incident.</p>


MCC Server Incident Summary

Community

Community Update


<p>Hello Everyone,</p>



<p>With Season 2 locked and loaded for May 3rd, we want to
share a roadmap of what we’re planning to deliver during Season 2 and for the
rest of the year. </p>



<p>More than anything else, we want this roadmap to be accurate
and reliable. Some items and dates are certain, and these are clearly listed.
Others are marked as “targets” or “goals” and they will become more certain
over time. The roadmap’s timeframe will also expand as we announce longer-term
initiatives. In other words, this roadmap is an evolving document, and we’ll
let you know when we make changes to it.</p>



<p>With all that in mind, here’s the Halo Infinite roadmap for 2022:</p>



<figure class="wp-block-image size-full"><a href="https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April.jpg"><img src="https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April.jpg" alt="Halo Infinite's 2022 roadmap as of April 22" class="wp-image-33672" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April.jpg 1920w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April-300x169.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April-1024x576.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April-768x432.jpg 768w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April-1536x864.jpg 1536w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April-1568x882.jpg 1568w" sizes="(max-width: 1920px) 100vw, 1920px" /></a></figure>



<p>This roadmap reflects the same studio priorities <a rel="noreferrer noopener" href="https://www.halowaypoint.com/news/halo-infinite-update-march-2022" target="_blank">that we shared back in March</a>…</p>



<ul><li>Addressing issues negatively impacting player experience.</li><li>Completing Season 2 and delivering it as promised on May 3. </li><li>Continuing work on Campaign Co-op, Forge, and Season 3.</li></ul>



<p>…with a “priority zero” of team health and getting ourselves into a sustainable development rhythm so that we can deliver great experiences to all of you while maintaining a healthy work/life balance.</p>



<p>We know we need to deliver more content and more features more quickly. Staying true to priority zero means that sometimes we need to slow down in order to stay healthy and move faster later. But we’re also aggressively looking at ways to accelerate.</p>



<p>For example, we want to get the Forge toolset into your hands sooner rather than later so that awesome community maps and modes can more rapidly make their way into the game. To make this happen, we’re targeting a Forge open beta later in Season 2. We’ve successfully had Forge in private flighting with a limited audience for some time, so we’ve decided to forgo a large-scale formal flighting program and go right to open beta. With this approach, we can get the core Forge tools out to everyone quicker while also remaining focused on our core priority of continuing to improve foundational aspects of <em>Halo Infinite</em>. Forge will persist from the open beta onwards, evolving and growing over time.</p>



<p>As we continue to improve and ramp up, please understand that all of us at 343 are fully committed to making <em>Halo Infinite</em> the best <em>Halo</em> experience ever. Indeed, the primary reason that we’re extending Season 2 is to give us time before shipping another Season to fully address “quality of life” improvements based on <a rel="noreferrer noopener" href="https://www.halowaypoint.com/news/season-1-outcomes-report-live" target="_blank">learnings from Season 1</a> as well as other data and feedback we’ve collected.</p>



<p>Quality of life (QoL) is shorthand for bug-fixes, new features and/or content to address issues negatively impacting player experience. We’ll be delivering QoL improvements as part of Season releases and as separate updates during Seasons.</p>



<p>Our player-facing term for QoL updates that land during Seasons is “Drop Pods.” And much like their ODST namesakes, these release vehicles are meant to drop in hot, just as soon as they’re ready. Our target is one Drop Pod every month, and you can expect Drop Pods to vary in scope and size. We’ll have more to share about what’s going into Season 2’s Drop Pods as soon as possible after Season 2 launch.</p>



<p>Speaking of which, Season 2 launch is now less than two weeks away, on May 3! If you haven’t already seen the previews for Season 2’s <a rel="noreferrer noopener" href="https://www.halowaypoint.com/news/season-2-map-previews-catalyst-and-breaker" target="_blank">new maps</a> and <a rel="noreferrer noopener" href="https://www.halowaypoint.com/news/season-2-modes-preview" target="_blank">new modes</a>, please have a look. During these preview videos, you’ll spy some of Season 2’s new armor customization options. And here are a few extra combinations to whet your appetite:</p>



<figure class="wp-block-image size-full"><a href="https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED.jpg"><img src="https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-scaled.jpg" alt="" class="wp-image-33643" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-scaled.jpg 2560w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-300x169.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-1024x576.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-768x432.jpg 768w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-1536x864.jpg 1536w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-2048x1152.jpg 2048w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-1568x882.jpg 1568w" sizes="(max-width: 2560px) 100vw, 2560px" /></a></figure>



<p>We’ll have a lot more to share about Season 2 including a deep dive into its Battle Pass rewards, events—including brand new “multiplayer narrative events” that place your personal Spartan in the center of an evolving Seasonal storyline—game modes, and much more during a community livestream scheduled for next Wednesday, April 27, at 1:00 PM PST on <a href="https://www.twitch.tv/Halo">Twitch.tv/Halo</a> and <a href="https://www.youtube.com/Halo/live">YouTube.com/Halo</a>. We’ll also be discussing the roadmap in more detail during this livestream.</p>



<p>In summary: we are committed to delivering more content faster; we will accelerate while prioritizing team health; you can expect QoL improvements during Seasons via Drop Pods—and you can expect more regular roadmap updates as our plans snap into sharper focus.</p>



<p>On behalf of the entire <em>Halo Infinite</em> team, thank you for your passion and patience as we embark on the road ahead. We’re fired up to deliver what’s already on the roadmap as well as to deliver other content and experiences that we’re not ready to reveal just yet… but that we can’t wait for you to play.</p>



<p>Joseph Staten<br>Head of Creative, <em>Halo Infinite</em></p>


The Road Ahead

Halo Infinite Update – April 2022

Fracture: Tenrai Event (Updated)


<p>Representing two decades of storytelling, the largest and most comprehensive official guide to the <em>Halo </em>universe ever to be created—the new <em>Halo Encyclopedia</em>—has finally been released into the wild. In this massive tome, Dark Horse and 343 Industries have partnered together to deliver a vast directory of <em>Halo </em>canon, allowing fans to explore the intricacies of an epic saga that spans over a hundred millennia and countless far-flung alien worlds. Delve into a vast archive of ships, vehicles, and weapons that defined the wars of the 26th century and come face-to-face with the major players of the UNSC, the Covenant, the Forerunners, and the Banished. The <em>Halo Encyclopedia</em> features stunning, never before seen art revealing characters, technology, and locations that fans have only ever imagined. </p>



<p>This extravagant oversized hardcover has something to offer both veteran fans of the <em>Halo </em>universe and those who are greeting it for the very first time. Pick up your own copy of the <em>Halo Encyclopedia</em> <a href="https://www.darkhorse.com/Books/3008-349/Halo-Encyclopedia-HC">here</a> or secure the impressive <em>Deluxe Edition</em> <a href="https://www.darkhorse.com/Books/3009-942/Halo-Encyclopedia-HC-Deluxe-Edition">here</a>, elegantly enshrined in a mysterious Forerunner cylix. </p>



<p>And whether you are totally new to <em>Halo </em>or perhaps coming back after some time, we know that with the release of <em>Halo Infinite</em>, many folks may be unfamiliar with some of the terminology we’ve accrued over the past twenty years. No worries, we’ve got you covered! Below is an online companion resource for the new <em>Halo Encyclopedia</em>—a brief, high-level glossary to help you navigate some of the names, terms, and acronyms that populate the universe’s lexicon. So no matter if you read this book from front to back or just dive into its 488 pages somewhere in the middle, feel free to use this directory as a friendly guide, especially if you’ve found yourself suddenly lost in the vast ocean of lore this new book holds. We really do hope you enjoy it!</p>



<p>For more information about the <em>Halo Encyclopedia</em>, check out the roundtable discussion we had with its writers, where we talk about its development process and showcase brand new art created specifically for this book:  </p>



<figure class="wp-block-embed-youtube wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
<iframe title="Halo Encyclopedia | Full Roundtable Discussion" width="640" height="360" src="https://www.youtube.com/embed/yNx3Vqv99t0?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div></figure>



<h2>Glossary</h2>



<h2><a href="#glossary-a">A</a> | <a href="#glossary-b">B</a> | <a href="#glossary-c">C</a> | <a href="#glossary-d">D</a> | <a href="#glossary-e">E</a> | <a href="#glossary-f">F</a> | <a href="#glossary-g">G</a> | <a href="#glossary-h">H</a> | <a href="#glossary-i">I</a> | <a href="#glossary-j">J</a> | <a href="#glossary-k">K</a> | <a href="#glossary-l">L</a> | <a href="#glossary-m">M</a> | <a href="#glossary-n">N</a> | <a href="#glossary-o">O</a> | <a href="#glossary-p">P</a> | <a href="#glossary-q">Q</a> | <a href="#glossary-r">R</a> | <a href="#glossary-s">S</a> | <a href="#glossary-t">T</a> | <a href="#glossary-u">U</a> | <a href="#glossary-v">V</a> | <a href="#glossary-w">W</a> | <a href="#glossary-x">X</a> | <a href="#glossary-y">Y</a> | <a href="#glossary-z">Z</a></h2>



<hr class="wp-block-separator"/>



<h2 id="glossary-a">A</h2>































<hr class="wp-block-separator"/>





<h2 id="glossary-b">B</h2>









<hr class="wp-block-separator"/>





<h2 id="glossary-c">C</h2>





























<hr class="wp-block-separator"/>





<h2 id="glossary-d">D</h2>

























<hr class="wp-block-separator"/>





<h2 id="glossary-e">E</h2>

























<hr class="wp-block-separator"/>





<h2 id="glossary-f">F</h2>





















<hr class="wp-block-separator"/>





<h2 id="glossary-g">G</h2>





























<hr class="wp-block-separator"/>





<h2 id="glossary-h">H</h2>

























<hr class="wp-block-separator"/>





<h2 id="glossary-i">I</h2>









<hr class="wp-block-separator"/>





<h2 id="glossary-j">J</h2>











<hr class="wp-block-separator"/>





<h2 id="glossary-k">K</h2>









<hr class="wp-block-separator"/>





<h2 id="glossary-l">L</h2>















<hr class="wp-block-separator"/>





<h2 id="glossary-m">M</h2>





























<hr class="wp-block-separator"/>





<h2 id="glossary-n">N</h2>







<hr class="wp-block-separator"/>





<h2 id="glossary-o">O</h2>













<hr class="wp-block-separator"/>





<h2 id="glossary-p">P</h2>





















<hr class="wp-block-separator"/>





<h2 id="glossary-q">Q</h2>





<hr class="wp-block-separator"/>





<h2 id="glossary-r">R</h2>











<hr class="wp-block-separator"/>





<h2 id="glossary-s">S</h2>







































<hr class="wp-block-separator"/>





<h2 id="glossary-t">T</h2>







<hr class="wp-block-separator"/>





<h2 id="glossary-u">U</h2>









<hr class="wp-block-separator"/>





<h2 id="glossary-v">V</h2>







<hr class="wp-block-separator"/>





<h2 id="glossary-w">W</h2>













<hr class="wp-block-separator"/>





<h2 id="glossary-x">X</h2>







<hr class="wp-block-separator"/>





<h2 id="glossary-y">Y</h2>





<hr class="wp-block-separator"/>



<h2 id="glossary-z">Z</h2>


343 INDUSTRIES

Latest