Alex Wakeford

Debrief 008

Silver Debrief: Reckoning

Games


<p>“So, who spilled coffee on the MCC servers on Friday 04/01?”</p>



<p>My name is Dana and I’m one of the engineers on the <em>Halo </em>Publishing Team who responded to the recent incident which affected matchmaking and custom games browser in <em>Halo: The Master Chief Collection</em>.</p>



<hr class="wp-block-separator"/>



<h2 class="wp-block-heading"><strong>MCC’s Dedicated Server Model</strong></h2>



<p>In order to contextualize the details shared in this blog, here are some basic details on MCC’s server model. MCC uses dedicated servers for matchmade games and custom game browser sessions. Other game modes in MCC use peer-hosted session (where one player within the game is designated the game host).</p>



<p>These servers are virtual machines hosted by another team at Microsoft which serves first- and third-party Xbox studios and that team is in turn served by Azure technologies. These servers run an application which we call the Unified Dedicated Server (UDS). The UDS is effectively the server counterpart to the MCC game client and contains a wrapper which can configure and launch server versions of any of the games within MCC.</p>



<p>The UDS is a “trusted” authority meaning that it can’t be tampered with like MCC game clients potentially can be on PC. As such, in addition to creating a fair and reliable game experience (where all players in a game have comparable ping, the host has a stable connection, and no one has “host advantage” or “host disadvantage”), it is also a trusted source for tracking match statistics. These stats and other trusted events are used for things like calculating competitive skill ranks and progressing challenges.</p>



<p>From an MCC client perspective (i.e. the version of MCC that runs on your local PC or Xbox), we look at the setup of the UDS in two phases: allocation and configuration.</p>



<p>To allocate a dedicated server, the MCC game client makes a request to our microservice which handles some other details of the allocation on the client’s behalf. A virtual machine is allocated in the optimal region (based on ping—or manual selection in the case of the custom game browser) with sufficient machines standing by and information about the allocated machine is communicated back to the game client. From there, we enter the configuration stage where a connection between the client is established and the client begins sending configuration data for the match (which <em>Halo </em>game to launch, what map and game variant to load, etc.)</p>



<hr class="wp-block-separator"/>



<h2 class="wp-block-heading"><strong>The Incident</strong></h2>



<p>On the afternoon of 04/01, our internal QA team escalated a potential matchmaking issue they were encountering on our release candidate (RC) build (the build expected to ultimately release to our players) for our upcoming MCC game update. Ensuring that this update is in good shape and able to ship by our target date has been my top priority in recent weeks, so I began investigating immediately.</p>



<p>After some initial investigation, the symptom became apparent. A server was being successfully allocated, but configuration was failing. It seemed surprising that this issue wouldn’t have been observed earlier in our RC testing, but as this was observed on a relatively new MCC build, the investigation was directed towards identifying what code changes since our last released game update (roughly a sixth month delta during which time numerous features were added and hundreds of bugs were fixed) could have introduced the issue.</p>



<p>However, it was around this time that reports started rolling in from MCC players that they were experiencing their own matchmaking issues in our production environment (the “real” environment where you play MCC as opposed to the “artificial” internal environment where we test iterative development changes for MCC). The issue we were observing in our internal environment was logically inferred to be caused by code churn from the past six months of development, so surely the production issue was not related. An issue actively impacting our players in the production environment takes priority of an internal issue, though, so our investigation retargeted that issue immediately.</p>



<p>It didn’t take long to confirm that the production issue was, in fact, the very same one impacting our internal RC build!</p>



<p>On one hand, it was a relief to confirm that the issue wasn’t something unique to our RC that would jeopardize our upcoming game update. But the fact that both builds (which were separated by six months of iterative development) were abruptly impacted in exactly the same way had concerning implications of its own. We had made two backend changes for MCC in the preceding days (in preparation for our upcoming game update), but these were logically incapable of introducing the behavior we were observing. Still, we rolled both changes back to entirely rule them out as potential causes.</p>



<p>Shortly after confirming this, we rolled out our first mitigation step for the issue which was to enable peer-to-peer fallback for MCC matchmaking. This allows matches where server configuration fails to fall back to a peer host (like custom games use). The peer-to-peer experience is not the ideal or intended experience and, with the lack of a trusted host we described earlier, doesn’t allow certain challenges to be progressed or skill ranks to be calculated. Still, it would at least increase matchmaking success rates and allow folks to get into matches more reliably for their Friday evening <em>Halo</em>-ing.</p>



<p>It was approaching midnight on Friday. Here’s what we knew so far…</p>



<p>Players were frequently failing to configure dedicated servers. From looking through logs corresponding to failures on the dedicated servers themselves, the primary configuration data was never being received from clients in the first place. Two wildly different MCC builds (one of which had been working as expected in the production environment for the past several months) were affected by the issue in exactly the same way. Recent service changes we’d made were not causal (but had been reverted just in case anyway). The primary mitigation steps we’d taken so far were enabling peer-to-peer fallback and disabling bans. Unfortunately, as we learned further into the weekend, this ban change failed to immediately take effect due to a completely unrelated issue.</p>



<p>On Saturday morning, now that we’d ruled out any MCC-specific changes being causal, we began engaging partner teams across Microsoft. We cast a gradually wider net as each team owning services along the path between client and server conducted their own investigation, slowly narrowing things down. On Saturday we also built a more complete understanding of the issue from a combination of data aggregation and manual reproduction of the issue aided by network traffic capture tools.</p>



<hr class="wp-block-separator"/>



<h2 class="wp-block-heading"><strong>On Thin ICE</strong></h2>



<p>The server configuration process is conducted by way of a series of packets sent between the client and server. The client-server connection itself is negotiated via <a rel="noreferrer noopener" href="https://datatracker.ietf.org/doc/html/rfc8445" target="_blank">Internet Connectivity Establishment (ICE)</a> in order to send STUN packets between them. STUN is a protocol for <a rel="noreferrer noopener" href="https://datatracker.ietf.org/doc/html/rfc8489" target="_blank">Session Traversal Utilities for Network Address Translator (NAT)</a>. In our case, these STUN packets are used for managing <a rel="noreferrer noopener" href="https://www.ietf.org/rfc/rfc768.txt" target="_blank">User Datagram Protocol (UDP)</a> flow between the client and server. The connection is negotiated via binding requests which allow the client to identify the address of the server. The client will only accept solicited traffic from the server (traffic sent along an established connection), but the server can accept unsolicited traffic (allowing the client to initially establish the connection).</p>



<p>This process looks like an initial establishment followed by a series of configuration messages telling the server which <em>Halo</em> game library to launch and optionally providing map and game variant data. What we observed was that the client would send the expected initial packets, the server would receive them and send the expected packets in response, but the client never received them!</p>



<p>Without receiving the initial acknowledgement, the client could never move on to sending the configuration data itself. After ten seconds of not receiving acknowledgment, the client would time out which would manifest as a failure in matchmaking or the custom game browser. 30 seconds after allocation with no configuration, the UDS would shut down (putting the virtual machine back into standby, ready to be allocated again).</p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="1100" height="791" src="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_01.jpg" alt="" class="wp-image-33806" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_01.jpg 1100w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_01-300x216.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_01-1024x736.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_01-768x552.jpg 768w" sizes="(max-width: 1100px) 100vw, 1100px" /></figure>



<p><em>Parallel Wireshark captures from the client (left) and server (right) when reproducing the issue with trace view (top) and UDP stream view (bottom). IP addresses have been anonymized. The server responses (blue in the UPP stream view) are not received by the client.</em></p>



<p>Interestingly, in the cases where the client did receive the response from the server and successfully configure, it would continue to succeed any subsequent attempts to configure that virtual machine for a period of about 15 minutes. Failures were also inconsistent; a client could fail to configure a given server and then succeed on its next attempt. In the retail environment, it is highly unlikely that a client would ever make two contiguous configuration requests to the same server, but this provided some further details for our investigation.</p>



<p>The number of virtual machines allocated rose to higher-than-normal levels because, in most cases, they would fail configuration and clients would move onto allocate another machine and repeat the process. Meanwhile, the amount of actual UDP traffic between clients and servers fell well below normal.</p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="1100" height="395" src="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_02.jpg" alt="" class="wp-image-33820" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_02.jpg 1100w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_02-300x108.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_02-1024x368.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_02-768x276.jpg 768w" sizes="(max-width: 1100px) 100vw, 1100px" /></figure>



<p> <em>MCC dedicated server machine allocations from 03/28 through 04/03</em></p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="1100" height="363" src="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_03.jpg" alt="" class="wp-image-33834" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_03.jpg 1100w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_03-300x99.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_03-1024x338.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_03-768x253.jpg 768w" sizes="(max-width: 1100px) 100vw, 1100px" /></figure>



<p> <em>MCC dedicated server outgoing User Datagram Protocol (UDP) bytes from 03/28 through 04/03</em></p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="1100" height="355" src="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_04.jpg" alt="" class="wp-image-33848" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_04.jpg 1100w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_04-300x97.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_04-1024x330.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_04-768x248.jpg 768w" sizes="(max-width: 1100px) 100vw, 1100px" /></figure>



<p><em>MCC dedicated server outgoing User Datagram Protocol (UDP) packets from 03/28 through 04/03</em></p>



<hr class="wp-block-separator"/>



<h2 class="wp-block-heading"><strong>Mitigation</strong></h2>



<p>Thankfully, we were able to reliably reproduce the issue ourselves which allowed us to set up and capture various network diagnostics in real-time from the client and server simultaneously. This allowed us to narrow down the issue somewhat as we tracked the returning packets on their way back to the client. However, the network topology between client and server is somewhat complex and some components of it are stateful—meaning that the same inputs can result in different outputs based on surrounding factors. As a result, identifying exactly where packets were getting lost along the way was an unfortunately time-consuming process.</p>



<p>In parallel to this investigation, we sought to simplify our test case. MCC and the UDS are each complex applications which can introduce noise or red herrings into this kind of investigation. To combat this, we built a simple test application to replicate our traffic scenario without any other implementation details getting in the way. We found that we could consistently reproduce the issue with our simple test app if the server sent an unsolicited packet to the client (before the client established a connection). This behavior which was consistently resulting in failure would have resulted in success prior to 04/01.</p>



<p>Armed with this information, we were able to make a targeted change to the traffic pattern on our UDS (i.e. a change which would not require a game update for MCC). We rolled this change out in parallel to two of our development branches and began testing the fix against our live MCC build and our upcoming release candidate at the same time. With this change, we observed our server configuration success rate go from something like 25% to 100%. This testing took place on Tuesday afternoon into Wednesday morning, and we deployed our new UDS build containing the change at scale early Wednesday afternoon.</p>



<p>Unfortunately, players currently need to relaunch MCC to receive the new server pairing information (this is something we’re looking to improve in the future). As a result, players saw a lot of false negatives throughout Wednesday afternoon where, even if they had relaunched, someone they matched with hadn’t and they attempted to configure a server from the old deployment (without the fix). Error rates declined as more and more of the population moved up to the new server deployment.</p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="1100" height="538" src="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_05.jpg" alt="" class="wp-image-33862" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_05.jpg 1100w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_05-300x147.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_05-1024x501.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/1100_MCC_MMBlog_05-768x376.jpg 768w" sizes="(max-width: 1100px) 100vw, 1100px" /></figure>



<p><em>MCC dedicated server configuration and reservation failures per hour from 03/31 through 04/08</em></p>



<hr class="wp-block-separator"/>



<h2 class="wp-block-heading"><strong>STUNning Conclusion</strong></h2>



<p>We finally had our fix, but what had caused this established traffic pattern (which adhered to the Internet Engineering Task Force (IETF) standard and had served us well for the past few years) to abruptly stop working? Investigation continued. Finally, it was identified that there had been some updates behind the scenes to the servers we use to relay STUN traffic as part of the ICE process which had resulted in misconfigurations.</p>



<p>We took other steps along the way such as adding more logging and other functionality to our UDS builds to make potential issues like this easier to diagnose in the future. We also implemented the traffic pattern change via a configurable switch (so we can reenable the old behavior at any time if needed). We were ultimately able to deploy a fix very quickly (under 24 hours from implementation to release) and without prompting a client-side MCC update (which requires certification and generally longer lead time), but the nature of the investigation to identify the issue in the first place meant that the whole process took much longer than we would’ve hoped.</p>



<p>Modern multiplayer games are complex and rely on a variety of technologies spanning multiple devices and networks. We’re always seeking to evolve and improve our own tech stack and how it interacts with dependent services. Thank you for joining us for this peek behind the curtain, and for your patience while we resolved the incident.</p>


343 INDUSTRIES

MCC Server Incident Summary

Community

Community Update


<p>Hello Everyone,</p>



<p>With Season 2 locked and loaded for May 3rd, we want to
share a roadmap of what we’re planning to deliver during Season 2 and for the
rest of the year. </p>



<p>More than anything else, we want this roadmap to be accurate
and reliable. Some items and dates are certain, and these are clearly listed.
Others are marked as “targets” or “goals” and they will become more certain
over time. The roadmap’s timeframe will also expand as we announce longer-term
initiatives. In other words, this roadmap is an evolving document, and we’ll
let you know when we make changes to it.</p>



<p>With all that in mind, here’s the Halo Infinite roadmap for 2022:</p>



<figure class="wp-block-image size-full"><a href="https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April.jpg"><img loading="lazy" decoding="async" width="1920" height="1080" src="https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April.jpg" alt="Halo Infinite's 2022 roadmap as of April 22" class="wp-image-33672" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April.jpg 1920w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April-300x169.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April-1024x576.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April-768x432.jpg 768w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April-1536x864.jpg 1536w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite-2022Roadmap-April-1568x882.jpg 1568w" sizes="(max-width: 1920px) 100vw, 1920px" /></a></figure>



<p>This roadmap reflects the same studio priorities <a rel="noreferrer noopener" href="https://www.halowaypoint.com/news/halo-infinite-update-march-2022" target="_blank">that we shared back in March</a>…</p>



<ul><li>Addressing issues negatively impacting player experience.</li><li>Completing Season 2 and delivering it as promised on May 3. </li><li>Continuing work on Campaign Co-op, Forge, and Season 3.</li></ul>



<p>…with a “priority zero” of team health and getting ourselves into a sustainable development rhythm so that we can deliver great experiences to all of you while maintaining a healthy work/life balance.</p>



<p>We know we need to deliver more content and more features more quickly. Staying true to priority zero means that sometimes we need to slow down in order to stay healthy and move faster later. But we’re also aggressively looking at ways to accelerate.</p>



<p>For example, we want to get the Forge toolset into your hands sooner rather than later so that awesome community maps and modes can more rapidly make their way into the game. To make this happen, we’re targeting a Forge open beta later in Season 2. We’ve successfully had Forge in private flighting with a limited audience for some time, so we’ve decided to forgo a large-scale formal flighting program and go right to open beta. With this approach, we can get the core Forge tools out to everyone quicker while also remaining focused on our core priority of continuing to improve foundational aspects of <em>Halo Infinite</em>. Forge will persist from the open beta onwards, evolving and growing over time.</p>



<p>As we continue to improve and ramp up, please understand that all of us at 343 are fully committed to making <em>Halo Infinite</em> the best <em>Halo</em> experience ever. Indeed, the primary reason that we’re extending Season 2 is to give us time before shipping another Season to fully address “quality of life” improvements based on <a rel="noreferrer noopener" href="https://www.halowaypoint.com/news/season-1-outcomes-report-live" target="_blank">learnings from Season 1</a> as well as other data and feedback we’ve collected.</p>



<p>Quality of life (QoL) is shorthand for bug-fixes, new features and/or content to address issues negatively impacting player experience. We’ll be delivering QoL improvements as part of Season releases and as separate updates during Seasons.</p>



<p>Our player-facing term for QoL updates that land during Seasons is “Drop Pods.” And much like their ODST namesakes, these release vehicles are meant to drop in hot, just as soon as they’re ready. Our target is one Drop Pod every month, and you can expect Drop Pods to vary in scope and size. We’ll have more to share about what’s going into Season 2’s Drop Pods as soon as possible after Season 2 launch.</p>



<p>Speaking of which, Season 2 launch is now less than two weeks away, on May 3! If you haven’t already seen the previews for Season 2’s <a rel="noreferrer noopener" href="https://www.halowaypoint.com/news/season-2-map-previews-catalyst-and-breaker" target="_blank">new maps</a> and <a rel="noreferrer noopener" href="https://www.halowaypoint.com/news/season-2-modes-preview" target="_blank">new modes</a>, please have a look. During these preview videos, you’ll spy some of Season 2’s new armor customization options. And here are a few extra combinations to whet your appetite:</p>



<figure class="wp-block-image size-full"><a href="https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED.jpg"><img loading="lazy" decoding="async" width="2560" height="1440" src="https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-scaled.jpg" alt="" class="wp-image-33643" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-scaled.jpg 2560w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-300x169.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-1024x576.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-768x432.jpg 768w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-1536x864.jpg 1536w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-2048x1152.jpg 2048w, https://wpassets.halowaypoint.com/wp-content/2022/04/HaloInfinite_SpartanLineup_S2_4k_WATERMARKED-1568x882.jpg 1568w" sizes="(max-width: 2560px) 100vw, 2560px" /></a></figure>



<p>We’ll have a lot more to share about Season 2 including a deep dive into its Battle Pass rewards, events—including brand new “multiplayer narrative events” that place your personal Spartan in the center of an evolving Seasonal storyline—game modes, and much more during a community livestream scheduled for next Wednesday, April 27, at 1:00 PM PST on <a href="https://www.twitch.tv/Halo">Twitch.tv/Halo</a> and <a href="https://www.youtube.com/Halo/live">YouTube.com/Halo</a>. We’ll also be discussing the roadmap in more detail during this livestream.</p>



<p>In summary: we are committed to delivering more content faster; we will accelerate while prioritizing team health; you can expect QoL improvements during Seasons via Drop Pods—and you can expect more regular roadmap updates as our plans snap into sharper focus.</p>



<p>On behalf of the entire <em>Halo Infinite</em> team, thank you for your passion and patience as we embark on the road ahead. We’re fired up to deliver what’s already on the roadmap as well as to deliver other content and experiences that we’re not ready to reveal just yet… but that we can’t wait for you to play.</p>



<p>Joseph Staten<br>Head of Creative, <em>Halo Infinite</em></p>


The Road Ahead

Halo Infinite Update – April 2022

Among Flood

Season 2: Modes Preview

Fracture: Tenrai Event (Updated)

Debrief 007

Silver Debrief: Homecoming


<p>Season 2 of <em>Halo Infinite</em>—“Lone Wolves”—is almost upon us!</p>



<p>Among the new additions in “Lone Wolves” are new story elements, new armor and cosmetics, new maps and modes. As we draw closer to release on May 3, we’re going to take a closer look at this upcoming content—starting with the two brand-new maps you’ll be playing, Catalyst and Breaker.</p>



<p>We’re fortunate to be joined today by Tyler Ensrude (he/him), Multiplayer Level Designer, to talk about the process of creating maps for <em>Halo</em> and some of the great experiences that players can look forward to having in Season 2.</p>



<p>But first, you want to take a look at these spaces, right?</p>



<p>Before we dive into the Q&A, check out this video flythrough put together by the fine folks on our Transmedia team, diving into the ancient and weathered ruins of Catalyst and the expansive, sand-blasted scrapyard site of Breaker…</p>



<figure class="wp-block-embed-youtube wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
<iframe loading="lazy" title="Halo Infinite | Catalyst & Breaker – Season 2 Map Previews" width="640" height="360" src="https://www.youtube.com/embed/aD88v-0IKJ0?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div></figure>



<hr class="wp-block-separator"/>



<p><strong>I think a good place to start is the question of what was the foundation for these maps and the process of making them? Do you start with a design goal, saying “We need a symmetrical map?” or is it more of an aesthetic one to visually diversify the new maps from what players are now very familiar with?</strong></p>



<p>We like to start with an idea that just gets us excited first and go from there. That idea could range from a location in the <em>Halo </em>universe, to an art palette, game mode, or encounter.</p>



<p>Every time we start something new though, it is important to consider what will make it stand out from the current selection of maps we already have. The context of experiences that players have already had in the game is definitely something we keep in mind when we look to deliver something exciting and new, and that extends across all of those factors—aesthetics, layout and design, and gameplay.</p>



<p>When you look at Catalyst and Breaker, it’s immediately apparent that they look very different from the maps players have become familiar with in <em>Halo Infinite</em>.</p>



<p><strong>How would you quickly summarize Catalyst and Breaker so <em>Halo</em></strong><strong> fans know what they’re in for with these maps?</strong></p>



<p>Catalyst is a symmetrical arena map nested high up inside of an expansive, tunnel-like Forerunner structure. You’ll find waterfalls pouring down around suspended catwalks, overgrown cathedral walls, and a light bridge connecting the sides of the map together.</p>



<p>Breaker is a pseudo symmetric BTB map set in a Banished scrapyard where each team’s base is a huge chunk of ship debris, divided by the plasma cutting beam structure in the middle. And yes, that beam is absolutely a dynamic element of the map that presents both opportunities and challenges in gameplay.</p>



<p><strong>Catalyst reminds me in some ways of Coliseum from <em>Halo 5</em></strong><strong>, which was a favorite for that symmetrical, competitive action. Is that same DNA part of this map, and was inspiration drawn from any other previous <em>Halo</em></strong><strong> arena maps?</strong></p>



<p>Even though the map is something completely new, there’s a ton of DNA from several legacy Forerunner locations in there. Most of that DNA is subtle, but there is quite a lot.</p>



<p>Some folks have definitely noted Haven vibes from <em>Halo 4</em> in the first screenshot we shared back in March, but I think the biggest influence for Catalyst though was Epitaph from <em>Halo 3</em>. I really wanted to evoke that architectural style because it left a huge impression on me after all these years.</p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="1920" height="1140" src="https://wpassets.halowaypoint.com/wp-content/2022/04/Catalyst_Arena_ConceptArt_Watermarked.jpg" alt="Concept art of Catalyst. " class="wp-image-33102" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/Catalyst_Arena_ConceptArt_Watermarked.jpg 1920w, https://wpassets.halowaypoint.com/wp-content/2022/04/Catalyst_Arena_ConceptArt_Watermarked-300x178.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/Catalyst_Arena_ConceptArt_Watermarked-1024x608.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/Catalyst_Arena_ConceptArt_Watermarked-768x456.jpg 768w, https://wpassets.halowaypoint.com/wp-content/2022/04/Catalyst_Arena_ConceptArt_Watermarked-1536x912.jpg 1536w, https://wpassets.halowaypoint.com/wp-content/2022/04/Catalyst_Arena_ConceptArt_Watermarked-1568x931.jpg 1568w" sizes="(max-width: 1920px) 100vw, 1920px" /></figure>



<p><strong>I think that naturally leads to the question of how you juggle drawing upon past inspiration and beloved maps from across the previous games versus the desire to create something fresh and new. How did that factor into the process of designing these maps?</strong></p>



<p><em>Halo </em>has provided us with so many inspirational places over the years. It’s easy to look at something already made and just do that again, but it’s not as creatively fulfilling—and some sandbox evolutions may not translate as well to the “remake” experience. I think it’s important to recognize aspects of what made those things so memorable and use that as a starting point.</p>



<p>Forerunner architecture is typically monolithic and ornate, which opens a lot of doors creatively. The Banished art palette is much newer, and its library of reference material is still something that is growing and evolving, so there’s a great opportunity for a bit more exploration for that. A lot of Banished stuff is scavenged from the UNSC and Covenant, but there’s also core design language unique to the Brutes as well, and often these elements are mixed together so it pulls from a lot of different areas of <em>Halo</em>’s aesthetics.</p>



<p>Expanding upon what the campaign had established, the team landed on leaning heavily into the scrapyard aspect of the map and had a lot of fun thinking about how that theme could be explored.</p>



<p><strong>We’ve obviously got some new modes coming in Season 2, which we’ll be doing a blog on next week so keep an eye out for that! To that point, was the design of these maps influenced by any particular modes? What can we really look forward to playing on Catalyst and Breaker?</strong></p>



<p>Catalyst was meant to expand our CTF (Capture The Flag) suite, but it proved to be a solid contender for several modes and was actually the first map that our new Land Grab mode was implemented on, which quickly became a favorite amongst the team.</p>



<p>Breaker started out as a CTF map as well. The whole idea was to create high-tension flag moments around the laser beam.</p>



<p>You can look forward to all the new modes releasing in Season 2, which were made from the ground up on these new maps. Of course, all the existing modes are supported on them as well.</p>



<figure class="wp-block-image size-full"><img loading="lazy" decoding="async" width="1920" height="1160" src="https://wpassets.halowaypoint.com/wp-content/2022/04/Breaker_BTB_ConceptArt_Watermarked_V2.jpg" alt="Concept Art of Breaker. " class="wp-image-33116" srcset="https://wpassets.halowaypoint.com/wp-content/2022/04/Breaker_BTB_ConceptArt_Watermarked_V2.jpg 1920w, https://wpassets.halowaypoint.com/wp-content/2022/04/Breaker_BTB_ConceptArt_Watermarked_V2-300x181.jpg 300w, https://wpassets.halowaypoint.com/wp-content/2022/04/Breaker_BTB_ConceptArt_Watermarked_V2-1024x619.jpg 1024w, https://wpassets.halowaypoint.com/wp-content/2022/04/Breaker_BTB_ConceptArt_Watermarked_V2-768x464.jpg 768w, https://wpassets.halowaypoint.com/wp-content/2022/04/Breaker_BTB_ConceptArt_Watermarked_V2-1536x928.jpg 1536w, https://wpassets.halowaypoint.com/wp-content/2022/04/Breaker_BTB_ConceptArt_Watermarked_V2-1568x947.jpg 1568w" sizes="(max-width: 1920px) 100vw, 1920px" /></figure>



<p><strong>Breaker definitely looks like one of our most ambitious BTB maps ever, what are some of the things that set this one apart from what players have experienced in <em>Halo Infinite</em>?</strong></p>



<p>We designed this map to embrace base versus base combat where territorial ownership is key to success. We love mixing environmental hazards with the <em>Halo </em>sandbox and Breaker offers an exciting variety of ways to melt your enemies. We’re also big fans of sniper battles, and Breaker’s sightlines and center lane offer risk and reward over key objectives.</p>



<p>The central feature of this map is the middle, which divides the spaces with ramps over a death pit which has given way to some pretty memorable vehicle runs.</p>



<p>We love dynamic map elements, which is something many players will no doubt remember from the remade maps in <em>Halo 2: Anniversary</em>’s multiplayer. For Breaker, we’ve got a giant laser that cuts through anything in its path as it moves across the map.</p>



<p>That death pit you’re going to try to jump your Warthog over to get your flag carrier back to your base? You’re definitely going to want to time that right because I don’t think insurance is going to cover the damage that laser will do to your Warthog and teammates. Alternatively, grab a Repulsor and time your usage wisely so you can ruin somebody else’s day!</p>



<p>But really, the artists and designers who brought this space to life did such an amazing job.</p>



<p><strong>That all sounds incredible, thank you for joining us today, Tyler! I can’t wait to get out onto the virtual battlefield myself to explore these spaces and get carri—I mean, </strong><em><strong>totally carry my team to victory!</strong></em></p>



<p>— </p>



<p>We’re super excited for all of you to jump into the new maps and modes of Season 2, speaking of which that’s the next thing we’ll be covering—check back next week for a look at King of the Hill, Land Grab, and Last Spartan Standing!</p>



<p><a href="https://www.halowaypoint.com/news/halo-infinite-update-march-2022" target="_blank" rel="noreferrer noopener" aria-label="Last month, Joseph shared our first look at Season 2’s content (opens in a new tab)">Last month, Joseph shared our first look at Season 2’s content</a> with details about these maps, the RAKSHASA and EAGLESTRIKE armor cores, the newest game modes, and more. Be sure to give that a read here on Waypoint!</p>



<p>Oh, and for fans who are keen to learn more about the story behind these maps, we’ll be doing a deeper dive into that in our next Canon Fodder.</p>



<p>See you online, Spartans.</p>


Season 2 Map Previews: Catalyst & Breaker


<p>Representing two decades of storytelling, the largest and most comprehensive official guide to the <em>Halo </em>universe ever to be created—the new <em>Halo Encyclopedia</em>—has finally been released into the wild. In this massive tome, Dark Horse and 343 Industries have partnered together to deliver a vast directory of <em>Halo </em>canon, allowing fans to explore the intricacies of an epic saga that spans over a hundred millennia and countless far-flung alien worlds. Delve into a vast archive of ships, vehicles, and weapons that defined the wars of the 26th century and come face-to-face with the major players of the UNSC, the Covenant, the Forerunners, and the Banished. The <em>Halo Encyclopedia</em> features stunning, never before seen art revealing characters, technology, and locations that fans have only ever imagined. </p>



<p>This extravagant oversized hardcover has something to offer both veteran fans of the <em>Halo </em>universe and those who are greeting it for the very first time. Pick up your own copy of the <em>Halo Encyclopedia</em> <a href="https://www.darkhorse.com/Books/3008-349/Halo-Encyclopedia-HC">here</a> or secure the impressive <em>Deluxe Edition</em> <a href="https://www.darkhorse.com/Books/3009-942/Halo-Encyclopedia-HC-Deluxe-Edition">here</a>, elegantly enshrined in a mysterious Forerunner cylix. </p>



<p>And whether you are totally new to <em>Halo </em>or perhaps coming back after some time, we know that with the release of <em>Halo Infinite</em>, many folks may be unfamiliar with some of the terminology we’ve accrued over the past twenty years. No worries, we’ve got you covered! Below is an online companion resource for the new <em>Halo Encyclopedia</em>—a brief, high-level glossary to help you navigate some of the names, terms, and acronyms that populate the universe’s lexicon. So no matter if you read this book from front to back or just dive into its 488 pages somewhere in the middle, feel free to use this directory as a friendly guide, especially if you’ve found yourself suddenly lost in the vast ocean of lore this new book holds. We really do hope you enjoy it!</p>



<p>For more information about the <em>Halo Encyclopedia</em>, check out the roundtable discussion we had with its writers, where we talk about its development process and showcase brand new art created specifically for this book:  </p>



<figure class="wp-block-embed-youtube wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
<iframe loading="lazy" title="Halo Encyclopedia | Full Roundtable Discussion" width="640" height="360" src="https://www.youtube.com/embed/yNx3Vqv99t0?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
</div></figure>



<h2 class="wp-block-heading">Glossary</h2>



<h2 class="wp-block-heading"><a href="#glossary-a">A</a> | <a href="#glossary-b">B</a> | <a href="#glossary-c">C</a> | <a href="#glossary-d">D</a> | <a href="#glossary-e">E</a> | <a href="#glossary-f">F</a> | <a href="#glossary-g">G</a> | <a href="#glossary-h">H</a> | <a href="#glossary-i">I</a> | <a href="#glossary-j">J</a> | <a href="#glossary-k">K</a> | <a href="#glossary-l">L</a> | <a href="#glossary-m">M</a> | <a href="#glossary-n">N</a> | <a href="#glossary-o">O</a> | <a href="#glossary-p">P</a> | <a href="#glossary-q">Q</a> | <a href="#glossary-r">R</a> | <a href="#glossary-s">S</a> | <a href="#glossary-t">T</a> | <a href="#glossary-u">U</a> | <a href="#glossary-v">V</a> | <a href="#glossary-w">W</a> | <a href="#glossary-x">X</a> | <a href="#glossary-y">Y</a> | <a href="#glossary-z">Z</a></h2>



<hr class="wp-block-separator"/>



<h2 class="wp-block-heading" id="glossary-a">A</h2>































<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-b">B</h2>









<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-c">C</h2>





























<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-d">D</h2>

























<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-e">E</h2>

























<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-f">F</h2>





















<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-g">G</h2>





























<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-h">H</h2>

























<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-i">I</h2>









<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-j">J</h2>











<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-k">K</h2>









<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-l">L</h2>















<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-m">M</h2>





























<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-n">N</h2>







<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-o">O</h2>













<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-p">P</h2>





















<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-q">Q</h2>





<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-r">R</h2>











<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-s">S</h2>







































<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-t">T</h2>







<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-u">U</h2>









<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-v">V</h2>







<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-w">W</h2>













<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-x">X</h2>







<hr class="wp-block-separator"/>





<h2 class="wp-block-heading" id="glossary-y">Y</h2>





<hr class="wp-block-separator"/>



<h2 class="wp-block-heading" id="glossary-z">Z</h2>


The New Halo Encyclopedia is Out Today

Modding Tools

Update on Halo: MCC