By ske7ch -
As the dust settles and we look fondly back upon our inaugural flight for the MCC Insider Program, we wanted to circle back with our community and share some of the results and outcomes. Today we’re going to breakdown how it all went and go over some of the game team’s key takeaways and we’ll hear from Dr. Jessie Thomas - our partner on the Xbox Research team who’s driving the MCC Insider Program and helping to capture all of your great feedback.
Before we dive into that though, we first want to extend a huge THANK YOU to everyone who jumped in and participated during our first flight. We really appreciate your support and the time you invested into helping us improve MCC!
But we also want to thank EVERYONE who has signed up for the MCC Insider Program – we’re equally appreciative of both your passion and commitment to help shape the future of MCC. We know the vast majority of our Insiders weren’t able to get into this first flight but don’t despair – there will definitely be more opportunities. As we progress down our development roadmap, there will be additional flights that will increase in both scope as well as participant pool. The team remains committed to ensuring that at some point before all is said and done, anyone and everyone who wants to help play an MCC flight and share feedback to be able to.
Flight #1 Spec & Goals
In case you missed this in our last development blog, here’s a quick catch-up on what the first flight entailed and what our key goals were:
The first public flight was designed to help stress test all the core matchmaking updates the team has been working on over the past few months. This major foundational overhaul and modernization work is really at the core of the biggest changes to MCC – many of the issues and areas of feedback fans have been most vocal about stem from the game's original matchmaking systems.
One important note is that this flight was intentionally setup to run with peer-to-peer networking. You may recall that we’ve already announced that following the official retail update, all MCC MP matchmaking games will run via dedicated server. However, for this flight, the dedicated servers weren’t quite fully ready for prime time and we still needed to validate and test peer-to-peer networking (which is what will be used for Custom Games, among other scenarios).
Additionally, this first flight also included several other changes such as the slightly updated main menu UI, the removal of the “vote” button, and the inclusion of “continuous matchmaking.” HDR video settings were also enabled just prior to the flight officially kicking off.
To help validate updates to our network transport layer and improvements to the matchmaking and MP experience for players, the team looked at two key data points:
- Matchmaking success rate
- Time-to-match
In both of these metrics, the team set baseline goals that would represent improvements over the current retail MCC offering and validate that these new updates are working as intended. To facilitate this data, the first flight was setup with a targeted content and experiential offering:
- Halo 2 Anniversary Multiplayer on four maps
- Team Slayer to 25 kills / 5 min game timer
The flight build was setup this way to really promote throughput above all else – the act of searching and joining games and continuously rolling from game to game was the most important aspect of the flight. Approximately 2,150 players were invited to participate in the first flight and were chosen based on a few different criteria (one of which being those who were most active in MCC MP in the past six months). We decided to leave the build accessible full time once it was available, but to help focus efforts and the population we landed on three official playtime windows:
- Friday, April 27: 7PM to 10PM PST
- Saturday, April 28: 10AM to 1PM PST
- Sunday, April 29: 4PM to 7PM PST
Additional Goals
Outside of the specific goals for the game itself, this first flight also carried some unique goals of its own:
- Establish the proper processes and procedures to release game builds (and rapidly iterate fixes/updates as needed)
- Ensure our MCC Message of the Day (rebuilt on new/updated internal services) worked as intended
- Validate our Xbox Insider Hub dependencies and toolsets
- Activate our MCC Insider Program participant pool and run the complete communications cycle (initial confirmation, build availability outreach, survey follow-up)
Flight #1 Results
Thanks to you all, we did it! The team has deemed the first flight a resounding success. Not only did we hit our key goals for time-to-match and matchmaking success, we also feel pretty good about the results of the other misc. goals noted above.
Let’s break it down:
- More than 14,000 total matchmaking attempts over the course of the three official playtest sessions
- Matchmaking success rates significantly exceeded Flight 1 goals
- Time-to-match results were significantly faster than our Flight 1 goals. The vast majority of players were into matches in less than 38s. (that’s pretty quick!)
From a purely technical level, metrics and the story the data told exceed expectations. We’re still waiting on a final apples-to-apples comparison between some of this flight data and the current retail MCC equivalent (it’s complex to align the various telemetry pieces) but it’s clear that the matchmaking and networking updates have already offered some big improvements. Our data folks also need to take into account things like population size and geographic locale, among other factors.
Like me, you may be wondering, “why would there ever be a case of an unsuccessful match?” It could really boil down to all sorts of things from legitimate errors (which we're actively hunting for), to NAT incompatibilities, to just the nature of the internet itself (particularly in the case of a few of our participants outside of North America who surely had some challenges connecting to a peer-to-peer lobby across the world).
Game Build Release Process
Overall the process of compiling and releasing builds went smoothly. You may not even realize this but from the initial time the first flight became available for download in the marketplace several updates were released before the weekend.
First, thanks to feedback from players who jumped in right away on Wednesday, the team was able to identify and resolve a ‘back fill’ issue that was sometimes resulting in an error with the 8th member of a lobby wasn’t correctly filled. This issue was quickly addressed with an internal build, tested, and then deployed quickly and flawlessly on Thursday.
The team also released another build update just prior to the Friday official playtest that enabled the new/updated HDR video settings in the main menu. That wasn’t so much driven by response to an issue as it was just a feature that was now ready and they were eager to get it out into the wild.
Lastly, we also received feedback and validated that somehow an audio issue was introduced that resulted in the main menu music being muted. This too was quickly fixed, tested, and released late Friday night following the first session, ensuring things were in good order for the weekend.
To get a sense of the types of feedback we received from the first flight, please welcome Dr. Jessie Thomas – a member of the Xbox Research Team who not only administers the MCC Insider Program for us but is also at the helm of the Halo Community Feedback Program. Take it away, Jessie!