News

MCC Insider Flight 2 Recap

By -

Public flights are an integral part of our MCC update plans as we roll out fixes, improvements, and features into work-in-progress builds so members of the MCC Insider Program can go hands-on to help validate all the great progress the team is making. Today, we’re looking back on the most recent flight to share some results and key takeaways from three action-packed days of MCC multiplayer mayhem. We’ll also hear from Dr. Jessie Thomas – our partner on the Xbox Research team – with a recap of some of the feedback she received via the official flight participant survey.

Before we dive in, we want to extend another huge THANK YOU to everyone who participated in flight #2! Your time and input truly are shaping the future of MCC and helping the team do the best work possible.

To everyone else who’s joined the MCC Insider program but hasn’t yet had an opportunity to participate in a flight – hang in there, we’ve got some news down below on what’s next and when you can likely expect to join in on the fun (soon!). And if you haven’t yet joined but would like to, we’d love to have you! Check out THIS PAGE for additional details and instructions on how to opt-in. And as always, if you’re interested in reading up on the work that’s been done to MCC so far, be sure to check out all of our prior MCC development update blogs.

Flight #2 Spec & Goals

The first public flight was all about validating foundational networking and matchmaking improvements at the core of MCC, which was measured via peer-to-peer playtesting of Halo 2: Anniversary multiplayer.

Following a successful test of that work, Flight #2 was designed to help validate continued updates and improvements to matchmaking but with a specific focus on dedicated server deployment and reliability across North America and Western Europe.

As you might recall from our prior blog updates, all multiplayer matchmaking games in MCC will utilize dedicated servers while customs games will rely on peer-to-peer connections.

To measure these matchmaking improvements and dedicated server operations the team focused on the following key data points (which were also the same metrics monitored during flight #1):

  • Matchmaking success rate
  • Time-to-match

The content offering for flight #2 was expanded with more variety across the three days of the flight (which was largely due to participant feedback from flight #1) and included the following:

  • Thursday, 6/7 : Halo 3 4v4
  • Friday, 6/8: Halo 3 Big Team Battle
  • Saturday, 6/9: Halo 3 Action Sack & Halo 3 Team Doubles

In addition to unique playlist offerings across each of the three days, each day included TWO official play sessions to help better facilitate participation outside of North America.

To help test dedicated servers beyond North America, and support larger population playlists like Big Team Battle, the overall number of people invited to participate was nearly double that of flight #1. Everyone who was included in the first flight was invited back again in addition to a new set of players that were chosen from the MCC Insider pool and had a recent history of Halo 2 and/or Halo 3 multiplayer in MCC (among other factors).

Additional Goals

In addition to the specific metrics used to validate the overall matchmaking experience and dedicated server deployment and reliability, this flight had a few other goals tied to it:

  • Continue to refine and hone the overall flighting program operations and build deployment process
  • Expand and increase participant feedback and discussion via the removal of the confidentiality restrictions and creation of daily forum threads on Halo Waypoint
  • Continue building best practices, expertise and tools to execute larger scale flights in the future
  • Update the MCC build to clients and dedicated servers during the flight
  • Experiment with additional methods to incentivize flight participation and player engagement
  • Continue driving positive qualitative feedback and positive participant sentiment

Overall the team gathered some great additional insights and learnings in these areas. Read on for the results below.

Flight #2 Results

Overall, thanks to our MCC Insider participants, flight #2 was a success!

The primary goal was to validate dedicated server deployment and reliability and we saw the desired performance from the dedicated servers and were able to work through the new deployment pipelines and processes successfully.

On the data side, matchmaking success rates were in line with our targets though we did see some expected regression during some sessions that was a result of a combination of high-player playlists (Big Team Battle) and a lower than desired active player population. Overall, cases where matchmaking success rates did not hit the target were directly attributable to low population and regional distribution, not technical issues.

Time-to-match metrics ended up meeting the desired targets overall though, as with matchmaking success, we did see some players and sessions falling outside the ideal range on account of lower than expected concurrent player population and/or regional distribution. Additionally, the added step of connecting a party to the dedicated server (which wasn’t a factor in flight #1), also slightly increased everyone’s time-to-match. When we account for that, and look closer at the sessions that had the highest peak concurrency, the data was positive and much more in line with targets. 

Now here’s Dr. Jessie Thomas to share some qualitative results based on MCC Insider survey feedback…

MCC Insider Survey Results

Hiya MCC Insiders!

Thank you again for taking time to fill out your post-flight surveys – you guys are amazing! As we have said from the beginning, we intend to maintain a high level of transparency with our Insider community. Therefore, I’d like to share a portion of the survey results from Flight 2.

Technical Quality

Following each flight, folks were asked to indicate how much they agree with a series of statements regarding the technical quality of the MCC Insider build. As shown below, Flight 2 showed improvement over Flight 1 across all five ratings for Technical Quality, particularly ratings for performance. While this is completely awesome news, it is possible this boost in ratings is due to slight differences across flight cohorts.

 

Program Quality

In general, Flight 2 respondents continued to feel very positive about the MCC Insider Program. Metrics regarding developer communication, expectations, developer response and build functionality received the highest scores with the majority of folks agreeing it was a positive experience. We also saw improvements in the rating for amount of content, likely due to the multiple rotation of playlists that occurred in Flight 2.

QUalitative Feedback

We received lots of wonderful qualitative feedback from our MCC Insiders on what they liked and disliked about their Flight 2 experience. Here are the Top 3 Likes and Dislikes as well as some quotes from our Insiders explaining their perspective.

Liked about Flight 2:

  • Dedicated servers

“The performance was excellent. No problems while in game and Battle Rifle felt way better.”

  • Fast matchmaking

“Matchmaking was the FASTEST and most secure I've ever seen up until this point. Was on and counting down to match start within 30 seconds of hitting start. Leaps and bounds better as far as speed and ease of connecting to multiplayer matches.”

  • Updated Graphics

“Halo 3 looked incredible on my Xbox One X in 4K! I'm very interested in the future of the MCC project and what other enhancements will come to the original games.”

Disliked about Flight 2:

  • Lack of skill based matchmaking

“Being matched with players of MUCH different skill”

  • Desire for other games/modes

“There were very limited maps/gametypes offered that made playing for up to 3 hours of time difficult, especially Thursday and Saturday (4v4's). For example on Thursday night session, I played so much High Ground assault/1 Flag CTF that I think I can run both offense and defense in my sleep! Same for Saturday with Heretic Rockets. The system’s ability to "mix it up" a little more frequently would make playing thru those 3 hour sessions much easier and I would be much more engaged to offer appropriate feedback.”

  • Longer search times in the UK

“Couldn't find games easily due to being in Ireland. Long wait times.”

Open Feedback

"4K graphic and dedicated servers make for a very nice multiplayer experience. The game feels snappier, load times are shorter, 4K graphics on my 65inch OLED allow to read the battlefield very quickly. I had only one crash out of 20-30 games, in the post-game lobby, so stability was pretty good. Not having to fear the dreaded Halo 3 host migration is AWESOME too.”

 “I saw a lot of improvement regarding the stability, searching for players, time for loading a match. But, at least from me (I'm from Italy), I had a lot of lag and latency issues (shots registration, people turning corners perfectly aiming at my head, people disappearing etc). And the graphics looked improved too

“It was very fun. I really liked the way you put constant upgrade - toward the playlist, etc - in reaction of feedbacks. Great job, can't wait to see flight 3. On multiple Halo games this time?”

My experience of Flight 2 was very good. I think it was a good idea to make two playlists one for casual and one for competitive, this resulted in very little frustration regarding my team. Furthermore, I would like to thank you all for the hard work on updating the MCC, I have been waiting a couple of years and are very happy that you got around to do it even though you guys have a tight schedule.”

“You guys are doing an amazing job as this truly feels like it's showing the amount of work and passion everyone is putting into the MCC. Even the original launch while not perfect still had the same spirit, this finally shines brightly all of the hard work that has be done.”

We also received feedback on MCC playlist preferences. We will be analyzing that data along with responses incorporated from the public survey data and publishing that in a separate post.

That’s all for now! Be sure to keep a lookout on your emails for invitations to future flight sessions, announcements, and surveys from the MCC Insider Program. You guys are awesome!

Your friendly Halo researcher,

Jessie (@Jessabirdy)

Key Takeaways & Learnings

Thanks Jessie! And thank you again to all our MCC Insiders who took the official survey and provided critical feedback to help the team!

This latest MCC flight helped the team learn a lot about the work they’ve been doing to the game itself as well as the work we continue to do around running a public flight. Here are some of the key takeaways the team has been discussing following the flight:

  • Engagement – One the one hand, players overall responded positively to the meta game we added during this flight that rewarded any players who happened to match with a 343 employee with a special Halo Waypoint avatar and badge along with a weapon skin in Halo 5. On the other hand, despite having nearly 2x the player audience than flight #1 and offering more variety over the course of the flight (which participants specifically asked for), participation was lower than expected and didn’t really mirror the level of engagement we saw during flight #1.

As we work on future flights and larger content packages/increased scope, we’re continuing to review player feedback and explore ways to keep players engaged and incentivized to play across every facet of the flight. We need the support and participation of the community – please share your input and vote with your play time – if you choose not to play, then you’ve effectively given up your right to complain later and you’ve denied someone else who does care the opportunity to make a real impact. We’ve still got work to do, if you signed up, please help us (or, no hard feelings at all, you can opt-out).

  • Pre Flight” Early Access – While the official flight session didn’t begin until Thursday, the build was actually available beginning on Monday. Though the population was small, we had enough players engaged in those early days to help identify an issue and ultimately improve the experience for everyone by the time things really kicked off. In general, the idea of a ‘soft launch’ a few days prior to make sure everything is working and getting in front of any issues proved to be incredibly helpful. The team is planning a similar cadence for the next flight as well. We also decided to leave the build "on" and leave the lights on following the final official flight session on Saturday. The population dropped dramatically but a few hardcore folks continued to play Halo 2 classic and we've gotten some great additional insights and feedback in new areas thanks to that effort!
  • Latency & Connection Difficulties – Some players reported difficulties connecting to games during the flight. In one instance, an issue was found early that was causing connection issues for players in North America who had their default language set to something other than English. This was quickly addressed in a new build that was deployed prior to the first official flight. If not for the early access “pre-flight” play time, we wouldn’t have been able to address this in time for the official tests.

    Given the lower than desired population, particularly outside of North America, it’s expected that the folks who did try to play in those regions had a difficult time finding a game (if at all). We also had only a fraction of the data centers setup for the flight so there were some real regional limitations that unfortunately led to some of you having a not-great experience. And to the one lone person who participated in Asia – thank you, and we’re sorry – Asia wasn’t officially supported and there were no dedicated servers or other players in your region to match with so it's understanable that you had a rough go of things.

  • Lack of Skill / Party Matching – Another area we’ve heard a lot about is the overall lack of skill-based matchmaking and the lack of any matchmaking logic that helped to curb solo players matching against full parties. Given that our population was so low for this flight, those systems and parameters were disabled to just make sure as many people as possible could successfully match and play. We heard similar feedback for flight #1 - many of us were on the receiving end of many an ass-whooping. We did course-correct for Saturday of Flight #2 by changing the playlists to feature "Action Sack" as a destination for those who were less competitively-oriented and "Team Doubles" for players who wanted to get as sweaty as possible. This is definitely an area the team will keep thinking about as we grow the flight program and we'll be looking to re-enable skill and party matchmaking systems as soon as we feel the population numbers can handle those extra hoops.
  • Shot Registration – While we did get a lot of good feedback that many players felt that Halo 3’s shot registration felt improved over the current retail MCC, it’s still an area the team will continue to monitor.

  • Poor Sportsmanship – Perhaps it was naïve to think that after such a great, positive first flight, that we’d double the population and have an even greater, more positive experience in flight #2. Truth is, some people seemed to make it their mission to harass and grief teammates throughout the flight. A few of those people were already removed from the program and going forward we won’t hesitate to permanently remove others who also make it their mission to just ruin the experience for everyone else. We're also looking into dusting off the banhammer tools and having it at the ready for future flights.That said, I’d like to say that I had an incredible time playing Halo 3 Big Team Battle all night on Friday and to everyone we played with and against – GGs my friends. What an awesome night that was!

  • Spawns – The team is also tracking some feedback around reports of some spawns being broken or not working as expected on a variety of maps. We believe this was traced back to a matter of using some incorrect map setups when the playlists were created but it’s definitely an area the team is continuing to look into.

  • Removal of Confidentiality Restrictions - Based on feedback from the first flight, we opted to lift the confidentiality restrictions for flight #2 to better facilitate feedback and the flow of information between the team and MCC Insiders. Many of you also decided to stream your Halo 3 exploits which was great fun to watch and participate in. Going forward, we expect a similar level of open-ness for future flights. It's great to see survey results echoing this positive response to loosening the confidentiality restrictions! (though I could personally do without screenshots of myself getting pwned or some very cringe-worthy post-game-carnage-reports)

Special Thanks - Over Achievers

We appreciate everyone who took the time to play and provide feedback but as always there were a few individuals who really rose to the occasion and blew us away with their commitment and passion. To these all-star over-achievers, an extra big double dose of THANK YOU!

Who played the most total games across the three official days of flight #2? These five folks did!

  1. Halo Pro
  2. xsv
  3. FRiCTiON just
  4. Fishmvn
  5. Luminously

This time around we left the flight active the entire time (in fact it’s still technically active now!) so we had a number of people who just kept on playing well beyond the mere three days we asked for. Here are the heroes who played the most games total across the entire build from 6/5 through 6/14:

  1. GregoR
  2. FRiCTiON just
  3. xsv
  4. Clutchy
  5. Fishmvn

In terms of overall wins, both during the three official sessions and across the duration of the flight build, that honor goes to GregoR!

Most kills across the entire flight? That honor goes to xsv!

And if I had to guess - the player who was on the receiving end of the most "celebratory crouches" - me!

Lastly, we’d also like to acknowledge the several hundred players who managed to match with or against members of the 343 team to earn some special rewards. We hope you enjoyed this additional activity and we’d love to hear your ideas and thoughts for similar meta games and fun engagement programs we could weave into future flights!

Next Flight

Right now, the team is hard at work on wrapping up a number of milestones and getting ever closer to the release of the first official update for MCC (which by the way, was officially confirmed to be coming to Game Pass later this year!). But, before that happens, we’re going to have more features and content that needs community testing and input.

If you were in flight #2, you may or may not already know that the build is actually still alive and functioning. Naturally, the population has diminished significantly but if you’d like to play a bit of classic Halo 2, it’s there for you, complete with a fresh FEEDBACK THREAD.

Exact details for flight #3 are still being finalized but the current target plan is that the next flight will essentially include the entirety of MCC and be opened to everyone in the MCC Insider Program to participate. We know many of you have been eagerly awaiting your chance to get hands-on and we’re happy to confirm that your time is nearly here. We’ll have more details as soon as things are locked but rest assured that we’re looking to go much bigger in scope and participation for round three and beyond.

Thanks again to everyone who supported this flight and to everyone who’s shared feedback, ideas, and input to help our team make MCC the best Halo experience it can be! And regardless of whether we’re in the midst of an active flight or not, your feedback is always welcomed!

Related


MCC Development Update #5

Halo: The Master Chief Collection

MCC Development Update #5

By ske7ch -

Welcome to the fifth MCC development update blog! In this latest installment, we’ve got exciting news about Flight #3, next steps for the MCC Insider Program, details on the crowd-sourced updated Matchmaking Playlist spec, more engineering updates in The Scoop, and more. Let’s go!

MCC Development Update #4

Halo: The Master Chief Collection

MCC Development Update #4

By ske7ch -

Welcome back MCC friends! For our latest update blog, we’re getting ready to pull the trigger on our second public flight and we’ve got some various odds and ends to catch up on including HDR Settings, Intelligent Delivery, and The Scoop from the engineering trenches. You’ve been patiently awaiting more details and we’ve got ‘em, so let’s go!

ForgeHub Community Contest Winners

Halo: The Master Chief Collection

ForgeHub Community Contest Winners

By Postums -

Starting back in May, the fine people over at ForgeHub kicked off the ForgeTacular contest for the amazing map creators to compete in making maps for Halo 2 Anniversary in the Master Chief Collection. Today, we take a look at the winners!