By 343 Industries -
Today, January 29th, there is a build update for Halo Wars 2 that includes rolling all the prior balance patches into this build, adds a handful of bug fixes, and adjusts some game files for better balancing in the future. The road to making this happen has been a passion project of several people in the studio to continue ongoing balance and development updates. With the Publishing team focused heavily on MCC, it has been a challenge, but a few on the team were very happy to take it on and make it happen. You can read more further down below in our first ever Halo Wars 2 edition of “The ScoopTM.”
Today’s update is on the larger side, clocking in at around 12GB, but there’s a lot of optimization and re-configuring occurring with this patch which will actually result in your overall Halo Wars 2 installation being SMALLER than it was prior. This ‘plumbing’ work will allow the team to better deliver updates in the future. This coming year is going to be a fun one with the 343 Publishing Team’s continued efforts to support Halo Wars 2 and its passionate community. The team has some cool plans in store for 2019 and we’re excited to share them with all of you in the coming months.
But for now, let’s take a peek at what’s in today’s Halo Wars 2 update and get the scoop on how we got here!
- Fixed hardlocks encountered while downloading a balance patch update
- Fixed hardlocks at the main menu while booting the title
- Removing duplicate files
- Fixing a desync issue that could occur after a balance patch
- All balance changes of the past year included into the content update
- Addition of 20+ MP tactic files to help balance units
- Game start delay at launch of Multiplayer match improved for those with an AMD graphics card
Our main man in the engineering trenches, Sean ‘Scoops’ Cooper, doesn’t just write about his work on MCC – today we’ve got the first ever The ScoopTM: Halo Wars 2 Edition:
The road to Halo Wars 2’s 17th Content Update (not to be confused with a patch - the game has seen many more than 17 of those) has been a long and winding one. It has been a rough eight months in the making. The active sustain development team is a group of passionate Halo Wars developers, QA staff, and community members that have helped us continue supporting the project. So, please bear with us! 😊
Halo Wars holds a special place both in and outside my professional life. I spent a few years reverse engineering it when it was only an Xbox 360 title, figuring out how to decrypt/unpack/repack its content files and modify save game files to do all sorts of crazy stuff. This work is how I ended up getting in contact with one of the core engineers on the first Halo Wars, Rich Geldreich.
By that time, Rich was no longer at Ensemble Studios but I got some invaluable face time with him before I eventually moved back to Dallas, Texas. Dallas happens to also be where Ensemble Studios was located back in the day, and where many studios sprung up from its ashes. Trivia: Dallas is also where Gearbox is located, whom worked on Halo PC. Rich would end up moving back down to Dallas to work as the CTO for Boss Fight Entertainment and vouched for me to start work there (as my first *real* industry job). The studio happened to have a sublicense for the Halo Wars engine, so I got to monkey around with it in my spare time. It was not in a state where you could build it, so there was a bunch of grunt work to get the engine and tools working in a Frankenstein Xbox 360 environment. They didn’t have any of the assets for Halo Wars either, so there was even more creativeness in getting it all running using retail content. So, while I’ve been reverse engineering Halo (for good!) since the dawn of its FPS days, the first game I got to touch the source of was Halo Wars. A few years later, I moved on to another really awesome studio made of ex-Ensemble Studios folks, Playful Corp. Before I could start planning a way to work at every studio that sprung up in Ensemble’s ashes, I was presented with a very unique opportunity with 343’s Publishing team.
So, a few months into my gig on the Publishing team, I offered to take on the responsibility of Halo Wars 2 on the side of our MCC work. We weren’t as actively developing Halo Wars and I already had exposure to the engine, albeit in the form of the first game (which the engine is based on), so it seemed reasonable. What came was a lot of upfront work. We were in the process of taking over all duties from Creative Assembly (CA). The game’s Perforce (Source Control) depot, once housed at CA, was shipped to us on a 10TB NAS. We ended up using it as an archive and started a new depot on another server (there was a couple of technical issues that made this the best and fastest course of action). The game’s build system, once housed at CA, was now being setup to run on-site at 343. I picked up where Joseph Cheng, the original 343 build engineer on the project, left off. There was a lot of things I had to learn about TeamCity (the build management system the game uses) and all the different build steps involved.
Since we started on a new Perforce server, that was not mirrored from the original, we didn’t start off with any of the attributes that many files had, and in some cases, required. These attributes include things like “this file is always writable” without a dev manually checking it out for edits. In one particular case, there was a file which a Visual C++ tool *expected* the file to be writable because it opened it for write, even when it is only using it in what is effectively a read-only mode. What made matters worse was that it did not give a clear error message.
The build farm that CA had was also setup to make use of cache databases for game content. That is, the build system would bake source content into game-ready formats then stash the outputs on these cache databases. It would then pull from these databases during the data-build instead of regenerating them. This also meant that a new build machine could be stood up and it would re-use the same shipped game data files. Some tools don’t generate the same *exact* (byte-for-byte) files even when the source content doesn’t change (ie, they have non-deterministic outputs). As it stands right now, if a single byte changes in an existing game file, the Xbox One update system requires the *entire* file to be downloaded instead of doing a more granular only-download-the-changes.
At 343, we are not making use of any cache databases. Instead, we have three build machines and I have one of them dedicated to building data files for Xbox One and another one handles Windows. These machines started the data builds from scratch. It turns out, to build all the game content from scratch for a single platform on the hardware we’re using is a 30-48hr process. But, so long as we don’t nuke the output folder (or let TeamCity ‘cleanup’ workspaces that haven’t been used in X days…), it will only check for changes to the source content and rebuild that subset. That process is relatively quick…so long as the project is housed on an SSD (this was not the case for the build machines we started with).
So, since the 343 build machines had to start the game data builds from scratch, we ended up generating files which do not match all of CU16’s files, leading us to an update on the Xbox One that’s ~13GB. However, for this update I did get a chance to remove around 500MB of data from what we ship to customers. What this all means is that, while you’ll see this update require a rather large download, the game will be smaller *overall* after. I’m still looking at some other changes/fixes to the data build pipeline that, if all goes well, could lead to an even bigger reduction of the game size.
There’s more that I could probably go into on the efforts to stand up a fully functioning build farm here at 343 but I’ve already written a book and haven’t even touched on some of the changes/fixes we’ve addressed with this update!
Shortly after The Yappening last year, Matt, Tyler, Pierre, and myself got together and planned out what bugs we deemed high priority and should be fixed in any future update. While we haven’t tackled that entire list in this CU, we believe we’ve conquered some of the more aggressive bugs. Two of them caused what are referred to as Hard Locks. This is where the game completely locks up and there’s no animations whatsoever. This is compared to a Soft Lock where the game or its UI may stop responding to your input but the background may still be animating. After investigating one of them, it became clear that the same problem was most likely leading to both hard locks. Hooray, two birds, one stone!
Another issue related only to the PC version of the game when the user had an AMD graphics card, where they load into MP games slower and then the game played at higher rate of speed to catch up to be in sync with everyone else. This related to a change that was made before the game actually launched, which should not have shipped, but we nonetheless shipped it. This resulted in some non-optimal usages of AMD’s DX12 implementation.
I happened to stumble upon the cause of another issue while getting familiar with the game code. It had been mentioned once or twice before that after game patches we would see a spike in Out of Sync reports in our session data. Well, it turned out that for crossplay playlists (PC playing with Xbox One) we did not factor in the patch data when determining if the players could play together. This is an issue shortly after updating the game because some people are online when the update rolls out and don’t get the new data while others log in and get the update. Then they both go into matchmaking, the game thinks everything is up to par, then they all get into a match and, depending on the changes, one of them performs an operation using patched data while the other doesn’t, causing the games to go out of sync.
While we still have some other code-based bugs we wish to address, that concludes what we’re targeting for this update. The rest will have to come when I’m not working on MCC updates, and other priority internal tasks 😊.
One thing we have learned while working on Halo Wars 2 is that the community is full of passionate individuals who are highly competitive. Last year we saw the Halo Wars Championship League run several seasons of tournaments which were highly successful. One piece of feedback that has stuck with us was that players wanted more content for Halo Wars 2. We have heard this and Wrensi, who has been quoted many times of saying “I went to art school,” has been working on something behind the scenes for the Halo Wars 2 community. He and the team can’t wait to show off what they have been working on later this year.
If you have feedback on these changes or other things you want to see come to Halo Wars 2, please visit the forums and let us know! Thank you for playing and we look forward to more awesomeness to come this year for Halo Wars 2!