Studio Shenanigans #2 – What is Dev-Ops, why is it so important?
What the hell is Dev-Ops?
Yeah so this was pretty much how we responded when we were first introduced to Dev-Ops. How can this big, vital component of game dev that we so desperately needed have gone unnoticed by us?
Unfortunately Dev-Ops (at least for us) went under the radar, we had never been properly introduced to Automation, but we are good friends now! Automation is the use or introduction of automatic processes in development or manufacturing, essentially, automatic. As the name suggests, automation takes the boring tasks from the developer and makes the computer do it!
There are relatively large components of game development that tend to go unnoticed, most definitely by a lot of end users and most likely even by a lot of people who are in the game (industry/Development). Components that if not implemented and done so correctly and with a solid workflow can massively impact development cycles. Before our introduction with Dev-ops, it would be like that section of a company building where people would walk past the door with a curious look on their face thinking “Huh, i wonder what it is they do in there?”
Dev-ops includes a relatively large array of different practises that enable more efficient and solidified workflows for the development of software and games. The most notable practice is the use of computers to automatically package your project for the end user. We, the developer, do not want to spend time compiling, packaging and uploading our 20gb game to Steam, when a computer can do it for us as we continue our day building new cool features!
You may already see the appeal here…
Okay, so what does it do?
We’re glad you asked! Simply put, Dev-Ops makes the computer do all the work when packaging our projects, instead of us. When we say packaging, what we mean is, taking it from inside the Unreal Engine, and making it into a playable application / game so we can play or test it. It’s super useful when trying to quickly find bugs or enhance user experience.
Let’s talk about something called 
Continuous Integration & Deployment So, Continuous Integration (CI) and Continuous Deployment (CD) are the two major parts in the Dev-Ops pipeline. One big advantage to Dev-Ops is being able to package our games without any user input, so we can sit back and relax. However, in reality, when a project is busy packaging at the end of each day, the last thing we are doing is relaxing, sadly.
As the developer, we want to ensure that each day, we update the product, so that when packaging time arrives, it rolls out a new build with new features! This could be a new game mechanic, or something for one of our clients. This whole process (Updating the project as we go and it does a package to test) is called Continuous Integration, as we are continually integrating new features as the product rolls out into staging.
You mentioned.. Versioning?
Ah yes, versioning! I’m sure you know how buggy software and games can be, well to mitigate issues with bugs popping up in builds we release or show to clients, we want to be able to hold onto any stable builds. To do this, we can test a deployed build, check if it is stable, and mark it down as a stable build. This means if our next build fails, we can easily and quickly rollback to our most recent stable build, saving us a lot of stress and time trying to fix anything on the fly!
Ahh, ok, we definitely need Dev-Ops…
Yes you do. Initially, back in our University days, we were blissfully unaware of what Dev-Ops was. As we progressed further into the end of our studies developing our Time Rivals IP, we quickly became aware of the sheer amount of time it takes to not only ‘package’ our projects to play. But to also package them for each target and platform.
For Time Rivals, we needed both a server, and the game itself to be packaged so we could quickly play and test it. Amazingly, we managed to get by without Dev-Ops in place, but it wasn’t easy! Manually packaging the game by pressing File > Package > Windows, then waiting an hour whilst your PC is out of service to package, was a tedious task. And then only to find an array of bugs in the build, making you need to go back, change things, then build again.
When we began diversifying into client based work, within the first milestone and deadline, we had delays and struggles handing over a packaged version of the project. Clearly this was very troublesome and unprofessional. We knew we needed to do better, but we needed to figure out how. We also had bugs which only showed themselves in meetings due to a lack of internal testing, which was very embarrassing! We wanted to avoid this.
Therefore, after all this pain, it had us beg the question “Is there an easier way to do this?”. There was, in fact, an easier, and better way to do this.
Why is it important (feat. Stuart Muckley)
About 6 months ago, we had a quick introductory chat with Stuart Muckley, managing director at Codewizards. At this point, still developing time rivals, and had just stepped out into the industry without the umbrella of university to shield us. One of the key points he made, and one we took away with us was, get Dev-Ops set up as soon as you can.
And so recently we have been connecting with Stuart more and we asked him to give us and all you lovely readers a nice insight into his mastermind on IT practises. Stuart asked initially if we wanted his COVID-sick hardcore response to the topic, we obviously said yes:
“If you don’t do CI for your game you’re a f****** i**** 😊”
Stuart Muckley 2022
Words to live by if you ask me, but of course he had more insight to give.
One of his main points was that ultimately, it’s all about the players’ experience, the end user, the gamers (even the sweaty ones 😉 ). Ensuring that your player base has a good experience playing your game.
“It may seem obvious but we want to ensure that players have the best experience, [CI/CD means] that testers can get hold of builds quickly and that we know when developers introduce errors. Players are essential, testers cost money when they’re waiting around and developers lose efficacy when they can’t trust the code to work upon.”
Stuart goes on to say that CI/CD “Cures” these issues as the centralisation of your build chain and means you are “documenting the process to build, you’re enabling test/QA to see the status of the work”. This all contributes to the process which saves your developer heaps of time and stress.
You’ll end up seeing lots of wasted time while in the run up to deadlines when you need builds and deployment without Dev-Ops. Stuart also mentioned that it’s “quicker to build centrally than locally” meaning you can hand over build to QA or clients much faster and “More importantly a developer can carry on working effectively instead of waiting for the CPU core % to drop below 80% 😊”.
The challenges of setting up Dev-Ops and how to do it right!
Unfortunately, setting up Dev-Ops is no simple task, it is one of those situations where all the effort pays off later down the line, and you will thank your past self. The most notable issue with heavy workload Dev-Ops such as Unreal Engine automation is that we need a fairly powerful computer and a build environment. There are a few ways we can go about achieving this setup.
Self hosted – It is not uncommon to see companies use in house, self hosted hardware for CI / CD operations as cloud based solutions are very expensive in comparison to on site hardware. For the same price as an always on cloud based solution, you could purchase a much more powerful on site machine to handle the task of Dev-Ops. However, this is where scalable Dev-Ops come in.
Scalable Dev-Ops is a cloud solution to the problems we just established above! What if it was possible to only boot up a virtual machine when the build is happening, and then shut it down immediately after? Well, thankfully there is a way, a few companies have this infrastructure, however AWS’ CodeBuild system is up there at the top.
What is really great about AWS CodeBuild is that you can use custom Docker images for your build pipeline, Docker is a container service, this means we can create an Unreal Engine Windows Docker container, which has all the tools we need built in for game packaging. This is also fully scalable, and builds can happen at the same time.
All of this put together saves us a massive amount of time as we can build our projects to our own specification and simultaneously, meaning we can work on multiple projects all at the same time!
So… What’s the verdict?
Like Stuart said… if you don’t do it, you’re an idiot.
As daunting of a task it is to get any form of Dev-Ops integrated into your project, it is 100% a necessity, as it will save you an enormous amount of time in the long term. A studio could easily lose a week of development every month due to building and deployment without Dev-Ops, and obviously time is money.
Big thanks to Stuart for getting involved with us and taking the time to give us some great info. Make sure you go check out CodeWizards and everything they do!
Hopefully you got something out of this blog, and see you in the next one.
Jamie. J
By jamie Jessop