List of changesets and updating
For a few years, Unity has used Team City from Jet Brains for automated building and testing.As the R&D team grew here at Unity, the demands on the build infrastructure grew on multiple axes (namely the number of users, the number of changesets, and the number of simultaneous branches).We reached a point where we needed to accommodate several thousand builds per day, and we started seeing performance problems on multiple fronts: servers became slow to respond, we encountered unexplained errors that we could not fix, new changes were processed very slowly causing delays, webpages took several minutes to load, etc. and both can be hard to get with off-the-shelf proprietary solutions).After a year of back-and-forth with the makers of Team City, and a progressively worsening state of our build infrastructure, I came to the conclusion that the best path forward for us was to switch to a solution that better suits our particular needs (obviously when you have a way of working that is as particular as ours, combined with our scale, extensibility and flexibility are a necessity in any tool . Being a long-time open-source enthusiast, I felt this was a particularly good scenario to leverage the power of open-source to fix our problems.After some research, I decided that we would build a custom solution on top of Buildbot — an open-source continuous integration framework used by Chromium, Mozilla, Python, and various other projects.Buildbot is written in Python on top of the Twisted event-driven networking engine.It was now September of 2012, and, luckily for me, we had just expanded the Build Engineering team at this time with a new hire — Maria — who already had previous experience working with Buildbot.We knew this would be a large project, so we started with a 2-month long prototyping/proof-of-concept phase where Maria explored various aspects of Buildbot to test its potential to scale in the future while maintaining the flexibility we needed for our complex build chains.
After around two months of prototyping and proof-of-concept work, we were confident the toolset we had chosen would work — with some serious investment.
The next phase of the project involved doing a feature comparison between Buildbot and Team City and an initial attempt to gather requirements for a system that could be used in production as a Team City replacement.
This part was hard and required some iteration, because we 1) were still learning about all of the capabilities and limitations of Buildbot, and 2) it was hard to figure out which features Team City had that were really useful to us and which ones we could live without.
We started with an initial project plan and schedule, which we revised along the way at regular intervals.
At this point, we brought our IT department in to provide estimates on the amount of hardware we would need to acquire to build a fully-functioning system without taking resources away from our production instances.
The version of Buildbot we forked from (0.8.7) does come with a user interface, but coming from Team City, it was practically impossible to use, especially with the number of build configurations and number of builds we have.