Should we be deploying code every day?

Many big names in tech are evangelists for frequent deployment. At one point Amazon claimed to deploy every 11.6 seconds and the likes of Google and Facebook aren't far behind [1]. However, many companies also choose to release updates to their software far less frequently. Are these teams lagging behind, or do they have a good reason for slower iteration?

Developers can feel hesitant to release updates with design changes, but with some good reason; research shows developers are wrong about user preferences about 90% of the time [2]. Any substantial changes to the UI of Instagram or Facebook prompts social-media outrage and guides to reversing changes [3] [4]. This could be solved by in-house user testing, but users are often equally unreliable at knowing what they want [5]. A better solution is A/B testing, which allow developers to monitor the real user experience and directly compare metrics against the existing choices. Skyscanner has built this type of experimentation into their culture, they use a custom tool, Dr. Jekyll, to simplify the process and empower developers without a statistics background to get quantitative feedback on their design decision [6].

Regardless of the quality, the way changes are delivered can also provoke reaction. Web services have the luxury of being able to deploy a new version invisibly, but native apps involve the user in the process. A recent Call of Duty update angered players by stopping them from playing until a 30GB update was installed [7]. The update was so large because Activation was following a release road-map designed to keep players engaged (and spending) over a longer period of time [8]. Tesla approaches this differently; instead of having a strict seasonal release schedule, they rely on small frequent updates. Once a commit has been through an automated testing pipeline, manual acceptance testing is done on the roads a few times a day, and then an update is deployed automatically over the air. These user updates are non-blocking and user-scheduled, to be minimally disruptive [9]. However, implementing this kind of release schedule did impose some obstacles. In 2013, Tesla had to roll back their recently released “smart air suspension” feature, as it was linked to the explosion of three cars [10]. Automating an extensive testing pipeline is only as good as the quality of the tests it runs. Fortunately, Tesla can roll-back changes as efficiently as they release them, which in 2013 prevented insufficient testing from causing further incidents.

In more established industries, regulation tries to prevent these kinds of accidents. The banking sector is a prime example; developers have to deal with data-residency laws and ensure the safety of their users' finances. Whilst important, these regulations have led to an over-reliance on safe legacy infrastructure. Many dependencies are embedded in financial systems; this makes it harder for software engineers to automate end-to-end testing and limits the rate of deployment. To address this bottleneck, multiple banks including Lloyds [11] have migrated some of their legacy services to a banking infrastructure by Thought Machine [12]. Vault is a cloud-native core banking platform built on a micro-service architecture, which supports build, testing and deployment automation. Whilst these solutions can't completely remove the burden of regulation, they can still accelerate the rate of deployment in banking.

Deploying infrequently can be caused by marketing strategies, legacy systems or fear of user reaction, but it is rarely motivated by improving the product. However, making small changes doesn't excuse carelessness; a fast rate of iteration needs to be supported by a solid testing pipeline and the facility to roll back changes. Avoiding Activation's large updates doesn't excuse Tesla blowing up cars; avoiding the UI outrage faced by Facebook doesn't mean you couldn't benefit from doing A/B testing like Skyscanner.

This article was co-authored by Joe Rackham and Julie Emile.

References

1. Velocity 2011: Jon Jenkins - Velocity Culture  [URL] [Accessed 25th January 2021]
2. The Evolution of Continuous Experimentation in Software Product Development: From Data to a Data-Driven Organization at Scale  [URL] Fabijan, A, Dmitriev, P, Olsson, H.H and Bosch, J, 2017. Proceedings of the 39th International Conference on Software Engineering, pp. 770–780. IEEE Press. DOI: 10.1109/ICSE.2017.76
3. Instagram launches risky redesign – and users aren't happy  [URL] [Accessed 24th January 2021]
4. How to get old Facebook back and switch to the ‘classic’ look  [URL] [Accessed 24th January 2021]
5. Myth #21: People can tell you what they want  [URL] [Accessed 25th January 2021]
6. Fostering a Culture of Experimentation at Skyscanner  [URL] [Accessed 24th January 2021]
7. Gamers are fed up after another massive Call of Duty: Warzone patch dropped  [URL] [Accessed 24th January 2021]
8. Call of Duty: Black Ops: Cold War: Season-1  [URL] [Accessed 25th January 2021]
9. Towards Continuous Integration and Continuous Delivery in the Automotive Industry  [PDF] [Accessed 22nd January 2021] Vöst, S and Wagner, S, 2016.
10. Steve Blank When Product Features Disappear – Amazon, Apple and Tesla and the Troubled Future for 21st Century Consumers  [URL] [Accessed 22nd January 2021] Blank, S, 2017.
11. Thought Machine nabs $83M for a cloud-based platform that powers banking services  [URL] [Accessed 24th January 2021]
12. Why microservices are the future of banking  [URL] [Accessed 24th January 2021]