
manage the risk of potentially deploying erroneous software,
including blue-green deployments [7], feature flags, and dark
launches [8]. As a worst case, stop-gap measure, it is always
possible to “roll back” any recently deployed update at the
push of a button, effectively undoing the deployment of the
target software module by restoring the module back to its
previous version.
In contrast with previous work, in this paper, we describe
and analyze how continuous deployment is applied to mobile
software at Facebook (FB). FB’s mobile software is used
by over a billion people each day. The key challenge in
deploying mobile software is that it is not possible to deploy
the software continuously in the same manner as for cloud-
based services, due the following reasons:
1. The frequency of software updates may be limited be-
cause (i) software updates on mobile platforms are not
entirely transparent to the end-user, and (ii) the time
needed for app reviews to take place by the platform
owner; e.g., Apple reviews for iOS apps.
2. Software cannot be deployed in increments one module
at a time; rather all of the software has to be deployed
as one large binary; this increases risk.
3. Risk mitigation actions are more limited; for example,
hot-fixes and roll-backs are largely unacceptable and
can be applied only in the rarest of circumstances as
they involve the cooperation of the distributor of the
software (e.g., Apple).
4. The end user can choose when to upgrade the mobile
software (if at all), which implies that different ver-
sions of the software run at the same time and need to
continue functioning correctly.
5. Many hardware variants — especially for Android —
and multiple OS variants need to be supported simul-
taneously. The risk of each deployment is increased
significantly because of the size of the Cartesian prod-
uct: app version × (OS type & version) × hardware
platform.
Given the above constraints, a compelling open question
that arises is: How close to “continuous” can one update
and deploy mobile software?
Facebook is striving to push the envelope to get as close as
possible to continuous deployment of mobile software. The
key strategy is to decouple software development and re-
leases from actual deployment; the former occurs frequently
in small increments, while the latter occurs only periodically.
In particular, developers push their mobile software updates
into a Master branch at the same frequency as with cloud
software and in similarly-sized increments. The developers
do this whenever they believe their software is ready for de-
ployment. Then, periodically, a Release branch is cut from
the Master branch — once a week for Android and once ev-
ery two weeks for iOS. The code in that branch is tested
extensively, fixed where necessary, and then deployed to the
general public. The full process used at FB is described in
detail in §2.
An important follow-on question is: How does this mobile
software deployment process affect development productivity
and software quality compared to the productivity and qual-
ity achieved with the more continuous cloud software deploy-
ment process?
We address this question in §5.2 where we show that pro-
ductivity, when measured either in terms of LoC modified
or added, or in terms of the number of commits per day, is
comparable with what it is for software as a whole at FB.
We show that mobile software development productivity re-
mains constant even as the size of the mobile engineering
team grows by a factor of 15X and even as the software
matures and becomes more complex. In fact, we show how
Android has gone from a deployment every 8 weeks to one
every week over a period of 4 years, with no noticeable effect
on programmer productivity.
Testing is particularly important for mobile apps given the
limited options available for taking remedial actions when
critical issues arise post-deployment (in contrast to the op-
tions available for cloud-based software). We describe many
of the testing tools and processes used at FB for mobile code
in §4. We show in §5.5 that the quality of FB mobile soft-
ware is better than what it is on average for all FB-wide
deployed software. And we show that the quality of the
mobile software code does not worsen as (i) the size of the
mobile software development team grows by a factor of 15,
(ii) the mobile product becomes more mature and complex,
and (iii) the release cycle is decreased. In fact, some of the
metrics show that the quality improves over time.
Finally, we present additional findings from our analysis
in §5. For example, we show that software updates pushed
on the day of a Release branch cut are of lower quality on
average. We also show that the higher the number of devel-
opers working on the same code file, the lower the software
quality.
A significant aspect of our study is that the data we base
our analysis on is extensive (§3). The data we analyzed cov-
ers the period from January, 2009 to May, 2016. It includes
all of the commit logs from revision control systems, all app
crashes that were captured, all issues reported by FB staff
and customers, and all issues identified as critical during the
release process.
The data prior to 2012 is noisier and less useful to draw
meaningful conclusions from. Those were the early days
of mobile software, and there were relatively few mobile-
specific developers working on the code base. In 2012, FB’s
CEO announced the company’s “Mobile First!” strategy to
the public at the TechCrunch Disrupt conference in San
Francisco. From that point on, the mobile development
team grew significantly, and the data collected became more
robust and meaningful. Hence, most of the data we present
is for that period of time.
In summary, this paper makes the following specific con-
tributions:
1. We are, to the best of our knowledge, the first to de-
scribe a process by which mobile software can be de-
ployed in as continuous a fashion as possible. Specifi-
cally, we describe the process used at FB.
2. This is the first time FB’s testing strategy for mobile
code is described and evaluated.
3. We present a full analysis with respect to productiv-
ity and software quality based on data collected over
a 7 year period as it relates to mobile software. In
particular, we are able to show that fast release cy-
cles do not negatively affect developer productivity or
software quality.
13