[whispersystems] TextSecure build times with gradle

1 minute 25 seconds is fast! Your machine must be way better than mine.

What you're seeing is endemic to Android development, not just TS.

- moxie

Post by Peter Goetz
Hi,
Is there anything I can do to speed up build+deploy times for TextSecure?
It currently takes 1min 25s on my machine to build and deploy. Build
alone takes 1min 12s (using gradle). I checked all the different options
in the gradle build settings, but couldn't really identify how I could
significantly improve it.
For an efficient workflow this is certainly too slow. How did you guys
solve this problem?
Thanks,
Peter
Task Duration Result
: 52.149s (total)
:dexDebug 28.186s
:proguardDebug 17.663s
:compileDebugJava 3.671s
:packageDebug 2.083s
Dependencies Duration
All dependencies 17.512s
:_debugCompile 4.060s
:_releaseCompile 3.294s
:_automationCompile 2.902s
:_debugApk 1.587s
:_automationApk 1.356s
:_devApk 1.247s
:_releaseApk 0.961s
:_devCompile 0.960s
:compile 0.960s
:classpath 0.147s
:_automationTestCompile 0.030s
:_automationTestApk 0.008s
:releaseWearApp 0s
:wearApp 0s
classpath 0s

--
http://www.thoughtcrime.org

johanw

2015-07-18 11:12:17 UTC

Post by Moxie Marlinspike
1 minute 25 seconds is fast! Your machine must be way better than mine.

I'm at build times around 2:50 for a TextSecure debug version. A signed
APK is even much slower (i5 with 6GB memory, no SSD).

Post by Moxie Marlinspike
What you're seeing is endemic to Android development, not just TS.

Not even that, server side Java development in Eclipse suffers from the
same fate. The fact that Android Studio and Eclipse try to build the
entire project after each small change does not help too.

Considering the amount of disk access and memory consumption I notice
during builds my first guess would be that a SSD would help reduce build
times.

--
With kind regards,

Johan Wevers

agrajaghh

2015-07-18 11:46:29 UTC

Post by johanw
Considering the amount of disk access and memory consumption I notice
during builds my first guess would be that a SSD would help reduce
build times.

I can confirm that, my build times significantly decreased once I
installed an SSD. "gradle assembleDebug" is around 1:45 for me now after
"gradle clean" on my old PC (Phenom II X4 955 and 4GB DDR2). Without the
SSD it was ~3 minutes...

But If I do "Build -> Clean Project" in Android Studio and then "Run" it
just takes 1:30, don't know why its faster there...

Peter Goetz

2015-07-19 14:00:31 UTC

Hm. But that seems crazy. I'm definitely not an expert in Android
programming, so for me there is a lot of trial and error. And that would
mean effectively that I spend ~75% of my time waiting for the the builds.
Do you really wait 2-3 minutes every time?

I do remember when I did some Android programming a couple of years ago
with Eclipse, the situation was a lot better. Eclipse's incremental
compiler would make sure that the waiting time was reduced to 20-30s, which
is kind of acceptable. I'm struggling to see why the push to Android Studio
was so successful, when the situation seems so unpleasant.

It's a real shame, but with these prospectives I'm not sure I'm up for it.
I'm used to code- build-deploy-cycles of 10s.

Thanks for your responses, guys. Appreciated it,
Peter

Post by agrajaghh

Post by johanw
Considering the amount of disk access and memory consumption I notice
during builds my first guess would be that a SSD would help reduce
build times.

I can confirm that, my build times significantly decreased once I
installed an SSD. "gradle assembleDebug" is around 1:45 for me now after
"gradle clean" on my old PC (Phenom II X4 955 and 4GB DDR2). Without the
SSD it was ~3 minutes...
But If I do "Build -> Clean Project" in Android Studio and then "Run" it
just takes 1:30, don't know why its faster there...

Johan Wevers

2015-07-19 15:13:48 UTC

Post by Peter Goetz
Hm. But that seems crazy. I'm definitely not an expert in Android
programming, so for me there is a lot of trial and error. And that would
mean effectively that I spend ~75% of my time waiting for the the
builds. Do you really wait 2-3 minutes every time?

I don't rebuild each time I've changed a single line of code when I
expect to change more.

--
Met vriendelijke groet,

Johan Wevers

Peter Goetz

2015-07-19 16:11:35 UTC

Post by Johan Wevers
I don't rebuild each time I've changed a single line of code when I
expect to change more.

I guess that can work. But I'm rusty with the Android API (that's actually
an understatement) and it was also extended a lot since I worked last with
it. So I *can't* really change several things at once, because I'll have to
try out a lot of small things. I want quick feedback loops.

And just finding out which line to comment out to get straight to the
MediaOverviewActivity without getting the register activity on the emulator
was a super painful process. (BTW, the emulator actually works quite well
and didn't hurt that process at all).

I don't rebuild each time I've changed a single line of code when I
expect to change more.
--
Met vriendelijke groet,
Johan Wevers

Peter Goetz

2015-07-19 22:38:32 UTC

Hi Joey,

I run my builds in the same way you do. Could you run one of your
incremental builds with the --profile option and post the breakdown here?
I'm curious how long the individual steps take in your setup. Also, how did
you enable incremental builds? I can't find any option for this and also
couldn't find anything on the Internet about it. If I see it correctly
TextSecure's default build settings don't do any magic here by default.

Also, what hardware do you have?

Thanks,
Peter

Hi Peter,
When I compile Android apps for running on my devices or the emulator to
test a change in code, I use the green arrow, highlighted in the screenshot
below. After the first build, it does incremental builds that only take
~5-10 seconds.
The builds are different from final builds that I published in Google
Play. Two main differences between the incremental builds and the official
builds that I publish are the incremental builds not signed with my keys
and are not run through proguard. Not running through proguard seems to
significantly reduce the compile time. The builds that I publish take
around 60-90 secs to fully compile, with signed keys and proguard.
I'm running a fairly default configuration of Android Studio as I haven't
made changes to compilation process, that I recall. Hope that helps and let
me know if you have any other questions.
[image: image.png]

Post by Johan Wevers
I don't rebuild each time I've changed a single line of code when I
expect to change more.

I guess that can work. But I'm rusty with the Android API (that's
actually an understatement) and it was also extended a lot since I worked
last with it. So I *can't* really change several things at once, because
I'll have to try out a lot of small things. I want quick feedback loops.
And just finding out which line to comment out to get straight to the
MediaOverviewActivity without getting the register activity on the emulator
was a super painful process. (BTW, the emulator actually works quite well
and didn't hurt that process at all).

Post by Peter Goetz
Hm. But that seems crazy. I'm definitely not an expert in Android
programming, so for me there is a lot of trial and error. And that

would

Post by Peter Goetz
mean effectively that I spend ~75% of my time waiting for the the
builds. Do you really wait 2-3 minutes every time?

I don't rebuild each time I've changed a single line of code when I
expect to change more.
--
Met vriendelijke groet,
Johan Wevers

Peter Goetz

2015-07-20 18:37:36 UTC

Sharing with list again.

Thanks for the exhaustive data, Joey. I think your hardware explains it: I
have a i5-3320M and no SSD. I think that could explain your 37s build time
versus my 52s (or about 1.5 times as fast).
I'll try the other option (of removing proguard and using multidexing)
too, to see if it makes my setup faster or slower.

Hi Peter,
As a caveat, I'm far from an expert on the newer gradle building system,
but hopefully the details below provide some more context. Let me know if
there are any other tests I should run and provide feedback to you.
Short answer: With TextSecure, the functions of dexDebug and
proguardDebug or packageAllDebugClassesForMultiDex appear to consume the
most time in the compile process, which I haven't seen in my other apps.
Although, these issues, such as the 64k method limit per dex file, appear
to be common for larger and more complex apps.
I'm not sure how best to tweak the build system for TextSecure to reduce
compile times. I'm not even sure if shorter compile times are possible
without faster hardware. Maybe others more familiar with the gradle build
systems and the issues with method limits of the dex files can share how
best to optimize?
For the below benchmarks, I'm using the command, ./gradlew --profile
assembleDebug. When I referred to incremental builds, I was referring to
the "UP-TO-DATE" references while compiling.
Using TextSecure version 2.23.0 from 7/17, without any changes, after
running a "clean project" command, and compiling the debug version, I see
consistent build times of: *~55 seconds*
Task Duration Result
:36.669s(total)
:dexDebug14.036s
:proguardDebug14.024s
:compileDebugJava6.072s
:packageDebug1.720s
Dependencies Duration
All dependencies16.540s
:_devCompile3.083s
:_automationApk2.109s
:_releaseApk1.991s
:_automationCompile1.548s
:_debugCompile1.515s
:_releaseCompile1.508s
:_devApk1.501s
Removing proguard (debug { minifyEnabled false}), leads to compile errors
with dexDebug. Removing proguard and enabling multiDex (multiDexEnabled =
true), which as a good write up and reference links here -
http://stackoverflow.com/a/26196397/1652110, shows consistent build
times of: *~65 seconds*
Task Duration Result
:45.946s(total)
:dexDebug24.908s
:packageAllDebugClassesForMultiDex7.744s
:compileDebugJava5.680s
:shrinkDebugMultiDexComponents3.239s
:packageDebug2.192s
Dependencies Duration
All dependencies15.891s
:_devCompile2.480s
:compile2.044s
:_debugCompile1.722s
:_releaseCompile1.601s
:_releaseApk1.567s
:_automationApk1.555s
:_automationCompile1.523s
:_devApk1.508s
Building a different app, which has many fewer dependencies and I was
referencing in my email to you, I see compile times around *~10 seconds*.
Task Duration Result
:app2.699s(total)
:app:packageDebug1.190s
:app:processDebugResources0.723s
:app:mergeDebugResources0.184sUP-TO-DATE
:app:crashlyticsGenerateResourcesDebug0.125s
:app:prepareComAndroidSupportAppcompatV72220Library0.116sUP-TO-DATE
:app:compileDebugJava0.078sUP-TO-DATE
:app:zipalignDebug0.044s
:app:dexDebug0.043sUP-TO-DATE
Dependencies Duration
All dependencies1.118s
:classpath0.728s
The desktop system I'm testing on is using an SSD and an i7-4770K, not
over-clocked.
Hope all those details are helpful!
Joey
joeykrim.com

Post by Peter Goetz
Hi Joey,
I run my builds in the same way you do. Could you run one of your
incremental builds with the --profile option and post the breakdown here?
I'm curious how long the individual steps take in your setup. Also, how did
you enable incremental builds? I can't find any option for this and also
couldn't find anything on the Internet about it. If I see it correctly
TextSecure's default build settings don't do any magic here by default.
Also, what hardware do you have?
Thanks,
Peter

Hi Peter,
When I compile Android apps for running on my devices or the emulator
to test a change in code, I use the green arrow, highlighted in the
screenshot below. After the first build, it does incremental builds that
only take ~5-10 seconds.
The builds are different from final builds that I published in Google
Play. Two main differences between the incremental builds and the official
builds that I publish are the incremental builds not signed with my keys
and are not run through proguard. Not running through proguard seems to
significantly reduce the compile time. The builds that I publish take
around 60-90 secs to fully compile, with signed keys and proguard.
I'm running a fairly default configuration of Android Studio as I
haven't made changes to compilation process, that I recall. Hope that helps
and let me know if you have any other questions.
[image: image.png]

Post by Johan Wevers
I don't rebuild each time I've changed a single line of code when I
expect to change more.

I guess that can work. But I'm rusty with the Android API (that's
actually an understatement) and it was also extended a lot since I worked
last with it. So I *can't* really change several things at once, because
I'll have to try out a lot of small things. I want quick feedback loops.
And just finding out which line to comment out to get straight to the
MediaOverviewActivity without getting the register activity on the emulator
was a super painful process. (BTW, the emulator actually works quite well
and didn't hurt that process at all).

Post by Peter Goetz
Hm. But that seems crazy. I'm definitely not an expert in Android
programming, so for me there is a lot of trial and error. And that

would

Post by Peter Goetz
mean effectively that I spend ~75% of my time waiting for the the
builds. Do you really wait 2-3 minutes every time?

I don't rebuild each time I've changed a single line of code when I
expect to change more.
--
Met vriendelijke groet,
Johan Wevers

Peter Goetz

2015-07-20 18:40:12 UTC

I've already used the daemon and that didn't help significantly, but I've
now enabled the offline mode and that did improve the situation
significantly: Overall build time down at 54s.

Thanks!

I'm not a Java buff, but it looks like there are some recent changes to
Gradle that have sped up some projects. Specifically, the options about
daemonizing and unchecking the update checker seem promising. Good luck!
http://stackoverflow.com/questions/16775197/building-and-running-app-via-gradle-and-android-studio-is-slower-than-via-eclips
Jeff
------------------------------
Sent: â7/â20/â2015 4:09
Subject: Re: [whispersystems] TextSecure build times with gradle
Hi Joey,
I run my builds in the same way you do. Could you run one of your
incremental builds with the --profile option and post the breakdown here?
I'm curious how long the individual steps take in your setup. Also, how did
you enable incremental builds? I can't find any option for this and also
couldn't find anything on the Internet about it. If I see it correctly
TextSecure's default build settings don't do any magic here by default.
Also, what hardware do you have?
Thanks,
Peter

Post by Johan Wevers
I don't rebuild each time I've changed a single line of code when I
expect to change more.

I guess that can work. But I'm rusty with the Android API (that's
actually an understatement) and it was also extended a lot since I worked
last with it. So I *can't* really change several things at once, because
I'll have to try out a lot of small things. I want quick feedback loops.
And just finding out which line to comment out to get straight to the
MediaOverviewActivity without getting the register activity on the emulator
was a super painful process. (BTW, the emulator actually works quite well
and didn't hurt that process at all).

Post by Peter Goetz
Hm. But that seems crazy. I'm definitely not an expert in Android
programming, so for me there is a lot of trial and error. And that

would

Post by Peter Goetz
mean effectively that I spend ~75% of my time waiting for the the
builds. Do you really wait 2-3 minutes every time?

I don't rebuild each time I've changed a single line of code when I
expect to change more.
--
Met vriendelijke groet,
Johan Wevers

Jake McGinty

2015-08-06 20:07:23 UTC

When iterating quickly, I have a âshortcutâ buildType and productFlavor that disables proguard, enables multidex, and sets the minSdkVersion to 21. This cuts my build times down to ~15s, but of course to test the final version you need to do a full proguarded build.

I've already used the daemon and that didn't help significantly, but I've now enabled the offline mode and that did improve the situation significantly: Overall build time down at 54s.
Thanks!
I'm not a Java buff, but it looks like there are some recent changes to Gradle that have sped up some projects. Specifically, the options about daemonizing and unchecking the update checker seem promising. Good luck!
http://stackoverflow.com/questions/16775197/building-and-running-app-via-gradle-and-android-studio-is-slower-than-via-eclips <http://stackoverflow.com/questions/16775197/building-and-running-app-via-gradle-and-android-studio-is-slower-than-via-eclips>
Jeff
Sent: â7/â20/â2015 4:09
Subject: Re: [whispersystems] TextSecure build times with gradle
Hi Joey,
I run my builds in the same way you do. Could you run one of your incremental builds with the --profile option and post the breakdown here? I'm curious how long the individual steps take in your setup. Also, how did you enable incremental builds? I can't find any option for this and also couldn't find anything on the Internet about it. If I see it correctly TextSecure's default build settings don't do any magic here by default.
Also, what hardware do you have?
Thanks,
Peter
Hi Peter,
When I compile Android apps for running on my devices or the emulator to test a change in code, I use the green arrow, highlighted in the screenshot below. After the first build, it does incremental builds that only take ~5-10 seconds.
The builds are different from final builds that I published in Google Play. Two main differences between the incremental builds and the official builds that I publish are the incremental builds not signed with my keys and are not run through proguard. Not running through proguard seems to significantly reduce the compile time. The builds that I publish take around 60-90 secs to fully compile, with signed keys and proguard.
I'm running a fairly default configuration of Android Studio as I haven't made changes to compilation process, that I recall. Hope that helps and let me know if you have any other questions.
<image.png>

Post by Johan Wevers
I don't rebuild each time I've changed a single line of code when I
expect to change more.

I guess that can work. But I'm rusty with the Android API (that's actually an understatement) and it was also extended a lot since I worked last with it. So I *can't* really change several things at once, because I'll have to try out a lot of small things. I want quick feedback loops.
And just finding out which line to comment out to get straight to the MediaOverviewActivity without getting the register activity on the emulator was a super painful process. (BTW, the emulator actually works quite well and didn't hurt that process at all).

Post by Johan Wevers
Hm. But that seems crazy. I'm definitely not an expert in Android
programming, so for me there is a lot of trial and error. And that would
mean effectively that I spend ~75% of my time waiting for the the
builds. Do you really wait 2-3 minutes every time?

I don't rebuild each time I've changed a single line of code when I
expect to change more.
--
Met vriendelijke groet,
Johan Wevers

Eric Hartsuyker

2015-08-07 07:59:04 UTC

Could you share this shortcut with those of us that are less-than-apt with
gradle?

-E

Post by Jake McGinty
When iterating quickly, I have a âshortcutâ buildType and productFlavor
that disables proguard, enables multidex, and sets the minSdkVersion to 21.
This cuts my build times down to ~15s, but of course to test the final
version you need to do a full proguarded build.
I've already used the daemon and that didn't help significantly, but I've
now enabled the offline mode and that did improve the situation
significantly: Overall build time down at 54s.
Thanks!

Hi Peter,
When I compile Android apps for running on my devices or the emulator to
test a change in code, I use the green arrow, highlighted in the screenshot
below. After the first build, it does incremental builds that only take
~5-10 seconds.
The builds are different from final builds that I published in Google
Play. Two main differences between the incremental builds and the official
builds that I publish are the incremental builds not signed with my keys
and are not run through proguard. Not running through proguard seems to
significantly reduce the compile time. The builds that I publish take
around 60-90 secs to fully compile, with signed keys and proguard.
I'm running a fairly default configuration of Android Studio as I
haven't made changes to compilation process, that I recall. Hope that helps
and let me know if you have any other questions.
<image.png>

Post by Johan Wevers
I don't rebuild each time I've changed a single line of code when I
expect to change more.

I guess that can work. But I'm rusty with the Android API (that's
actually an understatement) and it was also extended a lot since I worked
last with it. So I *can't* really change several things at once, because
I'll have to try out a lot of small things. I want quick feedback loops.
And just finding out which line to comment out to get straight to the
MediaOverviewActivity without getting the register activity on the emulator
was a super painful process. (BTW, the emulator actually works quite well
and didn't hurt that process at all).

Post by Peter Goetz
Hm. But that seems crazy. I'm definitely not an expert in Android
programming, so for me there is a lot of trial and error. And that

would

Post by Peter Goetz
mean effectively that I spend ~75% of my time waiting for the the
builds. Do you really wait 2-3 minutes every time?

I don't rebuild each time I've changed a single line of code when I
expect to change more.
--
Met vriendelijke groet,
Johan Wevers

johnny unitas

2015-08-07 14:58:11 UTC

Yes, please do. I was trying to work on a bug fix yesterday on my work
computer and one of my build times was 30 minutes. Damn you, resource
hungry work tasks.

:-X

Johnny

Post by Eric Hartsuyker
Could you share this shortcut with those of us that are less-than-apt with
gradle?
-E

When iterating quickly, I have a “shortcut” buildType and
productFlavor
that disables proguard, enables multidex, and sets the minSdkVersion to 21.
This cuts my build times down to ~15s, but of course to test the final
version you need to do a full proguarded build.
I've already used the daemon and that didn't help significantly, but I've
now enabled the offline mode and that did improve the situation
significantly: Overall build time down at 54s.
Thanks!

I'm not a Java buff, but it looks like there are some recent changes to
Gradle that have sped up some projects. Specifically, the options about
daemonizing and unchecking the update checker seem promising. Good luck!
http://stackoverflow.com/questions/16775197/building-and-running-app-via-gradle-and-android-studio-is-slower-than-via-eclips
Jeff
------------------------------
Sent: ‎7/‎20/‎2015 4:09
Subject: Re: [whispersystems] TextSecure build times with gradle
Hi Joey,
I run my builds in the same way you do. Could you run one of your
incremental builds with the --profile option and post the breakdown here?
I'm curious how long the individual steps take in your setup. Also, how did
you enable incremental builds? I can't find any option for this and also
couldn't find anything on the Internet about it. If I see it
correctly
TextSecure's default build settings don't do any magic here by default.
Also, what hardware do you have?
Thanks,
Peter

Hi Peter,
When I compile Android apps for running on my devices or the emulator to
test a change in code, I use the green arrow, highlighted in the screenshot
below. After the first build, it does incremental builds that only take
~5-10 seconds.
The builds are different from final builds that I published in Google
Play. Two main differences between the incremental builds and the official
builds that I publish are the incremental builds not signed with my keys
and are not run through proguard. Not running through proguard seems to
significantly reduce the compile time. The builds that I publish take
around 60-90 secs to fully compile, with signed keys and proguard.
I'm running a fairly default configuration of Android Studio as I
haven't made changes to compilation process, that I recall. Hope that helps
and let me know if you have any other questions.
<image.png>

Post by Johan Wevers
I don't rebuild each time I've changed a single line of code when I
expect to change more.

I guess that can work. But I'm rusty with the Android API (that's
actually an understatement) and it was also extended a lot since I worked
last with it. So I *can't* really change several things at once, because
I'll have to try out a lot of small things. I want quick feedback loops.
And just finding out which line to comment out to get straight to the
MediaOverviewActivity without getting the register activity on the emulator
was a super painful process. (BTW, the emulator actually works quite well
and didn't hurt that process at all).