Parallelize local base image compression #2022
Conversation
| progressEventDispatcher.newChildProducer(); | ||
|
|
||
| cachedLayers.add( | ||
| executorService.submit( |
There was a problem hiding this comment.
Ideally, the code here won't deal with concurrency and StepsRunner should be able to maximize concurrency by individually compressing and caching each layer. For example, right now it is possible that some registry base image layers are being pulled in while some base layers can be pushed simultaneously. (I am sure constructing a base image manifest can be delayed, since I believe the reason we need them at an early stage is to be able to download the layers from a registry.) Also this would be nice for the consistent async framework design where each step doesn't get an ExecutorService.
But I think this is OK for short-term simple improvement for a low-hanging fruit. We should continue to track for the ideal async handling.
There was a problem hiding this comment.
Yeah, I agree this should be iterated on to be more consistent/less self-contained. But I figured it'd be nice to at least get the small performance gain in without adding too much complexity at the moment.
There was a problem hiding this comment.
Yeah, I'm already getting a headache over all these. Sounds like a fun challenge!
| progressEventDispatcher.newChildProducer(); | ||
|
|
||
| cachedLayers.add( | ||
| executorService.submit( |
There was a problem hiding this comment.
Yeah, I'm already getting a headache over all these. Sounds like a fun challenge!
Towards #1913. This only really makes a difference for un-cached local base image layers, but it provides a decent speedup in that case. Here were the build times using maven + a clean cache on the master branch vs. these changes:
docker://gcr.io/distroless/javadocker://openjdk:8