Add build tips and tricks #364
Conversation
Signed-off-by: krishjainx <[email protected]>
|
|
||
| #### Current Situation | ||
|
|
||
| A full Flatcar build, starting from a clean environment and ending with a bootable image, takes around 2 hours on a modern build environment. On a state-of-the-art setup, this can be reduced to 1.5 hours. |
There was a problem hiding this comment.
More information and testing is required here, I think this time got decreased alot with the latest updates.
There was a problem hiding this comment.
Makes sense, I've converted this to a draft. I will update it as I or someone else tests, but it's good to have in this PR. This is particularly relevant to my personal project, so I'll update it as and when needed.
There was a problem hiding this comment.
https://github.com/flatcar/scripts/actions/runs/11147469386/job/31018310994?pr=2355 -> the last build took around 1h30mins to do build_packages(1h) + build_image (30m). Note that this job rebuilt the kernel.
|
|
||
| #### Ideal Future Situation | ||
|
|
||
| This document outlines the best ways to achieve the fastest builds possible in terms of hardware, software, and workflow. The Flatcar team is also considering instrumenting the build process to provide more detailed telemetry, using tools like `otel-cli`. |
There was a problem hiding this comment.
I would leave any future developments outside of the docs, as those do not bring any benefit to the user.
|
|
||
| ## Building Flatcar Efficiently | ||
|
|
||
| Flatcar is a Gentoo derivative, source-based operating system. Its immutability means that all packages must be built to create a production image. Here's how to achieve the fastest and most energy-efficient builds: |
There was a problem hiding this comment.
I would add the link to the official docs on how to build from source, as this document is an extension of it.
| Clone the repository onto the RAMDISK for best speedups. | ||
| 4. **Run the SDK Container**: | ||
| ```bash | ||
| ./run_sdk_container -t |
There was a problem hiding this comment.
add here the -U and document why
There was a problem hiding this comment.
Also, to speed up things, one can use the smaller image download for just one arch:
./run_sdk_container -t -U -a arm64
There was a problem hiding this comment.
./run_sdk_container -t -U -a arm64 - 6.58GB downloaded instead of 10GB for the full one.
|
|
||
| - **Binary Packages**: Use `-U` with `./run_sdk_container` if you want to use binary packages, though full builds generally rebuild everything. | ||
| - **Tmpfs**: Using Tmpfs can provide additional speedups. However, in some cases, using NVMe might not differ significantly. | ||
| - **Parallel Fetch**: Enable `parallel-fetch` in Portage to download files in the background during the build. |
| - **Binary Packages**: Use `-U` with `./run_sdk_container` if you want to use binary packages, though full builds generally rebuild everything. | ||
| - **Tmpfs**: Using Tmpfs can provide additional speedups. However, in some cases, using NVMe might not differ significantly. | ||
| - **Parallel Fetch**: Enable `parallel-fetch` in Portage to download files in the background during the build. | ||
| - **Rust/Go Module Download Bottleneck**: A significant bottleneck is the sequential download of Rust crates and Go modules. Parallel downloads or HTTP connection reuse could help speed this up. |
There was a problem hiding this comment.
ditto future dev work - remove Parallel downloads or HTTP connection reuse could help speed this up. Comment
| ``` | ||
| This command fetches all required packages. In one test, it took 24 minutes to fetch all 433 Rust crates. | ||
|
|
||
| **Discussion and Further Insights**: For ongoing discussions and further insights on optimizing Flatcar builds, refer to the [GitHub issue #1550](https://github.com/flatcar/Flatcar/issues/1550). |
There was a problem hiding this comment.
remove this please, as the issue might be closed in some time.
|
|
||
| #### Optimization Insights | ||
|
|
||
| - **Instrumenting Build Timeline**: Using tools like `otel-cli` can provide visibility into build delays. |
There was a problem hiding this comment.
future work should not be in the docs
| #### Example Builds | ||
|
|
||
| - **Equinix Metal**: A suitable machine can reduce build time to within 1 hour. Ideal setups include modern development environments like the m3 or a3 systems. | ||
| - **Build Log Insights**: A recent test build on a 24-core i9-13900HX laptop took less than 45 minutes for a full package build. Load is distributed unevenly, with some bottlenecks in dependency installation. Reordering builds could help maximize resource usage. |
There was a problem hiding this comment.
Hmm, this is rather a usage of the binary cached already built packages, and not a full or rather close-to full source build which contradicts the first 1.5h - 2h part, we need a clearer workflow on how to reach this time and what optimizations were used.
| - **Equinix Metal**: A suitable machine can reduce build time to within 1 hour. Ideal setups include modern development environments like the m3 or a3 systems. | ||
| - **Build Log Insights**: A recent test build on a 24-core i9-13900HX laptop took less than 45 minutes for a full package build. Load is distributed unevenly, with some bottlenecks in dependency installation. Reordering builds could help maximize resource usage. | ||
|
|
||
| For detailed build statistics, consider using `genlop` to track build times and dependencies. |
There was a problem hiding this comment.
future dev - this part is not documented, are there any outputs/screenshots of this tool that might help?
This PR adds a new document titled "Build Tips and Tricks" to the Flatcar website. The document provides various tips and tricks to optimize the build process for Flatcar. It includes sections on using binary packages, leveraging Tmpfs, enabling parallel fetch in Portage, and addressing bottlenecks in Rust and Go module downloads.
I spoke with @chewi (James Le Cuirot), who suggested that I should make this PR.