This small benchmark is intended to compare how different combinations of HTTP Client and HTTP Server will affect the binary size of applications.
The base application we will be covering here has some basic requirements that are based on needs that we have for our real-world app. The app also have some usage of OpenSSL, which is also a requirement of our app, as we need to handle RSA signature validation, so if any implementation make use of code defined there will have a valid advantage. This are the requirement for all binaries implementation:
- A running server to respond to requests from a local client;
- A client to make local requests to the server;
- A second client to make requests to a remote server;
- Clients needs to handle:
- JSON serialization and deserialization;
- HTTP Header extraction;
- Share state between the Remote Client and the App;
- Asynchronous type signature;
The RULES.md file describe how implementations should be made.
| Implementation | Server | Client |
|---|---|---|
| dummy | - | - |
| actix_full | actix-web | awc |
| actix_reqwest | actix-web | reqwest |
| gotham_reqwest | gotham | reqwest |
| hyper_full | hyper | hyper |
| hyper_reqwest | hyper | reqwest |
| warp_surf | warp | surf |
| tide_surf | tide | surf |
| warp_reqwest | warp | reqwest |
The total binary size and, mainly, the distribution of the size contribution by crate is generated using cargo bloat. Although not 100% precise, we believe it's estimations are reliable enough for the data we've collected and analysed.
For all contestants in this benchmark we have collected information with different build options. All builds are generated with cargo's release build plus a combination of the following flags:
| Parameter | Values |
|---|---|
| Optimization | [0, 1, 2, 3, s, z] |
| LTO | [thin, fat] |
| Codegen Units | [1, 16] |
The main lessons we've learned from checking the results of our benchmark can be summarised in three points:
- Don't be afraid of using micro-dependencies;
- Set some extra optimization flags;
- The most suitable pair for us is tide-surf;
Here on the repository README.md we will be presenting a feel graphs in order to illustrate our analysis; the full data, as well as some more interactive graphs, are available at this shinyapp for anyone who wants to have a more complete view of it.
It's easy to see how having multiple dependencies can largely affect build time of our projects.
But when it comes to binary size, the data we collected tells us that there is little to worry there.
Take for example the following plot for the major crates on our larger contestant, actix_full:
Figure: actix_full with top crates highlighted (Interactive graph).
Even after optimization, the plot shows that only 5 of it's total 76 crates accounts for over 50% of it's binary size.
The following box plot show the distribution of all crates size across all projects. This shows that the vast majority of dependencies contribute very little to the total binary size.
Figure: Crates size distribution (Interactive graph).
Therefore we find it much more meaningful to choose carefully the core dependency for or apps, then it is to trying to minimise the total number of dependencies.
Binary Size optimization is not the default goal that the rust compiler aims to achieve,
but it can get there if the developer requests it.
The following plot shows how different optimization options affects the Binary Size on our smaller contestant, tide_surf.
Figure: Optimization parameters on tide_surf (Interactive graph).
Cargo's default release build uses Optimization Level 3, LTO Thin and 16 Codegen Units. Comparing that to the smaller binary we've generated we can see a improvement of around 40%. All other projects have shown better results when using Optimization Z, LTO Fat and 1 Codegen Unit, so that will be our default flags for smaller binary size.
The async-std based pair, tide-surf,
is our new choice of HTTP client-server pair for embedded applications.
Despite having a high level API, it still had better results that more lower level APIs such as hyper_full.
If we were to choose a pair that's built fully on top of tokio, another of the main runtimes for rust's async ecosystem, we will definitely be considering warp. The crate has presented a small Binary Size overhead over hyper for a much more simpler API for building HTTP servers.
Figure: All projects size (Interactive graph)
New pairs of applications are welcome provided they meet the rules specified to compete.
Licensed under Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0).