[turbopack] Add `clusters` to chunking heuristics (#95157)
The goal of this PR increase the expected benefit of merging two chunks
when their overlapping chunk groups are in a "cluster" (a cluster is a
set of routes that are commonly visited together). This is because in
these cases the Z+Z case (one of six in the N=2 set of cases) is more
likely than others. To this I have increased the probability of the Z+Z
case.
At the moment, to do this I've done the following to the probabilities:
```rust
/*
MERGED CASE (N = 2):
case X + X (p = (a_rem/groups) * ((a_rem - 1)/rem_g)): size = a_size, requests = 1
case Y + Y (p = (b_rem/groups) * ((b_rem - 1)/rem_g)): size = b_size, requests = 1
case Z + Z (p = (o_groups/groups) * (o_groups - 1)/rem_g): size = (a_size + b_size), requests = 1
case X + Y (p = (a_rem/groups) * (b_rem/rem_g) + (b_rem/groups) * (a_rem/rem_g)): size = a_size + b_size, requests = 2
case X + Z (p = (a_rem/groups) * (o_groups/rem_g) + (o_groups/groups) * (a_rem/rem_g)): size = a_size + (a_size + b_size), requests = 2
case Y + Z (p = (b_rem/groups) * (o_groups/rem_g) + (o_groups/groups) * (b_rem/rem_g)): size = b_size + (a_size + b_size), requests = 2
Request count is different in this case: Z + Z (better)
Requests size is different (worse) in these cases: X + Z, Y + Z
Each cost / benefit is weighted by the probabilities above. There are cases when
we know that Z + Z is more likely due to common user behaviour. This is based on
the "cluster" chunking heuristic we provide. We increase P(Z + Z) when two or more
routes in a cluster request both chunk items together (ie. request Z). P(X + Z) and
P(Y + Z) are therefore less likely. To increase P(Z + Z) while maintaining a total
probability of 1, we do the following (when chunk items overlap in a cluster):
P'(X + Z) = (1/2) * P(X + Z)
P'(Y + Z) = (1/2) * P(Y + Z)
P'(Z + Z) = P(Z + Z) + (1/2) * P(X + Z) + (1/2) * P(Y + Z)
Otherwise:
P'(X + Z) = P(X + Z)
P'(Y + Z) = P(Y + Z)
P'(Z + Z) = P(Z + Z)
*/
```
This increases `P'(Z + Z)` while reducing `P'(X + Z)` and `P'(Y + Z)`.
It is a bit of a blunt tool for doing this, however. So if anyone has
other suggestions for how to change this probabilities while maintaining
existing behaviour - let me know!
