turbo-persistence: don't shrink the shard count before the first bottom merge (#100424)
## What
Defer shrinking shard counts of turbo-persistence families until there
has been a bottom merge. This reduces the maximum sizes of initial
compaction jobs at the cost of slightly more SST files until that
happens. The benefit is more parallelism and smaller job sizes for
initial bottom runs.
## Why
A 'bottom' merge is when we have performed a 'full compaction' of a
single shard of a family (aka table). Families are divided into shards
to accelerate lookups and reduce the size of compaction jobs. Part of
the compaction strategy is to occasionally compress the full shard into
a 'bottom' a set of hot and cold files that do not overlap (well hot and
cold can overlap, but hots cannot overlap with hots and colds cannot
with colds).
Before a family has a 'bottom', its shard count can shrink. Commits
compute the count too, so after a small first commit (e.g. the first
snapshot in dev) the next commits already wrote files spanning all of
the family's initial shards. Such files tie every shard into one
component, so the first bottom merge of the family was a single job over
all its files. e.g. for next-site TaskData went from 8 to 4 shards and
TaskMeta from 4 to 1, doubling and quadrupling the job sizes.
Now the count only grows until the family has bottom runs. Shrinking
afterwards is free: the bottom runs of the former shards are disjoint,
so together they are the bottom run of the combined shard and nothing is
rewritten. The cost is a less than optimal shard count, and so more,
smaller commit files, until the first compaction that is triggered by
our normal triggers. The benefit is more parallelism in those initial
bottom run jobs
Also, an intermediate merge that happens takes all files of a shard
without a bottom run now writes it as the shard's bottom run, since it
rewrites just as much.