cog

d343eb6a - feat(test-harness): add parallel execution, better error reporting, and lint fixes (#2944)

feat(test-harness): add parallel execution, better error reporting, and lint fixes (#2944) * feat(test-harness): add parallel execution, better error reporting, and lint fixes Add --concurrency flag (default 4) to run model builds/tests in parallel using errgroup with a configurable semaphore. All three commands (run, build, schema-compare) support parallel execution with: - Pre-allocated result slices to preserve manifest ordering - Per-model progress lines under a mutex - Quiet mode that captures build output instead of streaming to avoid interleaved output in parallel mode - Per-model isolated work directories so models sharing a repo don't clobber each other's setup commands Improve error reporting: - Final error message now includes per-model failure details (error message or failing test names) instead of just listing model names - Setup commands run with 'set -euo pipefail' so missing tools (e.g. yq) are caught immediately instead of silently producing empty files - Validate cog.yaml is non-empty after setup to catch silent failures Fix all 30 golangci-lint issues across the test-harness: - copyloopvar: remove unnecessary Go <1.22 loop variable copies - errcheck: handle intentionally ignored errors - gocritic: use 0o octal literals, rewrite if-else chains as switch - gosec: annotate intentional HTTP calls to known APIs - misspell: fix British/American spelling - modernize: use strings.SplitSeq, maps.Copy, built-in min * fix: use bash instead of sh for setup commands (dash lacks pipefail) * feat(test-harness): add requires_tools manifest field and Replicate model manifest Add requires_tools field to manifest model schema so models can declare CLI tools needed for setup commands. The harness checks tools exist on PATH before running setup and errors with a clear message listing the missing tools. Improve setup error reporting: both quiet and non-quiet modes now capture stderr and include it in the error message so users see the actual failure instead of a downstream error from an empty cog.yaml. Add manifest-replicate.yaml with 15 Replicate production models. * feat(test-harness): stream logs with model name prefix in parallel mode Replace the quiet/buffered approach with a prefixWriter that prepends each output line with the model name, like docker-compose: [flux-schnell ] Step 3/12: RUN pip install ... [chatterbox ] Building image cog-harness-chatterbox:test [flux-schnell ] Successfully built abc123 Output streams in real-time even in parallel mode. Sequential mode (--concurrency 1) streams without prefixes as before. * feat(test-harness): add timing output between build and test steps Print === banners with timing info at each phase boundary so users can see where time is being spent: === Preparing flux-schnell... === Building flux-schnell (timeout 900s)... === Build complete (62.3s) === Predict test 1/1: generate image from prompt (timeout 900s)... === Predict test 1/1 PASSED (48.7s) In parallel mode these are prefixed with the model name like all other output. * feat: add --skip-schema-validation flag to cog build Expose the existing SkipSchemaValidation option as a CLI flag. This is useful when building models that fail schema validation due to import-time side effects (e.g. accessing GPU/weights files) or when using a remote Docker context where the legacy schema validation container cannot access local resources. Also add skip_schema_validation field to the test harness manifest so models can opt out of schema validation per-model. * chore: rm manifest-replicate Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * fix(test-harness): pass --setup-timeout to cog predict/train cog predict has a default --setup-timeout of 300s which kills the container if model setup (weight downloads) takes longer. Pass the model's timeout value as the setup timeout so large models have enough time to download weights during first run. * fix(test-harness): add pass/fail feedback in sequential mode The sequential code paths (--concurrency 1) were missing per-model status output after each build/run/compare, unlike the parallel paths. Now both modes print consistent pass/fail/skip feedback. * chore: regen cli docs Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * fix(test-harness): address review findings in parallel execution and runner - Replace errgroup.WithContext with WaitGroup.Go + semaphore channel; goroutines never returned errors so errgroup semantics were misleading and g.Wait() was silently discarded - Extract generic runModels[T] helper to deduplicate the parallel/sequential loop that was copy-pasted across build.go, run.go, and schema_compare.go - Use singleflight.Group in cloneRepo instead of holding a mutex during the entire git clone operation, which serialized all parallel clones - Capture cloneRepo stderr into a buffer instead of writing directly to os.Stderr, preventing interleaved output in parallel mode - Add --concurrency flag validation (reject values < 1 to prevent panic) - Split modelOutput into modelLoggers + modelOutput so RunModel/BuildModel don't allocate capture buffers they never use - Cap prefixWriter line buffer at 64 KiB to prevent unbounded growth from long lines without newlines (e.g. progress bars) - Truncate runSetupCommands error output to last 2000 bytes, matching the pattern already used in buildModelWithEnv * chore: update llm docs Signed-off-by: Mark Phelps <mphelps@cloudflare.com> * feat(test-harness): unified-diff schema comparison and quiet builds Improve schema-compare output so differences are immediately readable: - Replace terse one-line-per-path diff format with unified-diff style output showing actual JSON values with --- static / +++ runtime headers and @@ path @@ hunks (like git diff) - Sort diff keys alphabetically for deterministic output - Suppress docker build log streaming during schema comparison builds (quietBuildModelWithEnv) so build output no longer intermingles with the diff report; build logs are still captured for error messages - Add spacing around diff blocks in console report for readability - Add table-driven tests for the new jsonDiff format * feat(test-harness): classify schema diffs as expected vs real failures Schema comparison now distinguishes between genuine mismatches (real failures) and known limitations of static schema gen (expected diffs): Expected (informational, not failures): - Training schemas/paths present in static but absent in runtime, since runtime schema gen only handles predict mode - Descriptions present in runtime but absent in static, since static analysis can't resolve dynamically-constructed descriptions (e.g. f-strings with conditional logic in class methods) Models with only expected diffs now pass with an informational note instead of failing. Real structural mismatches (wrong type, wrong default, missing fields) still fail as before. * chore: hide skip schema validation Signed-off-by: Mark Phelps <mphelps@cloudflare.com> --------- Signed-off-by: Mark Phelps <mphelps@cloudflare.com>