feat: add model test harness for validating cog SDK releases (#2851)
* feat: add model test harness for validating cog SDK releases
Add a declarative test harness (tools/test-harness/) that automates
building and running cog models against new SDK versions. Designed
for testing cog-examples against RC releases but extensible to any
model in any repo.
Models and their expected inputs/outputs are defined in manifest.yaml.
The harness clones repos, patches cog.yaml with the target sdk_version,
runs cog build + cog predict, and validates outputs using pluggable
validators (exact match, contains, file_exists, json_match, etc.).
Includes all 10 cog-examples models in the manifest and fixture images
for blur/resnet tests.
* docs: remove internal reference from README
* Potential fix for pull request finding 'Empty except'
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Signed-off-by: Mark Phelps <209477+markphelps@users.noreply.github.com>
* feat: auto-resolve latest stable cog CLI and SDK versions in test harness
Default to downloading the latest stable cog CLI from GitHub releases
and resolving the latest stable SDK version from PyPI, skipping any
alpha/beta/rc tags. Both can be overridden via --cog-version and
--sdk-version CLI flags, or pinned in manifest.yaml defaults.
---------
Signed-off-by: Mark Phelps <209477+markphelps@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>