feat: add weights build and push commands (#2683)
* feat(ociartifact): add weights artifact builder using go-containerregistry
Implements WeightsArtifactBuilder for creating OCI artifacts containing model
weights. The builder:
- Creates OCI-format manifests (application/vnd.oci.image.manifest.v1+json)
- Sets artifactType to application/vnd.cog.weights.v1
- Adds layers with custom media types and annotations for weight files
- Uses mutate.Addendum to properly attach layer annotations
Layer annotations include:
- vnd.cog.weights.name: original filename
- vnd.cog.weights.dest: container path
- vnd.cog.weights.source: origin URL (optional)
- vnd.cog.weights.digest.original: uncompressed digest
- vnd.cog.weights.size.uncompressed: uncompressed size
Test coverage: 93.8%
* feat(ociartifact): add AddLayersFromLock for file-based weight loading
* feat(model): add OCI index detection helpers in resolver
Add helper functions to detect and handle OCI indexes:
- isOCIIndex: checks if ManifestResult is an OCI Image Index
- findWeightsManifest: finds weights manifest by annotation
- findImageManifest: finds model image manifest with platform filtering
Also:
- Add Annotations field to PlatformManifest struct
- Populate annotations when reading OCI index manifests
- Add PushImage/PushIndex to registry Client interface for OCI index push support
* feat(model): consolidate OCI index builders and wire push flow
- Move WeightsArtifactBuilder, IndexBuilder, IndexFactory from pkg/ociartifact to pkg/model/index_factory.go
- Delete pkg/ociartifact/ package (code duplication eliminated)
- Add OCI index push flow in pkg/cli/push.go (COG_OCI_INDEX=1 env var)
- Add index detection helpers in resolver for loading OCI indexes
- Simplify BuildOptions: remove WeightsLockPath (hardcoded to weights.lock)
- Add Platform.Variant field to pkg/model/image.go
- Add push_test.go placeholder for registry push tests
* refactor(weights): remove Source field and use filePaths map for weight resolution
- Remove Source field from WeightFile - lockfile maps name→blob, not source
- Change Name semantics to identifier/handle (e.g., 'model-v1') not filename
- Add filePaths map[string]string parameter to AddLayersFromLock for file location
- Add context cancellation support to AddLayersFromLock for long operations
- Add AnnotationValueWeights constant for 'weights' annotation value
- Improve registry push error messages to include reference
- Remove binary test files (~109MB) that were accidentally committed
- Update all tests to use new API and identifier-style names
* feat(config): add WeightSource type and Weights field to Config
* feat(config): add weights section to cog.yaml JSON schema
* feat(model): add WeightsLockGenerator for processing weight files
* feat(cli): add 'cog weights build' command
* feat(cli): add 'cog weights push' command
* test(integration): add weights_build test
* refactor(cli): push weight blobs directly instead of as OCI artifact
Change 'cog weights push' to push weight files as individual blobs
using WriteLayer instead of building and pushing an OCI artifact.
Layers are pushed concurrently for better performance.
- Add WriteLayer method to registry.Client interface
- Implement concurrent blob uploads with progress reporting
- Update mock clients to implement WriteLayer
* feat: rename / slight refactor to weights-lock-gen -> weights-gen; add weights pushing and progress
Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
* chore: cleanup
Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
* feat: add retry logic
Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
* chore: clean
Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
* test(registry): remove string-based error checks from retry tests
Use typed errors (io.EOF, syscall.EPIPE, etc.) instead of string
matching for more robust error detection tests.
---------
Signed-off-by: Mark Phelps <mphelps@cloudflare.com>
Co-authored-by: Michael Dwan <mdwan@cloudflare.com>
