Support image descriptions using local AI model (#18475)
Resolves #16281
Summary of the issue:
NVDA currently lacks a built‑in, offline image captioning feature. Existing solutions require a reliable internet connection—raising privacy concerns, potential costs, and latency—and many NVDA users (especially in developing regions or on older hardware) have limited connectivity or constrained resources. There is no robust, integrated offline alternative.
Description of user facing changes:
Introduces device‑side image description directly within NVDA, requiring no cloud service.
Adds three global commands (with default shortcuts):
--NVDA+Windows+,--: Generate a caption for the current image under focus.
--NVDA+Windows+Shift+,--: Release the loaded model and free memory.
--NVDA+Windows+Ctrl+,--: Open the Model Manager GUI to download or manage models.
Extends NVDA’s settings panel to enable/disable offline captioning and configure model paths.
Description of developer facing changes:
New _localCaptioner module containing:
captioner.py: Core inference engine exposing generate_caption(image) for producing text descriptions.
panel.py: NVDA settings integration (lazy or on‑startup model loading, custom path).
modelDownloader.py: CLI tool to download ONNX models.
modelManager.py: GUI for selecting download paths and managing available models.
Uses the Hugging Face Xenova/vit-gpt2-image-captioning model in ONNX format (via onnxruntime) to balance accuracy, speed, and low resource usage.
Modular design allows for future extension to additional models or formats.
Description of development approach:
--Modular integration--: Keeps _localCaptioner self‑contained and compatible with NVDA’s plugin architecture.
--Lightweight inference--: Leverages ONNXRuntime for fast, local inference without heavy PyTorch or TensorFlow dependencies.
--Lazy loading--: Model is only loaded when first invoked (or at startup, if configured), minimizing initial memory footprint.
--Dual interfaces--: Provides both CLI scripts (captioner.py, modelDownloader.py) for quick tests and a GUI (modelManager.py) for end‑users.
--Extensible architecture--: Configuration files (e.g., config.json) conform to Hugging Face format for easy swapping of models.
