transformers

81fe4f16 - Add RF-DETR (#36895)

Add RF-DETR (#36895) * fix: moved variables to config for ScaleProjector * tests: remove deprecated tests * docs: remove unnecessary utf-8 * docs: add RF-DETR to toctree * style * tests: add cpu device in expectations * style: reordered classes * fix: removed unnecessary variable * style: make style * fix: removed backbone API related attributes from config * fix: added class_loss_coefficient to LWDetr config and loss * feat: add RfDetrForInstanceSegmentation model * docs: polish docs * fix: lwdetr unnecessary config attribute * chore: apply modular to rfdetr * fix: change from deformabledetr to detr image processor * Update docs/source/en/model_doc/rf_detr.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * docs: update convert script docs * feat: add new segmentation models * refactor: remove accelerate dependency for distributed * refactor: refactor BACKBONE and MODEL config dicts * refactor: remove large deprecated model * refactor: remove dependency to config for WeightMapping scale projectors * fix: remove unnecessary imports in loss file * tests: move object_detection and segmentation tests from convert script to tests * refactor: removed unnecessary argument * Update src/transformers/models/rf_detr/modular_rf_detr.py Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com> * chore: make style * refactor: remove einsum from segmentation_head method * refactor: cleaned group object queries refinement portion of code * refactor: inlined lists * refactor: replace output_hidden_states in Dino forward * docs: removed unnecessary arguments in pipeline * test: fix segmentation tests * refactor: use expand instead of repeat for efficiency * fix: added missing refactor in modular * refactor: removed no-op cat * fix: missing instead of mismatch * docs: fixed model names * refactor: inlined source and mask flattening * refactor: improved RfDetr forward legibility * refactor: grouped common logic in predict_encoder_class_logits and predict_class_and_boxes for object detection and segmentation models * docs: clarified object detection and segmentation forward with inline comments * fix: two num_h_patches_per_window * refactor: removed torch manual seed * refactor: removed multiscale projector because there is no config to use it * refactor: move weight mapping to modeling and conversion_mapping files to support both original and converted state dicts * test: fix segmentation models loss outputs * test: fix test_model_outputs_equivalence by setting manual seed * docs: remove unused argument in config docstring * docs: add docs mention that it is compatible with RfDetr trained model on Roboflow platform * refactor: renamde pwconv and dwconv modules * fix remaining test errors * fix docstring and set integration tests as slow test * refactor: loss improvements * refactor: removed unnecessary slow processor * fix: add warning for BC for mask_loss_coefficient in LwDetrConfig * docs: 2024 to 2026 copyright * refactor: convert script configs * docs: added auto_docstring decorator to config * refactor: removed LwDetrConfig as super class in modular * fix: reflected merge changes of LwDetr * chore: nits in variable names * chore: added comments for diff with parent class * refactor: removed LwDetrScaleProjector inheritance * chore: reflected Backbone output changes from main * fix: detach and undetach logits * docs: moved step documentations in docstring * docs: fix object detection docstring * refactor: use CLIPMLP * chore: changed rf_detr prefix in segmentation model * tests: remove unnecessary method * tests: add comments about test overwrite * tests: removed most of the integration tests to only include base models * chore: suggestions from code review * chore: added missing license * docs: improved model card * chore: added missing auto docstring to Config * refactor: moved window utilities inside modules * fix: wrong key renamming * fix: wrong variable name that replaced itself * fix: fixed copies in loss_rf_detr * fix: LwDetr config docstring * docs: moved steps to auto_docstring custom intro * docs: new date releases * chore: convert modular * skip all fa inference equivalence * refactor configs * feat: Add RfDetr initial commit * tests: RfDetr and RfDetrDinov2WithRegisters tests * docs: added RfDetr docs * refactor: apply lwdetr changes to rfdetr * refactor: apply changes after lwdetr rebase * tests: added missing _prepare_for_class method * refactor: removed registers and moved from DinoV2WithRegisters to simple DinoV2 for RF-Detr backbone * feat: updated RfDetr and RfDetrDinov2 to comply with original implementation * chore: make style * refactor: extracted window partition and unpartition in dedicated functions * refactor: changed from remove_window parameter to global_attention attribute in RfDetrDinov2Layer * chore: removed mistakenly committed file * refactor: added LwDetrConfig as superclass for RfDetrConfig * tests: add large RfDetr integration test * chore: make style * chore: make repo-consistency * chore: make quality * fix: rebased on lw_detr branch * chore: moved RfDetrDinov2 to a single modular_rf_detr.py file * fix: fixed backbone name in weight for convert script * refactor: fixed modular file after fix * docs: fix licences * fix: moved variables to config for ScaleProjector * tests: remove deprecated tests * docs: remove unnecessary utf-8 * docs: add RF-DETR to toctree * tests: add cpu device in expectations * style: reordered classes * fix: removed unnecessary variable * style: make style * fix: removed backbone API related attributes from config * fix: added class_loss_coefficient to LWDetr config and loss * feat: add RfDetrForInstanceSegmentation model * docs: polish docs * chore: apply modular to rfdetr * fix: change from deformabledetr to detr image processor * Update docs/source/en/model_doc/rf_detr.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * docs: update convert script docs * feat: add new segmentation models * refactor: remove accelerate dependency for distributed * refactor: refactor BACKBONE and MODEL config dicts * refactor: remove large deprecated model * refactor: remove dependency to config for WeightMapping scale projectors * fix: remove unnecessary imports in loss file * tests: move object_detection and segmentation tests from convert script to tests * refactor: removed unnecessary argument * Update src/transformers/models/rf_detr/modular_rf_detr.py Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com> * chore: make style * refactor: remove einsum from segmentation_head method * refactor: cleaned group object queries refinement portion of code * refactor: inlined lists * refactor: replace output_hidden_states in Dino forward * docs: removed unnecessary arguments in pipeline * test: fix segmentation tests * refactor: use expand instead of repeat for efficiency * fix: added missing refactor in modular * refactor: removed no-op cat * fix: missing instead of mismatch * docs: fixed model names * refactor: inlined source and mask flattening * refactor: improved RfDetr forward legibility * refactor: grouped common logic in predict_encoder_class_logits and predict_class_and_boxes for object detection and segmentation models * docs: clarified object detection and segmentation forward with inline comments * fix: two num_h_patches_per_window * refactor: removed torch manual seed * refactor: removed multiscale projector because there is no config to use it * refactor: move weight mapping to modeling and conversion_mapping files to support both original and converted state dicts * test: fix segmentation models loss outputs * test: fix test_model_outputs_equivalence by setting manual seed * docs: remove unused argument in config docstring * docs: add docs mention that it is compatible with RfDetr trained model on Roboflow platform * refactor: renamde pwconv and dwconv modules * fix remaining test errors * fix docstring and set integration tests as slow test * refactor: loss improvements * docs: 2024 to 2026 copyright * refactor: convert script configs * docs: added auto_docstring decorator to config * refactor: removed LwDetrConfig as super class in modular * fix: reflected merge changes of LwDetr * chore: nits in variable names * chore: added comments for diff with parent class * refactor: removed LwDetrScaleProjector inheritance * chore: reflected Backbone output changes from main * fix: detach and undetach logits * docs: moved step documentations in docstring * docs: fix object detection docstring * refactor: use CLIPMLP * chore: changed rf_detr prefix in segmentation model * tests: remove unnecessary method * tests: add comments about test overwrite * tests: removed most of the integration tests to only include base models * chore: suggestions from code review * chore: added missing license * docs: improved model card * chore: added missing auto docstring to Config * refactor: moved window utilities inside modules * fix: wrong key renamming * fix: wrong variable name that replaced itself * fix: fixed copies in loss_rf_detr * fix: LwDetr config docstring * docs: moved steps to auto_docstring custom intro * docs: new date releases * chore: convert modular * skip all fa inference equivalence * refactor configs * refactor: remove calculate_uncertainty function and inline logic in sample_points_using_uncertainty * refactor: inline bbox_cost computation using cdist with type casting * refactor: move target_masks type casting to a more appropriate line * refactor: replace NaN and Inf handling in cost matrix with maximum float value * refactor: extend RfDetrHungarianMatcher to inherit from HungarianMatcher * docs: add difference about mask cost to RfDetrHungarianMatcher * refactor: simplify RfDetrImageLoss initialization by using super() * refactor: inline source_masks in RfDetrImageLoss * fix: return zero tensors for loss_mask_ce and loss_mask_dice when source_masks is empty in RfDetrImageLoss * docs: add comment about num_points computation * docs: clarify handling of mask losses in auxiliary outputs for RfDetrImageLoss * docs: add clarification on auxiliary outputs for masks in _set_aux_loss of RfDetrImageLoss * chore: add spacing * fix: remove hardcoded num_labels for coco in get_model_config function * refactor: removed attribute_map in RfDetrConfig and renamed hidden_size occurences to d_model * refactor: removed unnecessary default config kwargs which don’t overwrite the default attribute in RfDetrDinov2 * refactor: set intermediate_size as config attribute and add coco labels * docs: clarify window partitioning and unpartitioning processes in RfDetrDinov2 classes * refactor: rename shortcut to residual and update residual connections in RfDetrDinov2Layer * refactor: update return type of forward method in RfDetrDinov2Layer to simplify signature * docs: add comment about required forced output hidden states for RfDetrDinov2Backbone * refactor: removed unnecessary split module attribute * chore: from gen to generate * refactor: norm_in into norm renaming * chore: rename features into hidden_states * docs: add docstring to get_mask_logits * tests: remove custom hidden states output test * chore: removed copied from statement from sample_points_using_uncertainty * chore: remove accelerate dependency from LwDetr loss * refactore: have RfDetrConfig inherit LwDetrConfig * chore: removed unnecessary comment * refactor: renamed features into hidden_states * tests: add cuda version * chore: fix missing import * Apply suggestions from code review Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * docs: changed comments into docstring * refactor: inlined attribute computation * refactor: renamed variable from hidden_states_tuple to hidden_states * tests: removed unnecessary override and added comments on override tests * fix: added specific config attributes to check config attribute file with a comment * docs: make fix-repo * tests: added different split percents * wip: conversion mapping * Fix conversion mapping rf detr + support nested prefixes * fix modular * fix: add .contiguous after split * chore: fix-repo * Fix conversion mapping rf_detr * fix modular and date --------- Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com> Co-authored-by: yonigozlan <yoni.gozlan@huggingface.co> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>