optimum
adding onnx support for Funnel Transformer
#1231
Open
Overview
Commits
4
Changes
View On GitHub

adding onnx support for Funnel Transformer #1231

sidistic wants to merge 4 commits into huggingface:main from sidistic:funnel-transformer
sidistic
sidistic1 year ago (edited 1 year ago)

What does this PR do?

Hello this is my first PR ever for an Open source project. I would love to hear your thoughts on how to test the code on my machine before committing any changes ! I've added a class for input and I've added a class in class. I've also added a few lines of code specifying the models to be used for tiny and large model tests in
optimum/tests/exporters/exporters_utils.py
and a NormalizedTextConfig in normalized_config.py. I've also added the mapping in
optimum/exporters/tasks.py
. I have been reading up the model specification in tranformer/models for funnel and would love to hear your advice in the next steps and how to test my code and if my class definition for onnxconfig is correct. I use Hugging face models for a lot of my projects and I would love to keep contributing to Hugging Face in the future. Looking forward to working with the Hugging face community.

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?
regisss
regisss commented on 2023-07-27
regisss1 year ago (edited 1 year ago)

@sidistic Nice PR! I left one comment that should be easily addressed.

There are also a few more things to do:

  • Could you rebase your branch on main? That way the PR documentation will be generated without error. Edit: not sure if this is linked to this actually, let's see.
  • The code style check failed. Here is what you need to run from the root of the repo to solve it: 
    pip install .[quality]
make style
  • Also, could you add "Funnel" right after "Flaubert" in this list of supported models?

Finally, to test your PR, you can try to export a model with Optimum CLI as follows:

optimum-cli export onnx --model funnel-transformer/large --task question-answering funnel_onnx/

You can of course try with other checkpoints and tasks.

Don't hesitate if you have any questions 馃檪

optimum/exporters/onnx/model_configs.py
12461246
12471247 return super().generate_dummy_inputs_for_validation(reference_model_inputs)
1248
1249
class FunnelTransformerOnnxConfig(TextEncoderOnnxConfig):
1250
 NORMALIZED_CONFIG_CLASS = NormalizedTextConfig
1251
 ATOL_FOR_VALIDATION = 1e-4
1252
1253
 @property
1254
 def inputs(self) -> Dict[str, Dict[int, str]]:
1255
 if self.task == "multiple-choice":
1256
 dynamic_axis = {0: "batch_size", 1: "num_choices", 2: "sequence_length"}
1257
 else:
1258
 dynamic_axis = {0: "batch_size", 1: "sequence_length"}
1259
 return {
1260
 "input_ids": dynamic_axis,
1261
 "attention_mask": dynamic_axis,
1262
 "token_type_ids": dynamic_axis,
1263
 }
regisss1 year ago
Suggested change
class FunnelTransformerOnnxConfig(TextEncoderOnnxConfig):
NORMALIZED_CONFIG_CLASS = NormalizedTextConfig
ATOL_FOR_VALIDATION = 1e-4
@property
def inputs(self) -> Dict[str, Dict[int, str]]:
if self.task == "multiple-choice":
dynamic_axis = {0: "batch_size", 1: "num_choices", 2: "sequence_length"}
else:
dynamic_axis = {0: "batch_size", 1: "sequence_length"}
return {
"input_ids": dynamic_axis,
"attention_mask": dynamic_axis,
"token_type_ids": dynamic_axis,
}
class FunnelTransformerOnnxConfig(BertOnnxConfig):
pass

If I'm not mistaken, it's the exact same config as BERT. So you can just make this class inherit from BertOnnxConfig and it will behave the same. There are actually quite a lot of models with such configs: https://github.com/huggingface/optimum/blob/bdc3ecc88b0796fbb7ae051ace55d74bb2edb81d/optimum/exporters/onnx/model_configs.py#L86C24-L86C38

sidistic1 year ago

I've added this suggestion and there was an error with the opset to i changed to default opset to 12. But now there's a different error and there's no exporting of the model itself. Before i changed the default opset I had this error

Framework not specified. Using pt to export to ONNX.
Some weights of FunnelForQuestionAnswering were not initialized from the model checkpoint at funnel-transformer/large and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Using framework PyTorch: 2.0.1
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:322: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  num_remove = shift * len(pooled_pos)
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:668: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  pooling_flag = pooling_flag and block_index > 0
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:497: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  shift = 2 if q_head.shape[1] != context_len else 1
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/_internal/jit_utils.py:306: UserWarning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/jit/passes/onnx/constant_fold.cpp:181.)
  _C._jit_pass_onnx_node_shape_type_inference(node, params_dict, opset_version)
================ Diagnostic Run torch.onnx.export version 2.0.1 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Traceback (most recent call last):
  File "/Users/siddharthsagar/Desktop/Open Source/hf/bin/optimum-cli", line 10, in <module>
    sys.exit(main())
  File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/commands/optimum_cli.py", line 163, in main
    service.run()
  File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/commands/export/onnx.py", line 219, in run
    main_export(
  File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/__main__.py", line 442, in main_export
    _, onnx_outputs = export_models(
  File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/convert.py", line 758, in export_models
    export(
  File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/convert.py", line 861, in export
    export_output = export_pytorch(
  File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/convert.py", line 578, in export_pytorch
    onnx_export(
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 506, in export
    _export(
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 1548, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 1117, in _model_to_graph
    graph = _optimize_graph(
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 665, in _optimize_graph
    graph = _C._jit_pass_onnx(graph, operator_export_type)
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 1891, in _run_symbolic_function
    return symbolic_fn(graph_context, *inputs, **attrs)
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 392, in wrapper
    return fn(g, *args, **kwargs)
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 306, in wrapper
    return fn(g, *args, **kwargs)
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/symbolic_opset11.py", line 625, in symbolic_fn
    padding_ceil = opset9.get_pool_ceil_padding(
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/symbolic_opset9.py", line 1501, in get_pool_ceil_padding
    return symbolic_helper._unimplemented(
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 607, in _unimplemented
    _onnx_unsupported(f"{op}, {msg}", value)
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/symbolic_helper.py", line 618, in _onnx_unsupported
    raise errors.SymbolicValueError(
torch.onnx.errors.SymbolicValueError: Unsupported: ONNX export of operator get_pool_ceil_padding, input size not accessible. Please feel free to request support or submit a pull request on PyTorch GitHub: https://github.com/pytorch/pytorch/issues  [Caused by the value '2491 defined in (%2491 : Float(*, 1, *, 1024, device=cpu) = onnx::Pad[mode="constant"](%2487, %2490), scope: transformers.models.funnel.modeling_funnel.FunnelForQuestionAnswering::/transformers.models.funnel.modeling_funnel.FunnelModel::funnel/transformers.models.funnel.modeling_funnel.FunnelEncoder::encoder
)' (type 'Tensor') in the TorchScript graph. The containing node has kind 'onnx::Pad'.] 

    Inputs:
        #0: 2487 defined in (%2487 : Float(*, 1, *, 1024, strides=[16384, 16384, 1024, 1], requires_grad=0, device=cpu) = onnx::Unsqueeze[axes=[1]](%2486), scope: transformers.models.funnel.modeling_funnel.FunnelForQuestionAnswering::/transformers.models.funnel.modeling_funnel.FunnelModel::funnel/transformers.models.funnel.modeling_funnel.FunnelEncoder::encoder # /Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:380:0
    )  (type 'Tensor')
        #1: 2490 defined in (%2490 : Long(8, strides=[1], device=cpu) = onnx::Constant[value= 0  0  0  0  0  0  0  0 [ CPULongType{8} ]](), scope: transformers.models.funnel.modeling_funnel.FunnelForQuestionAnswering::/transformers.models.funnel.modeling_funnel.FunnelModel::funnel/transformers.models.funnel.modeling_funnel.FunnelEncoder::encoder
    )  (type 'Tensor')
    Outputs:
        #0: 2491 defined in (%2491 : Float(*, 1, *, 1024, device=cpu) = onnx::Pad[mode="constant"](%2487, %2490), scope: transformers.models.funnel.modeling_funnel.FunnelForQuestionAnswering::/transformers.models.funnel.modeling_funnel.FunnelModel::funnel/transformers.models.funnel.modeling_funnel.FunnelEncoder::encoder
    )  (type 'Tensor')

This is what i get before adding DEFAULT_ONNX_OPSET = 12, This was the console log :

Framework not specified. Using pt to export to ONNX.
Downloading model.safetensors: 100%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻坾 1.54G/1.54G [01:09<00:00, 22.2MB/s]
Some weights of FunnelForQuestionAnswering were not initialized from the model checkpoint at funnel-transformer/large and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Using framework PyTorch: 2.0.1
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:322: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  num_remove = shift * len(pooled_pos)
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:668: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  pooling_flag = pooling_flag and block_index > 0
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/transformers/models/funnel/modeling_funnel.py:497: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  shift = 2 if q_head.shape[1] != context_len else 1
/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/_internal/jit_utils.py:306: UserWarning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/jit/passes/onnx/constant_fold.cpp:181.)
  _C._jit_pass_onnx_node_shape_type_inference(node, params_dict, opset_version)
================ Diagnostic Run torch.onnx.export version 2.0.1 ================
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 1 ERROR ========================
ERROR: operator-supported-in-newer-opset-version
================================================
Exporting the operator 'aten::einsum' to ONNX opset version 11 is not supported. Support for this operator was added in version 12, try exporting with this version.
None
<Set verbose=True to see more details>


Traceback (most recent call last):
  File "/Users/siddharthsagar/Desktop/Open Source/hf/bin/optimum-cli", line 10, in <module>
    sys.exit(main())
  File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/commands/optimum_cli.py", line 163, in main
    service.run()
  File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/commands/export/onnx.py", line 219, in run
    main_export(
  File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/__main__.py", line 442, in main_export
    _, onnx_outputs = export_models(
  File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/convert.py", line 758, in export_models
    export(
  File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/convert.py", line 861, in export
    export_output = export_pytorch(
  File "/Users/siddharthsagar/Desktop/Open Source/optimum/optimum/exporters/onnx/convert.py", line 578, in export_pytorch
    onnx_export(
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 506, in export
    _export(
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 1548, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 1117, in _model_to_graph
    graph = _optimize_graph(
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 665, in _optimize_graph
    graph = _C._jit_pass_onnx(graph, operator_export_type)
  File "/Users/siddharthsagar/Desktop/Open Source/hf/lib/python3.9/site-packages/torch/onnx/utils.py", line 1901, in _run_symbolic_function
    raise errors.UnsupportedOperatorError(
torch.onnx.errors.UnsupportedOperatorError: Exporting the operator 'aten::einsum' to ONNX opset version 11 is not supported. Support for this operator was added in version 12, try exporting with this version.

Could you point me toward what files I have to look at next to fix this issue or where I could have gone wrong in my implementation?

regisss1 year ago

It seems to be an issue with this line: https://github.com/huggingface/transformers/blob/4692d2619433f1eb064f3da4f3573f060a115eac/src/transformers/models/funnel/modeling_funnel.py#L385
According to what I see here, the issue has not been solved yet 馃檨

sidistic1 year ago

Hello @regisss , Could I work on the bug? I would love to help in any way I can.

regisss1 year ago

Unfortunately there is not much we can do I think. We basically need:

  • This PR to be merged in ONNXRuntime
  • Some change in Torch too to align on this

Maybe you can reach out in this issue: pytorch/pytorch#101397

sidistic1 year ago

Sure I will look into that! thank you for your help.

sidistic first additions added
bbf66583
sidistic inheretic bert config into funnel onnx
f26d2337
sidistic added support in docs
a64db490
sidistic Changed Default Opset for Funnel
ee28c09f
sidistic sidistic force pushed from be80e0a2 to ee28c09f 1 year ago
github-actions
github-actions3 days ago

This PR has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.

github-actions github-actions added Stale

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
Stale
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum