Further reduce work load for Mac CI pipeline (#24197)
### Description
Further reduce work load for Mac CI pipeline
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
