Megatron-DeepSpeed
WIP: Shared t5 code
#286
Open
Overview
Commits
25
Changes
View On GitHub

WIP: Shared t5 code #286

thomasw21 wants to merge 25 commits into main from thomas/add_shared_t5
thomasw21
thomasw21 thomasw21 force pushed from 9b4a7415 to 48ed32f9 3 years ago
thomasw21 WIP
cc6dd1d3
thomasw21 Fix shared T5
fbc1d040
thomasw21 WIP
91a20aa6
thomasw21 Shared t5 tests
e55b6031
thomasw21 Woops
b5568fba
thomasw21 Woops
50cf6d9c
thomasw21 Hack my way into fix attn_mask
dcb2d610
thomasw21 DS has poor default
04cdc2e0
thomasw21 Maybe this is better
ffbe3bf5
thomasw21 Maybe this is better
90ee34c6
thomasw21 SEP is only defined for HFTokenizer
ed3dbeac
thomasw21 add --vocab-extra-ids 100
c93b5639
thomasw21 Hopefully this fixes MLM
ef205ddf
thomasw21 Hopefully this fixes MLM
c71b3799
thomasw21 Fix MLM datasets length
9da75889
thomasw21 thomasw21 force pushed from 5fd74567 to 9da75889 3 years ago
thomasw21 I think GPT2 tokenizer might be missing <sep> token
07f0ce57
thomasw21 I think GPT2 tokenizer might be missing <sep> token
12626f61
thomasw21 Flatten to pass tuples of tensor instead
52c42b25
thomasw21 Lambda function can't unpack tuples
b25803bf
thomasw21 Woops
26d9dce1
thomasw21 More dirty fixes
fc0560ac
thomasw21 Woops
8578ab36
thomasw21 Removing erroneous trailing layers
216a3f5f
thomasw21 Add TODO for loading previous checkpoint
f557523e
thomasw21 WIP: conversion script:
e73aa58e

Login to write a write a comment.

Login via GitHub

Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum