Repository navigation
Commit c58bd96
committed
fix(Llama): use output embedding dimensions and respect non-causal microbatch limits
- Read embeddings using n_embd_out instead of the hidden dimension to avoid
truncating projected outputs, such as EmbeddingGemma 2's 768-dimensional vectors.
- Limit non-causal embedding batches to n_ubatch so each input fits in a single
microbatch, preventing native assertions and preserving full-context attention.
Signed-off-by: JamePeng <jame_peng@sina.com>1 parent 6740fbe commit c58bd96
2 files changed
Lines changed: 6 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3138 | 3138 | | |
3139 | 3139 | | |
3140 | 3140 | | |
| 3141 | + | |
| 3142 | + | |
| 3143 | + | |
3141 | 3144 | | |
3142 | 3145 | | |
3143 | 3146 | | |
| |||
3149 | 3152 | | |
3150 | 3153 | | |
3151 | 3154 | | |
3152 | | - | |
| 3155 | + | |
3153 | 3156 | | |
3154 | 3157 | | |
3155 | 3158 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
149 | 149 | | |
150 | 150 | | |
151 | 151 | | |
152 | | - | |
| 152 | + | |
| 153 | + | |
153 | 154 | | |
154 | 155 | | |
155 | 156 | | |
| |||
0 commit comments