model-conversion : add token ids to prompt token output [no ci] (#17863)
This commit adds the token ids to the printed prompt outputs. The motivation for this is that is can be useful to see the actual token ids alongside the token strings for debugging.
This commit is contained in:
parent
951520ddb0
commit
2fa51c19b0
1 changed files with 1 additions and 1 deletions
|
|
@ -144,7 +144,7 @@ int main(int argc, char ** argv) {
|
|||
return 1;
|
||||
}
|
||||
std::string s(buf, n);
|
||||
printf("%s", s.c_str());
|
||||
printf("%s (%d)", s.c_str(), id);
|
||||
}
|
||||
printf("\n");
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue