How to count consumed and inferred tokens? #922
-
|
Subj Currently I'm using Really though it might make more sense to do everything explicitly instead of using higher-level ops. I need to get initial tokens, count them, then generate new ones one by one, and stream back. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
If you'd prefer to work at the level of inidividual token, you're probably better off using the |
Beta Was this translation helpful? Give feedback.
-
|
is there a way to get the consumed tokens count and the remaining context length in the InteractiveExecutor? |
Beta Was this translation helpful? Give feedback.
-
|
Not as far as I'm aware :( |
Beta Was this translation helpful? Give feedback.
If you'd prefer to work at the level of inidividual token, you're probably better off using the
BatchedExecutor. That's a lower level executor (tokens in, logits out) so you have more control over the details (but it's harder to use than the higher level executors which are strings in, strings out).