🔙 Expectation-aware generation

[hejny]

|Language| models| generate| one| token| at| a| time|.|

The official SDKs give or stream you the result as a whole by default. You can specify max number of tokens or/and JSON mode in OpenAI, but thats it.

But if we can somehow control the flow and picking of the tokens one-by-one and return back on each wrong step, we can ensure result expectations #30 in a much more granular scale, kind of a "SuperJSON" mode.

Counts

For example

Generate 20-30 words long sentence:

|______________________________________________________|
                                    ↑                  ↑
                                   Min                Max

Now try 1:
|----|---|------|-------|-------|----------|---|--|------------|-----|STOP|

Now try 2:
|--|----|------------|------|STOP|

Now try 1:
|----|-----|--------|--------|--------|--------|STOP|

With expectation-aware generation:
|-----|-----------|---------|--------|---|-----|-------|--
|-----|-----------|---------|------------|--|--------|----------
|-----|-----------|---------|------------|--|---|----|STOP|

Formats

• See more 📑 Formats
• Fix invalid JSON
• Partial format check (can this text result in valid JSON / XML / Whatrever), if so, continue, if not, return a token

Format + Schema

• Heal valid JSON with extra stuff not defined in structure (JSON schema)
• Partial structure check (can JSON pass this schema), if yes, continue, if no, return a token back or heal

---

See also ⏳ Just-in-time fine-tuning

[hejny]

What we tried

(Describe our experience here)

[hejny]

🔎 Existing solutions

(Do some research / discussion)

• Has anyone else had the same problem?
• Has a project come up with the solution
• Is there a research paper about it?
• Is there an article, video, podcast about it?
• Is this even a good idea?

[hejny]

See OpenAI`s Structured Outputs vs JSON mode

[hejny]

11:11