call_me_maybe

Python project · featured · LLM, constrained decoding

Turns a sentence into a JSON function call with a 0.6B model using constrained decoding — parseable by construction, not by luck.

• 0.6Bparameters
• 2choices when picking the name
• 198tokens the model never computed
• 25 sfor 11 calls

The problem

Turn a sentence like "What is the sum of 40 and 2?" into a function call in JSON, using Qwen3-0.6B: a model so small that asking it nicely for JSON only works some of the time.

What I built

• Constrained decoding: at every step, tokens that would break the JSON or the function schema get their logits set to -inf, so the output is parseable by construction.
• A grammar written as a Python generator, because the parameters depend on which function name the model picks.
• Fast legality checks: fixed text uses vocabulary prefix lookups, and JSON values use a byte automaton lifted to token masks and cached per state.
• Forced tokens: when only one continuation is possible, it is emitted without running the model at all (198 of 296 tokens in a run).

The hard part

The first version forbade tokens that spanned two grammar segments. It looked harmless, but a byte-level vocabulary holds "}} as one token, so the model had no natural way to close a string and started rambling. Letting a token cross into the text that follows fixed it: the score went from 15/22 to 17/22 and the run time from 51 s to 25 s. It also made the anti-rambling hacks I had added unnecessary, so I removed them.

What it looks like

$ uv run python -m src --verbose
[1/11] {"name": "fn_add_numbers", "parameters": {"a": 2, "b": 3}}  (6 chosen + 18 forced tokens)
[3/11] {"name": "fn_greet", "parameters": {"name": "shrek"}}  (6 chosen + 13 forced tokens)
11/11 calls generated in 27.0s (98 tokens chosen by the model, 198 filled in by the grammar)

Built with

• Python 3.10+
• Qwen3-0.6B
• uv
• mypy --strict
• flake8

More projects

• metro · Lisbon metro sim in pixel art: watch the four lines run, step into any station, board a train and ride it.
• codexion · Multithreaded twist on the Dining Philosophers: coders compete for shared dongles, with FIFO/EDF scheduling.
• fly-in · Drone fleet routing: Dijkstra pathfinding, turn-based scheduling and a pygame replay.
• a-maze-ing · Maze generator with multiple algorithms, pathfinding solvers and terminal visualization.
• push_swap · Highly optimized sorting with two stacks and a tiny set of allowed operations.
• ft_printf · printf rebuilt from scratch: chars, strings, pointers, decimals, unsigned and hex.
• get_next_line · Reads a file descriptor one line at a time with careful dynamic memory handling.
• Libft · My own C standard library — the foundation every later 42 project is built on.