LLMs get more reliable when they switch between plain English and structured code. A new write-up from ServiceNow AI via Hugging Face shows how “code-switching” improves tool calls, grounding, and evals (source).
What is code-switching for LLMs?
Instead of answering entirely in natural language, the model alternates between short explanations (NL) and compact, machine-checkable code or JSON (CODE). Think: reasoning in English, then acting through a function call or DSL.
Why it works
- Grounded actions: The model uses whitelisted tools, not free-form prose.
- Predictable structure: JSON/DSL outputs are easy to parse, validate, and log.
- Lower hallucinations: The model must “commit” actions as code, reducing vague claims.
- Faster iteration: You can unit test the CODE part and benchmark quality over time.
How to implement it (fast)
- 1) Define a minimal protocol: Require the model to label lines as NL (reasoning), CODE (structured action), and VERIFY (post-check).
- 2) Constrain actions: Offer a small set of tool functions or a DSL, and enforce a JSON schema.
- 3) Add runtime checks: Validate JSON, simulate or dry-run risky calls, then execute.
Prompt starter (drop-in template):
Protocol:
– NL: Brief reasoning in plain English (1-2 lines).
– CODE: One line JSON for the selected tool. Use keys: tool, arguments (object).
– VERIFY: True/False plus a short note.
Example:
User: "Show top 5 customers by revenue last quarter."
Assistant:
NL: We need Q4 dates and a limit of 5.
CODE: {“tool”:”get_top_customers”,”arguments”:{“limit”:5,”start_date”:”2024-10-01″,”end_date”:”2024-12-31″}}
VERIFY: true – arguments match the request.
Example pattern
User: "Summarize this ticket and assign the right team."
Assistant:
NL: Extract product, severity, and component to route correctly.
CODE: {“tool”:”assign_ticket”,”arguments”:{“summary”:”Login failure on mobile app after update”,”severity”:”high”,”component”:”auth”,”team”:”Identity”}}
VERIFY: true – high severity + auth maps to Identity team.
Guardrails that matter
- Schema first: Provide a JSON schema and reject outputs that don’t validate.
- Tool whitelist: Only allow known functions (e.g., search_knowledge_base, assign_ticket, escalate_case).
- No raw SQL by default: Prefer a safe DSL or parameterized calls.
- Timeouts & retries: Wrap tool calls with circuit breakers and observability.
- Red-team prompts: Stress-test edge cases (missing params, conflicting constraints).
Measure the uplift
- Action success rate: % of CODE calls that execute without errors.
- First-pass accuracy: % of tasks solved with one tool call.
- Correction rate: How often VERIFY flips from false to true after self-correction.
- Latency budget: NL+CODE+VERIFY vs. baseline chain length.
Learn more
Deep dive: ServiceNow AI on Hugging Face (blog). Related research: ReAct by Yao et al. (arXiv) and function calling patterns (OpenAI docs).
Takeaway
Force the model to “think in NL, act in CODE, and check via VERIFY.” You’ll cut hallucinations, gain observability, and ship safer LLM features faster.
Enjoy Nuggets like this? Subscribe to our free weekly newsletter for sharp, no-fluff AI tactics: theainuggets.com/newsletter.
