Skip to content

Commit 272f863

Browse files
cursoragentxxg1413
cursoragent
and
xxg1413
committed
courses: add Building LLM features in Python (validate, allowlist tools, retry, honest eval)
Co-authored-by: P小二 <xxg1413@gmail.com>
1 parent a8d24ac commit 272f863

30 files changed

Lines changed: 2508 additions & 0 deletions

‎courses/llm-features-in-python/COURSE.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
---
2+
id: course-llm-features
3+
type: course
4+
title: "Build LLM features in Python: structured output, tool calls, and evals"
5+
summary: Turn a demo-day model call into a feature you can ship — validate structured output, allowlist tool calls, retry with the error fed back, fall back to a human instead of guessing, and measure it with an offline eval that divides by every case.
6+
lang: en-US
7+
content_version: 1
8+
status: reviewed
9+
reviewed_on: 2026-10-11
10+
badge:
11+
id: course-llm-features
12+
name_en: Model output is untrusted input
13+
name_zh: 模型输出是不可信输入
14+
requires: All five checkpoints claimed (L01–L05)
15+
course_id: course-llm-features
16+
---
17+
18+
# Build LLM features in Python: structured output, tool calls, and evals
19+
20+
> TL;DR: install the FlyPython Skill in your coding agent and let it fetch
21+
> this course, then say **"start lesson 1"**. (Lesson files arrive via the
22+
> Skill — you download nothing by hand.) You finish with `ticket_triage.py`:
23+
> a support-ticket triage feature that treats every model reply as
24+
> untrusted input, plus an offline eval with a pass/fail threshold. No API
25+
> key, no network, no third-party packages — a replay client stands in for
26+
> the model.
27+
28+
## What you build
29+
30+
A triage feature for support tickets: the model reads a ticket and replies
31+
with a category, a priority, a one-line summary, and whether a human must
32+
look at it. It may call a lookup tool first. Three scenario "skins" ship
33+
with the course, each with labelled cases and recorded model replies:
34+
35+
| Skin | You are… | Tool the model may call |
36+
| --- | --- | --- |
37+
| `scenario/shop-support/` | routing an online shop's support inbox | `lookup_order(order_id)` |
38+
| `scenario/it-helpdesk/` | sorting internal IT requests | `lookup_device(asset_tag)` |
39+
| `scenario/saas-feedback/` | classifying in-app product feedback | none |
40+
41+
The recorded replies are hand-written fixtures that imitate failure shapes
42+
models really produce — JSON in a Markdown fence, a category that is not on
43+
the list, `"priority": "high"`, an extra field, a tool that does not exist,
44+
a tool argument of the wrong type, chatty prose instead of JSON. They are
45+
**not** captured from any model, and the course makes no claim about how
46+
often any real model produces them.
47+
48+
## Teaching contract (read this first, agent)
49+
50+
- **Audience:** a Python developer who has called a model API in a script
51+
or notebook and now has to put that call inside a product — where a
52+
malformed reply, a hallucinated tool call, or an eval that looks better
53+
than it is becomes somebody's incident. Not a prompt-engineering course
54+
and not an introduction to Python.
55+
- **Prerequisites:** Python 3.11+ on PATH and any coding agent that can run
56+
commands. Standard library only; no API key and no network access are
57+
needed for the exercise.
58+
- **Tool:** tool-agnostic. The course has not yet been taught in a tool;
59+
`REVIEW.md` records the mechanical verification and the pending
60+
agent-taught run-through, which will name the exact tool and version.
61+
- **Lesson order:** L01 → L02 → L03 → L04 → L05, one lesson per session.
62+
Never skip the checkpoint at the end of a lesson.
63+
- **Teaching style:** work from the files in this folder, not from general
64+
knowledge about any provider's API. Quote the `TASK.md` line you are
65+
satisfying. Make the smallest change that moves a test from red to
66+
green. Never add dependencies, never edit `solution/`, `tests/`, or
67+
`scenario/`, and ask before touching any file not named in the current
68+
lesson. Never weaken validation to make a recorded reply pass.
69+
- **When to stop:** a lesson is done when its checkpoint command runs and
70+
the learner can say, in their own words, what failed and why. If the
71+
learner cannot, re-teach from the failing test, not from the solution.
72+
- **How to use `verify.py`:** the learner's command is plain
73+
`python verify.py` — it runs the suite against `starter/`, prints
74+
per-checkpoint status and earned claim codes, and exits non-zero until
75+
every checkpoint is passed (l03, l04) or attested (l01, l02, l05). `starter --expect-failure`
76+
and `solution` remain maintainer checks.
77+
- **Honesty rules:** an offline eval over a handful of recorded replies
78+
says nothing about how a live model will behave on live traffic — say so.
79+
Do not claim the feature is production-ready. Do not promise learning or
80+
career outcomes.
81+
82+
## What this course does NOT cover
83+
84+
Any specific provider SDK or its structured-output mode, API keys and
85+
secrets handling, prompt wording, retrieval (RAG) and embeddings,
86+
fine-tuning, agent frameworks, streaming, cost and latency budgets, or
87+
online evaluation on production traffic. Lesson 5 shows where a real
88+
client plugs in; the companion repository has guides for the rest — link,
89+
don't improvise.
90+
91+
## Badge contract
92+
93+
- Badge: **Model output is untrusted input Badge** (badge id `course-llm-features`) — earned by claiming all five checkpoints.
94+
- Challenges: L01–L05 checkpoints, 10 points each; +50 course-badge bonus when all five are claimed on flypython.com.
95+
- Evidence: `python verify.py` — L03 (validation, tool calls, retries) and L04 (honest eval) are objectively gated by the suite; L01/L02/L05 are learner-attested.
96+
- Submission: each test-passed checkpoint prints a deterministic claim code; a reflection checkpoint prints one only after you answer its questions and run `python verify.py --attest ID`; record it on flypython.com against your account. Self-reported evidence, never a certificate.
97+
98+
## Folder map
99+
100+
```
101+
COURSE.md / COURSE_cn.md this file (EN / 中文)
102+
lessons/L01.md … L05.md lessons (each has an _cn.md pair)
103+
scenario/<skin>/ scenario.json, cases.json, recordings.json, lookup data
104+
TASK.md / TASK_cn.md the task contract the change must satisfy
105+
starter/ticket_triage.py the demo-day implementation
106+
solution/ticket_triage.py the reviewed solution (do not copy in lesson 3)
107+
tests/test_ticket_triage.py the contract suite (15 tests, read-only)
108+
verify.py objective pass/fail evidence
109+
REVIEW.md maintainer run-through record
110+
```
111+
112+
## Evidence and licensing
113+
114+
`REVIEW.md` records the run-through state with dates, tool versions, and
115+
observed deviations. Code in this folder is MIT-licensed; lesson prose is
116+
CC BY 4.0 (see repository `LICENSE`). Report teaching drift or unclear
117+
lessons via the repository's `course-feedback` issue form.

‎courses/llm-features-in-python/COURSE_cn.md‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
---
2+
id: course-llm-features
3+
type: course
4+
title: "用 Python 构建 LLM 功能：结构化输出、工具调用与评测"
5+
summary: "把演示日的一次模型调用变成能上线的功能——校验结构化输出、给工具调用设白名单、带着错误信息重试、拿不准就转人工而不是瞎猜，并用分母覆盖全部用例的离线评测来度量。"
6+
lang: zh-CN
7+
content_version: 1
8+
status: reviewed
9+
reviewed_on: 2026-10-11
10+
badge:
11+
id: course-llm-features
12+
name_en: Model output is untrusted input
13+
name_zh: 模型输出是不可信输入
14+
requires: 认领全部五个检查点（L01–L05）
15+
course_id: course-llm-features
16+
---
17+
18+
# 用 Python 构建 LLM 功能：结构化输出、工具调用与评测
19+
20+
> 一句话：在你的编码 Agent 里装上 FlyPython Skill，让它取回本课程，然后
21+
> 说**「开始第 1 课」**。（课程文件经 Skill 送达——你不手动下载任何东西。）
22+
> 完成时你会得到 `ticket_triage.py`：一个把每条模型回复都当作不可信输入的
23+
> 工单分诊功能，外加一个带通过/失败阈值的离线评测。不需要 API key、不需要
24+
> 联网、不需要第三方包——由一个回放 client 代替模型。
25+
26+
## 你要做的东西
27+
28+
一个工单分诊功能：模型读一张工单，回复分类、优先级、一句话摘要，以及
29+
是否必须由人来看。它可以先调用一个查询工具。课程自带三个「场景皮肤」，
30+
每个都有带标注的用例和录好的模型回复：
31+
32+
| 皮肤 | 你是…… | 模型可调用的工具 |
33+
| --- | --- | --- |
34+
| `scenario/shop-support/` | 给网店客服收件箱分流 | `lookup_order(order_id)` |
35+
| `scenario/it-helpdesk/` | 整理公司内部 IT 请求 | `lookup_device(asset_tag)` |
36+
| `scenario/saas-feedback/` | 给应用内产品反馈分类 | 无 |
37+
38+
录好的回复是手写的练习数据，模仿模型真实会出现的失败形状——包在
39+
Markdown 围栏里的 JSON、不在列表里的分类、`"priority": "high"`、多出来的
40+
字段、不存在的工具、类型不对的工具参数、用闲聊散文代替 JSON。它们**不是**
41+
从任何模型采集的，课程也不对任何真实模型出现这些问题的频率做任何声明。
42+
43+
## 教学契约（先读，Agent）
44+
45+
- **受众：** 在脚本或 notebook 里调过模型 API、现在要把这个调用放进产品
46+
的 Python 开发者——在产品里，一条格式错误的回复、一次幻觉出来的工具
47+
调用、一个看起来比实际好的评测，都会变成某个人的事故。这不是提示词
48+
工程课，也不是 Python 入门。
49+
- **前置条件：** PATH 上有 Python 3.11+，以及任意一个能执行命令的编码
50+
Agent。只用标准库；练习不需要 API key，也不需要联网。
51+
- **工具：** 与工具无关。本课程尚未在任何工具里实际授课；`REVIEW.md`
52+
记录机械验证结果和待完成的 Agent 授课实跑，实跑时会写明确切的工具与
53+
版本。
54+
- **课序：** L01 → L02 → L03 → L04 → L05，一次课一个会话。
55+
绝不跳过课尾的检查点。
56+
- **教学风格：** 从本文件夹里的文件出发，不要凭任何厂商 API 的通用知识
57+
发挥。引用你正在满足的那条 `TASK.md` 原文。做让测试从红变绿的最小
58+
改动。绝不加依赖，绝不改 `solution/`、`tests/` 或 `scenario/`，动当前
59+
课程未点名的任何文件前先询问。绝不为了让某条录好的回复通过而放松校验。
60+
- **何时停下：** 一节课的检查点命令跑通、且学习者能用自己的话说清
61+
哪里坏了、为什么，这节课才算完。学习者说不出就从失败测试重新教，
62+
不要从 solution 教。
63+
- **如何使用 `verify.py`：** 学员命令就是 `python verify.py`——它针对
64+
`starter/` 跑套件、打印逐检查点状态与已获认领码，直到每个检查点都
65+
通过（l03、l04）或已自报确认（l01、l02、l05）之前都以非零退出。`starter --expect-failure` 与 `solution` 留作
66+
维护者检查。
67+
- **诚实规则：** 在少量录好的回复上跑的离线评测，说明不了真实模型在真实
68+
流量上的表现——要讲明白。不要声称功能达到生产可用。不要承诺学习或
69+
职业结果。
70+
71+
## 本课程不覆盖什么
72+
73+
任何具体厂商的 SDK 或其结构化输出模式、API key 与密钥管理、提示词措辞、
74+
检索增强（RAG）与 embedding、微调、Agent 框架、流式输出、成本与延迟
75+
预算，以及在生产流量上的在线评测。第 5 课展示真实 client 从哪里接入；
76+
其余内容由配套仓库的指南承载——给链接，不即兴发挥。
77+
78+
## 徽章契约
79+
80+
- 徽章：**模型输出是不可信输入徽章**（badge id `course-llm-features`）——认领全部五个检查点获得。
81+
- 挑战：L01–L05 检查点，各 10 分；在 flypython.com 上集齐五个再加 50 分课程徽章奖励。
82+
- 证据：`python verify.py`——L03（校验、工具调用、重试）与 L04（诚实评测）由套件客观把关；L01/L02/L05 为学习者自我报告。
83+
- 提交：测试通过的检查点打印确定性认领码；自报检查点要先回答课后问题，再运行 `python verify.py --attest ID` 才打印码；在 flypython.com 上记入你的账号。这是自我报告的证据，从来不是证书。
84+
85+
## 文件夹地图
86+
87+
```
88+
COURSE.md / COURSE_cn.md 本文件（EN / 中文）
89+
lessons/L01.md … L05.md 课文（每课都有一个 _cn.md 对）
90+
scenario/<skin>/ scenario.json、cases.json、recordings.json、查询数据
91+
TASK.md / TASK_cn.md 改动必须满足的任务契约
92+
starter/ticket_triage.py 演示日版本的实现
93+
solution/ticket_triage.py 评审过的解答（第 3 课不要抄）
94+
tests/test_ticket_triage.py 契约套件（15 个测试，只读）
95+
verify.py 客观的通过/失败证据
96+
REVIEW.md 维护者实跑记录
97+
```
98+
99+
## 证据与许可
100+
101+
`REVIEW.md` 记录实跑状态——日期、工具版本、观察到的偏差。文件夹内代码
102+
采用 MIT 许可；课文文字采用 CC BY 4.0（见仓库 `LICENSE`）。发现教学漂移
103+
或课程不清楚之处，请通过仓库的 `course-feedback` issue 表单反馈。

‎courses/llm-features-in-python/REVIEW.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Maintainer run-through record
2+
3+
## 2026-10-11 — contract verification (mechanical)
4+
5+
- Environment: Linux x86_64, Python 3.12.3, standard library only.
6+
- The scenario recordings (`scenario/*/recordings.json`) are hand-written
7+
fixtures that imitate common reply failures (Markdown fences, invented
8+
categories, string priorities, prose, out-of-allowlist tool calls). They
9+
were not captured from any model, and no model API was called while
10+
authoring or verifying this course.
11+
- Commands and results:
12+
- `python verify.py starter --expect-failure` — exit 0; the twelve
13+
expected failure names reproduced; three tests pass on the starter.
14+
- `python verify.py solution` — exit 0; 15/15 tests pass.
15+
- `python verify.py` — exit 1 on the shipped starter; l03/l04 `[open]`,
16+
l01/l02/l05 `[pending]` with no code.
17+
- Solution eval CLI:
18+
19+
| Skin | Output | Exit |
20+
| --- | --- | --- |
21+
| `shop-support` | `accuracy=0.750 correct=6/8 fallbacks=1 min_accuracy=0.800` | 1 |
22+
| `it-helpdesk` | `accuracy=0.833 correct=5/6 fallbacks=1 min_accuracy=0.800` | 0 |
23+
| `saas-feedback` | `accuracy=0.800 correct=4/5 fallbacks=0 min_accuracy=0.800` | 0 |
24+
25+
The starter prints `accuracy=1.000 correct=5/6` on `it-helpdesk`, the
26+
contradiction Lesson 1 is built on.
27+
- Lesson 3's failing-count table (12 → 9 → 7 → 3 → 0) was measured by
28+
swapping the solution's functions into a scratch copy of the starter
29+
one group at a time; `test_tool_result_is_fed_back_to_the_model` turns
30+
red after the parser group, as the lesson says.
31+
- The Lesson 3 exercise one-liner prints the retry feedback message
32+
after the `triage` turn.
33+
- `python tools/verify_courses.py` — contract satisfied.
34+
35+
## Pending before this course can be called taught-in-tool
36+
37+
- One full agent-taught run-through ("start lesson 1" through the Lesson 5
38+
checkpoint) in a named coding agent and version, recording observed
39+
deviations from `COURSE.md` here. The course is tool-agnostic; no
40+
agent-taught session has happened yet.
41+
- Lesson 5 asks learners to apply the pattern to their own model-backed
42+
feature with a real provider. Whether the contract transfers cleanly to a
43+
provider's native structured-output or tool-calling API is unverified
44+
here.
45+
46+
## Deviation log
47+
48+
### 2026-10-11 — agent solvability run (challenge mode, not taught mode)
49+
50+
- Agent: a Cursor Cloud Agent subagent (Claude Opus 5.5), Linux, Python
51+
3.12.3, standard library only.
52+
- Method: worked in a copy of the course with `solution/` and this file
53+
removed; implemented `starter/ticket_triage.py` from `TASK.md` and the
54+
tests only, without access to the reference solution.
55+
- Result: green after one implementation pass (14/14 on the revision it
56+
was given); the eval CLI printed the same three lines as the solution
57+
table above.
58+
- Observed deviations and the changes they triggered:
59+
- `TASK.md` did not define `attempts` in the fallback; the agent inferred
60+
"number of rejected replies" from a test. `TASK.md`/`TASK_cn.md` now
61+
state it.
62+
- `TASK.md` did not say whether a tool call beyond `max_tool_calls` runs,
63+
or whether refused calls count as `tool_calls`, and no test exercised
64+
the budget. The contract now says only calls that ran are counted and
65+
the over-budget call is not run; the new
66+
`test_tool_call_budget_ends_in_the_fallback` enforces it (suite 14 → 15
67+
tests, starter failures 11 → 12, lessons updated). The agent's
68+
implementation passes the new test unchanged.
69+
- `COURSE.md` said plain `python verify.py` exits non-zero "while l03 and
70+
l04 are open"; it actually exits non-zero until l01/l02/l05 are also
71+
attested. Wording corrected in `COURSE.md`/`COURSE_cn.md`.
72+
- Limitations: challenge-mode run (TASK.md → tests), not the taught
73+
COURSE.md walkthrough; teaching quality remains pending. The agent
74+
implemented all four functions in one pass, so it did not walk Lesson 3's
75+
per-turn sequence (that sequence was measured mechanically, above).

0 commit comments

Comments
 (0)