Add foundry-hosted-agent-copilotkit skill#2090
Open
lordlinus wants to merge 14 commits into
Open
Conversation
github-actions
Bot
added
agent
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
main, but PRs should target staged.
The main branch is auto-published from staged and should not receive direct PRs.
Please close this PR and re-open it against the staged branch.
You can change the base branch using the Edit button at the top of this PR,
or run: gh pr edit 2090 --base staged
github-actions
Bot
added
branched-main
Contributor
✅ External plugin PR checks passed
Per-plugin quality summary
No changed external plugin entries were detected in this PR. |
lordlinus
force-pushed
the
add-foundry-hosted-agent-copilotkit-skill
branch
from
3cab3d5 to
13e44cf
Compare
github-actions
Bot
added
agent
Contributor
🔍 Skill Validator Results⛔ Findings need attention
Summary
Full validator output
|
Contributor
🔒 PR Risk Scan ResultsScanned 25 changed file(s).
|
lordlinus
force-pushed
the
add-foundry-hosted-agent-copilotkit-skill
branch
from
13e44cf to
0214dad
Compare
… agent Adds a skill and companion agent for building complete agentic web apps on the Azure AI Foundry hosted-agent + AG-UI + CopilotKit stack, with native human-in-the-loop approval on consequential tools. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lordlinus
force-pushed
the
add-foundry-hosted-agent-copilotkit-skill
branch
from
0214dad to
5973adb
Compare
lordlinus
changed the title
Comment on lines
+3
to
+6
| model: 'gpt-5' | ||
| tools: ['codebase', 'terminalCommand'] | ||
| name: 'foundry-hosted-agent-copilotkit' | ||
| --- |
| ALL @tools + @tool(approval_mode="always_require") HITL + history server-side | ||
| ``` | ||
|
|
||
| ## Validated live (deployed agent agentic-copilot-foundry, swec-proj-default) |
Ran the skill through a full, independent end-to-end build (a fresh
Copilot CLI session with only this skill installed, no other context)
that scaffolded an expense-approval agentic app and verified it against
a real local Foundry hosted agent (structural check, smoke test, and a
real Playwright browser E2E of HITL pause/approve/reject — all passed
with screenshot evidence). The build surfaced several inaccuracies this
commit fixes:
- The skill described a bridge (`HostedProxyAgent`, `bridge_app.py`,
two ag-ui patches) as proven/shipped code to reuse. No code ships with
this skill. Reframed as a design with two valid implementation
strategies (native add_agent_framework_fastapi_endpoint shim, or a
verified-working hand-rolled AG-UI translation), and added an
up-front notice that this is a rulebook, not a template.
- Added the real bootstrap command for hosted/: `azd ai agent init -m
<manifest-url>` / `azd ai agent sample list` — previously undocumented,
discovered by the test build.
- Corrected CopilotKit-side guidance that live testing proved wrong:
the provider component is `CopilotKit` (not `CopilotKitProvider`);
the current client can default to single-route mode (opposite of
what troubleshooting.md said); the HITL resolve payload shape
(`{accepted, steps}`) is a convention you define yourself, not a
CopilotKit-enforced contract; the CopilotKit-facing agent registry
key (often literally "default") is a separate identifier from the
Foundry hosted agent's own name — the two do not have to match.
- Flagged that the newest CopilotKit npm prerelease can be a broken/
deprecated CI accident; recommend checking for deprecation warnings
and pinning a known-good stable line.
- Removed made-up-sounding Makefile targets (`make up`/`make local`)
that don't exist without a shipped template; replaced with plain
`azd`/script guidance.
- Added a note that `python3 -m venv` can fail in sandboxed dev
environments without sudo, and `uv` is a reliable fallback.
- Softened all 'proven'/'native' claims to explicitly point at
confirming behavior against the reader's own installed package
versions, since this stack's packages move fast.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…kit-skill' into add-foundry-hosted-agent-copilotkit-skill
The skill's first real end-to-end test build (see prior commit) took
~1 hour, with roughly half that time spent re-deriving code and package
APIs from scratch that would be eliminated by shipping a proven
starting point. Added skills/foundry-hosted-agent-copilotkit/references/snippets/
with a minimal, VERIFIED-WORKING implementation for a generic "records"
domain:
- src/agent.py: build_hosted_agent() + one read tool + one
approval_mode="always_require" gated tool.
- backend/bridge_app.py, hosted_proxy.py, hosted_client.py: the
hand-rolled AG-UI bridge strategy (architecture.md strategy 2) - FastAPI
endpoint translating the hosted agent's Responses stream to AG-UI
events and forwarding mcp_approval_response on approve.
- hosted/responses/main.py: the ResponsesHostServer entry point shape
(prefer generating the rest of hosted/ with azd ai agent init).
- frontend/{app,components,lib}: CopilotKit route.ts, providers.tsx,
useHumanInTheLoop/useRenderTool components, and the agent-id constant
with its "separate identifier" note; package.json.snippet with
last-known-good pinned versions (copilotkit 1.61.2, ag-ui/client
0.0.57, next 15.5.19).
- scripts/verify.sh, smoke.py, browser_e2e.js: structural check, bridge
smoke test, and Playwright browser E2E (read, HITL pause, approve,
reject), generalized from a passing run.
Re-verified the generalized snippets independently in a fresh scratch
directory against a real local Foundry hosted agent (aif-swec-01):
hosted agent read + HITL-pause confirmed via curl, then the full bridge
plus smoke.py passed 12/12 (read, HITL pause, reject leaves state
unchanged, approve re-executes and changes state). Also fixed one bug
found only by this re-verification: hosted_client.py's Authorization
header had been corrupted by an output-redaction filter during the
original test run (a credential-shaped literal got masked in the file
itself, not just terminal output) - corrected to send a proper Bearer
token. This path (platform/deployed mode) was never exercised in the
original local-only test, so the bug had gone undetected until this
re-verification.
Wired the snippets into SKILL.md: "1. Scaffold" now says to copy
references/snippets/ first and rename the domain, "load-bearing rules"
links the hand-rolled bridge strategy to the concrete files, "3. Prove
it" points at the three scripts, and the CopilotKit section points at
the pinned package.json snippet instead of npm-installing latest blind.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment on lines
+3
to
+6
| model: 'gpt-5' | ||
| tools: ['codebase', 'terminalCommand'] | ||
| name: 'foundry-hosted-agent-copilotkit' | ||
| --- |
Comment on lines
+118
to
+121
| check( | ||
| "C2 the underlying tool has NO TOOL_CALL_RESULT yet (paused, not executed)", | ||
| find(events2, type="TOOL_CALL_RESULT", toolCallName="approve_record") is None, | ||
| ) |
… file Move src/agent.py -> hosted/responses/agent.py (no other consumer, no more sys.path hack) and merge the 3-file backend/ bridge into one bridge_app.py. Updated all skill docs + verify.sh to match; re-verified end-to-end against a real local hosted agent (12/12 smoke checks, verify.sh all pass). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lordlinus
force-pushed
the
add-foundry-hosted-agent-copilotkit-skill
branch
from
7dedb47 to
7327862
Compare
- Restore human-readable agent name per AGENTS.md's checklist (name field should read e.g. 'Address Comments', not the file-name slug); an earlier commit had wrongly changed it to match the filename based on a one-time, apparently-since-superseded validator comment. Regenerated docs/README. - smoke.py C2 checked TOOL_CALL_RESULT with toolCallName=approve_record, but ToolCallResultEvent never carries a toolCallName field at all, so the assertion trivially passed regardless of whether the tool executed. Verified live against a real hosted agent that a function_call item (the model's intent) DOES appear for a gated tool before its mcp_approval_request, so checking for the absence of TOOL_CALL_START was also wrong (it's expected). Fixed to match by tool_call_id: find the TOOL_CALL_START for approve_record (if any), then assert no TOOL_CALL_RESULT exists for that same id - the real signal that the gated tool never executed. - bridge_app.py returned an empty StreamingResponse on auth failure, which looks like a silently-empty successful SSE run when debugging. Now returns a plain 401 with a body. Re-verified all three fixes live: 401 returns 'Unauthorized' as text/plain, and smoke.py passes 12/12 with the corrected (and now actually discriminating) C2 assertion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment on lines
+2
to
+3
| name: foundry-hosted-agent-copilotkit | ||
| description: "Build a complete agentic web app on the Azure AI Foundry hosted-agent + AG-UI + CopilotKit stack: a Next.js/CopilotKit v2 chat UI over a light FastAPI/AG-UI bridge that forwards every turn to ONE Microsoft Agent Framework agent hosted in Azure AI Foundry, with native human-in-the-loop approval on consequential tools. Requires an Azure AI Foundry project (paid). Triggers: agentic app, CopilotKit app, AG-UI bridge, Foundry hosted agent, Microsoft Agent Framework, human-in-the-loop/HITL approval, approval_mode always_require, confirm_changes. Also for fixing the known traps: HITL approve-resume 400 'No tool output found', confirm_changes mis-wired, AG-UI snapshot cards vanishing, CopilotKit catch-all route 404/422, useSingleEndpoint, keyless Foundry 401 audience, Docker Hub rate-limit on ACR build." |
Comment on lines
+20
to
+22
| | 3 | HITL approval | `@tool(approval_mode="always_require")` | `useHumanInTheLoop({name,render})` → `respond(<your chosen shape>)` | bridge surfaces an approval-request card and forwards the decision as `mcp_approval_response` | | ||
| | 5 | Tool-Based Generative UI | `FunctionTool(func=None)` (declaration-only) + `tool_choice="required"` | `useFrontendTool({name,handler,render,followUp:false})` | native — stream tool-call args to the renderer | | ||
| | 4 | Agentic Generative UI | `predict_state_config` + `require_confirmation=False`; stream step status via tool args | `useAgent({updates:[OnStateChanged]})` → `agent.state` | roadmap through a pure proxy bridge — see below | |
Comment on lines
+49
to
+50
| | 5 | Tool-Based Generative UI | `FunctionTool(func=None)` | `useFrontendTool` render | stream tool-call args | | ||
| | 4 / 6 / 7 | Agentic Generative / Shared / Predictive State | `state_schema` + `predict_state_config` | `useAgent` + `setState` | bridge would need to relay text/tool-arg deltas as state events and forward `setState`; this is roadmap/unverified through a pure proxy bridge (native only when the adapter wraps an in-process agent) | |
Comment on lines
+137
to
+139
| Keep one agent-name token (`AGENT_NAME`, the hosted yaml) consistent within | ||
| `hosted/responses/` (`agent.py` and `agent.yaml`). The CopilotKit-facing | ||
| agent registry key is a **separate** identifier — see the CopilotKit section |
Root cause found by installing @microsoft/vally-cli locally and reading its source: extractFileReferences' regex handles literal brackets in a link URL fine (it only excludes parens/whitespace), but orphan-files.js resolves ref.normalized directly against the filesystem with no decodeURIComponent step — so my earlier percent-encoded route.ts link (%5B%5D) never matched the real path. Fixed by using the raw literal path (brackets and all) with a bracket-free link label instead. Verified locally: vally lint now reports 3/3 checks passed, orphan-files 18/18 reachable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ran a second independent end-to-end build test (fresh Copilot CLI session, different domain: a "PTO approver" instead of the first test's "expense approver") to verify the skill's snippets/simplification/bug-fixes actually help. Result: 12/12 smoke checks, all 7 browser E2E scenarios passed with screenshots, and verify.sh all green - and per the build's own report, the bridge snippet worked "completely unmodified except for the AGENT_NAME constant", azd ai agent init worked exactly as documented, and verify.sh/smoke.py/browser_e2e.js needed only mechanical domain renames. Total wall clock was roughly half the first test run's ~1 hour. That run also surfaced one real, valuable gap: the skill's frontend snippets only shipped providers.tsx (the <CopilotKit> wrapper + hook registrations) but never the actual chat WIDGET every consumer needs. The builder had to discover CopilotChat by reading the installed package's .d.ts files. Added: - frontend/app/page.tsx - the CopilotChat widget, generalized from this run's real, working, screenshot-verified code. - frontend/app/layout.tsx - minimal root layout wiring globals.css + Providers. - frontend/app/globals.css - plain CSS (no Tailwind dependency) for the .hitl-card/.tool-card class names, so the approval card/tool cards render legibly instead of just being present in the DOM unstyled. Verified these three files by swapping them into the actual test project and confirming `npm run build` succeeds and the page renders correctly in a headless browser (screenshot taken, matches expected UI). Also added 3 troubleshooting.md entries and 1 HITL-section entry for gaps this run's own REVIEW_NOTES.md flagged: in-memory hosted-agent state is process-lifetime (restart between verification passes or approvals leak across runs), a stale process can leave the hosted-agent port bound (Address already in use on the next `azd ai agent run`), an occasional transient DeploymentNotFound 404 on the very first request after startup (retry/restart resolves it), and keeping a renamed gated-tool parameter name in sync between the Python tool and the frontend's parsed-args cast. Verified locally with `npx @microsoft/vally-cli lint`: 3/3 checks pass, 21/21 reference files reachable from SKILL.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment on lines
+213
to
+216
| `ToolCallResultEvent`, `RunFinishedEvent`, `RunErrorEvent`). Translate the | ||
| hosted agent's raw Responses SSE stream | ||
| (`response.output_item.added/done`, `response.function_call_arguments.delta`, | ||
| `response.completed`, `mcp_approval_request`) into these events 1:1, and |
Comment on lines
+371
to
+373
| @app.get("/health") | ||
| async def health() -> dict: | ||
| return {"status": "ok", "agent": AGENT_NAME} |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
| @@ -0,0 +1,331 @@ | |||
| --- | |||
| name: foundry-hosted-agent-copilotkit | |||
| description: "Build a complete agentic web app on the Azure AI Foundry hosted-agent + AG-UI + CopilotKit stack: a Next.js/CopilotKit v2 chat UI over a light FastAPI/AG-UI bridge that forwards every turn to ONE Microsoft Agent Framework agent hosted in Azure AI Foundry, with native human-in-the-loop approval on consequential tools. Requires an Azure AI Foundry project (paid). Triggers: agentic app, CopilotKit app, AG-UI bridge, Foundry hosted agent, Microsoft Agent Framework, human-in-the-loop/HITL approval, approval_mode always_require, confirm_changes. Also for fixing the known traps: HITL approve-resume 400 'No tool output found', confirm_changes mis-wired, AG-UI snapshot cards vanishing, CopilotKit catch-all route 404/422, useSingleEndpoint, keyless Foundry 401 audience, Docker Hub rate-limit on ACR build." | |||
| 404s. Do not assume which default is current; verify empirically against a | ||
| real request. | ||
| - **Agent identifier is two separate things:** the Foundry hosted agent's own | ||
| name (`hosted/responses/agent.py` `AGENT_NAME`, `hosted/responses/agent.yaml`) is independent from |
Comment on lines
+18
to
+23
| echo "== Layout ==" | ||
| check_file "hosted/responses/agent.py" | ||
| check_file "hosted/responses/main.py" | ||
| check_file "hosted/responses/agent.yaml" | ||
| check_file "hosted/responses/Dockerfile" | ||
| check_file "hosted/azure.yaml" |
Comment on lines
+82
to
+85
| AGENT_PY_NAME=$(grep -oE 'AGENT_NAME = "[^"]+"' "$AGENT_PY" | head -1 | sed -E 's/AGENT_NAME = "([^"]+)"/\1/') | ||
| YAML_NAME=$(grep -oE '^name: [a-zA-Z0-9_.-]+' "$ROOT/hosted/responses/agent.yaml" | head -1 | sed -E 's/name: //') | ||
| echo " hosted/responses/agent.py AGENT_NAME = $AGENT_PY_NAME" | ||
| echo " hosted/responses/agent.yaml name = $YAML_NAME" |
|
|
||
| app.add_middleware( | ||
| CORSMiddleware, | ||
| allow_origins=os.environ.get("ALLOW_ORIGINS", "*").split(","), |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
| @@ -0,0 +1,331 @@ | |||
| --- | |||
| name: foundry-hosted-agent-copilotkit | |||
| description: "Build a complete agentic web app on the Azure AI Foundry hosted-agent + AG-UI + CopilotKit stack: a Next.js/CopilotKit v2 chat UI over a light FastAPI/AG-UI bridge that forwards every turn to ONE Microsoft Agent Framework agent hosted in Azure AI Foundry, with native human-in-the-loop approval on consequential tools. Requires an Azure AI Foundry project (paid). Triggers: agentic app, CopilotKit app, AG-UI bridge, Foundry hosted agent, Microsoft Agent Framework, human-in-the-loop/HITL approval, approval_mode always_require, confirm_changes. Also for fixing the known traps: HITL approve-resume 400 'No tool output found', confirm_changes mis-wired, AG-UI snapshot cards vanishing, CopilotKit catch-all route 404/422, useSingleEndpoint, keyless Foundry 401 audience, Docker Hub rate-limit on ACR build." | |||
Comment on lines
+214
to
+216
| hosted agent's raw Responses SSE stream | ||
| (`response.output_item.added/done`, `response.function_call_arguments.delta`, | ||
| `response.completed`, `mcp_approval_request`) into these events 1:1, and |
Comment on lines
+46
to
+50
| if grep -q 'approval_mode="never_require"' "$AGENT_PY"; then | ||
| pass "has at least one read (no side-effect) tool" | ||
| else | ||
| fail "no read-only tool found" | ||
| fi |
aaronpowell
reviewed
Jul 2, 2026
aaronpowell
left a comment
aaronpowell
left a comment
Contributor
There was a problem hiding this comment.
Having another read through this skill after the updates and I'll admit that I'm still pretty apprehensive about including this skill. It's very heavily geared towards the idea of building an initial sample application, with the included sample template and everything to scaffold out.
I'd argue that's actually not a great use of an agent - there's plenty of scaffolding engines out there for setting up an initial project.
This expands into where I think there becomes a larger gap in continual work with the agent. Since it focuses heavily on getting you setup, how do you continue development with CopilotKit and Foundry Hosted Agents? That would be where model uplift via a skill would be most valuable.
Comment on lines
+32
to
+39
| 1. **Scaffold** by copying `references/snippets/` (see its README) and | ||
| bootstrapping `hosted/` with `azd ai agent init -m <manifest-url>` | ||
| (`azd ai agent sample list` to discover manifests) rather than | ||
| hand-writing either. Keep the layout flat: the agent brain | ||
| (`agent.py`) lives right next to `main.py` inside `hosted/responses/` | ||
| (no separate top-level `src/`), and the bridge is ONE file | ||
| (`backend/bridge_app.py`), not split into several — don't add files or | ||
| import indirection the app doesn't need. |
Contributor
There was a problem hiding this comment.
The Agent has no guarantee that these files will exist as they are part of the Skill.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A skill and a companion agent for building a complete agentic web app on the Azure AI Foundry hosted-agent + AG-UI + CopilotKit stack — a Next.js/CopilotKit v2 chat UI over a light FastAPI/AG-UI bridge that forwards every turn to ONE Microsoft Agent Framework agent hosted in Azure AI Foundry, with native human-in-the-loop (HITL) approval on consequential tools.
skills/foundry-hosted-agent-copilotkit/(SKILL.md+ 4 reference docs)agents/foundry-hosted-agent-copilotkit.agent.md("Forgewright App Builder")Why it adds value (gap it fills)
This is a specialized, hard-to-discover workflow, not generic coding the model already handles. It encodes live-verified knowledge that frontier models get wrong by default:
add_agent_framework_fastapi_endpoint(FoundryAgent)path cannot complete hosted HITL (no client-sidemcp_approval_response), and the minimal hand-rolled forwarder that fixes exactly that gap — tracked upstream as microsoft/agent-framework#6652.400 "No tool output found"(useFoundryChatClient, not the ResponsesOpenAIChatClient), the{ accepted, steps }HITL contract, CopilotKit v2 catch-all[[...slug]]route +useSingleEndpoint={false}, AG-UI multi-tool snapshot split, and MCR base images to dodge Docker Hub ACR rate-limits.The full runnable template and scaffolding scripts live in the companion repo lordlinus/forgewright; this skill is the focused build recipe + references.
Paid service disclosure
This stack targets Azure AI Foundry (paid). The prerequisite is stated up front in both
SKILL.mdand the agent. Per CONTRIBUTING, flagging for the paid-services guidance.Validation
npm run skill:validate→✅ foundry-hosted-agent-copilotkit is validnpm run build→ README tables regenerated (docs/README.skills.md,docs/README.agents.md)namematches folder, lowercase-hyphen;descriptionwithin limits; references < 5 MB