NeMo Guardrails
NVIDIA's open-source toolkit for adding programmable guardrails to LLM applications. You define conversational rails in Colang to control topics, safety, and output checks, and the toolkit places them between the user and the model.
Appendix · References
References andLinks
The open-source tools, standards, stack pieces, benchmarks, prompting resources and books behind the talk. Each entry has a short description and its official links.
Part 3 · ADLC
1. The open-source toolbox
Tools grouped by the phase of the Agent Development Lifecycle they support.
Phase 1 · Goal: guardrails
Guardrails AI
A Python framework that validates LLM inputs and outputs against schemas and reusable validators from its hub. When an output fails validation, it can re-ask the model to correct the response.
Phase 2 · Design: agent frameworks
LangGraph
LangChain's low-level orchestration library for stateful agents. Agents are modeled as graphs with nodes, edges, and cycles, and they include checkpoints and human-in-the-loop interrupts for long-running workflows.
CrewAI
A framework for building teams of role-based agents. You define agents, tasks, and crews, and the crew collaborates either sequentially or under a manager agent.
AutoGen
Microsoft's multi-agent framework built around conversing agents. It is now in maintenance mode, and Microsoft points users to its successor, Microsoft Agent Framework, which was not checked for this page.
Phase 3 · Select: models and serving
Ollama
A simple tool for downloading and running open-weight models on your own machine. It offers a command-line interface and a local API, which makes it a common starting point for private experiments.
vLLM
A high-throughput inference and serving engine for LLMs. It uses PagedAttention and continuous batching, and it exposes an OpenAI-compatible server for production workloads.
LiteLLM
A Python SDK and proxy that provides one OpenAI-style interface to more than 100 model providers. It also handles routing, fallbacks, and cost tracking.
Phase 4 · Build: MCP servers, documents, no-code
FastMCP
A Python framework for building MCP servers and clients. Decorators let you expose tools, resources, and prompts with very little boilerplate.
LlamaIndex
A data framework for connecting LLMs to documents. It covers ingestion, indexing, and retrieval, and it is widely used for retrieval-augmented generation (RAG).
Langflow
An open-source visual builder for AI workflows and agents. Components are arranged by drag and drop, and the flows can be exported and run as code.
Phase 5 · Finetuning (optional)
TRL
Hugging Face's library for training language models with supervised fine-tuning, preference optimization such as DPO, reinforcement learning methods, and reward modeling.
PEFT
Hugging Face's library for parameter-efficient fine-tuning, including LoRA and adapter methods. It trains a small set of added weights while the base model stays frozen, which lowers cost and memory needs.
Phase 6 · Evaluate
promptfoo
A CLI and library for testing prompts, models, and agents with declarative test cases. It supports side-by-side comparisons and red-team scans for security issues.
DeepEval
An open-source LLM evaluation framework with pytest-style metrics for answer relevance, hallucination, and agent task success. It can run in continuous integration pipelines.
Ragas
A library of metrics for evaluating RAG pipelines, including faithfulness, context precision, and answer relevancy.
Phase 7 · Observe
Langfuse
An open-source LLM engineering platform for tracing, evaluation, prompt management, and cost analytics. It records what each agent step did and why.
OpenTelemetry
A vendor-neutral standard and set of SDKs for collecting traces, metrics, and logs from applications. Agent telemetry can be sent through it to any compatible backend.
Grafana
An open-source platform for dashboards, visualization, and alerting across metrics, logs, and traces.
Part 4 · Open standards
2. Standards
Protocols and conventions that let agents, tools and interfaces work together, and the bodies that govern them.
Slide: MCPMCP (Model Context Protocol)
An open protocol for connecting AI applications to tools, data, and prompts through a standard client-server interface. The deck describes it as a USB-C port for AI.
Slide: MCPMCP for Beginners
A learning course from Microsoft that introduces MCP with lessons and code samples, suited to readers who are new to the protocol.
Slide: A2AA2A (Agent2Agent)
An open protocol for agents to discover one another and exchange tasks. Agents publish capability cards that other agents can read. The deck uses it for the idea that agents talk to agents.
Slide: A2UIA2UI
A project for letting an agent describe a user interface declaratively, so the host application renders it with its own native components. The deck summarizes it as: the agent asks, the app draws.
Slide: AGENTS.mdAGENTS.md
A plain Markdown file that gives coding agents project-specific instructions, such as build steps and conventions. It works as the agent counterpart to a README.
GovernanceAAIF (Agentic AI Foundation)
An open governance body for the agent ecosystem, which hosts shared projects and standards so that no single vendor controls them.
GovernanceLF AI & Data Projects
The Linux Foundation's umbrella for open AI and data projects. It provides neutral governance for projects in the agent stack, including Docling.
Part 4 · Stack pieces
3. Stack pieces
Components that fill specific roles in the architecture: how the standards work together and how context is engineered.
agentgateway
An open-source gateway that sits in front of agent traffic, handling MCP and A2A connections. It enforces security and permissions and records logs.
Docling
An open-source document parser that converts PDFs and other files into structured formats for LLMs and RAG. In the deck, it reads supplier PDF invoices. It is an LF AI & Data project.
pgvector
A PostgreSQL extension that stores and searches vector embeddings, with both exact and approximate nearest-neighbor indexes. It lets teams keep vector search inside an existing database.
LangChain
A framework of components for LLM applications, with integrations for models, tools, retrievers, and agents. It is the framework the deck names alongside LangGraph.
Model selection
4. Model selection and benchmarks
Places to compare model quality, speed and price before choosing a model.
Arena leaderboardarena.ai/leaderboard
A public ranking built from blind, head-to-head votes on model answers, shown as Elo-style scores. It reflects human preference more than fixed test scores.
OpenRouter benchmarksopenrouter.ai
Benchmark and usage information on OpenRouter model pages, which also list the providers that serve each model. The linked page is for one specific model.
Artificial Analysisartificialanalysis.ai
An independent benchmarking site that compares models on quality, speed and price across providers. It is useful for cost-versus-quality decisions.
Prompt engineering
5. Prompt engineering and system prompts
Guides and collections for writing prompts, and for studying how production tools structure their system prompts.
Getting the most out of Claude Opus 5.5claude.dev/blog
A blog post with practical tips for prompting the Claude Opus 5.5 model. It was saved from the project notes and was not reviewed in detail for this page.
Claude prompting best practicesplatform.claude.com
Anthropic's official guide to prompting Claude. It covers clear instructions, examples, structure, and how to ask for step-by-step reasoning.
System prompts and models of AI toolsgithub.com/x1xhlol
A GitHub collection of system prompts from popular AI tools, useful for studying how real products set behavior, tool use and guardrails.
prompts.chatgithub.com/f/prompts.chat
A community project for sharing, browsing and reusing prompts, with a GitHub repository behind it.
NVIDIA NeMo organizationgithub.com/NVIDIA-NeMo
The GitHub organization that hosts NVIDIA's NeMo projects. The NeMo Guardrails repository is linked in section 1.
BenchLMbenchlm.ai
A site that compares LLM benchmark scores. The link was taken from a Google redirect, and the destination was not verified.
Further reading
6. Books
Longer reading on building agents and prompting LLMs.
An Illustrated Guide to AI Agents
Listed under this title in the references file. The linked Amazon page is for a book titled Hands-On Large Language Models, so the link or title needs to be checked before sharing.
Build an AI Agent (From Scratch)
By Jungjun Hur and Younghee Song, published by Manning. It walks through building an agent step by step with code, covering the core pieces of tool use, memory and reasoning.
Prompt Engineering for LLMs
An O'Reilly book (ISBN 9781098153427) on designing prompts for large language models. It is a reference for prompt design and testing, and it fits with the prompt engineering section above. The page could not be opened for this entry, so the author and summary are unverified.