AI Safety Team Structure and Roles

As the number of AI agents in an organization grows, safety cannot remain a side responsibility of individual developers. It needs dedicated ownership. The structure of that ownership depends on the organization's size and the number of agents in production.

Small Teams (1 to 3 agents)

A dedicated safety function is not necessary at this scale. Instead, assign safety responsibilities to existing roles:

Agent developer writes and maintains the safety policy for their agent. They run red team tests before each deployment and monitor the agent's behavior.

Engineering lead reviews safety policies before deployment, approves changes to approval workflows, and serves as the escalation point for safety incidents.

On-call engineer includes AI agent safety incidents in the standard on-call rotation. The incident response runbook covers agent-specific scenarios.

Medium Teams (4 to 15 agents)

At this scale, a part-time safety role becomes necessary.

AI Safety Engineer (part-time or rotational) is responsible for:

• Maintaining organization-wide policy templates
• Reviewing agent-specific policies before deployment
• Conducting quarterly safety reviews
• Managing the red team exercise schedule
• Analyzing incident trends and recommending systemic improvements

This role can rotate among senior engineers on a quarterly basis to build safety expertise across the team.

Large Teams (15+ agents)

Dedicated safety staffing is warranted.

AI Safety Lead sets the safety strategy, defines standards, and represents the safety function to leadership and regulators. Reports to the CTO or VP of Engineering.

Safety Engineers (1 per 10 to 15 agents) handle day-to-day policy management, incident investigation, scanner tuning, and monitoring configuration.

Red Team Specialist conducts adversarial testing across all agents. Maintains the attack library and develops new test scenarios.

Compliance Analyst maps regulatory requirements to technical controls, prepares audit documentation, and tracks regulatory changes.

Cross-Cutting Responsibilities

Regardless of size, these functions must be covered:

• Policy review: Someone other than the policy author reviews every policy before deployment
• Incident response: A named person is responsible for safety incidents during each shift
• Audit trail verification: Regular integrity checks on the cryptographic audit trail
• Regulatory tracking: Awareness of relevant regulatory developments

Scale the structure to match your needs. Overbuilding the safety function creates bureaucracy. Underbuilding it creates risk. The right size is the smallest team that can consistently execute all cross-cutting responsibilities.