Designing a Safety-Critical AI System for Crisis Response

Save the Children | US + Kenya | AWS Partner
Designing a Safety-Critical AI System for Crisis Response

I defined a context-aware AI system that delivers real-time guidance during emergencies, where incorrect responses can lead to real-world harm.

Most AI systems are optimized for fluency and completion. In crisis environments, those priorities are wrong. This work reframed the problem from a chatbot into a safety-critical decision-support system, where the goal is not to answer every question, but to ensure the system behaves safely under uncertainty.

Overview

My Contribution

I led the definition of how this system operates across context, risk, and infrastructure constraints. The work moved beyond interface design into system behavior—defining how the AI interprets ambiguous inputs, retrieves knowledge, responds under uncertainty, and escalates when it cannot safely proceed.

A central challenge was translating real-world complexity into structured system logic. This included modeling user context, grounding responses in verified child-protection knowledge, and defining thresholds where the system defers to human support. The outcome was not a feature set, but a clear specification of how the system should behave when it is incomplete, uncertain, or at risk of being wrong.

AI Design
Research

The Problem

Image Caption: Example of real-world context shaping system requirements (caregiver navigating post-crisis trauma and resource gaps)

Crisis environments expose a fundamental mismatch in how most AI systems are designed.

Inputs are often incomplete, emotional, and context-dependent. Infrastructure is unreliable. Users range from caregivers to teachers to public officials, each with different needs. At the same time, the cost of incorrect guidance is significantly higher.

Standard approaches fail because they assume well-formed inputs and prioritize response completion. In this context, that leads to systems that sound helpful but cannot be trusted.

The problem is not generating better answers. It is designing systems that know when not to answer.

Defining the System

The system was designed as a coordinated set of behaviors rather than a single interface. It interprets who the user is, what stage of crisis they are in, and how urgent the situation is before generating a response.

Image caption: System architecture showing context interpretation, knowledge retrieval, and human-in-the-loop escalation

Instead of relying on generative output, it retrieves guidance from verified, expert-reviewed sources. This constrains the system, but ensures that responses remain grounded and auditable.

Equally important, the system establishes clear boundaries. When uncertainty or risk exceeds defined thresholds, it escalates to human support rather than attempting to complete the interaction. This shifts the role of AI from authority to support within a broader human system.

Because the system needed to function in real-world conditions, delivery extends beyond a single interface. It supports SMS, low-bandwidth environments, and use through community intermediaries, allowing access even when traditional digital channels fail.

Key Decisions

The system is shaped by a set of deliberate tradeoffs. Retrieval was chosen over generative flexibility to ensure accuracy and verifiability. Escalation was treated as a core behavior, even though it interrupts the interaction, because completion without certainty introduces risk. Context-aware response logic was introduced despite added complexity, because generic responses fail in crisis scenarios. Delivery was optimized for SMS and low-bandwidth environments, prioritizing access over interface richness.

Across these decisions, the pattern is consistent: prioritizing safety and real-world reliability over completeness and polish.

Handling Failure

A central part of the design was defining how the system behaves when it is wrong, uncertain, or incomplete.

Rather than masking these conditions, the system responds conservatively, avoids speculation, and escalates when necessary. It surfaces limitations instead of hiding them behind confident language. This reflects a broader principle: in high-risk systems, failure should be explicitly managed, not implicitly concealed.

The system is designed to fail safely, not optimally.

Example of System Behavior

When a caregiver asks, “My child is scared after the flood and won’t sleep alone,” the system interprets both the user’s role and the post-crisis context. It retrieves trauma-informed guidance from verified sources and delivers simple, actionable steps. If the situation exceeds what guidance can safely address, it introduces escalation to human support.

This is not a generated answer. It is a defined response pattern shaped by context, risk, and verified knowledge.

Driving Alignment

This work required changing how the problem was understood across the team. The initial framing centered on building a chatbot. I pushed to reframe it as a system, which shifted both design and technical direction.

I worked across research, product, engineering, and domain experts to align on this model. Research insights were translated into system logic, and domain expertise was embedded into how knowledge is structured and retrieved. This alignment enabled the team to move from abstract intent to implementable behavior.

Executive Alignment & Influence

This work was presented to the Save the Children Tech & Data Impact Council, where it informed how leadership evaluated the role of AI in crisis response systems.

A key part of my contribution was anchoring these discussions in user reality—ensuring that decisions remained grounded in what families, teachers, and community leaders actually need during emergencies.

This introduced a consistent tension between ideal system behavior and MVP constraints. Rather than optimizing for speed or completeness, the work influenced how tradeoffs were evaluated, raising the standard for safety, reliability, and contextual relevance in the system.

Organizational Impact & Adoption

This work influenced how Save the Children evaluates and designs AI systems for crisis response, shifting the approach from chatbot-based interactions to safety-critical decision-support systems.
It introduced a model for structuring and distributing knowledge through a context-aware system, enabling more consistent and reliable guidance across crisis scenarios.
The system also established a feedback loop where real user interactions inform content development, improving the organization’s ability to respond to evolving needs.
By integrating into existing community and inter-agency workflows, the system extends beyond a standalone tool into a scalable support layer within broader crisis response ecosystems.

Responsible AI

Responsible AI is implemented through system behavior rather than principle statements. Responses are grounded in verified knowledge, high-risk scenarios trigger escalation, and the system avoids reliance on sensitive personal data.

This reflects a broader view: responsible AI is not a layer added after the fact. It is a function of how the system behaves under real conditions.

Broader Implication

This work highlights a gap in how AI systems are typically designed.

Most systems optimize for interaction quality—fluency, responsiveness, and completion. In high-risk environments, these are secondary concerns. The primary requirement is that the system behaves predictably, safely, and within its limits.

This requires shifting from designing interfaces to designing behavior under constraint, and from optimizing answers to defining when not to answer.

Closing

I focus on defining how AI systems operate in environments where ambiguity, risk, and real-world constraints are the norm.

If you are building systems where correctness and safety matter more than coverage, I can help define how they should behave.

Let’s collaborate

Related case studies

Designing an AI Capability System for Real-World Adoption
Hands-on workshops
Prompt engineering
Ethical AI principles
Collaborative learning
Designing an Inclusive Payment System for Global Commerce
Research
Strategy
Concept Design
Designing an Inbound Visibility System for Seller Operations at Scale
Research
Strategy
UX/UI Design
Prototyping
Usability Testing