Raahat Scenario Harness

Curated conversation replay against the real runtime with stepwise diagnostics, readable deltas, and deterministic checks.

Eye Trauma

medical_eye_trauma · medical · 1/1 turns executed

Scenario Summary

IDmedical_eye_trauma
TitleEye Trauma
Categorymedical
Turns1/1
Run Statecompleted
urgent eye

Seed State

UserUnknown user
Phone919900000006
Seed Workflow
Seed State Machine

Current Runtime State

Routeconcierge
Workflowconcierge
State Machine
Stagecompleted

Checks And Lifecycle

Expectations Passed0
Expectations Failed1
Heuristic Warnings0
Stage advanced
Final assessmentLikely follow-up needed: at least one explicit check failed during this run.

Final Outcome

Completed

Completed vs partial run state.

Final Route

concierge

Current route at the end of the replay.

Final Workflow / State

concierge

Workflow preferred; state machine shown if no workflow is active.

Final Stage

completed

Resolved from workflow progress or active snapshot.

Expectation Totals

Pass 0 Fail 1

Heuristic Totals

Warnings 0

Missing Fields

No unresolved workflow fields.

Lifecycle Events Seen

Stage advanced
Turn 1
concierge concierge completed turn_interpretation_domain
Stage advanced New slots filled Memory updated Expectation failure
LIKELY BUG
User
Her eye is injured and swollen after a fight
Assistant
I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.

What Changed This Turn

  • Workflow changed none -> concierge.
  • Stage advanced none -> completed.
  • New slot filled: stage=completed.
  • New slot filled: topic=service_discovery.
  • New slot filled: user_goal=information.
  • Memory updated with last_non_medical_domain.
  • Snapshot version incremented to 1.

Expectation And Heuristic Results

Expectations
LIKELY BUG: route_type expected 'medical_triage' but got 'concierge'
Routing And Interpretation
{
  "confidence": 0.35,
  "reason_codes": [
    "turn_interpretation_domain"
  ],
  "route_type": "concierge",
  "workflow_name": "concierge"
}
{
  "confidence": 0.35,
  "entities": {},
  "is_vague": false,
  "primary_domain": "general",
  "reason": "deterministic interpretation is ambiguous; llm fallback unavailable: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-proj-**************************************************************************************************************************************************************hhwA. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'code': 'invalid_api_key', 'param': None}, 'status': 401}",
  "secondary_domains": [],
  "should_continue_active_workflow": false,
  "should_preempt_active_workflow": false,
  "urgency_signal": "none"
}
Workflow Progress And Intents
{
  "missing_fields": [],
  "next_question": null,
  "notes": [],
  "stage": "completed",
  "state_payload": {
    "conversion_target_workflow": null,
    "last_question_key": null,
    "location": null,
    "service_interest": null,
    "stage": "completed",
    "subtopic": null,
    "topic": "service_discovery",
    "user_goal": "information"
  },
  "status": "completed",
  "summary": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
  "superseded_by": null,
  "workflow_name": "concierge"
}
[
  {
    "action_type": "recommend_service",
    "idempotency_key": "2af935ce0e5404d0",
    "metadata": {
      "missing_fields": [],
      "normalized_slots": {
        "location": null,
        "service_interest": null,
        "topic": "service_discovery",
        "user_goal": "information"
      },
      "snapshot_version": null,
      "workflow_name": "concierge",
      "workflow_stage": "completed"
    },
    "payload": {
      "location": null,
      "service_interest": null,
      "topic": "service_discovery",
      "user_goal": "information"
    },
    "priority": "normal",
    "reason": "concierge answered a service discovery or informational request",
    "requires_admin_review": false,
    "source_workflow": "concierge",
    "status": "proposed",
    "user_visible": true
  }
]
Conversation Before And After
{
  "active_state_machine": null,
  "active_workflow": null,
  "current_state": null,
  "entities": {},
  "messages": [],
  "phone": "919900000006",
  "service_actions": [],
  "started_at": "2026-03-16T22:32:48.204332+00:00",
  "triage_state": null,
  "updated_at": "2026-03-16T22:32:48.204410+00:00"
}
{
  "active_state_machine": null,
  "active_workflow": {
    "last_question_key": null,
    "missing_fields": [],
    "next_question": null,
    "stage": "completed",
    "started_at": "2026-03-16T22:32:48.510739+00:00",
    "state_payload": {
      "conversion_target_workflow": null,
      "last_question_key": null,
      "location": null,
      "service_interest": null,
      "stage": "completed",
      "subtopic": null,
      "topic": "service_discovery",
      "user_goal": "information"
    },
    "status": "completed",
    "summary": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
    "superseded_by": null,
    "updated_at": "2026-03-16T22:32:48.510742+00:00",
    "version": 1,
    "workflow_name": "concierge"
  },
  "current_state": "COMPLETED",
  "entities": {
    "last_non_medical_domain": "general"
  },
  "messages": [
    {
      "role": "user",
      "source": null,
      "text": "Her eye is injured and swollen after a fight",
      "timestamp": "2026-03-16T22:32:48.510781+00:00"
    },
    {
      "role": "assistant",
      "source": "B",
      "text": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
      "timestamp": "2026-03-16T22:32:48.510783+00:00"
    }
  ],
  "phone": "919900000006",
  "service_actions": [],
  "started_at": "2026-03-16T22:32:48.510778+00:00",
  "triage_state": null,
  "updated_at": "2026-03-16T22:32:48.510800+00:00"
}
Memory And Trace
{
  "active_state_machine": null,
  "backend": "inmemory",
  "conversation_key_used": "919900000006",
  "conversation_keys": [
    "919900000006",
    "9900000006"
  ],
  "current_state": null,
  "loaded": true,
  "ping_ok": null,
  "redis_enabled": false,
  "thread_message_count": 0,
  "ttl_seconds_remaining": null
}
{
  "active_state_machine": null,
  "backend": "inmemory",
  "conversation_key_used": "919900000006",
  "conversation_keys": [
    "919900000006",
    "9900000006"
  ],
  "current_state": "COMPLETED",
  "loaded": true,
  "ping_ok": null,
  "redis_enabled": false,
  "thread_message_count": 2,
  "ttl_seconds_remaining": null
}
[
  {
    "component": "brain",
    "payload": {
      "channel": "scenario_harness",
      "conversation_key": "919900000006",
      "message_text": "Her eye is injured and swollen after a fight",
      "normalized_phone": "919900000006",
      "phone": "919900000006"
    },
    "stage": "input",
    "summary": "Received normalized message envelope",
    "timestamp": "2026-03-16T22:32:48.219687+00:00"
  },
  {
    "component": "memory_store",
    "payload": {
      "active_state_machine": null,
      "backend": "inmemory",
      "current_state": null,
      "loaded": true,
      "message_count": 0,
      "storage_key": "919900000006"
    },
    "stage": "memory",
    "summary": "Loaded conversation memory",
    "timestamp": "2026-03-16T22:32:48.219691+00:00"
  },
  {
    "component": "brain",
    "payload": {
      "has_prior_triage_state": false,
      "prior_state_error": null
    },
    "stage": "state",
    "summary": "Parsed prior working state",
    "timestamp": "2026-03-16T22:32:48.219693+00:00"
  },
  {
    "component": "brain",
    "payload": {
      "compact_payload_keys": [
        "incoming_message"
      ],
      "has_conversation": true,
      "has_customer": false,
      "has_icp": false,
      "has_profile": false,
      "recent_import_count": 0,
      "storage_key": "919900000006"
    },
    "stage": "context",
    "summary": "Assembled SessionContext for the turn",
    "timestamp": "2026-03-16T22:32:48.219695+00:00"
  },
  {
    "component": "contact_resolver",
    "payload": {
      "confidence": 0.0,
      "customer_id": null,
      "display_name": null,
      "matched_phone": null,
      "status": "UNKNOWN"
    },
    "stage": "decision",
    "summary": "Resolved sender contact status",
    "timestamp": "2026-03-16T22:32:48.219699+00:00"
  },
  {
    "component": "turn_interpretation",
    "payload": {
      "active_workflow": null,
      "deterministic_interpretation": {
        "confidence": 0.35,
        "is_vague": false,
        "primary_domain": "general",
        "reason": "deterministic interpretation is ambiguous",
        "urgency_signal": "none"
      },
      "message": "Her eye is injured and swollen after a fight",
      "model": "gpt-5.4"
    },
    "stage": "llm_request",
    "summary": "Calling small interpretation model because deterministic interpretation was ambiguous",
    "timestamp": "2026-03-16T22:32:48.219898+00:00"
  },
  {
    "component": "turn_interpretation",
    "payload": {
      "confidence": 0.35,
      "contact_status": "UNKNOWN",
      "entities": {},
      "is_vague": false,
      "primary_domain": "general",
      "reason": "deterministic interpretation is ambiguous; llm fallback unavailable: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-proj-**************************************************************************************************************************************************************hhwA. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'code': 'invalid_api_key', 'param': None}, 'status': 401}",
      "secondary_domains": [],
      "should_continue_active_workflow": false,
      "should_preempt_active_workflow": false,
      "urgency_signal": "none"
    },
    "stage": "decision",
    "summary": "Fell back to deterministic interpretation because LLM-assisted interpretation was unavailable",
    "timestamp": "2026-03-16T22:32:48.510379+00:00"
  },
  {
    "component": "profile_context",
    "payload": {
      "phone": "919900000006"
    },
    "stage": "context",
    "summary": "No customer matched provided phone",
    "timestamp": "2026-03-16T22:32:48.510402+00:00"
  },
  {
    "component": "entry_policy",
    "payload": {
      "allow_identity_question": true,
      "bypass_onboarding": false,
      "defer_existing_workflow": false,
      "preempt_active_workflow": false,
      "reason": "unknown contact with a clear non-urgent request can allow one light identity question later",
      "run_welcome_orientation": false,
      "selected_posture": "light_onboarding"
    },
    "stage": "decision",
    "summary": "Determined entry posture for this turn",
    "timestamp": "2026-03-16T22:32:48.510404+00:00"
  },
  {
    "component": "intent_parser",
    "payload": {
      "continue_active_workflow": false,
      "primary_intent": "general_inquiry",
      "reason": "turn_interpretation_domain_route",
      "resolved_state_machine": "concierge",
      "workflow_source": "intent_match"
    },
    "stage": "decision",
    "summary": "Resolved primary intent and state machine",
    "timestamp": "2026-03-16T22:32:48.510485+00:00"
  },
  {
    "component": "context_builder",
    "payload": {
      "active_workflow": "concierge",
      "background_fact_count": 0,
      "current_step": null,
      "foreground_fact_count": 1,
      "prompt_family": "general",
      "purpose": "workflow_response"
    },
    "stage": "context",
    "summary": "Built workflow-response context slice for prompt assembly",
    "timestamp": "2026-03-16T22:32:48.510513+00:00"
  },
  {
    "component": "workflow",
    "payload": {
      "action_intents": [
        {
          "action_type": "recommend_service",
          "idempotency_key": "2af935ce0e5404d0",
          "metadata": {
            "missing_fields": [],
            "normalized_slots": {
              "location": null,
              "service_interest": null,
              "topic": "service_discovery",
              "user_goal": "information"
            },
            "snapshot_version": null,
            "workflow_name": "concierge",
            "workflow_stage": "completed"
          },
          "payload": {
            "location": null,
            "service_interest": null,
            "topic": "service_discovery",
            "user_goal": "information"
          },
          "priority": "normal",
          "reason": "concierge answered a service discovery or informational request",
          "requires_admin_review": false,
          "source_workflow": "concierge",
          "status": "proposed",
          "user_visible": true
        }
      ],
      "missing_fields": [],
      "next_question": null,
      "stage": "completed",
      "status": "completed",
      "summary": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
      "workflow_name": "concierge"
    },
    "stage": "decision",
    "summary": "Executed deterministic workflow handler",
    "timestamp": "2026-03-16T22:32:48.510765+00:00"
  },
  {
    "component": "memory_store",
    "payload": {
      "active_state_machine": null,
      "current_state": "COMPLETED",
      "message_count_after": 2,
      "storage_key": "919900000006"
    },
    "stage": "mutation",
    "summary": "Persisted updated conversation memory",
    "timestamp": "2026-03-16T22:32:48.511295+00:00"
  }
]