Raahat Scenario Harness

Turn 1

concierge concierge completed turn_interpretation_domain

Stage advanced New slots filled Memory updated Expectation failure

LIKELY BUG

User

My male cat is straining in the litter box and nothing is coming

Assistant

I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.

What Changed This Turn

Workflow changed none -> concierge.
Stage advanced none -> completed.
New slot filled: stage=completed.
New slot filled: topic=service_discovery.
New slot filled: user_goal=information.
Memory updated with last_non_medical_domain.
Snapshot version incremented to 1.

Expectation And Heuristic Results

Expectations

LIKELY BUG: route_type expected 'medical_triage' but got 'concierge'

Routing And Interpretation

{
  "confidence": 0.35,
  "reason_codes": [
    "turn_interpretation_domain"
  ],
  "route_type": "concierge",
  "workflow_name": "concierge"
}

{
  "confidence": 0.35,
  "entities": {},
  "is_vague": false,
  "primary_domain": "general",
  "reason": "deterministic interpretation is ambiguous; llm fallback unavailable: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-proj-**************************************************************************************************************************************************************hhwA. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'code': 'invalid_api_key', 'param': None}, 'status': 401}",
  "secondary_domains": [],
  "should_continue_active_workflow": false,
  "should_preempt_active_workflow": false,
  "urgency_signal": "none"
}

Workflow Progress And Intents

{
  "missing_fields": [],
  "next_question": null,
  "notes": [],
  "stage": "completed",
  "state_payload": {
    "conversion_target_workflow": null,
    "last_question_key": null,
    "location": null,
    "service_interest": null,
    "stage": "completed",
    "subtopic": null,
    "topic": "service_discovery",
    "user_goal": "information"
  },
  "status": "completed",
  "summary": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
  "superseded_by": null,
  "workflow_name": "concierge"
}

[
  {
    "action_type": "recommend_service",
    "idempotency_key": "2af935ce0e5404d0",
    "metadata": {
      "missing_fields": [],
      "normalized_slots": {
        "location": null,
        "service_interest": null,
        "topic": "service_discovery",
        "user_goal": "information"
      },
      "snapshot_version": null,
      "workflow_name": "concierge",
      "workflow_stage": "completed"
    },
    "payload": {
      "location": null,
      "service_interest": null,
      "topic": "service_discovery",
      "user_goal": "information"
    },
    "priority": "normal",
    "reason": "concierge answered a service discovery or informational request",
    "requires_admin_review": false,
    "source_workflow": "concierge",
    "status": "proposed",
    "user_visible": true
  }
]

Conversation Before And After

{
  "active_state_machine": null,
  "active_workflow": null,
  "current_state": null,
  "entities": {},
  "messages": [],
  "phone": "919900000005",
  "service_actions": [],
  "started_at": "2026-03-16T22:34:06.190779+00:00",
  "triage_state": null,
  "updated_at": "2026-03-16T22:34:06.190784+00:00"
}

{
  "active_state_machine": null,
  "active_workflow": {
    "last_question_key": null,
    "missing_fields": [],
    "next_question": null,
    "stage": "completed",
    "started_at": "2026-03-16T22:34:06.498275+00:00",
    "state_payload": {
      "conversion_target_workflow": null,
      "last_question_key": null,
      "location": null,
      "service_interest": null,
      "stage": "completed",
      "subtopic": null,
      "topic": "service_discovery",
      "user_goal": "information"
    },
    "status": "completed",
    "summary": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
    "superseded_by": null,
    "updated_at": "2026-03-16T22:34:06.498277+00:00",
    "version": 1,
    "workflow_name": "concierge"
  },
  "current_state": "COMPLETED",
  "entities": {
    "last_non_medical_domain": "general"
  },
  "messages": [
    {
      "role": "user",
      "source": null,
      "text": "My male cat is straining in the litter box and nothing is coming",
      "timestamp": "2026-03-16T22:34:06.498310+00:00"
    },
    {
      "role": "assistant",
      "source": "B",
      "text": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
      "timestamp": "2026-03-16T22:34:06.498312+00:00"
    }
  ],
  "phone": "919900000005",
  "service_actions": [],
  "started_at": "2026-03-16T22:34:06.498307+00:00",
  "triage_state": null,
  "updated_at": "2026-03-16T22:34:06.498327+00:00"
}

Memory And Trace

{
  "active_state_machine": null,
  "backend": "inmemory",
  "conversation_key_used": "919900000005",
  "conversation_keys": [
    "919900000005",
    "9900000005"
  ],
  "current_state": null,
  "loaded": true,
  "ping_ok": null,
  "redis_enabled": false,
  "thread_message_count": 0,
  "ttl_seconds_remaining": null
}

{
  "active_state_machine": null,
  "backend": "inmemory",
  "conversation_key_used": "919900000005",
  "conversation_keys": [
    "919900000005",
    "9900000005"
  ],
  "current_state": "COMPLETED",
  "loaded": true,
  "ping_ok": null,
  "redis_enabled": false,
  "thread_message_count": 2,
  "ttl_seconds_remaining": null
}

[
  {
    "component": "brain",
    "payload": {
      "channel": "scenario_harness",
      "conversation_key": "919900000005",
      "message_text": "My male cat is straining in the litter box and nothing is coming",
      "normalized_phone": "919900000005",
      "phone": "919900000005"
    },
    "stage": "input",
    "summary": "Received normalized message envelope",
    "timestamp": "2026-03-16T22:34:06.210030+00:00"
  },
  {
    "component": "memory_store",
    "payload": {
      "active_state_machine": null,
      "backend": "inmemory",
      "current_state": null,
      "loaded": true,
      "message_count": 0,
      "storage_key": "919900000005"
    },
    "stage": "memory",
    "summary": "Loaded conversation memory",
    "timestamp": "2026-03-16T22:34:06.210035+00:00"
  },
  {
    "component": "brain",
    "payload": {
      "has_prior_triage_state": false,
      "prior_state_error": null
    },
    "stage": "state",
    "summary": "Parsed prior working state",
    "timestamp": "2026-03-16T22:34:06.210037+00:00"
  },
  {
    "component": "brain",
    "payload": {
      "compact_payload_keys": [
        "incoming_message"
      ],
      "has_conversation": true,
      "has_customer": false,
      "has_icp": false,
      "has_profile": false,
      "recent_import_count": 0,
      "storage_key": "919900000005"
    },
    "stage": "context",
    "summary": "Assembled SessionContext for the turn",
    "timestamp": "2026-03-16T22:34:06.210041+00:00"
  },
  {
    "component": "contact_resolver",
    "payload": {
      "confidence": 0.0,
      "customer_id": null,
      "display_name": null,
      "matched_phone": null,
      "status": "UNKNOWN"
    },
    "stage": "decision",
    "summary": "Resolved sender contact status",
    "timestamp": "2026-03-16T22:34:06.210046+00:00"
  },
  {
    "component": "turn_interpretation",
    "payload": {
      "active_workflow": null,
      "deterministic_interpretation": {
        "confidence": 0.35,
        "is_vague": false,
        "primary_domain": "general",
        "reason": "deterministic interpretation is ambiguous",
        "urgency_signal": "none"
      },
      "message": "My male cat is straining in the litter box and nothing is coming",
      "model": "gpt-5.4"
    },
    "stage": "llm_request",
    "summary": "Calling small interpretation model because deterministic interpretation was ambiguous",
    "timestamp": "2026-03-16T22:34:06.210332+00:00"
  },
  {
    "component": "turn_interpretation",
    "payload": {
      "confidence": 0.35,
      "contact_status": "UNKNOWN",
      "entities": {},
      "is_vague": false,
      "primary_domain": "general",
      "reason": "deterministic interpretation is ambiguous; llm fallback unavailable: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-proj-**************************************************************************************************************************************************************hhwA. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'code': 'invalid_api_key', 'param': None}, 'status': 401}",
      "secondary_domains": [],
      "should_continue_active_workflow": false,
      "should_preempt_active_workflow": false,
      "urgency_signal": "none"
    },
    "stage": "decision",
    "summary": "Fell back to deterministic interpretation because LLM-assisted interpretation was unavailable",
    "timestamp": "2026-03-16T22:34:06.497940+00:00"
  },
  {
    "component": "profile_context",
    "payload": {
      "phone": "919900000005"
    },
    "stage": "context",
    "summary": "No customer matched provided phone",
    "timestamp": "2026-03-16T22:34:06.497963+00:00"
  },
  {
    "component": "entry_policy",
    "payload": {
      "allow_identity_question": true,
      "bypass_onboarding": false,
      "defer_existing_workflow": false,
      "preempt_active_workflow": false,
      "reason": "unknown contact with a clear non-urgent request can allow one light identity question later",
      "run_welcome_orientation": false,
      "selected_posture": "light_onboarding"
    },
    "stage": "decision",
    "summary": "Determined entry posture for this turn",
    "timestamp": "2026-03-16T22:34:06.497965+00:00"
  },
  {
    "component": "intent_parser",
    "payload": {
      "continue_active_workflow": false,
      "primary_intent": "general_inquiry",
      "reason": "turn_interpretation_domain_route",
      "resolved_state_machine": "concierge",
      "workflow_source": "intent_match"
    },
    "stage": "decision",
    "summary": "Resolved primary intent and state machine",
    "timestamp": "2026-03-16T22:34:06.498046+00:00"
  },
  {
    "component": "context_builder",
    "payload": {
      "active_workflow": "concierge",
      "background_fact_count": 0,
      "current_step": null,
      "foreground_fact_count": 1,
      "prompt_family": "general",
      "purpose": "workflow_response"
    },
    "stage": "context",
    "summary": "Built workflow-response context slice for prompt assembly",
    "timestamp": "2026-03-16T22:34:06.498074+00:00"
  },
  {
    "component": "workflow",
    "payload": {
      "action_intents": [
        {
          "action_type": "recommend_service",
          "idempotency_key": "2af935ce0e5404d0",
          "metadata": {
            "missing_fields": [],
            "normalized_slots": {
              "location": null,
              "service_interest": null,
              "topic": "service_discovery",
              "user_goal": "information"
            },
            "snapshot_version": null,
            "workflow_name": "concierge",
            "workflow_stage": "completed"
          },
          "payload": {
            "location": null,
            "service_interest": null,
            "topic": "service_discovery",
            "user_goal": "information"
          },
          "priority": "normal",
          "reason": "concierge answered a service discovery or informational request",
          "requires_admin_review": false,
          "source_workflow": "concierge",
          "status": "proposed",
          "user_visible": true
        }
      ],
      "missing_fields": [],
      "next_question": null,
      "stage": "completed",
      "status": "completed",
      "summary": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
      "workflow_name": "concierge"
    },
    "stage": "decision",
    "summary": "Executed deterministic workflow handler",
    "timestamp": "2026-03-16T22:34:06.498295+00:00"
  },
  {
    "component": "memory_store",
    "payload": {
      "active_state_machine": null,
      "current_state": "COMPLETED",
      "message_count_after": 2,
      "storage_key": "919900000005"
    },
    "stage": "mutation",
    "summary": "Persisted updated conversation memory",
    "timestamp": "2026-03-16T22:34:06.498604+00:00"
  }
]

Turn 2

concierge concierge completed turn_interpretation_domain

Stayed on concierge Expectation failure

LIKELY BUG

User

This has been happening since this morning

Assistant

I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.

What Changed This Turn

Active workflow remained concierge.
Snapshot version incremented to 2.

Expectation And Heuristic Results

Expectations

LIKELY BUG: route_type expected 'medical_triage' but got 'concierge'

Routing And Interpretation

{
  "confidence": 0.8,
  "reason_codes": [
    "turn_interpretation_domain"
  ],
  "route_type": "concierge",
  "workflow_name": "concierge"
}

{
  "confidence": 0.8,
  "entities": {
    "time_text": "morning"
  },
  "is_vague": false,
  "primary_domain": "general",
  "reason": "looks like a follow-up to the active workflow",
  "secondary_domains": [],
  "should_continue_active_workflow": true,
  "should_preempt_active_workflow": false,
  "urgency_signal": "none"
}

Workflow Progress And Intents

{
  "missing_fields": [],
  "next_question": null,
  "notes": [],
  "stage": "completed",
  "state_payload": {
    "conversion_target_workflow": null,
    "last_question_key": null,
    "location": null,
    "service_interest": null,
    "stage": "completed",
    "subtopic": null,
    "topic": "service_discovery",
    "user_goal": "information"
  },
  "status": "completed",
  "summary": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
  "superseded_by": null,
  "workflow_name": "concierge"
}

[
  {
    "action_type": "recommend_service",
    "idempotency_key": "2af935ce0e5404d0",
    "metadata": {
      "missing_fields": [],
      "normalized_slots": {
        "location": null,
        "service_interest": null,
        "topic": "service_discovery",
        "user_goal": "information"
      },
      "snapshot_version": null,
      "workflow_name": "concierge",
      "workflow_stage": "completed"
    },
    "payload": {
      "location": null,
      "service_interest": null,
      "topic": "service_discovery",
      "user_goal": "information"
    },
    "priority": "normal",
    "reason": "concierge answered a service discovery or informational request",
    "requires_admin_review": false,
    "source_workflow": "concierge",
    "status": "proposed",
    "user_visible": true
  }
]

Conversation Before And After

{
  "active_state_machine": null,
  "active_workflow": {
    "last_question_key": null,
    "missing_fields": [],
    "next_question": null,
    "stage": "completed",
    "started_at": "2026-03-16T22:34:06.498275+00:00",
    "state_payload": {
      "conversion_target_workflow": null,
      "last_question_key": null,
      "location": null,
      "service_interest": null,
      "stage": "completed",
      "subtopic": null,
      "topic": "service_discovery",
      "user_goal": "information"
    },
    "status": "completed",
    "summary": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
    "superseded_by": null,
    "updated_at": "2026-03-16T22:34:06.498277+00:00",
    "version": 1,
    "workflow_name": "concierge"
  },
  "current_state": "COMPLETED",
  "entities": {
    "last_non_medical_domain": "general"
  },
  "messages": [
    {
      "role": "user",
      "source": null,
      "text": "My male cat is straining in the litter box and nothing is coming",
      "timestamp": "2026-03-16T22:34:06.498310+00:00"
    },
    {
      "role": "assistant",
      "source": "B",
      "text": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
      "timestamp": "2026-03-16T22:34:06.498312+00:00"
    }
  ],
  "phone": "919900000005",
  "service_actions": [],
  "started_at": "2026-03-16T22:34:06.498307+00:00",
  "triage_state": null,
  "updated_at": "2026-03-16T22:34:06.498327+00:00"
}

{
  "active_state_machine": null,
  "active_workflow": {
    "last_question_key": null,
    "missing_fields": [],
    "next_question": null,
    "stage": "completed",
    "started_at": "2026-03-16T22:34:06.498275+00:00",
    "state_payload": {
      "conversion_target_workflow": null,
      "last_question_key": null,
      "location": null,
      "service_interest": null,
      "stage": "completed",
      "subtopic": null,
      "topic": "service_discovery",
      "user_goal": "information"
    },
    "status": "completed",
    "summary": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
    "superseded_by": null,
    "updated_at": "2026-03-16T22:34:06.502128+00:00",
    "version": 2,
    "workflow_name": "concierge"
  },
  "current_state": "COMPLETED",
  "entities": {
    "last_non_medical_domain": "general"
  },
  "messages": [
    {
      "role": "user",
      "source": null,
      "text": "My male cat is straining in the litter box and nothing is coming",
      "timestamp": "2026-03-16T22:34:06.498310+00:00"
    },
    {
      "role": "assistant",
      "source": "B",
      "text": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
      "timestamp": "2026-03-16T22:34:06.498312+00:00"
    },
    {
      "role": "user",
      "source": null,
      "text": "This has been happening since this morning",
      "timestamp": "2026-03-16T22:34:06.502157+00:00"
    },
    {
      "role": "assistant",
      "source": "B",
      "text": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
      "timestamp": "2026-03-16T22:34:06.502158+00:00"
    }
  ],
  "phone": "919900000005",
  "service_actions": [],
  "started_at": "2026-03-16T22:34:06.498307+00:00",
  "triage_state": null,
  "updated_at": "2026-03-16T22:34:06.502172+00:00"
}

Memory And Trace

{
  "active_state_machine": null,
  "backend": "inmemory",
  "conversation_key_used": "919900000005",
  "conversation_keys": [
    "919900000005",
    "9900000005"
  ],
  "current_state": "COMPLETED",
  "loaded": true,
  "ping_ok": null,
  "redis_enabled": false,
  "thread_message_count": 2,
  "ttl_seconds_remaining": null
}

{
  "active_state_machine": null,
  "backend": "inmemory",
  "conversation_key_used": "919900000005",
  "conversation_keys": [
    "919900000005",
    "9900000005"
  ],
  "current_state": "COMPLETED",
  "loaded": true,
  "ping_ok": null,
  "redis_enabled": false,
  "thread_message_count": 4,
  "ttl_seconds_remaining": null
}

[
  {
    "component": "brain",
    "payload": {
      "channel": "scenario_harness",
      "conversation_key": "919900000005",
      "message_text": "This has been happening since this morning",
      "normalized_phone": "919900000005",
      "phone": "919900000005"
    },
    "stage": "input",
    "summary": "Received normalized message envelope",
    "timestamp": "2026-03-16T22:34:06.501682+00:00"
  },
  {
    "component": "memory_store",
    "payload": {
      "active_state_machine": null,
      "backend": "inmemory",
      "current_state": "COMPLETED",
      "loaded": true,
      "message_count": 2,
      "storage_key": "919900000005"
    },
    "stage": "memory",
    "summary": "Loaded conversation memory",
    "timestamp": "2026-03-16T22:34:06.501686+00:00"
  },
  {
    "component": "brain",
    "payload": {
      "has_prior_triage_state": false,
      "prior_state_error": null
    },
    "stage": "state",
    "summary": "Parsed prior working state",
    "timestamp": "2026-03-16T22:34:06.501688+00:00"
  },
  {
    "component": "brain",
    "payload": {
      "compact_payload_keys": [
        "incoming_message"
      ],
      "has_conversation": true,
      "has_customer": false,
      "has_icp": false,
      "has_profile": false,
      "recent_import_count": 0,
      "storage_key": "919900000005"
    },
    "stage": "context",
    "summary": "Assembled SessionContext for the turn",
    "timestamp": "2026-03-16T22:34:06.501691+00:00"
  },
  {
    "component": "contact_resolver",
    "payload": {
      "confidence": 0.0,
      "customer_id": null,
      "display_name": null,
      "matched_phone": null,
      "status": "UNKNOWN"
    },
    "stage": "decision",
    "summary": "Resolved sender contact status",
    "timestamp": "2026-03-16T22:34:06.501695+00:00"
  },
  {
    "component": "turn_interpretation",
    "payload": {
      "confidence": 0.8,
      "contact_status": "UNKNOWN",
      "entities": {
        "time_text": "morning"
      },
      "is_vague": false,
      "primary_domain": "general",
      "reason": "looks like a follow-up to the active workflow",
      "secondary_domains": [],
      "should_continue_active_workflow": true,
      "should_preempt_active_workflow": false,
      "urgency_signal": "none"
    },
    "stage": "decision",
    "summary": "Resolved turn interpretation deterministically",
    "timestamp": "2026-03-16T22:34:06.501904+00:00"
  },
  {
    "component": "profile_context",
    "payload": {
      "phone": "919900000005"
    },
    "stage": "context",
    "summary": "No customer matched provided phone",
    "timestamp": "2026-03-16T22:34:06.501909+00:00"
  },
  {
    "component": "entry_policy",
    "payload": {
      "allow_identity_question": false,
      "bypass_onboarding": true,
      "defer_existing_workflow": false,
      "preempt_active_workflow": false,
      "reason": "existing active workflow should continue unless urgency preempts it",
      "run_welcome_orientation": false,
      "selected_posture": "continue_active"
    },
    "stage": "decision",
    "summary": "Determined entry posture for this turn",
    "timestamp": "2026-03-16T22:34:06.501911+00:00"
  },
  {
    "component": "intent_parser",
    "payload": {
      "continue_active_workflow": false,
      "primary_intent": "general_inquiry",
      "reason": "turn_interpretation_domain_route",
      "resolved_state_machine": "concierge",
      "workflow_source": "intent_match"
    },
    "stage": "decision",
    "summary": "Resolved primary intent and state machine",
    "timestamp": "2026-03-16T22:34:06.501954+00:00"
  },
  {
    "component": "context_builder",
    "payload": {
      "active_workflow": "concierge",
      "background_fact_count": 0,
      "current_step": "COMPLETED",
      "foreground_fact_count": 2,
      "prompt_family": "general",
      "purpose": "workflow_response"
    },
    "stage": "context",
    "summary": "Built workflow-response context slice for prompt assembly",
    "timestamp": "2026-03-16T22:34:06.501978+00:00"
  },
  {
    "component": "workflow",
    "payload": {
      "action_intents": [
        {
          "action_type": "recommend_service",
          "idempotency_key": "2af935ce0e5404d0",
          "metadata": {
            "missing_fields": [],
            "normalized_slots": {
              "location": null,
              "service_interest": null,
              "topic": "service_discovery",
              "user_goal": "information"
            },
            "snapshot_version": null,
            "workflow_name": "concierge",
            "workflow_stage": "completed"
          },
          "payload": {
            "location": null,
            "service_interest": null,
            "topic": "service_discovery",
            "user_goal": "information"
          },
          "priority": "normal",
          "reason": "concierge answered a service discovery or informational request",
          "requires_admin_review": false,
          "source_workflow": "concierge",
          "status": "proposed",
          "user_visible": true
        }
      ],
      "missing_fields": [],
      "next_question": null,
      "stage": "completed",
      "status": "completed",
      "summary": "I can help with boarding, transport, clinic guidance, and general cat-care planning. Tell me which part you need help with and I’ll route it properly.",
      "workflow_name": "concierge"
    },
    "stage": "decision",
    "summary": "Executed deterministic workflow handler",
    "timestamp": "2026-03-16T22:34:06.502144+00:00"
  },
  {
    "component": "memory_store",
    "payload": {
      "active_state_machine": null,
      "current_state": "COMPLETED",
      "message_count_after": 4,
      "storage_key": "919900000005"
    },
    "stage": "mutation",
    "summary": "Persisted updated conversation memory",
    "timestamp": "2026-03-16T22:34:06.502574+00:00"
  }
]

Raahat Scenario Harness

Male Cat Straining To Pee

Scenario Summary

Seed State

Current Runtime State

Checks And Lifecycle

Final Outcome

Final Route

Final Workflow / State

Final Stage

Expectation Totals

Heuristic Totals

Missing Fields

Lifecycle Events Seen

What Changed This Turn

Expectation And Heuristic Results

Expectations

What Changed This Turn

Expectation And Heuristic Results

Expectations