Coming soon. The get_user_memory API is not yet available. This page
describes the planned API, the full JSON schema of what will be returned, and
how user memory works internally so you can design your integration ahead of
time.
Overview
The current SDK gives your agent live, in-session context — the UI actions
a user is taking right now, plus a rolling LLM summary of those actions.
User Memory extends this with historical context: a cross-session
profile per user that records which product workflows they have completed, how
far they got on the ones they haven’t finished, and where they keep getting
stuck.
The core job of user memory is to answer one question for your copilot:
What does this specific user already know — and what don’t they know yet?
Without this, a copilot can only make generic suggestions. It might walk a
power user through basics they completed weeks ago, or recommend a feature they
already use every day. With user memory, the copilot can filter out everything
the user has mastered and focus exclusively on the gaps — the workflows they
haven’t started, the steps they keep missing, the features they’ve never
touched.
The role of user memory in a proactive copilot
A proactive copilot has three signals available to it (see
Proactive copilot):
| Signal | Answers |
|---|
| Real-time events | What is the user doing right now? |
| Golden paths | What should the user be doing at this point in their journey? |
| User memory | What has this user already done, and what have they struggled with? |
Golden path for feature X
− Workflows already mastered by this user ← user memory filters these out
− Workflows the user is actively stuck on ← user memory surfaces these first
= Relevant, targeted suggestion
- Don’t suggest what they already know. If
knowledge_state.mastered
contains the workflow, skip it entirely.
- Pick up where they left off. If
knowledge_state.in_progress shows an
incomplete workflow, surface the specific missed step — not the whole flow
from the beginning.
- Lead with untouched, high-value workflows.
knowledge_state.untouched
combined with the golden path tells you exactly which adoption gaps are worth
addressing next.
- Adapt if a struggle is recurring.
struggle_patterns signals that a
different approach is needed — not just a repeat of what already didn’t work.
From raw events to LLM-ready context
Before any of this analysis happens, Autoplay processes the raw browser events
your product emits. Raw events — clicks, page views, input changes — carry no
inherent meaning for a language model. A selector like #btn-connect-slack at
coordinates (412, 830) tells an LLM nothing useful.
Autoplay’s pipeline transforms these low-level signals into structured,
human-readable actions that fit efficiently inside a context window:
- Extraction — raw DOM events are normalised and deduplicated
- Labelling — each action is given a descriptive title and inferred intent
- Grouping — related actions are collected into a coherent session with an inferred goal
- Compression — redundant or low-signal events are dropped so only meaningful actions reach the model
The result is a compact, numbered list of actions that an LLM can reason over
directly — without the noise, repetition, or coordinate-level detail of the
original event stream. This format is intentionally optimised for context window
efficiency: a full user session that might produce hundreds of raw events is
distilled to a handful of labelled steps.
How it works: from sessions to memory
Every session a user has with your product is run through Autoplay’s AI
pipeline. The pipeline analyses the processed event stream and produces a
session analysis document — a structured JSON that captures what the user
did, which golden-path workflows they attempted, and how far they got.
User session
└── Raw events (clicks, page views, scrolls)
└── Autoplay event processor (extraction → labelling → compression)
└── LLM-ready action list
└── Autoplay AI pipeline (Gemini 2.5 Flash)
└── session_analysis.json
└── User Memory aggregate (cross-session)
What a session analysis looks like
Each session produces a document like this. This is the raw material that
feeds into user memory.
{
"meta": {
"id": "019cd7f7-aaaa-bbbb-cccc-ddddeeee0001",
"product_id": "499c2d1b-bb39-4d3f-a122-3dd39f38eb8a",
"duration_seconds": 142.5,
"analyzed_at": "2025-01-15T10:02:12Z",
"pipeline": {
"version": "2.0",
"model": { "provider": "vertex_ai", "name": "gemini-2.5-flash" }
}
},
"insights": {
"title": "User successfully set up Slack integration",
"summary": "The user navigated from the Dashboard to Settings > Integrations with clear intent to connect Slack. They found the Slack integration card, initiated the Connect flow, and completed the OAuth authorization. No friction points encountered.",
"intent": {
"goal": "Connect Slack integration to the product",
"actions": [
"Navigate to integration settings",
"Initiate Slack connection",
"Complete OAuth authorization"
]
},
"friction": []
},
"tags": {
"hesitation": "Low",
"knowledge": "High",
"product_sections": ["Settings", "Integrations"],
"intent": ["connect-integration"],
"friction": []
},
"tasks": {
"status": "completed",
"matched": [
{
"task_id": "task_gp_a1b2c3d4",
"task_title": "Setup Slack Integration",
"completion_pct": 85.0,
"matched_actions": [
{
"action_title": "Click Integrations Tab",
"expected_action_title": "Navigate to Integrations page",
"score": 0.88
},
{
"action_title": "Click Connect Slack",
"expected_action_title": "Click Connect on Slack card",
"score": 0.95
},
{
"action_title": "Click Authorize",
"expected_action_title": "Authorize OAuth connection",
"score": 0.91
}
],
"missed_steps": [
{
"expected_action_title": "Verify connection success message",
"expected_action_description": "Confirm the success message appears"
}
]
}
]
}
}
tasks section is the most important input to user memory. It records:
- Which golden-path workflow was attempted (
task_title)
- How much of it was completed (
completion_pct)
- Which steps were matched and at what similarity score
- Which steps were missed
How user memory shapes agent behaviour
When a session starts, the profile is serialised into a compact block and injected into the agent’s system prompt:
USER LEARNING PROFILE (cross-session memory):
- Returning user (4 sessions, last: 2026-04-07)
- Mastered workflows: connect_data_source, create_dashboard
- In progress:
• setup_slack_integration (2 attempts, best: 85% complete, last issue: "missed final step: verify connection success message")
• invite_teammates (1 attempt, best: 40% complete, last issue: "did not find the team settings page")
- Never started: set_up_alerts, configure_webhooks, export_to_csv
- Journey completion: 29% (2 of 7 workflows completed)
- Struggle pattern: oauth_flows → Has not confirmed success screens in 2 sessions involving OAuth
- Last session [2026-04-07]: Connect Slack integration — completed OAuth flow but missed the final confirmation step.
| User memory state | Copilot behaviour |
|---|
Workflow is in mastered | Do not suggest or explain it — the user already knows it |
Workflow is in in_progress | Surface the specific missed step, not the whole flow again |
Workflow is in untouched | Proactively introduce it when the user is in a relevant context |
Area appears in struggle_patterns | Change approach — repetition of what failed before is noise |
journey_completion.completion_rate is low | Prioritise high-value untouched workflows over minor refinements |
Planned use cases
| Use case | Signal used | Why it matters for relevance |
|---|
| Skip already-mastered content | knowledge_state.mastered | Removes noise — the user doesn’t need to hear about flows they’ve completed |
| Resume an incomplete workflow at the right step | knowledge_state.in_progress[].last_issue | Picks up exactly where they got stuck, instead of replaying the whole flow |
| Surface untouched high-value features | knowledge_state.untouched | Proactively fills adoption gaps the user hasn’t discovered yet |
| Show overall onboarding progress | journey_completion.completion_rate | Frames suggestions in context — “you’re 29% through, here’s what’s next” |
| Change approach on repeated struggles | struggle_patterns | Detects when repetition won’t help and triggers a different explanation |
| Match communication style | interaction_style.hesitation_level, prefers_concise | Ensures the delivery of suggestions fits how this user prefers to receive help |
| Cross-session RAG enrichment | recent_sessions[].intent + workflows_attempted | Grounds retrieval in this user’s history so results aren’t generic |
Stay updated
User Memory is in active development. Watch the GitHub repository or check back here for updates.
