Question detection

Overview

Question detection finds questions in your workflow. on screen, in speech, and in structured UI. and retrieves candidate answers from historical captures using time-filtered semantic search. You review suggestions in a question inbox: accept when a candidate fits, dismiss when it does not, or leave items pending while you decide.

The pipeline runs through background workers. On macOS, overlays can show answers near the source with live re-anchoring as windows move or resize. Behaviour is gated by a feature flag in builds that expose it.

Question sources

Source	What is analysed	Typical signals
OCR screen text	Visible text from captured frames	Written questions in editors, browsers, documents, chat panes
Audio transcriptions	Speech-to-text from capture windows	Spoken questions in meetings and solo work
UI accessibility snapshots	Accessibility trees where enabled	Labels, headings, and control text that encode questions or prompts

Together these cover most knowledge-work surfaces where questions appear explicitly; quality tracks your capture settings, app coverage, and transcription configuration.

How candidate answers are found

For each detected question, the system runs semantic search over historical captures with time bounds so suggestions favour recent, relevant context rather than decade-old noise. Answer similarity scoring ranks candidates so the inbox and overlays prioritise stronger matches.

Similarity and ranking

Embedding-based similarity compares the question’s representation to chunks of past captures. Higher scores indicate phrasing or topic overlap; the pipeline still applies time filters so a strong but ancient document does not displace a moderate match from last week without justification. Weak matches may be suppressed entirely to avoid false confidence.

Relationship to full-text search

Semantic retrieval complements keyword search: synonyms and rephrased questions still find related notes, while exact phrases benefit from lexical overlap in the underlying index where configured. The question worker focuses on short-lived, contextual suggestions rather than replacing the main search experience.

Why time filtering matters

Knowledge work is seasonal: last month’s project matters more than a generic match from years ago. Time filtering keeps suggestions aligned with how you are working now, while still allowing deep history when it scores highly.

Question lifecycle

Questions move through controlled states so you can triage without losing auditability:

State	Meaning
Detected / pending	A question was found; candidates may be loading or awaiting your review.
Resolved	You accepted a candidate answer or the system treated the thread as closed in line with product rules.
Dismissed	You rejected suggestions for this question; it leaves the active review queue.

The inbox is the canonical place to accept or dismiss; overlays on macOS are a presentation layer for timely answers, not a substitute for reviewing edge cases in the full list.

Deduplication

Without deduplication, small UI changes or repeated phrasing would spam the inbox. Overshow deduplicates using:

• A content hash for exact or near-exact repeats.
• Embedding similarity to collapse paraphrases and minor edits.
• An app-scoped window so the same phrase in unrelated apps can still surface separately when appropriate.
• A time window so legitimate revisits across days are not incorrectly suppressed.

The result is a steadier stream of distinct questions worth reviewing.

Native answer overlays (macOS)

Overshow renders answer overlays close to where the question appeared. Live binding re-anchors overlays when the source window moves or resizes, so the hint stays visually associated with the task.

Overlay behaviour

Behaviour	Detail
Queue	A limited number of overlays visible at once to avoid clutter.
Expiry	Queued items expire after a short window if not shown, keeping stale prompts from filling the stack.
Positioning	Prefers placement beside the source question region when geometry is known.
Live binding	As the source window moves or resizes, overlay geometry re-anchors so the answer stays associated with the same on-screen task.
Permission degradation	If screen or accessibility permissions are incomplete, the system falls back to safer default positioning rather than failing silently.

Overlays versus the inbox

Use overlays for fast confirmation while you work. Use the question inbox when you need full context, multiple candidates, or to dismiss misleading suggestions without reaching for on-screen chrome.

When permissions are limited

Reduced permissions change how confidently the product can place overlays and tie answers to specific controls. Fallback positioning still shows useful text, but anchoring may be coarser. Improving Accessibility and screen capture permissions (where your platform and policy allow) restores full live binding and sharper source association.

Background processing

Question processing uses background workers that advance jobs through a multi-stage pipeline. Stages separate detection, retrieval, scoring, and notification so failures in one stage can retry without corrupting the whole pipeline.

This architecture keeps the desktop responsive while still scanning new captures continuously.

Stage	Responsibility
Ingest	New screen text, transcript segments, and UI snapshots become searchable.
Detect	Heuristics flag interrogative or prompt-like spans per source.
Deduplicate	Checks collapse repeated or paraphrased questions within a time window.
Retrieve and score	Semantic search with time bounds; answer similarity ordering.
Notify	Inbox updates; overlay enqueue subject to queue and expiry rules.

Why polling instead of only push triggers

The worker tolerates bursty capture (many frames or transcript windows in seconds) without over-subscribing resources. Stages can back off under load while preserving order.

Configuration and feature availability

• Ensure the question detection feature is enabled in your build if questions do not appear at all.
• Ignored windows and excluded apps reduce signals: questions in those surfaces may not be detected.
• Transcription stride and audio devices affect spoken questions; weak or muted audio produces fewer candidates.

How quality improves over time

Question detection is index-dependent. As you capture more OCR, transcripts, and UI context, semantic search has richer targets, deduplication becomes more meaningful, and similarity scoring separates strong from weak matches more clearly. Early adopters often see a noticeable step change after a few dense workdays with capture enabled.

Tips for useful questions and answers

• Work in captured apps for the material you might need again; if it was never indexed, no candidate can surface.
• Speak clearly in meetings when you want spoken questions picked up; background noise and overlap reduce transcription quality.
• Dismiss wrong candidates promptly; that keeps mental load low and prevents mistaking stale overlays for current truth.
• Accept strong matches to build confidence in retrieval; the product does not “train” on your clicks like a cloud ranker, but your habits align expectations with the index.
• Review the inbox after large imports or new projects so deduplication windows do not hide genuinely new threads you care about.

Review workflow in practice

Step	You	Product
1. Capture	Work normally in indexed apps	OCR, audio windows, and AX snapshots accumulate
2. Detect	(no action)	Workers find question-like spans and enqueue review items
3. Suggest	Glance at overlay or open inbox	Candidates appear ranked by similarity and recency
4. Decide	Accept or dismiss	State moves to resolved or dismissed; pending clears when you act
5. Continue	Repeat	Deduplication prevents the same wording from refilling the queue all day

Accepting does not send content to the cloud by itself; it records your choice against local state so the product reflects what you found useful.

Related concepts

Question detection complements meetings, search, and daily summaries: meetings supply long-form context, search is deliberate retrieval, and question detection is proactive nudging at the moment phrasing looks like a question.