Decision-grade finance

The decision problem

Financial teams do not need a plausible paragraph. They need the applicable policy, the correct version of a document, the relevant transaction evidence and a traceable explanation of how those sources support a recommendation. A retrieval system that returns a fragment without its entity, date, jurisdiction or exception can produce an answer that sounds confident but is not decision-grade.

There is public evidence that contextual retrieval improves the search step. Anthropic’s 2024 evaluation tested codebases, fiction, research papers and other knowledge domains. Its method prepends chunk-specific explanatory context before creating embeddings and a BM25 index. Anthropic reports that contextual embeddings reduced the top-20 retrieval failure rate from 5.7% to 3.7%; combining contextual embeddings with contextual BM25 reduced it to 2.9%, a 49% reduction in failure rate. Adding reranking reduced the reported failure rate to 1.9%, a 67% reduction. These are Anthropic’s experiments across its selected datasets and configurations, not a fintech production benchmark.

Financial-services deployments show what governance looks like in practice. Morgan Stanley reports that more than 98% of advisor teams use its internal AI Assistant for knowledge retrieval. The firm says it expanded from answering 7,000 questions to handling questions across a corpus of 100,000 documents, while document access increased from 20% to 80%. Its workflow includes daily regression testing, expert feedback, retrieval-method refinement and human review of generated meeting outputs. These are figures published by OpenAI and Morgan Stanley’s collaboration, not an independent audit; the Assistant is an internal wealth-management tool rather than a lending-decision engine.

Rogo provides a more fintech-like example of agentic financial research. OpenAI reports that the platform serves more than 5,000 bankers, saves analysts up to 10 hours per week on research and diligence tasks, and searches more than 50 million financial documents. The figures are vendor-reported. Rogo’s use case is analyst research and diligence, not automatic underwriting or investment approval.

A PySmith decision-support workflow

PySmith can host the persistent Python workspace around a controlled retrieval and analysis system. Persistence is useful when a review spans multiple sessions: the workspace can retain a versioned evidence set, intermediate calculations, model outputs, reviewer questions and the decision audit trail. It can also keep common Python data and document-processing libraries warm. Persistence does not make retrieval correct, confer permission to access confidential data or guarantee that a model will follow policy.

The first use should be low-risk decision support such as policy interpretation, regulatory-change triage, transaction-monitoring investigation or credit-file preparation. Every answer should expose source passages, document versions, timestamps and access decisions. Generated code should run in a restricted environment with allow-listed dependencies and no unrestricted network access.

What a pilot should prove

Evaluate retrieval recall and citation correctness separately from answer quality. Use a labelled set of real questions with known authoritative sources. Measure top-k retrieval failure, source-version accuracy, unsupported-claim rate, analyst correction rate, time to decision, latency, cost and performance under permission changes.

Test adversarial cases: near-duplicate policies, superseded guidance, missing documents, ambiguous entities and conflicting evidence. Compare a stateless baseline with persistent execution so any PySmith benefit is measured rather than assumed.