Most pharmaceutical document repositories — SharePoint, Documentum, Veeva Vault, and dozens of legacy systems — provide basic keyword search as their only discovery mechanism. Replacing these systems is a multi-year, high-risk programme that organisations are understandably reluctant to initiate. The good news is that a semantic search layer can be added on top of existing repository infrastructure, without replacing anything, in a timeframe of three to six months for a well-scoped initial deployment.
Architecture: The Overlay Pattern
The overlay approach treats the existing document repository as an opaque source system and adds a semantic layer that independently processes, annotates, and indexes the documents it contains. The overlay reads documents from the repository via its API (or via scheduled export), runs them through an NLP annotation pipeline that extracts and normalises concept mentions against the target ontology, and stores the resulting annotations in a separate semantic index. Search queries are issued against the semantic index, which returns document identifiers that are then resolved against the original repository for rendering. The source system's storage, access control, and audit trail remain unchanged.
Index Design
The semantic index must support both concept-level queries (find all documents mentioning SNOMED CT concept X or any of its descendants) and combined queries that filter by concept and by metadata (document type, date range, study phase, product identifier). An inverted index mapping concept identifiers to document lists — augmented with descendant pre-computation for the is-a hierarchy — provides efficient query execution for most pharmaceutical document retrieval use cases. For repositories larger than a few million documents, Elasticsearch or Apache Solr with custom ontology-aware query expansion plugins provides the necessary scalability.
Incremental Annotation
Documents added to the repository after the initial annotation run must be processed automatically. A repository event listener — triggered by document creation or modification events — places new documents in an annotation queue and updates the semantic index as each annotation job completes. For repositories without event streaming, a scheduled polling mechanism achieves the same result with slightly higher latency. The annotation pipeline itself should be idempotent: re-running it on a document that has already been processed produces the same result, making it safe to re-process documents when the underlying NLP models or ontologies are updated.
