The journey from a collection of raw pharmaceutical data sources to a queryable, AI-ready knowledge graph is not a single technical step — it is a pipeline of five distinct stages, each with its own technical and organisational requirements. Understanding the full pipeline before starting prevents the common failure mode of investing heavily in early stages and discovering that a critical later stage was never designed for.
Stage 1: Source Inventory and Profiling
Before any extraction begins, every relevant data source must be inventoried and profiled: its format, volume, update frequency, access method, data quality characteristics, and the domain concepts it contains. Profiling reveals the practical realities that drive pipeline design — the 20% of fields that are systematically null, the date fields stored as free text, the drug names entered in seven different formats. A source inventory document produced at this stage saves weeks of debugging in later stages.
Stage 2: Concept Extraction and Normalisation
Raw data values are extracted and normalised to standard concept identifiers. Free-text drug names are mapped to pharmaceutical ontology identifiers. Diagnosis codes in various coding systems are mapped to a canonical representation. Free-text notes are processed by NLP pipelines to extract entity mentions and map them to ontology concepts. The output of this stage is not a knowledge graph — it is a set of typed, normalised concept mentions with provenance records pointing back to their source.
Stage 3: Relation Assembly
Normalised concept mentions are assembled into subject-predicate-object triples using the relation types defined in the target ontology. Some triples are derived directly from the source data structure (a record linking a patient identifier, a drug identifier, and an administration date yields an administration event triple). Others are extracted from text by the relation extraction pipeline described in earlier sections.
Stage 4: Validation and Quality Scoring
Automatically extracted triples are validated against reference data, ontology constraints, and statistical expectations. Triples that violate ontological constraints (asserting that a disease is a type of drug, for example) are flagged for human review. Triples with low extraction confidence scores are separated for expert curation. Each triple receives a provenance record and a quality score that downstream applications can use to filter their queries.
Stage 5: Deployment and Query Layer
The validated knowledge graph is loaded into a triple store (Apache Jena, GraphDB, Amazon Neptune, or similar) and exposed via a SPARQL endpoint or a graph query API. A query layer translates application-level requests into graph queries, handles caching, and manages access control. Monitoring for drift — cases where the source data changes in ways that invalidate existing graph assertions — is ongoing from this point forward.
