Vector search isn't the hard part. Deciding what should be searched isOver the last few weeks I've been redesigning the retrieval pipeline for an AI knowledge system.Initially, the architecture was fairly typical:User Question │ ▼ Vector Search │ ▼ Top K Chunks │ ▼ LLMIt worked well while the knowledge base was small.As more documents were added, I started seeing a few recurring problems:More irrelevant chunks being retrieved.Larger prompts and increasing token costs.Multiple documents discussing the same topic competing with each other.Vector search returning semantically similar chunks from documents that weren't actually the best source of truth.I realized the problem wasn't vector search itself.It was deciding what should be searched before semantic retrieval even began.Instead of treating every document equally, I separated the system into two independent stages.IngestionDuring document upload, every document is processed once.The pipeline extracts structured metadata including:document typebusiness roledepartmentstopicsplanner summaryretrieval keyw