Prompting with Internal Documents (RAG) — Use Your Company Knowledge with AI

What is RAG and Why Does It Matter?

RAG stands for Retrieval-Augmented Generation. In simple terms, it means giving AI access to your company's documents so it can answer questions based on your actual information — not just its general training data.

Without RAG, AI can only draw from public knowledge. With RAG, AI becomes an expert on your company: your policies, your products, your processes, and your market data.

How Business Teams Can Use RAG Today

You don't need to build a custom AI system to benefit from RAG concepts. Here are practical approaches available now:

Method 1: Copy-Paste Context

The simplest form of RAG. Paste relevant document content directly into your prompt.

Based on the following excerpt from our Employee Handbook: [paste relevant section]

Answer this question: Can employees work from home on Fridays? Only use information from the provided text. If the answer is not in the text, say "Not covered in the provided document."

Method 2: Custom GPTs (ChatGPT)

Create a custom GPT with your documents uploaded. The AI will reference these documents when answering questions.

Use cases:

• Company policy Q&A bot
• Product knowledge assistant
• Onboarding guide
• Process documentation helper

Method 3: Microsoft Copilot with M365

If your company uses Microsoft 365, Copilot can access your SharePoint, Teams, and email data.

Use cases:

• "Summarise the key decisions from last Tuesday's project meeting"
• "What does our procurement policy say about vendor approval for purchases over $5,000?"
• "Find all emails about the Singapore expansion project from the past month"

Method 4: Enterprise AI Platforms

Tools like Azure OpenAI Service, AWS Bedrock, or dedicated platforms allow companies to build secure RAG systems over their document repositories.

Effective Prompting with Document Context

Rule 1: Be Specific About Sources

Based ONLY on the provided document, answer this question. Do not use any outside knowledge. If the information is not in the document, state "Not found in the provided document."

Rule 2: Identify the Most Relevant Section

Read this document and identify the sections most relevant to [question]. Quote the relevant text, then provide your analysis.

Rule 3: Cross-Reference Multiple Documents

I am providing 2 documents: Document A: Our AI Usage Policy Document B: Singapore PDPA Guidelines

Compare them and identify:

1. Areas where our policy meets PDPA requirements
2. Gaps where our policy does not address PDPA requirements
3. Recommended additions to our policy

Rule 4: Summarise for Different Audiences

Summarise this 20-page policy document in 3 versions:

1. Executive summary (1 paragraph, 100 words)
2. Manager briefing (5 bullet points, key actions)
3. Employee quick-reference (10 FAQ-style Q&As)

Common Business RAG Use Cases

Policy Q&A

Upload your employee handbook, HR policies, IT policies, and compliance documents. Employees can ask questions and get answers grounded in your actual policies.

Product Knowledge

Upload product documentation, feature specs, pricing guides, and competitive comparisons. Sales and customer service teams get instant, accurate product information.

Training and Onboarding

Upload training materials, SOPs, and process guides. New employees can ask questions and get answers based on your actual procedures.

Research and Analysis

Upload market research, industry reports, and competitive intelligence. Strategy teams can query across multiple documents.

Meeting Intelligence

Use AI to search across meeting notes, action items, and decisions. Find "What did we decide about [topic] in the Q2 planning meeting?"

Data Safety Considerations

What Documents Are Safe to Use with AI

• Internal process documentation (SOPs, workflows)
• Published company policies
• Public-facing marketing materials
• Non-confidential training materials
• General industry research

What Requires Enterprise AI Platforms

• Customer data and contracts
• Financial records
• Employee personal information
• Proprietary algorithms or IP
• Legal documents

Never Upload to Consumer AI

• Personally identifiable information
• Trade secrets and source code
• Pre-release financial data
• Legally privileged communications

Getting Started with RAG

1. Start simple: Use copy-paste context for immediate needs
2. Build a Custom GPT: Upload 5-10 key company documents for a team Q&A assistant
3. Evaluate Copilot: If your company uses M365, explore Copilot's document access capabilities
4. Plan enterprise RAG: For larger-scale needs, work with IT to evaluate enterprise AI platforms

The progression from copy-paste to enterprise RAG typically takes 3-6 months, with each step delivering immediate value.

Related Reading

• Copilot M365 Use Cases — Use Microsoft Copilot with your internal M365 documents
• ChatGPT Data Leakage Prevention — Keep sensitive internal data safe when using AI
• Prompting Structured Outputs — Get consistent results when working with document data

How Retrieval-Augmented Generation Architecture Has Evolved Since 2024

The foundational concept of grounding language model responses in organizational documents matured substantially between early 2024 and March 2026. What began as experimental proof-of-concept implementations evolved into production-grade architectures supporting enterprise-scale document retrieval across thousands of concurrent users.

Vector Database Landscape Consolidation. The vector storage ecosystem consolidated around several dominant platforms: Pinecone maintained market leadership for fully managed deployments; Weaviate gained adoption among organizations preferring open-source infrastructure hosted on their own Kubernetes clusters; Qdrant emerged as a strong contender for high-throughput retrieval workloads; and pgvector extensions brought vector similarity search into existing PostgreSQL deployments used by teams wanting to avoid introducing additional infrastructure components. ChromaDB retained popularity among development teams building prototypes and smaller-scale applications.

Embedding Model Selection. OpenAI's text-embedding-3-large became the default choice for English-language document corpora, while Cohere's embed-multilingual-v3 demonstrated superior performance for organizations maintaining document repositories spanning Bahasa Indonesia, Thai, Vietnamese, Mandarin, and Bahasa Malaysia. Sentence Transformers models from Hugging Face provided cost-effective alternatives for organizations processing high document volumes where API-based embedding costs became prohibitive.

Chunking Strategies That Actually Work for Corporate Documents

Document chunking — splitting source materials into retrieval-appropriate segments — remains the most impactful architectural decision affecting response quality. Pertama Partners evaluated five chunking methodologies across engagements with organizations in Singapore, Malaysia, Thailand, and Indonesia:

Fixed-Size Chunking (512 tokens). Simple implementation through libraries like LangChain or LlamaIndex. Adequate for homogeneous document collections like customer service knowledge bases but produces poor results when applied to complex documents containing tables, hierarchical headings, and cross-referenced sections.

Recursive Character Splitting. Attempts to respect document structure by splitting at paragraph boundaries, then sentence boundaries, then character boundaries. LangChain's RecursiveCharacterTextSplitter implements this approach and performs acceptably for most corporate document types including policy manuals, procedure guides, and internal communications.

Semantic Chunking. Groups content by topical coherence rather than arbitrary size boundaries. Greg Kamradt's semantic chunking approach, implemented through embedding similarity analysis between adjacent paragraphs, produces retrieval-optimal segments but requires additional computational overhead during ingestion. Effective for legal contracts, regulatory filings, and technical specification documents where topic boundaries carry significant meaning.

Parent-Child Document Hierarchies. Stores both summary-level parent chunks and detailed child chunks, retrieving children for specificity while providing parent context for comprehension. This architecture, implemented through LlamaIndex's recursive retrieval modules or custom pipelines built with Haystack framework, delivers the strongest results for large document repositories exceeding ten thousand pages.

Prompt Engineering Patterns for Context-Aware Responses

Once documents are retrieved, the prompting layer determines whether responses accurately synthesize source material or hallucinate plausible-sounding but unsupported content:

1. Citation Enforcement Pattern — append to system prompt: "Every factual claim must include a bracketed reference to the source document title and section number. If no retrieved context supports a claim, explicitly state that the information is not available in the provided documents."
2. Confidence Calibration Pattern — instruct the model: "Rate your confidence for each answer on a three-tier scale: HIGH (directly stated in retrieved documents), MEDIUM (reasonably inferred from multiple retrieved passages), LOW (partially supported or requiring assumptions beyond retrieved context)."
3. Multi-Document Synthesis Pattern — when retrieved contexts span multiple source documents: "Synthesize information from all provided context passages. When sources present conflicting information, present both perspectives and identify the more recently dated source."
4. Scope Boundary Pattern — critical for preventing hallucination: "If the retrieved context does not contain sufficient information to answer the question completely, respond with what is available and clearly identify which aspects of the question cannot be addressed from the provided documents."