Explain an end‑to‑end workflow for building a chatbot that can search text‑based information from PDF documents. How would you ingest the PDFs, extract text, embed, store, and query the data?

An end‑to‑end workflow for a PDF‑search chatbot typically follows these steps: 1) Ingest PDFs into a storage layer (S3, Blob). 2) Extract text using OCR or PDF parsers (Apache PDFBox, Tika). 3) Chunk the text into manageable pieces (200–500 tokens) and optionally add metadata (document ID, page number). 4) Generate embeddings for each chunk using a transformer model (OpenAI, Cohere, or a local model). 5) Store the embeddings in a vector database (Pinecone, Weaviate, Qdrant). 6) Build a retrieval layer that, given a user query, generates an embedding and performs a similarity search to retrieve the top‑k relevant chunks. 7) Pass the retrieved chunks as context to an LLM (OpenAI GPT‑4, Anthropic Claude) to generate a response. 8) Optionally, cache frequent queries and use a prompt template to keep the LLM’s output consistent. This pipeline ensures that the chatbot can answer questions based on the content of arbitrary PDFs with low latency and high relevance.