AI Engineering

AI on Confidential Documents Without Sending Them to OpenAI: Private LLM and RAG Options

Back to Articles
2026-11-13
9 min read
MAAuthor Marius Andronie
AI on Confidential Documents Without Sending Them to OpenAI: Private LLM and RAG Options

Short answer: you can use AI on confidential documents without sending them to OpenAI by running open models and a search index on infrastructure you control, so the files never leave it. That is a private LLM with retrieval-augmented generation (RAG): the system finds the relevant passages in your own documents, and a model that you host writes the answer from them, with a citation back to the page. It is not the only safe option. A hosted API can also be acceptable when the terms, the contract and the data you send are right. The choice depends on how sensitive the documents are, who is allowed to see them, and what your contracts say.

Key facts:

  • Hosted frontier models and a private setup are two ends of a scale, with a middle option: private hosting for the sensitive parts and a hosted model only for what is safe to send.
  • OpenAI's documentation says data sent to its API is not used to train its models unless you opt in, and that abuse-monitoring logs are retained for up to 30 days unless the law requires longer. Zero data retention is subject to prior approval (OpenAI, Data controls, read 13 November 2026).
  • Under GDPR Article 28, a processor that handles personal data for you must work under a contract, and may not engage another processor without your written authorization (GDPR Article 28, read 13 November 2026).
  • Private does not mean automatically accurate. A private RAG system still needs source citations and a rule for when it must say "I do not know".

The options in one table

OptionWhere your documents goStrengthWeak spotFits when
Hosted chat product (consumer or team plan)To the provider, under its product termsFastest to startLeast control over retention and access; hard to auditLow-sensitivity material
Hosted API with a contract (and zero data retention where approved)To the provider's API, under its API termsStrongest models, no hardware to runData still leaves your environment; approval and contract workSensitive but not secret material, with a processor agreement in place
Private cloud or your own server, open modelsStays inside infrastructure you controlYou decide retention, access and loggingYou run it, and open models are smaller than the top hosted onesConfidential material that must not leave
HybridSensitive text stays private; only cleared or redacted text goes to a hosted modelQuality where it is safe, control where it is notMore design work: the boundary has to be written down and testedMixed material, one pipeline
Nothing leaves the building (air-gapped)Never on a networkMaximum controlSlowest to update; more work to operateDefence, regulated or deal-critical files

No row is the "right" one. Each is a trade between control, model quality and the work of running it.

What "private LLM" and "RAG" mean here

A private LLM is a language model you run yourself instead of calling someone else's service. Open-weight models can be run locally with tools such as Ollama, whose own site describes running AI models locally and in the cloud (read 13 November 2026). The model itself does not learn from your files while it answers. It only reads what you give it for that question.

RAG is the way you give it the right pages. Your documents are cut into passages, converted to numeric vectors (embeddings) and stored in an index. When someone asks a question, the system finds the closest passages and passes only those to the model, which answers from them. For the index, a common building block is pgvector, described in its repository as "Open-source vector similarity search for Postgres", which keeps the vectors in a database you already know how to back up and secure.

The part that is easy to forget is the first stage. Your documents have to be turned into clean text before anything can be searched. For PDFs and scans, a parser that runs locally matters: the open-source Docling project lists local execution for sensitive data and air-gapped environments among its features (MIT licensed, read 13 November 2026). If the scan is bad, the text is bad, and the answer is bad, however private the model. We wrote about that in why a scan is not a document and in document data extraction methods compared.

What private does not fix

Keeping documents inside your environment answers the question "who else sees them". It does not answer "is the answer right". Two things still need design:

  1. Retrieval can fail quietly. A system that finds the wrong passage will answer confidently from it. We measured a case where a retrieval score could not tell a real question from an off-topic one, and wrote it up in why retrieval scores cannot gate a RAG system. The fix there was to check the answer against its sources, not to trust a confidence number.
  2. Access rules must follow the documents. If everyone who can ask a question can retrieve every passage, the assistant becomes a way around your folder permissions. Retrieval should respect who is asking.

A decision order that works

  1. Classify the documents. What is public, what is internal, what is confidential, what is regulated (personal data, health, deal or client secrets)? Do this per document set, not per company.
  2. Check what you have promised. Client contracts, NDAs and your own privacy notices may already limit where files can go. That settles some choices before any technical comparison.
  3. Decide the boundary. For each class, write down what may reach a hosted model, what must stay private, and what must be redacted first.
  4. Pick the smallest private setup that does the job. Often one document type and one question type is enough to prove it.
  5. Test with your real documents, including the awkward ones: scans, tables, a file in the wrong language, a question the documents cannot answer.
  6. Write down the operating rules. Who updates the index, who can query it, how long logs are kept, what happens when a document is withdrawn.

What a private RAG build costs with us

Devaland builds private document assistants as fixed-price projects, as described on the AI and Python development page. That page lists a paid proof pilot from $2,500 and fixed-scope builds typically from $8,000 to $25,000, all excluding VAT. The pilot runs on your real documents for about one to two weeks, and if you continue, the pilot fee is credited toward the full build. The code goes into your own repository. Prices are USD and exclude VAT, which is added where it applies. What reaches a hosted model, if anything, is agreed in writing before the build starts. Intake is a short written form, and scope, price and delivery are confirmed in writing.

We do not promise a result. A pilot tells you whether a private assistant answers your questions correctly from your documents, and what it would take to run it, before you commit to more.

What to do this week

  1. List your three most sensitive document sets and who is allowed to read each.
  2. Check the contracts and notices that cover them.
  3. Pick one set and ten real questions people ask about it.
  4. If you want a written scope and a fixed price for a private assistant on that set, send the intake form on the AI and Python development page or use the contact page.

Frequently asked questions

How can I use AI on confidential documents without sending them to OpenAI? Run an open model and a search index on infrastructure you control, so retrieval and answering both happen inside it. The model reads only the passages retrieved for each question. If you still want a hosted model for some tasks, decide in writing what may be sent to it and what may not.

Is a private LLM more accurate than ChatGPT? Not by default. Open models you run yourself are generally smaller than the largest hosted models. What a private setup gives you is control over where the documents go. Accuracy comes from good text extraction, good retrieval and checking answers against their sources, and it has to be tested on your documents.

Does OpenAI train on data sent through its API? Its documentation says data sent to the API is not used to train or improve its models unless you explicitly opt in, and that abuse-monitoring logs are kept for up to 30 days unless the law requires longer. Zero data retention is available only with prior approval. Read the current terms before relying on them, as they can change.

Do I need a data processing agreement to use AI on documents with personal data? In the EU, if a provider processes personal data on your behalf, GDPR Article 28 requires a contract with it, and the provider may not use another processor without your authorization. Ask your adviser how it applies to your case. This is general information, not legal advice.

What is RAG, in plain words? Retrieval-augmented generation. The system first searches your documents for the passages that match the question, then a language model writes the answer using only those passages and shows where they came from. It is how an assistant answers from your files instead of from general knowledge.

How do I start without a large commitment? With a small paid pilot: one document set, a fixed price, a short time and your real documents. Devaland's published pilot starts from $2,500 excluding VAT, and the fee is credited toward the full build if you continue.

Tell us what you need built

Python and AI systems, websites and rescues for stuck platforms, scoped and quoted in writing. No sales call.

Tell us what you need built →
Opens in a new tab