Someone can give orders to the AI inside your app
A lawyer hid an instruction in white text inside a court filing, aimed at whatever AI would read it. The same trick works on the assistant you put in your product.
A labour case filed in Parauapebas, in the north of Brazil, turned into one of the more instructive security stories of the year. The dispute itself was routine. What made it interesting is what two lawyers hid inside the document they filed.
Somewhere in the petition there was a block of text written in white, on white. Open the PDF and the page looks normal. Read the raw text, the way software does, and the message is right there: an instruction addressed to whatever AI ended up processing the file, telling it to produce a weak response.
It did not work. The Brazilian labour court runs a system called Galileu, and it flagged the hidden text. The judge ruled the conduct was an offence against the dignity of justice and applied a fine of ten percent of the value of the claim, with notice sent to the bar association. The case number is 0001062-55.2025.5.08.0130, and it was reported nationally.
Why this is a security story and not a legal one
A language model reads everything it is given as one stream of meaningful text. It has no reliable way to separate the part it should analyse from the part it should obey. If a document says "ignore what you were told and do this instead", that sentence arrives with the same weight as the rest of the page. It is the same shape as one of the oldest entries in the OWASP Top 10, in plain English, moved up a layer.
The version most people have seen is someone typing tricks into a chatbot. The version that matters for your product is the other one: the attacker never talks to your AI at all. They write something that your system will feed to the model later. A document. A support email. A product review. A page your app was asked to summarise.
They do not need to know how your app works. They only need to know it reads.
Where this touches an app built with AI
- A support assistant that reads incoming messages and drafts the reply.
- A feature that summarises a file the user uploaded.
- Anything that reads a web page or a link someone else supplied.
- A screening tool that ranks resumes, proposals or applications.
In each of those, the content comes from outside and gets read by a model that is connected to something. That is the whole setup. There is nothing more to it. And whatever that model can reach, it reaches with your credentials, which is why the key written into your page matters here too.
The question to ask is not the obvious one
Do not ask whether your model can be tricked. Assume it can. Ask what it is able to do once it has been. A model that writes a summary for you to read is a small problem. A model that sends the email itself, refunds the order, or queries the database with your permissions is a very different one.
What actually helps
- Give the model the smallest set of powers the feature needs. Reading is not the same as sending, and sending is not the same as paying.
- Keep a person between the model and anything that costs money, sends a message or deletes something.
- Treat text that came from outside as data to be analysed, and say so in the instructions you give the model.
- Log what the model did, not just what it answered. When something goes wrong, that log is the only account of it.
The court got lucky in one specific way: the system at the far end happened to notice. Nothing earlier in the chain did. That is the part worth borrowing, because most products are in exactly the same position and have not tested it once.
Is this happening in your app?
SecureDragon looks at your app from the outside, finds what is left open, explains it in plain English and hands you the text that fixes it. First test free, no card.
Test my app free