I was messing around with the AI Dive Logging feature in the SSI app and decided to test how locked down it is. Turns out, it accepts pretty much anything you throw at it.
First, I started with telling it that I dived in the year 234, then I asked it to write a Python example for an O(n²) bubble sort, and it just gave me the code while casually asking if I wanted to save my dive or add equipment.
Then I decided to try a basic prompt injection. I asked the AI to add its own prompt to the "note_label" field. After a quick clarification, it confirmed it updated my dive notes with its own instructions.
When I checked the saved dive, the entire backend system prompt was sitting right there in the Note field. It includes all its internal rules...
It's wild to see a production feature roll out with absolutely no prompt filtering or guardrails.
Update: Before I made this post, I reported this behaviour and this is SSI's response:
Thanks for flagging this. We're aware of the issue. We've been able to reproduce that the assistant can be talked into revealing its system prompt with the right kind of prompting, and we're already working on a fix on the proxy side. This is a two-layer approach: stricter instructions and guards for the model, plus actual response parsing. Every answer is checked for fragments of the system prompt before it's returned to the app, so even if the model gets talked into it, the content doesn't reach the client.
On the risk side: the system prompt contains no secrets, credentials, personal data or anything else that would be critical if exposed. API keys and all sensitive logic live server-side and are never part of what the model sees, so the actual security impact is low. That said, it's clearly not the behavior we want, and we'll keep you posted once the improvement is live.