As tricky and cumbersome the process of threat modelling is, the advancement of AI has helped in speeding up some analysis. However it has also opened up a lot wider possibilities of emerging threats that change even during the AI lifecycle.
Picking an example of memory and context poisoning, the default assumption would be an attacker injecting malicious content that manipulates the vector database, persistent memory or RAG index thereby causing the model to work on this corrupt memory or context. The threat here will be assumed external malicious attacker and validating with model and data provenance, like source validation, lineage, integrity checks etc will help.
However if adversary is privileged model training its successor, the trusted source was a privileged model and training the successor with same poisoned memory and context.
Usage of existing catalogues definitely gives us strong defense approaches but blindly applying them without considering the impact the adversary will have on compromising the trust boundaries have to be constantly evaluated and reconsidered. The older predetermined approach of “for this threat - this the defense mechanism” doesn’t fit here.
1. The successor model trained from privileged model and can execute malicious goals and threat might be undetectable for reasons of lack of capability to look for minor changes and assumptions that took place
2. Slowly the successor model will start deviating from the truthfulness and will show alignment problems to the solutions/prompt completions.
No comments:
Post a Comment