I'm trying not to oversimplify things here, but it feels like we're at a point where the genie is already out of the bottle, right? Sure, there's room for government action to rein things in or soften the impact – and the framework you've laid out is super valuable for that. What I'm really pondering is whether we've reached a stage where we should just assume that any new tech is bound to spread everywhere online sooner or later. Whether it's companies from other countries (like Mistral) or even state players getting involved, the incentive to push boundaries seems inevitable.
I am intrigued by your choice to develop and adopt a new risk assessment ‘methodology’. I tried to review your (many) co-authors and I could have missed it - but setting aside the impressive academic pedigrees - did your group include anyone with risk analysis and mitigation competencies, background, or experience? Thank you in advance.
I agree with what I think you’re implying, which is that it’s going to be really, really hard — if not impossible — to create models that will only write “good” emails according to the current LLM paradigm. Trying to govern at this stage is, if not hopeless, likely to come with a bunch of unfavourable tradeoffs.
Having this capacity seems pretty important from a governance POV, especially as the SOTA improves to the point where models can do far more than just write realistic emails.
Thanks for sharing, Sayash. However - It seems so evident that closed models are safer than open models that I don't understand why we are still discussing this. Objectively, they are. There are tradeoffs, yes, but we will have enormous innovation in safety only if it is encouraged by rules. Letting developers create technical debt and ignore the harmful effects their products cause will kill safety innovation. Leaving externalities for others to deal with is a tech cop-out.
Also - I don't think the victims will agree that risks are low. Ask the company that lost $25 million to a deep fake CFO.
An election can be swayed by one vote. A repetition of the New Hampshire primary deep fake in the general election could determine who is US President. Is that low risk?
Some concrete examples below (link) prove closed models are safer than open source. I'm happy to hear any pushback and be proven wrong.
The National Telecommunications and Information Administration (NTIA) has a tough task getting consensus on what seems obvious.
it's worth reading the Microsoft & OpenAI reports on State-linked threat actors & projecting out what risks such groups could create once AI agents that can work within systems are prevalent, not to mention AGI. It seems naive to say the risk is low. We are not talking about regulating the past.
The absence of "authoritative evidence" of marginal risk seems disputed by AI-assisted attack instances.
Reputable schools seek frontier AI research access, but why not manage it responsibly instead of handing access to all terrorists, rogue states, & criminal organizations?
This approach makes a lot of sense. Looking at spear phishing, however, it seems that Hazell's paper does show that LLMs can help scale spear phishing attacks significantly. How does this translate to no evidence of marginal risk, as shown in the harvey ball table?
Fascinating read, thanks for sharing!
I'm trying not to oversimplify things here, but it feels like we're at a point where the genie is already out of the bottle, right? Sure, there's room for government action to rein things in or soften the impact – and the framework you've laid out is super valuable for that. What I'm really pondering is whether we've reached a stage where we should just assume that any new tech is bound to spread everywhere online sooner or later. Whether it's companies from other countries (like Mistral) or even state players getting involved, the incentive to push boundaries seems inevitable.
Thanks again!
An interesting initiative, Sayash.
I am intrigued by your choice to develop and adopt a new risk assessment ‘methodology’. I tried to review your (many) co-authors and I could have missed it - but setting aside the impressive academic pedigrees - did your group include anyone with risk analysis and mitigation competencies, background, or experience? Thank you in advance.
Important work.
I agree with what I think you’re implying, which is that it’s going to be really, really hard — if not impossible — to create models that will only write “good” emails according to the current LLM paradigm. Trying to govern at this stage is, if not hopeless, likely to come with a bunch of unfavourable tradeoffs.
But it doesn’t seem like it will be *that* difficult for developers who rent access to their models via an API to at least (ex-post) detect whether users are misusing their models for things like writing spear phishing emails at scale (e.g., see https://openai.com/blog/disrupting-malicious-uses-of-ai-by-state-affiliated-threat-actors).
Having this capacity seems pretty important from a governance POV, especially as the SOTA improves to the point where models can do far more than just write realistic emails.
Thanks for sharing, Sayash. However - It seems so evident that closed models are safer than open models that I don't understand why we are still discussing this. Objectively, they are. There are tradeoffs, yes, but we will have enormous innovation in safety only if it is encouraged by rules. Letting developers create technical debt and ignore the harmful effects their products cause will kill safety innovation. Leaving externalities for others to deal with is a tech cop-out.
Also - I don't think the victims will agree that risks are low. Ask the company that lost $25 million to a deep fake CFO.
An election can be swayed by one vote. A repetition of the New Hampshire primary deep fake in the general election could determine who is US President. Is that low risk?
Some concrete examples below (link) prove closed models are safer than open source. I'm happy to hear any pushback and be proven wrong.
The National Telecommunications and Information Administration (NTIA) has a tough task getting consensus on what seems obvious.
#GoPU
https://www.linkedin.com/posts/maciejko_the-case-for-closed-ai-models-addressing-activity-7168292478977048576-LaXL
PS.
it's worth reading the Microsoft & OpenAI reports on State-linked threat actors & projecting out what risks such groups could create once AI agents that can work within systems are prevalent, not to mention AGI. It seems naive to say the risk is low. We are not talking about regulating the past.
The absence of "authoritative evidence" of marginal risk seems disputed by AI-assisted attack instances.
Reputable schools seek frontier AI research access, but why not manage it responsibly instead of handing access to all terrorists, rogue states, & criminal organizations?
This approach makes a lot of sense. Looking at spear phishing, however, it seems that Hazell's paper does show that LLMs can help scale spear phishing attacks significantly. How does this translate to no evidence of marginal risk, as shown in the harvey ball table?