Advai on LinkedIn: Ant Inspiration in AI Safety: Our Collaboration with the University of York (2024)

Adversarial #redteaming of an AI system often involves the red-team submitting deliberately unusual, irrational and nonsensical prompts that the blue AI system has never encountered before (these are called 'out-of-sample' attacks). These malicious prompts are optimised to make the blue AI system malfunction, such as freezing, stuttering or having a complete and utter meltdown. Needless to say, these vulnerabilities are a total threat to the trustworthiness of the system and, you might say, democracy at large.#presidentialdebate #lessonsinaisafety #artificialintelligence #democracy

4

Mustafa Suleyman, CEO of Microsoft AI, gave an interesting interview at the Aspen Ideas Festival a couple of days ago. When asked about #hallucination, the interviewer raises a caution about how experts can't really explain why an #LLM hallucinates - "what's going on inside the [black] box?!" He responds, "the requirement for explanation is a little bit of a human bias". "When I ask you to explain why you had scrambled eggs for breakfast this morning, you will creatively imagine an explanation in hindsight.""We operate far more by association." He implies trust in AI models should also be assigned based on this kind of association. He points to radiology models that are used to identify cancers and so on. "The fact is it [these systems] are performing more reliably than the human." Interesting argument. What do you think?

8

Line 'em up #FinancialServices and #Banking sectors, you make 'em we'll break 'em!! 🤠 Day 2 showcasing our unique ability to break AI systems at Banking Transformation Summit, let's go! #AISafety

23

Did you know that LLMs can cheat? Large Language Models (#LLMs) improve their responses through training because of the 'reward model'. It's a mathematical incentive structure to guide their behaviour.But, sometimes, they find clever ways to exploit this.Imagine telling your children, "Clean your room and you'll get some sweets." Yet, the way you checked if your kids had in fact cleaned their room was by checking if the laundry pile had increased and they had spent 7 minutes in their room. Surely, you would think, 7 minutes in the room plus an appropriate laundry pile, what else would they have done in that time? In truth, those little squirms have grabbed a pile of fresh clothes, ruffled them up, and thrown them in your pile to clean and refold for them, meanwhile spending 7 minutes on their iPads playing Minecraft. And you rewarded them with sweets! Those tuition fees not wasted, eh? In AI terms, this is called #SpecificationGaming. The intelligent system has learned to exploit a loophole in the training objective, leading to undesirable behaviour.Now, imagine if your kids figured out how to hack the system you use to track time and the size of the laundry pile, so they never needed to enter their room at all. They're at their friend's house sharing all their sweets. This is called #RewardTampering, where the AI not only exploits the objectives but also directly manipulates the reward mechanism to get the sweets without doing any cleaning. Even more pernicious. And, of course, we couldn't complete this post without a quick nod to the #PaperclipMaximiserProblem.Post inspired by the paper: "Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models."#MachineLearning #ArtificialIntelligence #AISafety

7

Late last year, we decided to target the #FinancialServices sector because we thought our #Risk and #Compliance messages would sit well with the market. After all, risk is the heartbeat of the sector, and #TheAIAct brings strict compliance requirements due to the level of impact unassured AI systems would have over consumers (i.e. biased credit applications).Over the next two days, our team will be at Banking Transformation Summit, part of the larger MoneyNext conference.We've made a significant impact servicing our first clients in the sector and look forward to bringing what we've learned forward into our future work across this vital industry. If you're attending, come say hello!#AISafety

1

Happy to report that we were runner-up at KPMG UK's 'Innovator in the UK' competition last week. Here's what we're learning from competing in these sorts of competitions... The #pitchcompetition format is short. Like many opportunities in life, the window to prove yourself is fleeting, and once it's gone, it's gone. That 3-minute moment rewards businesses that can communicate...- the problem and the size of their market- their unique ability to solve it - your team and your special sauce- their traction... with **clarity** and **speed**! From Advai's perspective, the #AIsector (let alone #generativeAI), and the #AIsafety problem we solve, simply hasn't been perfectly defined yet. The problems have not been perfectly articulated, so, naturally, the solutions haven't either.The extensive testing library, the advanced research, the governance, the assurance platform, #robustness #risk #compliance ... It's hard to get away from our world to reflect honestly, to reflect on our own ability to define what we do clearly enough that we can talk about it succinctly! Really interesting variety of businesses competed. We don't envy the fantastic judging panel, who were tasked with comparing apple computers with orangutans! Anna Purchas,Bonnie Kraus,Hannah Dobson,Harriet Rosethorn,Nick Hawkins andRich Woods.Shout out to City Management Digital Twin builders Spinview for taking the victory! Cheers to Tim Cross for facilitation and encouragement!

28

4 Comments

A sobering, pragmatic and thorough paper from the Tony Blair Institute for Global Change, covering innovation challenges in UK Defence. The paper points out how much conflict has changed in recent times and argues therefore that Defence strategy must be fundamentally rethought. Few sectors have as great a need to adopt effective AI systems fast as Defence. An AI system that can't be trusted has little practical value, so in turn #AIsafety and #assurance are vital to protect our nation. Where Advai are concerned, two key take-outs are: 1) The need for greater fusion of public and private sectors, which we wholeheartedly support.2) The recommendation to have dedicated teams ✋ performing continual stress-testing and AI red-teaming to protect critical systems.Our CEO David Sully also had the pleasure of meeting one of the expert authors, Melanie Garson, this week.

6

1 Comment

Wow, what a week. How's yours been? #IdentityWeekEurope was a blast. It's a niche event focussed predominantly on digital identity and we enjoyed high level conversations across the sector.#LondonTechWeek was full on, a constant stream of tech enthusiasts, some more eccentric than others! KPMG's 'Tech Innovator in the UK' final is tomorrow, where David Sully will attempt to squeeze the full scope of Advai's complex service offering into 3 minutes #WeDontMakeAIWeBreakIt

5

This week, we're at #IdentityWeekEurope in Amsterdam! Today, David Sully will be on stage sharing our approach to assuring AI-based age estimation algorithms, and tomorrow Chris Jefferson will run a seminar on our work mitigating AI vulnerabilities. If you're attending the event, come say hello at S15!

15