《CETaS:2023对抗性人工智能研究报告:发展成熟还是过度炒作?(英文版)(12页).pdf》由会员分享,可在线阅读,更多相关《CETaS:2023对抗性人工智能研究报告:发展成熟还是过度炒作?(英文版)(12页).pdf(12页珍藏版)》请在三个皮匠报告上搜索。
1、Adversarial AIComing of age or overhyped?The views expressed in this article are those of the authors,and do not necessarily represent the views of The Alan Turing Institute or any other organisation.Patrick HintonIntroductionThis article explores developments in adversarial artificial intelligence(
2、AAI)and machine learning,examining recent research,practical realities for the deployment of adversarial attacks,and the pursuit of secure and robust AI.Invisible to the digital eyeIn an underground command centre,an intelligence analyst sits at a computer terminal.The analyst is presented with a se
3、ries of aerial photographs taken by uncrewed air systems and satellites of potential targets ammunition dumps,vehicle parks,and defensive positions.Due to the huge volume of imagery and videos being produced by the suite of aerial sensors,target recognition software sifts through the millions of fra
4、mes searching for objects of potential interest.The software has been trained to identify armoured vehicles,aircraft,and command posts.The analyst then works through the pile of indications and passes them to targeting specialists to decide on further action.However,the software fails to flag a squa
5、dron of enemy fighter aircraft sitting on a rural airfield,which continue to target friendly troops and destroy vehicles and equipment.Coloured patches designed to trick target recognition software and present false negatives have been attached to the jets.Consequently,they are not flagged to the an
6、alyst and remain hidden among the noise of the gigabytes of aerial footage.This is one potential threat which is raised by the spectre of adversarial AI.AI systems are becoming increasingly critical assets in commerce,transportation,and the military.As the role of military AI increases to manage eve
7、r-growing volumes of data,a potential vulnerability presents itself.Instead of targeting physical infrastructure with missiles and bombs,it is possible to attack the algorithms,machine learning models and networks which support the military decision-making process.Adversarial AI:Coming of age or ove
8、rhyped?1AI security concerns2Adversarial AI:Coming of age or overhyped?Adversarial attacks are a class of techniques that analyse the parameters of a machine learning model(such as a neural network)to calculate a series of changes to an input that causes a misclassification.1In other words,they are
9、attacks which are designed to lead the model to make a mistake.Some have argued that the secret to winning the AI war might rest not in making the most impressive weapons but in mastering the disquieting treachery of the software.2The proliferation of defence and security AI use cases has garnered m
10、uch more attention than the potential vulnerabilities in the software.Developers are prioritising getting their AI systems to work in the first place,with security and adversarial activity taking a back seat.This is not an advisable strategy.Adversarial AI was first discussed and identified as a thr
11、eat in 2004.3At this time,the focus was not on the defence or security realm,but the more innocuous subject of email spam.In this case,a machine learning algorithm was pitted against a spam filter and was able to learn how to write spam emails which would get through the filter by using identified g
12、ood words.3Adversarial AI:Coming of age or overhyped?The first conference on AI security followed in 2007.There was a dearth of activity between 2008 and 2014 with a spike of research papers published on the subject,ostensibly because of the first successful attacks on deep learning algorithms.4Sinc
13、e 2015,research into adversarial AI has risen substantially,with more than 1000 papers published in 2019 and more than 1500 published in 2020.Multiple papers are being published almost every day on the subject.5In the worst cases,AI systems may be tricked into targeting the wrong people,or causing u
14、ncrewed systems to malfunction and stop dead in their tracks.More widely,artificial intelligence is being used for administrative and organisational tasks within the national security apparatus as well as in cyber security.These are all areas where if a machine learning model learns the wrong thing,
15、does the wrong thing,or reveals the wrong thing,there may be very damaging consequences.6 AI security concerns4Attack vectorsAdversarial AI:Coming of age or overhyped?AI systems that process images are the most commonly attacked,although others such as speech recognition,malware detection,and sentim
16、ent analysis have also been victim.7There are several ways machine learning models can be attacked.8These attacks may be designed with intimate knowledge of a system,which are known as white-box attacks.Attacks designed without knowledge of the internal workings of the systems are black-box attacks.
17、PoisoningPoisoning attacks see intentionally malicious training data fed into machine learning models ahead of deployment.Only a very small amount of data needs to be affected to influence the whole model,making this a significant threat.An example of poisoning would be mislabelling a series of harm
18、ful images as benign whilst adding a physical identifier such as a small red square in one corner.9The model then learns that images with a red square are safe,and they will make it through the filter even if they are not safe.EvasionEvasion attacks are similar to poisoning attacks but take place af
19、ter deployment at test time.Neural networks have been shown time and again to be easily fooled by changes to images that are often imperceptible to the human eye,but will mean that the AI system classifies objects incorrectly.This may be changing a few pixels in an image resulting in a system classi
20、fying a cat as a dog.In a defence context,an armoured vehicle being classified as a civilian car,or vice versa,may have catastrophic ramifications if a targeting decision is made without meaningful human input.Another well-known example is researchers at McAfee putting a small sticker on a 35mph spe
21、ed limit sign which tricked a Tesla into believing the limit was 85mph and accelerating to 50mph above the speed limit.10A similar experiment where two bits of tape were put onto a stop sign led the autonomous driving software to read it as a 45mph road sign instead.Extraction and inference attacksE
22、xtraction attacks seek to replicate a machine learning model by feeding it inputs and logging the outputs.11In other words,malicious queries will be used to expose details of the models internal details.Attackers may be targeting the model itself,or the data on which it has been trained allowing sen
23、sitive information to be extracted.12In the case of businesses this may be proprietary information and in the security sphere,it may be classified or otherwise sensitive information.Successful extraction attacks may then lead to carefully crafted evasion attacks,moving from a black box to a white bo
24、x scenario.5How worried should we be?Adversarial AI:Coming of age or overhyped?There has been increasing focus on the subject of adversarial AI with many publications highlighting particular vulnerabilities with machine vision,large language models,and neural networks.However,the move from laborator
25、y setting to deploying AAI in the real world(especially a battlefield)is very difficult.Most research to date on the topic of adversarial camouflage,such as the colourful patches mentioned in the introductory vignette,has taken place in a sterile environment.AI has been fed static images with the pa
26、tches pasted on top they have not been placed on real aircraft and tested,as the authors themselves admit.13Patches would need to be effective whilst the airborne sensor flies all the way over,which means it gets a view of the target at numerous angles.Real world research has been more problematic,f
27、or both friendly and enemy forces.Some image recognition models failed when presented with a desert environment.Equally,the altitude and standoff distance of the sensor and the size of the target vehicle in the cameras aperture also affected how effective adversarial patches were.One experimentfound
28、 that even with adversarial patches woven into camouflage paint,AI models would be able to correctly identify a mobile vehicle every 3.3 seconds in full-motion video,enough to track it accurately.14Indeed,the same research concluded that the adversary would need to print or paint adversarial patches
29、 the size of football fields to be truly deceptive,which limits the tactic to stationary high-value targets.Models can be tricked by feeding them minutely perturbed static images and audio,but this becomes much more difficult in the real world where changing perspectives and different types of noise
30、 make it difficult to keep up a ruse.It is not practical for an adversary to inject noise directly into a sensor.Indeed,adversarial attacks appear to be much more suited to disguising a static object,rather than one that is moving,like a person.The development of adversarial examples requires overco
31、ming several other challenges.There is difficulty in developing a generalisable model that will work on numerous vectors.It is not economical to make bespoke solutions every single time,unless a target is particularly valuable.Controlling the size of perturbations can be difficult as it is not known
32、 how large the aperture is.If the perturbation is too small it will not work,and if it is too large it may get spotted.Nevertheless,these adversarial examples could present a significant vulnerability to future AI systems.6Lack of focus and regulationAdversarial AI:Coming of age or overhyped?AI deve
33、lopers are striving to get their models to simply work,with little consideration for the robustness of the model.Moreover,there are no agreed standards for the robustness or security of machine learning models.There are several private endeavours such as the Microsoft and Mitre Adversarial ML Threat
34、 Matrix and the Cleverhans Python library on GitHub used to benchmark the robustness of ML models.15Looking to the future,agreed standards and regulations for AI security will help shape the research field,allow best practices to be shared,and give users peace of mind and trust in the systems.The UK
35、 Governments AI White Papernotes the importance of systems functioning in a robust and secure way throughout their lifecycle.16The International Organisation for Standardisation is also developing a series of standards for the safety,transparency,and robustness of ML models.The UKs National Cyber Se
36、curity Centre has also published guidance on the security of ML,offering a number of principles for practitioners,decision makers and IT security professionals.17In the US,the DefenseAdvanced Research Projects Agency(DARPA)created the Guaranteeing AI Robustness against Deception(GARD)programme which
37、 aims to create broad-based defences that work against numerous attack vectors.18Previous defences were designed for specific,pre-defined attacks which limits their efficacy.What can be done?7Adversarial AI:Coming of age or overhyped?How might governments and militaries look to prevent adversarial a
38、ttacks and mitigate their effects should they take place?In traditional cybersecurity,vulnerabilities can be patched and continue to be used by customers.This approach does not work for machine learning models.If a model is poisoned,it will have to be retrained from an earlier untainted version whic
39、h can be very costly.Equally,hosting the model on an encrypted cloud is no use if the model has been poisoned during development.Adversarial robustness is the term used to describe a models ability to resist being tricked or exploited.When models move from using training data to new data,the models
40、performance can change.As such,exposing models to adversarial examples when they are being developed can allow them to be strengthened against such attacks.One difficulty is trying to conceive of every different type of attack.AI models are coded by humans who define their parameters.If the attack i
41、s something that has not been foreseen,the ML model will struggle to reconcile this with what it knows.Training data might also be sanitised.Ensuring that the collection and labelling of data is thorough and accurate is a way to prevent the insertion of poisoned data.However,many larger models rely
42、on massive scale data scraping from the internet.Ensuring that models contain no bad data is a huge ask and may undermine the point of training the model in the first place if it becomes a very human-centric and analogue endeavour.In some cases,another AI system might be used as a filter.Moreover,ex
43、tensive testing on a series of discrete datasets can help make a model more robust.8Adversarial AI:Coming of age or overhyped?The resource cost of attacks should also be considered.A white box attack is less costly for an adversary than a black box attack.Without prior knowledge of the system,an adv
44、ersary must develop its own version of the model which takes time,effort,and money.Time conducting reconnaissance on another partys model also increases the chance of such efforts being noticed,which will cause a defensive reaction.However,it is likely that the adversary will be part way between the
45、 two.There is value in trying to make attacks uneconomical to the adversary,by denying them information about systems.Actively looking for adversarial attacks by monitoring models for failure patterns can also be a productive route.Importantly,research has found that there is no defence that cannot
46、be overcome by a specialised attack.19The field is moving quickly so keeping abreast of developments is important and will give situational awareness to developers and users.However,adversarial attacks are inherently brittle and appropriate pre-processing and well-designed models can effectively mit
47、igate most effects.20A growing library of resources to counter adversarial attacks can be found on GitHub.21The Alan Turing Institute has also published guidance on responsible design for AI systems in the public sector which remains relevant.22What can be done?9There is still time to address the ri
48、sks posed by adversarial AI.As is often the case,most progress is being made in the private sector,but it is government who will need to legislate or mandate appropriate standards in close partnership with industry and academia.Defence and security professionals should be alive to the threats posed
49、by adversarial AI,and the responsibility of mitigating those risks should not rest with developers alone.About the AuthorPatrick Hinton is the British Armys Visiting Fellow at the Royal United Services Institute(RUSI).He is an artillery officer with a professional background in missiles and uncrewed
50、 systems.Moving forwardAdversarial AI:Coming of age or overhyped?ReferencesThe photo on page 1 is from Defence Imagery.CC BY-NC 2.01.Li Ang Zhang et al.,Operational Feasibility of Adversarial Attacks Against Artificial Intelligence(RAND Corporation:2022),https:/www.rand.org/pubs/research_reports/RRA
51、866-1.html.2.Will Knight,“Military artificial intelligence can be easily and dangerously fooled,”MIT Technology Review,October 21,2019,https:/ Graham-Cumming,“How to beat an adaptive spam filter,”MIT Spam Conference,2004,https:/blog.jgc.org/2023/07/how-to-beat-adaptivebayesian-spam.html.4.Eugene Nee
52、lou and Alex Polyakov,“The History of Adversarial AI,”Hack in the Box Security Conference,2021,https:/conference.hitb.org/hitbsecconf2021ams/materials/D1%20COMMSEC%20-%20The%20History%20of%20Adversarial%20AI%20-%20Alex%20Polyakov%20&%20Eugene%20Neelou.pdf.5.“A Complete List of All(arXiv)Adversarial
53、Example Papers,”Nicholas.C,last modified June 15,2019,https:/ Pereira,“The Challenges of and Defending Against Adversarial Machine Learning,”Ooda Loop,last modified June 9,2023,https:/ Machado,Eugenio Silva and Ronaldo Goldschmidt,“Adversarial Machine Learning in Image Classification:A Survey Toward
54、s the Defenders Perspective,”ArXiv(September 2020).8.Kangyi Ding et al.,“A low-query black-box adversarial attack based on transferability,”Knowledge-Based Systems 226(August 2021).9.Payal Dhar,“Protecting AI Models from“Data Poisoning”:New ways to thwart backdoor control of deep learning systems,”I
55、EEE Spectrum,March 24,2023,https:/spectrum.ieee.org/ai-cybersecurity-data-poisoning.10.Patrick Howell ONeill,”Hackers can trick a Tesla into accelerating by 50 miles per hour,”MIT Technology Review,February 19,2020,https:/ AI:Coming of age or overhyped?References11.Federal Office for Information Sec
56、urity,Security of AI-Systems:Fundamentals-Adversarial Deep Learning(BSI:2022),https:/www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/Security-of-AI-systems_fundamentals.pdf.12.Adam C and Richard Carter,“Large Language Models and Intelligence Analysis,”CETaS Expert Analysis(July 2023).13.Ajaya Adhikar
57、i et al.,“Adversarial Patch Camouflage against Aerial Detection,”ArXiv(August 2020).14.Li Ang Zhang et al.,Operational Feasibility of Adversarial Attacks Against Artificial Intelligence(RAND Corporation:2022),https:/www.rand.org/pubs/research_reports/RRA866-1.html.15.Mitre,“Adversarial ML Threat Mat
58、rix,”GitHub,June 18,2021,https:/ 31,2023,https:/ Government,Policy Paper:A pro-innovation approach to AI regulation(DSIT,OAI:2023),https:/www.gov.uk/government/publications/ai-regulation-a-pro-innovation-approach/white-paper.17.National Cyber Security Centre,“Principles for the security of machine l
59、earning,”last modified August 31,2022,https:/www.ncsc.gov.uk/collection/machine-learning.18.Defense Advanced Research Projects Agency(DARPA),“Defending Against Adversarial Artificial Intelligence,”DARPA,February 6,2019,https:/www.darpa.mil/news-events/2019-02-06.19.Austin Short,Trevor La Pay and Apu
60、rva Gandhi,Defending Against Adversarial Examples(Sandia National Laboratories:2019),https:/www.osti.gov/servlets/purl/1569514.20.Li Ang Zhang et al.,Operational Feasibility of Adversarial Attacks Against Artificial Intelligence(RAND Corporation:2022),https:/www.rand.org/pubs/research_reports/RRA866
61、-1.html.21.Trusted AI,“Adversarial Robustness Toolbox,”GitHub,https:/ Leslie,Understanding artificial intelligence ethics and safety:A guide for the responsible design and implementation of AI systems in the public sector(The Alan Turing Institute:2019),https:/www.turing.ac.uk/sites/default/files/2019-06/understanding_artificial_intelligence_ethics_and_safety.pdf.11Adversarial AI:Coming of age or overhyped?