1、 The Future of Online SafetyA data-centric approachThe views expressed in this article are those of the authors,and do not necessarily represent the views of The Alan Turing Institute or any other organisation.Bertie VidgenIntroduction1The Future of Online SafetyOnline threats are increasingly varie

2、d,challenging,and widespread ranging fromhate speech to terrorism,from disinformation to child abuse.These disruptive,unwanted and often illegal activities present a clear risk to the safetyand wellbeing of individuals,platforms and societies.Despite incoming regulation inmany territories,increasing

3、 public support for action,and a growing economy ofvendors who provide products,these problems remain fundamentally difficult to solve.All of the solutions currently available raise issues of performance,free speech,proportionality,privacy and technological capability.There are no silver bullets whe

4、n dealing with a complex problem like online safety,butArtificial Intelligence(AI)has real potential to drive a step change in detecting andresponding to online threats.It can make law and policy enforcement more efficient andeffectiveby supporting,replacing and advancing on human-led interventions.

5、In recent years,AI has drastically improved and workflows for its use and deploymenthave been overhauled with the widespread use of large pre-trained models and transferlearning.However,these changes have not been fully leveraged in how AI is used foronline safety.In particular,the critical role pla

6、yed by data,and as such the uniqueposition and importance of data owners,has not been fully recognised.This articlediscusses these changes and their implications,and explores what this means for thefuture online safety sector.2How AI is used to tackle online threatsIn the past ten years there has be

7、en a surge inboth commercial and academic research inAI,leadingtoincreasinglysophisticatedmodels,entirely new techniques for trainingsystems,andmorerobustevaluationprocesses.This has resulted in the release of jaw-droppingimage generation models like Dall-E and StableDiffusion,powerful speech transc

8、ription modelssuchasWhisper,andNaturalLanguageUnderstanding models such as GPT,BERT andMegatron.Most of these models can be used toboth classify and generate content,achievinghuman and super-human levelsof performance.AI can also augment,support and advancehuman responses to online threats.It can be

9、used to automatically find threatening contentand malign accounts,tackle and mitigate theharmful effects of their behaviour,and monitorthem to better understand patterns,dynamicsand motivations.There are four features thatmake AI particularly suitable for tackling onlinethreats:The Future of Online

10、Safety1.Speed.AI can process content in milliseconds,which is near real-time.This means that users do not have to wait for the result,allowing the AI to be embedded into a range of automated monitoring,moderation and evaluation processes.2.Scale.AI can handle a huge volume of content.Even a small se

11、rver can process millions of items every day,with no upper limit.This is particularly important when there are unexpected events which lead to sharp spikes in content volume.AI can handle this easily,whereas human-only approaches struggle to scale.3.Consistency.Production-ready AI models behave dete

12、rministically.Given the exact same inputs,they will return the same output.Although some AI is brittle,meaning it is very sensitive to minor changes in the input data,in principle this means that a model can be trusted to behave in the same way each time,and crucially the AIs behaviour can be invest

13、igated post-hoc.4.Performance.Properly trained AI can be better than humans,particularly untrained humans,at making difficult decisions about content.This is not always the case and depends on the complexity and difficulty of the task.How AI is used to tackle online threats3These features mean that

14、AI offers two main advantages over human-onlyapproaches.First,it can increase the efficiency of online safety measures,savingmoney and time.For instance,a social media platform might identify extremistactivity by having human moderators review reports from users.It could makethis process more effici

15、ent by using AI to triage content and give an initialassessment of whether it is likely to be extremist.Second,AI can increaseeffectiveness by creating new ways of keeping people safe and secure,such asbeing embedded into the design of their products.For instance,many platformsuse AI to shape users

16、experiences by populating timelines,and downrankingpotentially harmful content.Other novel applications include using AI to powerbots that automatically generate counter speech,and to provide real-timewarnings and nudges to stop people from engaging in unsafe behaviour.Despite these recent improveme

17、nts,AI is far from perfect,and any system willhave weaknesses and flaws.AI struggles to handle context,nuance andintention,which are key limitations when dealing with online threats.Otherconcerns include the ethical and societal implications of using AI for automateddecision-making,and the need for

18、effective human governance at all levels.Arelated challenge is the environmental impact of training and deploying AImodels,which can use huge amounts of energy.i The suitability of using AIshould be assessed in the context of the application,and it may ultimately beconsidered inappropriate or unsafe

19、 meaning human-only approaches will stillbe preferred in some contexts.4AI workflows for tackling online threatsAI workflows have been transformed by thewidespreaduse of transfer learning.Today,most practitioners do not train their ownmodels from scratch but optimise and thendeploy extremely large m

20、odels(i.e.models withbillions of parameters)which have been trainedby big teams at well-funded organisations likeOpenAI and DeepMind,and then open sourced.These models are often called foundationmodels,and are distinct to specific appliedmodels,which we call classifiers.ii Foundationmodelsaretrained

21、throughself-supervisedlearningtasks,suchasmaskedlanguagemodelling,to develop an understanding ofcontent.Large language models,for example,are trained over thousands of hours on hugedatasets which comprise billions of entries,such as the Common Crawl Corpus.iii Originallydeveloped for English,variant

22、s have now beentrained for most major languages,and fullymultilingual models,such as XLM-R,have alsobeenintroduced.Effortsareunderwaytoimprove coverage of languages from the GlobalSouth,which have historically been under-represented in the AI community and are“lowresource”-meaning there is a limited

23、 volume oftraining data available when compared withmore dominant languagessuch as English.ivThrough transfer learning,practitioners canoptimise these foundation models to create aclassifier for a specific use case,such as findinghate speech,identifying extremist groups,ormapping the activity of bot

24、 networks.This articlefocuses primarily on models for analysing text but similar arguments apply to models for otherapplications for tackling online threats,such asimage models.Practitioners have a range of ways to improvemodel performance when training classifiers,ofwhich three are particularly imp

25、ortant.1.Continuedpre-training.Off-the-shelfmodels can be optimised by continuing theirtraining.The original task(e.g.maskedlanguagemodelling)isrestarted,usingeither randomly sampled in-domain data orunlabelled task data.vFor instance,a hatespeech classifier could be created by takinga pre-trained B

26、ERT model and showing itseveral million posts from social mediacommunities that host large volumes ofhateful material.This would give it a farbetter understanding of both social mediadata and toxic content.2.Fine tuning.Large models can be adapted toa specific task by retraining them on a smalldatas

27、et of labelled examples,which adjusts theweights and parameters of the upper layers ofthe network.viFine tuning is the most commonway in which large models are used and hasbeenshowntoachievestate-of-the-artperformance on a wide range of tasks,even withrelatively few examples.vii3.Model prompting.In-

28、context learning iswhere models learn a task by conditioning oneither no examples(“zero shot”),or just a smallnumber(“few shot”),and without optimising anyparameters.viiiThis is an incredibly quick andeasy way of training classifiers,although it isoften not suitable for complex tasks involved intack

29、ling online threats.Practitioners can use all of these techniques(and others),combining them as needed.Forinstance,a team could take an off-the-shelfmodel and continue pre-training on a largecorpus of messy and toxic social media data tocreate a new foundation model.They can thenfine tune it on task

30、-specific datasets to createnew classifiers,such as tools for detectingextremist,hateful or illegal content.The Future of Online SafetyAI workflows for tackling online threats5The Future of Online SafetyThe widespread use of transfer learning has created a split between AI researchers whoare creatin

31、g foundation models and practitioners who are applying these models tocreate classifiers.Nearly all work in online safety and security is conducted bypractitioners creating classifiers who are benefiting from the rising tide of increasinglypowerful and increasingly adaptable foundation models.Apart

32、from the largest andmost innovative teams,it does not make sense to swim against this tide.Three factorscontribute to making this the most viable approach for tackling online threats.1.Cost.The cost of training new foundation models is huge.GPT-3 is reported to havecost$12 million to train,ixand the

33、 open science model Bloom cost$7 million.x Thesefigures do not include the costs of the research teams,which typically have very highsalaries.For a team to start from scratch and not use these models is to effectivelythrow away the millions of dollars already spent on their development.2.Risk of obs

34、olescence.Model architectures are constantly improving because offierce competition amongst the big players,motivated as much by research glory asfinancial benefit.A small team could spend a small fortune to create their own AI model,only to find that within a year or two it is rendered obsolete by

35、an open sourced modelfrom big tech.3.Low switching costs.Off-the-shelf foundation models are now very easy to accessthrough services such as Hugging Face xiand no-code solutions such as Data Robot.xiiGenerally,it is as easy to set up the code and workflow to evaluate many models as justone.In practi

36、ce,therefore,teams can easily switch between foundation models andassess multiple models at once.They can also consider much smaller distilled modelswhich are typically much faster to run.xiii6Data data dataWith transfer learning and widespread accessto large models,the biggest challenge nowfacing p

37、ractitioners is how to acquire,labeland use the right data.In light of this,there has been resurgent interestin the role and curation of datasets as a crucialpart of the AI development process,with leadingexperts calling for a shift from a model-centricapproach to a data-centric approach to AIdevelo

38、pment.xivIn many ways,this reflects alongstandingmantraincomputerscience:GarbageInwillleadtoGarbageOut.xvNumerous studies show that a shockingly smallamountofgooddatacancreateahighperformingAIclassifier,whereaslargequantities of low quality data only result in a veryweak model.In nearly all applicat

39、ions of AI for tackling onlinethreats,there is no single way of determining“good”data.What counts as“good”datadepends on the context and the task at hand:good data for you might not be good data foranyone else.And,often your data is the bestdata it is the most relevant,in-domain and willbest reflect

40、 the task that motivated you to createan AI classifier.For instance,if you run aplatform and want to apply AI to detect usersexpressions of intent to self-harm,there is nobetter data to train it on than your data i.e.datataken from the platform and then labelled in linewith this task.Or,if you are a

41、 security agencythat wants to identify expressions of support foraspecificterroristgroup,basedonyouranalystsqualitativeanalyses,thenyourqualitative data is your best starting point as itreflects the type of content that you actuallycare about.There are two reasons for this.In-domain data.Your data w

42、ill be in-domain,which means that it is selected from the samepool of content that you will apply the classifierto,and therefore has similar features.xviIn-domain data is important because even fairlysmall differences between settings can radicallyalter the performance of classifiers.Considerthe dif

43、ferences between a gaming chat,repliesto a tweet,and a Facebook post or even justthe differences between a Twitch livestream foragaminginfluencercomparedwiththelivestream for a fashion influencer.All of themcould contain text,but the topics of the contentwill be very different,as well as the style o

44、fexpression,use of unusual symbols such asemoji,demographics of the content creators,and the norms in using external links andshorthands.Thisisalsowhyoff-the-shelfclassifiers that cannot be customised,such asstatic models provided by a third party vendor,may give reasonable performance but are veryu

45、nlikely to be optimal.Similarity of task.You can label the data in linewith the exact task that you want the classifier todeliver.This is true whether the data is in-domain or not,and whether or not you havecollected it new.Indeed,we have often foundvalue in reannotating datasets provided by otherpe

46、ople to reflect the categories that we careabout.Stating that the task is important maysoundobvious,orevenatruismbutpractitioners routinely use imperfect data thatdoes not quite meet all of their specifications asinonlinesafetythereareveryfewwell-established taxonomies and categories,and thefield is

47、 constantly changing.The advantage oflabelling data from scratch is that you canspecify the exact task you want the AI classifierto deliver.This means encoding the classifiersexpecteddecisionboundaryxviiinthedatathrough consistent labelling.xviiiThe Future of Online Safety7Data data dataThere is als

48、o no single best way of labellingdata,but it is crucial to be aware of thelimitations of different approaches.We havefound that some“market leading”data labellingproviders create datasets with serious errorsand inconsistencies,and at Rewire we have ateam of trained data labellers.They typicallyperfo

49、rm far better and more consistently thancrowdsourcedworkers,butarealsomoreexpensiveand require more coordination.In many cases,well-labelled in-domain data isnotavailable,availableonlyinverysmallquantities,orneedsahugeamountofprocessing to be usable for machine learning.To address the problem of not

50、 having enoughgood data,a range of data-centric techniqueshave been adapted specifically for online safety.In our work,we have used adversarial datageneration through the Dynabench platform,xixand active learning to find the most relevant andinformative cases when you have millions ofunlabelled entr

51、ies to select from but a limitedannotation budget.xxOther techniques includeusingdataaugmentationandsyntheticgeneration techniques,leveraging AI to actuallycreate new data to train future AI classifiers,andhandcrafting challenging perturbations.xxiAll ofthese approaches can be used to gain moreusefu

52、l data points or maximise the utility of thedata that practitioners already have.Finally,the shift towards data-centric AI hasclear implications for the online safety andsecurityeconomy.Increasingly,realvalueresides with the organisations that own the bestdataassets,whethertheyareplatforms,vendors,c

53、ivil society organisations or securityagencies.For most tasks,high quality data ishard to acquire requiring special relationshipswith platforms and a team of data analysts.Ofcourse,emphasising the importance of data isnot to trivialise the other significant challengesin building trustworthy AI class

54、ifiers,but simplyto argue that your best chance of building sucha classifier is to start with good data.Anyonewithout data assets,and a resilient data pipelineto ensure new data keeps coming in,willstruggle to deliver best-in-classresults.The implications are clear:organisations shouldtake a deep lo

55、ok at what data assets they haveand their capacity to build AI expertise.Onlythenshouldtheyassess(1)whattheirexpectations for their AI actually are;and(2)whether they want to bring in third partyvendors and,if so,in what ways.If good data isthe real“secret sauce”behind powerful AI andwhatcountsasgoo

56、ddependsonyourapplication and context you should thinkcarefully about who can add real value.The Future of Online SafetyConclusion8Online threats present a fundamentally difficult problem and will not be solved by any single technological innovation.But,given the huge scale and complexity of activit

57、y online,some form of automatedtechnology is now essential.In 2022,the real question is not should AI be used totackle online threats?but,instead,how should it be used?and,increasingly,whoshould implement it?The answers to these questions will determine whether wesee high-performing AI being widely

58、developed and used across the sector,keepingpeople safe by increasing efficiency and effectiveness of online safety,or an under-utilisation of overpriced and under-performing AI in the future fight against onlinethreats.About the AuthorDr Bertie Vidgen is CEO and co-founder of Rewire,a tech startup

59、building sociallyresponsible AI for online safety.He is also a visiting researcher at The Alan TuringInstitute and the University of Oxford,part of the World Economic Forums ExpertAdvisory Group on Digital Safety,and has over twenty peer reviewed papers on AI andonline safety.Previously,Bertie was H

60、ead of Online Safety at The Alan Turing Instituteand has been a specialist adviser to the Online Safety Bill Joint Committee in UKparliament(2021).He completedhisPhD at the Universityof Oxford.The Future of Online SafetyReferencesi Strubbell,E.,Ganesh,A.,and McCallum,A.2019.Energy and policy conside

61、rations for Deepl Learning in NLP.Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,p.3645-3650.Available at:https:/aclanthology.org/P19-1355.pdfii Bommassani,R.et al.2021.On the opportunities and risks of foundation models.Arxiv.Available at:https:/arxiv.org/ab

62、s/2108.07258iii Luccioni,A.,and Viviano,J.2021.Whats in the box?A preliminary analysis of undesirable content in the common crawl corpus.Arxiv.Available at:https:/arxiv.org/pdf/2105.02732v3.pdfiv Joshi,P.,Santy,S.,Budhiraja,A.,Bali,K.,Choudhury,M.2020.,The state and fate of linguistic diversity and

63、inclusion in the NLP World.Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,p.6282-6293.Available at:https:/aclanthology.org/2020.acl-main.560/v Gururangan,S.et al.2020.Dont stop pretraining:Adapt language models to domains and tasks.Proceedings of the 58th Ann

64、ual Meeting of the Association for Computational Linguistics,p.8342-8360.Available at:https:/aclanthology.org/2020.acl-main.740.pdfvi Merchant,A.,Rahimtoroghi,E.,Pavlick,E.,Tenney,I.2004.What happens to BERT embeddings during fine-tuning?Available at:https:/arxiv.org/pdf/2004.14448.pdfvii Devlin,J.,

65、Chang,M.,Lee,K.,and Toutanova,K.2019.BERT:Pre-training of deep bidirectional transformers for language understanding.Available at:https:/arxiv.org/pdf/1810.04805.pdfviii Xie,S.M.and Min,S.2022.How does in-context learning work?A framework for understanding the differences from traditional supervised

66、 learning.Available at:http:/ai.stanford.edu/blog/understanding-incontext/ix Wiggers,K.2020.OpenAIsmassive GPT-3 model is impressive,but size isnt everything.VentureBeat.Available at:https:/ Wiggers,K.2022.A year in the making,BigSciences AI language model is finally available.TechCrunch.Available a

67、t:https:/ Future of Online Safety9Referencesxi Hugging Face.N.d.The AI community building the future.Available at:https:/huggingface.co/xii Data Robot.N.d.Data Robot AI Cloud Platform.Available at:https:/ Sanh,V.,Debut,L.,Chaumond,J.,and Wolf,T.2019.DistilBERT,a distilled version of BERT:smaller,fas

68、ter,cheaper and lighter.Available at:https:/arxiv.org/abs/1910.01108xiiv Landing AI.N.d.Data-centric AI.Available at:https:/landing.ai/data-centric-ai/xv Vidgen,B.,and Derczynski,L.2020.Directions in abusive language training data,a systematic review:Garbage in,garbage out.Available at:https:/journa

69、ls.plos.org/plosone/article/authors?id=10.1371/journal.pone.0243300 xvi Li,Y.,Baldwin,T.,and Cohn,T.2018.Whats in a domain?Learning domain-robust text representations using adversarial training.Available at:https:/aclanthology.org/N18-2076.pdfxvii Gardner,M.et al.Evaluating models local decision bou

70、ndaries via contrast sets.Findings of the Association for Computational Linguistics:EMNLP 2020,p.1307-1323.Available at:https:/aclanthology.org/2020.findings-emnlp.117.pdfxviii Rottger,P.,Vidgen,B.,Hovy,D.,and Pierre,H.2022.Two contrasting data annotation paradigms for subjective NLP tasks.Available

71、 at:https:/aclanthology.org/2022.naacl-main.13/xix DynaBench.N.d.Hate speech.Available at:https:/dynabench.org/tasks/hsxx Kirk,H.R.,Vidgen,B.and Hale,S.A.2022.Is more data better?Re-thinking the importance of efficiency in abusive language detection with transformers-based active learning.Arxiv.Available at:https:/arxiv.org/abs/2209.10193xxi Kirk,H.,Vidgen,B.,Rottger,P.,Thrush,T.and Hale,S.2022.Hatemoji:a test suite and adversarially-generated dataset for benchmarking and detecting emoji-based hate.Available at:https:/aclanthology.org/2022.naacl-main.97/The Future of Online Safety10