《OpenText&凯捷:2023-24年世界质量报告(英文版)(96页).pdf》由会员分享,可在线阅读,更多相关《OpenText&凯捷:2023-24年世界质量报告(英文版)(96页).pdf(96页珍藏版)》请在三个皮匠报告上搜索。
1、IN ASSOCIATION WITH:15th Edition 2023-24The future up closeWORLD QUALITY REPORTWORLDQUALITYREPORTRegional and country reports are available online from or from Capgemini,Sogeti,and OpenText local offices covering:North America,United Kingdom,France,Germany,Benelux,Southern Europe(covering Italy,Spai
2、n,and Portugal)Eastern Europe(covering Poland,Romania,Hungary,and the Czech Republic)China,Australia and New Zealand,Japan,and the Nordic region.2 0 2 3-2 4CONTENTSINTRODUCTION 04EXECUTIVE SUMMARY 06KEY RECOMMENDATIONS 08CURRENT TRENDS IN QUALITY ENGINEERING&TESTING 11 Business Assurance 12 Agile Qu
3、ality Management 17 QE Lifecycle Automation 22 AI,the Future of QE 28 Quality Ecosystem 35 Digital Core Reliability 42 Intelligent Product Testing 48 Quality&Sustainability 53SECTOR ANALYSIS 60 Automotive 62 Consumer Products,Retail,and Distribution 65 Energy,Utilities,Natural Resources,and Chemical
4、s 68 Financial Services 71 Healthcare and Life Sciences 73 Manufacturing 76 Public Sector 79 Telco&Media 82ABOUT THE STUDY 87ABOUT THE SPONSORS 92Welcome to the very special 15th edition of the World Quality Report,which is recognized as the industrys largest research study on quality engineering(QE
5、).The World Quality Report provides a comprehensive assessment of the current state of quality engineering practices around the world and across different industries.Over the last 15 years,this report has examined the most important trends and developments in quality engineering and testing by surve
6、ying 1,750 executives to put together the most interesting firsthand insights.Its a great honor for us here at Capgemini and Sogeti to publish this report,along with our strategic technology partner OpenText.We have made every effort to cover topics that are as wide and far-reaching as possible to g
7、ive you an in-depth view of the latest trends,challenges,transformative initiatives,and disruptions shaping them.In this report,you will see our key findings and recommendations for several focus areas:business assurance,agile quality management,QE lifecycle automation,AI,quality ecosystem,intellige
8、nt product testing,and sustainability.The expert findings are further accentuated by commentary,examples,and best practices from 10 senior executives from various Fortune 500 organizations,who participated in our deep-dive interview series around these topics.CHANGING TIMES AGAINIn the past editions
9、,we saw an immense drive for faster and more impactful IT developments and digital transformation.We also saw an even greater acceleration of that agenda on a global scale across all industries.Development of unified digital platforms,Agile solution delivery,hyper-automation,advanced analytics and A
10、I,and business and IT fusion were required to push organizations forward.Having the right level of focus on quality was thus even more critical.This year we see a continuation of these needs plus the emergence of a true game changer in the field of software and quality engineering:Generative AI adop
11、tion to augment our engineering skills,accelerated like never before.The lack of focus on quality seen in the last few years is becoming more visible now and has brought back the emphasis on the Hybrid Testing Center of Excellence(TCoE)model,indicating somewhat of a reversal trend.Business Assurance
12、 is the name of the game,and QE evangelists need to truly step up and elevate their real purpose to focus on assuring business outcomes.TOP,DOWN,ALL AROUND The future and what it holds for quality engineering and testing is essentially two-dimensional.One view is of the big picture macro dynamics an
13、d interdisciplinary synergies.The other is a deep dive into the nuances and micro shifts in the QE space.This years World Quality Report offers insights through both lenses:a pull-back panorama of the future as well as the inner workings of minor developments and moving pieces.A TEAM EFFORT This rep
14、ort wouldnt have been possible without the significant contributions of many people.If you are one of the 1,750 executives across 32 countries who took part in this years survey,we would like to thank you for your time and contribution.Your input was invaluable in helping us gauge the prevailing moo
15、ds and trends.We also have a special callout to the industry expert panel,whose insights have been valuable in illustrating the broader themes.We thank our partners at OpenText,and our lead authors,sector,and regional subject matter experts(SMEs)at Capgemini and Sogeti,who together INTRODUCTIONMARK
16、BUENEN Global Leader,Quality Engineering&Testing,Capgemini&SogetiSATHISH NATARAJAN Group Vice President,Quality Engineering&Testing,Capgemini&Sogeti,Americas4World Quality Report I 2023-24analyzed,interpreted,and provided expert commentary on the research data and interviews to build this report.In
17、addition,we thank the reports production team:much work takes place behind the scenes to ensure this annual exercise bears fruit.Finally,we thank you,our readers.Its your own experience and interest that gives the World Quality Report its reason for being.As ever,we hope this years edition makes a r
18、ewarding contribution to your continuing efforts in software quality assurance,and that you can take advantage of these findings and recommendations to shape your quality engineering strategy,and perhaps even to challenge some of your current thinking.OpenText is excited to deliver the 15th edition
19、of the World Quality Report.The ever-evolving global environment we face has highlighted the importance of emerging trends showcased in prior editions of the World Quality Report,from the rapid organizational adoption of cloud-based quality ecosystems to widespread investment in Artificial Intellige
20、nce(AI)solutions,all of which promise an exponential increase in productivity and efficiency.This edition shines a light on AI-the future of quality engineering.More than half of organizations anticipate that AI adoption will lead to higher productivity and increased velocity of development,and they
21、 plan to continue the investment towards execution of scaling the quality engineering process accordingly.Of course,AI also comes with challenges,such as 33%of respondents believe sustainability concerns and lack of privacy will present roadblocks to maximizing the value of AI in their organizations
22、.The organization that can overcome these roadblocks will be well positioned to be an industry leader going forward.Once organizations overcome the challenges and mitigate the risks associated with AI,the opportunities and applications are endless.From performance engineering and autonomous testing
23、to data generation and root cause analysis,AI is poised to permanently alter our understanding of what is possible.No matter how you decide to implement AI,the key will be making AI a part of your organizations DNA,which will require careful thought and analysis for prioritization along with unambig
24、uous KPIs measuring tangible outputs to drive successful quality engineering transformation with AI.As AI becomes ubiquitous,it will create the need for organizations to build holistic quality ecosystems in which data is simultaneously highly visible and secure for every part of the organization to
25、see and interact with.Moreover,64%of respondents recognize quality ecosystem sustainability as a priority,not just for themselves but for customers and investors as well,with the economic,social,and environmental benefits being profound.It is critical that organizations build a robust quality ecosys
26、tem now,because soon,having one will not be optional,it will be a necessity.Organizations must invest in workforce transformation by building subject matter expert and chaos engineering capabilities within quality engineering organizations to embrace and support the emerging trends in digital and cl
27、oud migration strategies of the customers.They will also have to accelerate the inclusion of cloud and infrastructure testing as part of the software development lifecycle to improve on resiliency,security,redundancy,and data recovery.Lastly,they will also need to build systems resistant and resilie
28、nt to failure by adopting stringent practices to test the impact of failure events on cloud and continue to give extreme high priority to the performance and resiliency of applications on cloud.We invite you to delve into the report to discover the many challenges and opportunities that organization
29、s will face in their endless pursuit of quality excellence and get a glimpse of the future up close.Finally,I would like to express my appreciation and thanks to Capgemini,and to everyone who has worked so diligently to create this edition of the World Quality Report.MUHI MAJZOUBChief Product Office
30、r,OpenText5EXECUTIVE SUMMARYWorld Quality Report 2023-24In the constantly changing world of business,the pursuit of quality has become a key factor in achieving sustainable,long-term success.Its now more important than ever to develop a systematic approach to identifying business risks and crucial p
31、riority areas.This 15th edition of the World Quality Report explores new and evolving approaches in quality engineering(QE),and provides an in-depth view of the latest trends,challenges,transformative initiatives,and disruptions shaping them.Quality is no longer a mere checkpoint;its a serious board
32、room agenda and part of a culture that organizations are increasingly embracing.As 38%of businesses shift their focus from volume to value,theres a growing recognition of the importance of quality in delivering value to customers.Notably,67%have incorporated quality assurance(QA)into the core of the
33、ir business operations,ensuring that their technology investments align with quality standards.Furthermore,almost all organizations have transitioned from conventional testing to agile quality management.Evidently,they understand the necessity of adapting to the fast-paced digital world.An agile qua
34、lity culture is permeating organizations,albeit often at an individual level rather than at a holistic program level.Many organizations are adopting a hybrid mode of Agile.In fact,70%of organizations still see value in having a traditional Testing Center of Excellence(TCoE),indicating somewhat of a
35、reversal trend.What is clear is the extended knowledge and skills that are required from the QE experts who operate in agile teams.Coding skills in particular(C#,Java,SQL,Python),and business-driven development(BDD)and test-driven development(TDD)competencies,are in demand.EXECUTIVE INTRODUCTION:EMB
36、RACING QUALITY TRANSFORMATION FOR SUSTAINABLE SUCCESSASHWINI NADIGERVice President,Portfolio COE Head for QE&T,Capgemini North AmericaJEAN-BAPTISTE BONNETOffer Leader,Intelligent Testing for Connected Products,Capgemini Engineering,FranceJEFF SPEVACEKHead of QE&T,Financial Services,Capgemini North A
37、mericaKAVITHA VELCHALPortfolio Manager,Testing Deal Architect for Global Sales Support Team,QE&T,Capgemini Financial ServicesANISH BEHANANHead of QE&T,Capgemini UKJEBA ABRAHAMPractice Vice President,QE&T,Sogeti USAANDREW FULLENHead of Innovation and Technology,Sogeti UK6World Quality Report I 2023-2
38、4It is also evident that operating in a full agile model does still create some serious challenges for many organizations.The most important agile quality challenges are a lack of knowledge of techniques and inadequate support for testing processes.To address these challenges,a shift towards product
39、-based quality engineering setup is recommended,with a focus on customer-centric design thinking,artificial intelligence and machine learning(AI/ML)adoption,and aligning to a business assurance mindset.Objectives such as improved customer experience,risk reduction,increased QA efficiency,and reduced
40、 testing effort drive automation in quality.While quality automation is certainly on the rise,organizations still face critical challenges.An average of 27%report challenges related to dealing with legacy systems and the fast-changing application landscape.Artificial intelligence(AI),and especially
41、the application of generative AI,is poised to be a game-changer in quality engineering.It is fundamentally redefining the art of the possible.The expected outcomes of generative AI are sure to drive higher productivity and efficiency gains.Almost 77%of organizations are reportedly investing in AI so
42、lutions to drive the quality transformation agenda.However,concerns related to security,privacy,and biased outcomes need to be addressed.A significant percentage(31%)remains skeptical about the value of AI in QA,emphasizing the importance of an incremental approach.Quality ecosystems are evolving ra
43、pidly,with a decisive shift in cloud adoption for test environments and the emergence of site reliability engineering(SRE).We are witnessing the metamorphosis of cloud testing,with over 82%of the organizations asserting the need for a cloud testing strategy as they modernize their application portfo
44、lios.Still,challenges persist in areas around test data provisioning for CI/CD pipelines without reliance on production data.For most industries,if not all,the digital core is at the heart of how you do business.More than ever before,the strategic purpose of testing is about delivering the best cust
45、omer experiences and digital happiness,protecting the brand image,and ultimately assuring successful business outcomes.This requires more focus on end-to-end customer journey validation and a purposeful collaboration with business to maximize impact.We will soon start to see newer approaches such as
46、 test isolation and contract testing become mainstream to sidestep the roadblocks in end-to-end business process testing.The interconnected product landscape brings a level of complexity never seen before.Demanding customer experiences,interoperability needs,regulatory pressures,constantly changing
47、guidelines,and cyber threats warrant a much more rigorous testing focus.There is a genuine need to reinvent the product testing approach for the globally interconnected spaces we interact in.Hyper-personalization for unique user experiences necessitates exhaustive testing to validate all possible co
48、mbinations,which is impossible to achieve.This is where AI takes center stage to drive maximum value,a key lever being an intelligent testing strategy.Lastly,organizations recognize the role of QA in driving the sustainability agenda.It is very encouraging to learn that 73%of the respondents showed
49、positive intent to become active in driving sustainability best practices in their organizations.A good first step towards making an impact is to own the process of bringing in transparency in measuring,reporting,and monitoring.They must act as custodians of sustainability and not allow greenwashing
50、 practices,which we are unfortunately starting to witness.In conclusion,succeeding in quality engineering and assurance requires embracing change,fostering a culture of quality,and leveraging advancements such as AI and automation.As organizations continue to evolve,a commitment to quality ensures s
51、ustainable success now and in the future.Ready to see the future up close?Lets take a look.7World Quality Report 2023-24KEY RECOMMENDATIONS Leverage the testing function to deliver business outcomes.This will ensure that business operation processes are aligned with business standards and objectives
52、.Use value stream mapping to strengthen business assurance if you want to create the right conditions for improved business results and long-term success.Embrace change and foster a culture of continuous improvement.Prioritize continuous improvement to drive incremental change for sustainable succes
53、s.BUSINESS ASSURANCE AGILE QUALITY MANAGEMENT Invest in robust training programs:Organizations must adopt a build-or-buy mentality.Developing a robust training program reduces overall costs while setting your organization up to continue to adapt to the industrys ever-changing technology and skillset
54、 requirements.Integrate with DevOps/DevSecOps:Forging a seamless integration with DevOps/DevSecOps practices through close collaborations can ensure the smooth incorporation of quality engineering assets into CI/CD pipelines.Modernize your QE organization:To empower individual agile aligned teams to
55、 be more self-sufficient.8World Quality Report I 2023-24 QE LIFECYCLE AUTOMATION Continue to expand on what works dont adopt new automation technologies or platforms just because they are new or trendy.Automation can be prioritized against legacy systems focus on automating the test scripts and tech
56、nologies which yield a larger and faster return on investment first.Upskill teams to use AI and to evaluate AI systems Develop training plans and career pathways so that your organization knows how to leverage and assess these systems.Set up an Automation Marketplace platform Let your automation eng
57、ineers own the development and maintenance of the scripts while other teams integrate them into their daily routines.AI,THE FUTURE OF QE Organizations need to continue the investment towards scaling the Quality Engineering process,skills,and bandwidth accordingly.Implementation of AI use cases in Qu
58、ality Engineering will require careful thought and analysis for prioritization along with unambiguous KPIs measuring tangible outputs to drive successful transformation with AI.An iterative approach towards chosen prioritized use cases,along with an MVP strategy is recommended as there is going to b
59、e considerable experimentation around AI solutions in the near future with multiple unknowns,that will get uncovered during the journey.QUALITY ECOSYSTEM Accelerate the inclusion of cloud and infrastructure testing as part of the software development lifecycle,to improve on resiliency,security,redun
60、dancy,and data recovery.Continue to consider intelligent integrations to make the best use of commercial off-the-shelf and cloud-native tooling as a hybrid strategy for non-production environment provisioning.Adopt stringent practices to assess the impact of failure events on cloud,and continue to g
61、ive extremely high priority to the performance and resiliency of applications on cloud.Invest in workforce transformation in building SRE and Chaos Engineering capabilities within quality engineering organizations.Use newer approaches like test isolation,contract testing etc.,to drive more segmentat
62、ion and higher automated test execution.Utilize service virtualization to overcome challenges around multiple applications,environments,and data dependencies.QA teams of the future will need to have a mix of skills strong testing skills need to be augmented with some domain and platform skills to dr
63、ive efficiencies.INTELLIGENT PRODUCT TESTING Invest in AI solutions for test prioritization and test case selection to drive maximum value from intelligent testing.Focus on an E2E testing approach to ensure seamless customer experience through effective utilization of abstraction of various architec
64、ture layers.Give equal priority to the non-functional aspects of the system as the functional aspects as they equally influence the end-user experience.QUALITY&SUSTAINABILITY Have a set of key KPIs or metrics that can be easily understood across the organization.Ensure regular reviews of the KPIs,an
65、d ensure they remain effective in driving improvements.KPIs need to be considered not just in QE&T but through requirements gathering,production support,and working with external vendors.Sustainability needs to be part of the whole lifecycle,included as a non-functional requirement,tested-for as a r
66、equirement,and reported on against the success(or not)of that compliance.Know your green success factors and ensure they are part of your strategy,your education,your procurement,and your hiring processes.DIGITAL CORE RELIABILITY910World Quality Report I 2023-24CURRENT TRENDS IN QUALITY ENGINEERING&
67、TESTINGBusiness Assurance 12Agile Quality Management 17QE Lifecycle Automation 22AI,the Future of QE 28Quality Ecosystem 35Digital Core Reliability 42Intelligent Product Testing 48Quality&Sustainability 5311BUSINESS ASSURANCE ASHWINI NADIGER Vice President,Portfolio CoE Head for Quality Engineering&
68、Testing,Capgemini North America KYLE ABRAHAM Associate Vice President,Quality Engineering&Testing,Capgemini USA KAUSHIK MUKHERJEE Associate Vice President,Quality Engineering&Testing,Capgemini USA VICTORIA VOINIGESCU Sr.Product Manager,OpenText OREN ZIV Product Manager,OpenText“The first thing is,re
69、ally listening to our product teams that represent the business.These product teams not only have product owners,but also our business transformation teams that do the end-to-end program-level introduction.Everything for us is about our guest experience at each of our destinations.”FABIAN RODRIGUEZV
70、ice President,Application Delivery&Strategy|Digital&Technology,Universal Destinations&Experiences,NBC Universal“Business results are assessed by ensuring the reliability of the orchestration layer,cost-effectiveness in resource allocation,automation,and defect prevention strategies.Some of the chall
71、enges in achieving these objectives are:keeping up with rapidly evolving technologies and platforms,the scale and complexity of the orchestration layer (as we continue to grow rapidly),and releasing new features quickly while also ensuring quality standards.”VYSHALI REDDYSenior Director,Mobile Quali
72、ty Engineering(MQE),Charter Communications12World Quality Report I 2023-24The last few decades have transformed the scope of testing,as well as the definition of business assurance practices.Business assurance has now become an integral part of strategic business functions and is considered imperati
73、ve for long-term organizational success.Business assurance encompasses a systematic approach to determining business risks and focusing on what really matters the most.It consists of a comprehensive plan,methodologies,and approaches to ensure that business operation processes and outcomes are aligne
74、d with business standards and objectives.In a volatile business landscape that demands full control,business assurance can help new products and services launch ahead of the competition with a consistent level of quality.Organizations are finally recognizing business assurance as a necessity,and we
75、think that is driving a fundamental shift in business culture today.We see more and more businesses adopting comprehensive business assurance practices to mitigate risks,ensure quality,promise reliability,maintain compliance,and safeguard the organizations reputation and financial stability.Business
76、es are even diverging from traditional,set ways of functioning to adapt to business assurance principles,which is reassuring.A SHIFT IN MINDSET:VALUE OVER VOLUME Todays dynamic digital age has steered a host of new opportunities toward us.However,managing multiple streams in complex business project
77、s and delivering the highest quality in omnichannel customer experiences is a herculean task.A business assurance practice is a necessity today.We live in a world where a single social media post could cause an irreversible dent in an organizations brand image and reputation.And one lapse in busines
78、s judgment could hamper customer satisfaction at large,and undo achievements gathered over the years.Its a tightrope,and businesses must maintain a fine balance.BUSINESS ASSURANCE FOR BRAND PROTECTION,RELIABILITY,DIGITAL EXPERIENCE,AND POSITIVE BUSINESS OUTCOMES Business AssuranceWhich of the follow
79、ing best describes how much use you make of Value Stream Mappingand related techniques in optimizing the IT lifecycle?Optimization of IT lifecycleConducting a pilot14%12%2%1%0%71%Currently usingPlanning stage(pre-pilot)Tried and no longer usingOn the roadmap in the next 12 monthsNot consideringTOTAL
80、 n=1750WQR 2023-Global ResultsFIG 113One of the key trends of this years survey is the rapid adoption of a business outcome-driven approach across enterprises.Many businesses are now shifting from mere output to a results-driven mindset with value stream mapping(VSM).This has brought an increased fo
81、cus on knowledge that helps businesses carve out their niche and differentiate from competitors.Value stream mapping is one of the key business drivers focusing on customer-centricity,adding value to the customer,and enhancing customer experience.In this years WQR survey,we find a sharp increase in
82、the adoption of VSM and related techniques in optimizing the IT life cycle-71%,which is up by 11%from last year.How do you drive business outcome in your projects?Base:Those Who Selected Chief Information Officer AND Chief Marketing Officer ORChief Digital Officer AND Chief Technology Officer/Produc
83、t HeadBuild delivery pods and drive large-scale transformation58%49%46%45%38%64%Drive the behavior skill or capability toget a measurable business resultBuild and sustain customer trustBuild capabilities aligned with customer needsEnable responsive Business GovernanceDrive value over volumeTOTAL n=7
84、00Driving business outcomesWQR 2023-Global ResultsFIG 2Organizations are prioritizing value over volume,indicating that they are harnessing non-linear growth.VSM helps deliver business outcomes through a systematic approach.It defines,visualizes,and analyzes processes,and optimizes them by eliminati
85、ng inefficiencies.It aligns all operational activities with customer needs and strategic goals.This preference for value has also been key for over 64%of organizations actively undertaking product-oriented delivery models.This is driving large-scale transformations and has enabled 46%of organization
86、s to build specialized capabilities aligned with customer needs.Importantly,another positive step in this direction:theres more acknowledgment that business assurance is not a one-step effort,but a continuous journey to ensure long-term organizational success.The fact that 58%of organizations are ex
87、periencing a shift in mindset and behaviors to get measurable business results,and 49%are building and sustaining customer trust by going beyond conventional testing KPIs,affirms this.14World Quality Report I 2023-24WQR 2023-Global ResultsWhat is the involvement of QE in performing Business Assuranc
88、e testing?QE involvement in Business Assurance testingBase:Those Who Selected QA Testing/Testing Manager ANDQuality Engineers.Dont know excluded from base.Question sample size=314OwnerInvolvedNot involvedBusiness Operations-driven:UAT done by Operationsteam as proxy for BusinessBusiness&Testing Coll
89、aboration:Balanced UAT donecollaboratively by Business,BAs and TestersTesting-driven:UAT done by Testers&IT BAs with limitedBusiness participationBusiness-driven:UAT done primarily by Business who have astrong understanding of the applications11%14%17%19%54%56%54%53%34%30%29%28%FIG 3IS THE CULTURE S
90、HIFT AFFECTING THE ROLE OF A QE FUNCTION?Interestingly,along with a change in priorities,the culture shift is also affecting the role of quality engineering(QE)and its function within the business.We found that 56%of organizations are now successfully fostering collaboration between their business a
91、nd testing teams to achieve a balanced approach to user acceptance testing(UAT).This shows that both business stakeholders and testers are working together during the UAT process to drive value.These findings also indicate how QE has evolved and integrated with business to successfully deliver busin
92、ess outcomes by meeting business standards and objectives and raising overall quality standards.Business AssuranceTAKING A RELENTLESS ROAD TO CONTINUOUS IMPROVEMENTIt is encouraging to see that organizations are embracing the culture of QE and continuous improvement to remain effective,relevant,and
93、aligned with the changing needs of the business environment.Having said that,driving this improvement is not only about making incremental changes but its also a strategic approach.Interestingly,this years survey shows that 56%of the organizations are aligning strategic business goals,products,and v
94、alue streams,and 53%of organizations are leveraging fact-based,measurable methods and monitoring.Overall,49%of the organizations are measuring the numerous benefits of continuous improvement,including increased efficiency,quality,innovation,and customer satisfaction.Additionally,38%of organizations
95、are prioritizing continuous improvement by changing their business mindset and driving incremental change to be better positioned to adapt,thrive,and achieve their long-term goals.15RECOMMENDATIONS AND SUMMARY1.Leverage the testing function to deliver business outcomes.This will enable a fundamental
96、 shift in the organizations culture and mindset to ensure that business operation processes are aligned with business standards and objectives.2.Use value stream mapping to strengthen business assurance.Align processes with customer objectives to mitigate risks,protect reputation,and further drive q
97、uality,reliability,customer satisfaction,and financial stability.Collectively,these factors create the conditions for improved business results and long-term success.3.Embrace change and foster a culture of continuous improvement.Prioritize continuous improvement to drive incremental change for sust
98、ainable success.Set goals,evaluate solutions,and iterate for ongoing growth.The scope of business assurance is massive and there is no one-size-fits-all solution when it comes to its application in enterprise processes.However,with mindful strategies that are hyper-focused on business outcomes,we ca
99、n create thriving ecosystems that are ready to take on the future.FIG 453%49%38%38%36%31%56%What are the top three most important considerations when drivingcontinuous improvement?Top three considerations when driving continuous improvementAbility to align between strategic business goals,products,v
100、alue streams or portfoliosLeverage fact-based,measurable methods and monitoringAbility to measure efficiency of processBusiness mindset and thinkingDrive incremental changeContinuous transformation and innovationSustainable and repeatable process improvementTOTAL n=1750WQR 2023-Global Results16World
101、 Quality Report I 2023-24 JEFF SPEVACEK Head of Quality Engineering&Testing,Financial Services,Capgemini North America KAVITHA VELCHAL Portfolio Manager,Testing Deal Architect for Global Sales Support Team,Quality Engineering&Testing,Capgemini Financial Services RAMA YAMMANURU Principal,Quality Engi
102、neering&Testing,Capgemini USA PARINITA PATANKAR Vice President,Quality Engineering&Testing,Sogeti India BARRY WESTON Delivery Head of Quality Engineering&Testing,Sogeti UK HITESH PATEL Region Practice Manager,Quality Engineering&Testing,Sogeti USA AYAL COHEN Director,Product Management Functional Te
103、sting,OpenText MOTTI FINE Product Manager,OpenText OREN ZIV Product Manager,OpenTextAGILE QUALITY MANAGEMENT“Quality Engineering needs to take on a“Quality as Code”mentality.Instilling a higher degree of development capabilities into the team,so that they can more closely integrate with the overall
104、development lifecycle”.IFTEKHAR IVAANHead of Quality Engineering of a Major Global Bank“Quality requires more attention than ever because you are now on a massive growth path.People are moving towards more digitalization and you must ensure what you release or what you give to customers is of the hi
105、ghest quality”.SHIVAPRAKASH MOOLYAManager,Quality Engineering,BUPA(UK)17IntroductionThe adoption of agile principles across the industry is beyond nascent stages and most organizations today are well into their agile and DevOps journey.However,the approach quality engineering organizations have take
106、n varies;some have taken a ground-up approach,focusing first on workforce transformation,while others have taken a top-down approach,first choosing to re-structure their quality engineering(QE)organization from a process standpoint.Organizations are now prioritizing development skills over tradition
107、al testing skills as the most critical skills for quality engineers.Development-focused skills like C#/Java/SQL/Python and CI/CD are all ranked in the top 5,while traditional testing skills like automation and performance tooling ranked at the bottom of the results.Many organizations are undergoing
108、some form of workforce transformation,evolving their traditional quality engineers into full-stack quality engineers.When looking at QE organizations through the lenses of organization setup and critical skills,it can be interpreted that most organizations are taking a grassroots or reactionary appr
109、oach choosing to upskill their team first and then focusing on change at an overall organization level.FIG 5No formalstructure/ad hocTraditional TestingCenter of Excellence70%Product Aligned QE(combination ofCapability Units withPOD aligned teams)28%Fully DevAligned2%TOTAL n=300Base:Those Who Select
110、ed IT DirectorsWQR 2023-Global ResultsHow is your Quality Engineering organization set up to support agile projects today?Setup of QE organization to support agile projectsThe concept of agile organizations has been a part of boardroom conversation over the past decade.Businesses continue to pursue
111、the goal of remaining relevant in volatile,uncertain,complex,and ambivalent environments.Over time,this concept has evolved into much more than a mere development methodology.It has become a way of thinking and a mindset that emphasizes continuous improvement,adaptability,and customer-centricity.Thi
112、s evolution is clearly represented in the trends shaping agile quality management(AQM)practices today.Last year,we noticed a paradigm shift in AQM through the emphasis placed on embracing agile principles rather than merely implementing agile methodologies.This year,we try to uncover how deep this u
113、nderstanding of agility runs.ARE BUSINESSES AGILE READY?18World Quality Report I 2023-24Agile Quality Management OfficeWE SEE THE QUALITY ENGINEERS ECOSYSTEM EVOLVINGThere is a clear trend that suggests a parallel evolution in the skills makeup of quality engineers.Initially automation-centric,the r
114、ole has shifted towards a more development-oriented approach.A quality engineer today is expected to support in-sprint development activities,with respondents citing development skills such as SQL/Python,CICD/Orchestration,and C#/Java as the most critical skills for quality engineers today.The globa
115、l survey of agile-oriented organizations revealed that 42%of the respondents felt scripting language like C#/Java/SQL/Python is the most critical skill required by todays quality engineer,39%voted for CI/CD and orchestration closely followed by BDD/TDD at 38%;compared to just 28%for automation tooli
116、ng and 24%for performance tooling.While skills like ETL,open-source testing solutions,Artificial Intelligence(AI)/Machine Learning(ML),cloud,and tooling were considered important;the above-mentioned three stood ahead of the rest.We feel this is in alignment with the industrys continued focus on spee
117、d to market;as quality engineers continue to adopt more of a developer mindset,their ability to introduce automation earlier in the lifecycle increases as is their ability to troubleshoot and even remediate defects on a limited basis.FIG 6Base:Those Who Selected VP Applications/Product OwnerAND QA T
118、esting/Testing Manager AND Quality EngineersTOTAL n=58024%Performance Test tooling28%Automation Test tooling29%Cloud fundamentals(OCP,AWS,GCP or Azure)32%AI/ML33%Open source testing solutions35%ETL/Big Data e.g.Hadoop fundamentals38%BDD/TDD39%CI/CD,Orchestration42%C#/Java/SQL/PythonPlease select the
119、 five(5)most critical skills for your Quality Engineering Associates.Top five most critical skills for Quality Engineering AssociatesWQR 2023-Global Results19HOWEVER,WITH EACH INDUSTRY TREND,A NEW SET OF CHALLENGES EMERGEEach region appears to have its own set of challenges which aligns with how the
120、 QE organizations are set up today.The US and Canada face similar primary challenges,with 65%of respondents confirming a general lack of supporting test processes;although 86%in Canada confirming a lack of coding skills,and 70%of US respondents confirming a limited career path.This is indicative tha
121、t organizations are attempting to operate in traditional Testing Center of Excellence(TCoE)models even today.To stay competitive in the market,organizations are pushing to get new features and enhancements out faster,which puts a greater focus on speed to market instead of a long-term view on mainta
122、inability.On the contrary,other regions like the UK reported a lack of knowledge of agile techniques(77%)as the most common challenge.This could indicate the speed at which these organizations moved into a product-aligned QE model while still trying to utilize their traditional testers.Global survey
123、 results indicate all the above-mentioned challenges need to be addressed as more and more organizations“embrace”agility in the true sense.SHIFTING TO A PRODUCT-BASED QUALITY ENGINEERING SETUP MIGHT BE KEY TO OPTIMAL AGILE QUALITY MANAGEMENT PRACTICESMany quality engineering organizations predicted
124、the evolution and took a more proactive stance by shifting from a traditional Testing Center of Excellence approach to a product-based quality engineering setup;however,most organizations are still in the process of truly integrating their quality engineering teams into an agile-centric model.As sho
125、wn in the diagram above,only 4%of respondents report that more than 50%of their quality engineering teams are operating in a more agile-centric pod-based model.FIG 7Base:Those Who Selected QA Testing/Testing Manager AND Quality Engineers At S1What are the major challenges faced by quality engineers
126、in your Agile project?Major challenges faced by quality engineers in Agile projectsLack of time totest or automate63%64%59%77%41%54%79%64%64%68%Lack of codingskills68%68%60%86%62%73%68%57%36%50%18%63%Lack of supportingtesting processes60%71%52%36%64%68%64%79%82%32%Limited careerpath for testers59%29
127、%62%59%50%70%57%71%73%Lack ofknowledge ofAgile techniques57%50%66%55%77%50%64%36%64%TOTALn=315Canadan=14Francen=29Germanyn=22UKn=22USAn=56Braziln=14Aus+NZn=14Chinan=11Netherlandsn=19WQR 2023-All Countries Results20World Quality Report I 2023-24CONSIDERING THE CURRENT LANDSCAPE,WE HAVE A FEW RECOMMEN
128、DATIONS FOR ENSURING AGILE ADHERENCE IN BUSINESS OPERATIONS.Invest in robust training programs:Organizations must adopt a build-or-buy mentality either looking for new talent in the market or upskilling their current workforce to meet the evolving demands of the business.Replacing the existing workf
129、orce will yield faster results but comes at the drawback of increased cost and reduced sustainability.Developing a robust training program reduces overall costs while setting your organization up to continue to adapt to the industrys ever-changing technology and skillset requirements.Integrate with
130、DevOps/DevSecOps:Forging a seamless integration with DevOps/DevSecOps practices through close collaborations can ensure smooth incorporation of quality engineering assets into CI/CD pipelines by identifying and rectifying issues promptly,enabling rapid and secure software delivery,and enhancing over
131、all product quality and customer satisfaction.Modernize your QE organization:Establish lightweight capability units for test process and governance,test automation,performance engineering,test data management,and test environment management to establish enterprise frameworks and tooling platforms.Th
132、is will enable a capability for all approaches,empowering individual agile aligned teams to be more self-sufficient.Now,more than ever,the role of quality engineering is elevating quality engineering teams are asked to take on a more development-focused persona and build utilities for the rest of th
133、e organization to leverage rather than focusing solely on safeguarding quality.The evolution of agile practices,the integration of AI and ML,and the synergy between DevOps and agile are transforming quality engineering in infinitely futuristic ways.The question is,when will organizations embrace the
134、se changes en-masse,adopt proactive strategies,and make them a norm?Next years report will probably unveil the answer.The product-aligned QE organization setup enables organizations to diverge from centralized automation,test data,and performance testing teams to nimbler capability units,relieving t
135、he overarching burden on quality engineers.Apart from this,businesses might be able to maximize outputs through:1.Customer-centric design thinking,which emphasizes rapid feedback and improves a businesss agility to inculcate an innovation mindset,aligned with customer needs.2.Incorporating Artificia
136、lIntelligence and Machine Learning in test automation in a business tech ecosystem can also directly contribute to higher-quality outcomes by predicting issues,analyzing data,and enhancing testing efficiencies.3.Embracing agile practices across the organization to promote better alignment and enabli
137、ng seamless collaboration to achieve goals.FIG 84%37%59%What percentage of your Quality Engineering associates are Agile POD aligned?Percentage of QE associates who are Agile POD alignedWQR 2023-Global ResultsBase:Those Who Selected VP Applications/Product Owner AND QA Testing/Testing ManagerAND Qua
138、lity Engineers AND Chief Information Officer”TOTAL n=1005TOTAL n=1005Average%29%0%None1-25%25-50%50-75%More than 75%Agile Quality Management Office21 ANDREW FULLEN,Head of Innovation and Technology,Sogeti UK JEFF SPEVACEK,Head of Quality Engineering&Testing,Financial Services,Capgemini North America
139、 CHAITALI LAMBAT,AI&Automation Lead,Quality Engineering&Testing,Sogeti India AYAL COHEN,Director,Product Management Functional Testing,OpenText JULIAN FISH,Director,Product Management,OpenText VICKY GIAVELLI,Director,Product Management,Performance Engineering and Virtualization,OpenTextQE LIFECYCLE
140、AUTOMATION Humankind has always been in the pursuit of simplifying work,and automation is probably one of the best fruits of this pursuit.Today,as we notice accelerated paces of delivery that need lightning-fast development speeds and adherence to quality benchmarks-QE Automation holds the promise t
141、o deliver seamless results.This year,we decided to probe its potential and extend our focus beyond functional automation and encompass the entire quality engineering(QE)lifecycle.We attempted to analyze the challenges in non-traditional automation areas to mirror those in conventional ones and to as
142、sess the far-reaching benefits of automation.And even though we are charting unfamiliar territories like Artificial Intelligence(AI)and Machine Learning(ML)we would echo what the English author Douglas Adams once said,“Dont panic,”because we believe that with automation,we learn,we adapt,we benefit,
143、and we will deliver.Let us dive into the findings of this years World Quality Report(WQR)survey and its revelations for the near future.WHAT IS EVERYONE EXPECTING OUT OF AUTOMATION?54%52%51%50%50%42%RiskreductionImprove customerexperienceReduce livedefectsReducetesting effortImprove testefficiencyRe
144、duce timeto marketWhat are the top three benefits you need automation to deliverto your organization?Top benefits needed from automationWQR 2023-Global ResultsTOTAL n=1750FIG 922World Quality Report I 2023-24ALL EYES ON AIFIG 10Which are the top three focus areas for automation in yourorganization o
145、ver the 12 months?Top three focus areas for automationWQR 2023-Global Results55%40%36%30%27%26%25%23%20%Testing AIMobileSecurityAudit and CompliancePerformanceAccessibilityEnvironment ProvisioningData creationRequirements18%FunctionalTOTAL n=1435Base:Those Who Selected Chief Information Officer AND
146、VP Applications/Product OwnerAND IT Directiors AND Chief Marketing Officer OR Chief Digital Officer AND Chief Technology Officer/Product Head AND VP OR Director Of Research AND Development/EngineeringQE Lifecycle AutomationIt is interesting to note that the top benefits from automation this year inc
147、lude risk reduction(54%of respondents),improved test efficiency(52%of respondents),and reduced live defects(51%of respondents).We were surprised to learn that reduced time to market held the least priority at just 42%of respondents while it had topped the list last year.We think the reason behind th
148、is could be that organizations are already moving as fast as they can and seeing less benefit in cutting time further.It is also interesting to note that in a year where many organizations are freezing or reducing their IT spending,improved test efficiency and reduced testing effort came up as key p
149、riorities.This shift is a clear indication that organizations are starting to achieve their speed-to-market goals,and now more focused on reducing technical debt and overall spending on IT instead.So,where are organizations placing their bets now?23BUT EVEN AS AUTOMATION CONTINUES TO RUN DEEP IN ORG
150、ANIZATIONAL ECOSYSTEMS,THERE ARE DILEMMAS TO DECODEFIG 1134%30%18%18%16%15%13%13%12%12%11%6%Legacy systems and decisionsLack of skillsCollaboration issuesLack of toolsComplexityLack of a strategyLack of dataRequirementsLack of timeLack of requirementsLack of environmentsLack of moneyWhen it comes to
151、 doing more with automation,what are the two mainthings holding your teams back?Main roadblocks to automationWQR 2023-Global ResultsTOTAL n=615Base:Those Who Selected IT Directors AND QA Testing/Testing Manager AND Quality EngineersWe were expecting AI to steal the limelight this year since it is th
152、e buzzword in the boardroom these days.We think that AI is the new tide that needs to be ridden with caution,which means quality engineering and testing(QE&T)teams need to understand how AI-based tools work for them,how they can do their jobs better,and bring better outcomes for their customers.More
153、 than 50%of the respondents were eager to see testing AI with automation,which is well ahead of the other top focus areas like security and mobile.Interestingly,we found that functional(18%of respondents)and requirements(20%of respondents)automation were of the least focus for organizations,presumab
154、ly because of the challenges in automating and regular updating involved in these areas.Perhaps,this is an area where we can expect to see AI tools becoming a key part of automation toolkits.Closely following this,were security and mobile.With 36%of the respondents voting for security to remain a ke
155、y focus area,we predict that risk reduction will continue to be a key benefit of automation.In a globally digitized economy,we are seeing end-user interactions dominating mobile devices,and mobile-first platforms taking precedence,as they will continue to do for the foreseeable future.Another notewo
156、rthy trend was how respondents chose non-functional(performance and accessibility)over functional automation features,indicating more focus on customer experience.24World Quality Report I 2023-24QE Lifecycle AutomationThe WQR survey suggested that legacy systems and decisions(34%of respondents),foll
157、owed by the lack of skills(30%of respondents)as the two things holding back automation.There is a continued need for developing technical skills to deal with both existing and upcoming technologies.But it was heartening to see that only 6%of the respondents claimed lack of money as an issue,making i
158、t evident that organizations are ready to invest in automation when the strategy is right and return on investment is attractive.FIG 12WQR 2023-Global ResultsIntegration of automation test suites into pipeline68%29%0%3%0%1-25%25-50%50-75%More than 75%TOTAL n=315Q10.Average%31%What percentage of your
159、 automation test suites(smoke+regression)are integrated into yourOrchestration/lifecycle/delivery pipeline?Base:Those Who Selected QA Testing/Testing Manager AND Quality EngineersTOTAL n=315WE ALSO SEE AN URGENCY TO OPTIMIZE TEST SUITES Overall,the survey revealed that only 3%of respondents organiza
160、tions have more than half of their suites integrated which may be due to diminishing returns or lack of QE access to orchestration instances.This comes as a surprise since most of the organizations now have some automated test suites integrated into their pipelines for smoke and regression testing.“
161、This shift-left mentality has also given testers the opportunity to shift their testing farther left and lean more on automation as an early to potentially only solution.The idea of in-sprint automation vs.regression-only automation has become a reality”.CHAD FETTERDirector of Quality Strategy and D
162、elivery,Medica25FIG 1358%Quality Engineering team doesnthave access to enterprise CI instance51%We are still defining our DevOps/DevSecOps roadmap44%Test Data availability37%Lack of skillset36%Test environment availability36%Limited Automation36%Management supportWhat are the most impactful challeng
163、es preventing your Quality Engineering organization fromintegrating into the DevOps/DevSecOps pipeline?Challenges preventing integration into DevOps/DevSecOps pipelineWQR 2023-Global ResultsBase:Those Who Selected VP Applications/Product Owner AND QA Testing/Testing Manager AND Quality Engineers AND
164、 Chief Technology Officer/Product HeadTOTAL n=75058%of the respondents stated that quality engineering teams still do not have access to an enterprise CI for instance.This may be due to a lack of knowledge of legacy systems,stability issues of test scripts and high maintenance.Especially,since we kn
165、ow that increasing QE integrations into DevOps/DevSecOps lifecycle requires greater collaboration between the development,quality engineering,and DevOps/DevSecOps teams.What was worth noting was that the more senior respondents(those furthest from the actual automation)reported higher levels of inte
166、gration than those solely responsible for the work.“If you look at productivity,hyper-automation,zero-touch,and full end-to-end lifecycle automation,people still struggle to get there.I look at AI-augmented tool ecosystems that really help us automate and make your testing life cycle faster and bett
167、er from a productivity side”.Director-Quality Engineering Group at one of the worlds leading financial organizationsINABILITY TO ADOPT ADVANCED AUTOMATION SOLUTIONSDespite the promises of low code/no code and automation-based frameworks,only 19%of respondents are currently using such tools,with near
168、ly 70%of respondents at the stage of testing a pilot.While low-code type tools can allow an easier entry point into automation,there is still a long way to go before they truly challenge more traditional and feature-rich automation options,especially in terms of productivity and efficiency.GANESH PR
169、ACTOOR26World Quality Report I 2023-24RECOMMENDATIONS AND SUMMARYWhile delivering good systems with quality is a priority,it also must be done in a timely manner.Quality engineering and testing teams must adapt to working with AI and testing AI systems.Even as lawmakers address the challenges of AI
170、systems,there are already legal requirements to prove AI systems have been tested and are fair.Considering legacy systems and decisions still have a major impact on automation teams everywhere.It is important to equip teams with the right skills to understand legacy systems and technologies,while al
171、so adapting and adopting AI systems which by no means will be easy.Our research leads us to the following recommendations to increase the value from your quality automation initiatives:Continue to expand on what works dont adopt new automation technologies or platforms just because they are new or t
172、rendy.There is elegance in simplicity;continue using the frameworks and solutions that are providing value.When new solutions arise,evaluate them first before pushing for full adoption.Automation can be prioritized against legacy systems focus on automating the test scripts and technologies which yi
173、eld a larger and faster return on investment first.Once you have achieved the desired coverage,capability and speed to market goals;then move on to the more challenging technologies.Upskill teams to use AI and to test AI systems AI-based systems are here to stay,whether used for testing or the focus
174、 of testing.Develop training plans and career pathways so that your organization knows how to leverage and test these systems.Set up an Automation Marketplace platform enables an“automation as an asset”approach,empowering development,infra and business communities to leverage automation test scripts
175、 which have been created.Your automation engineers own the development and maintenance of the scripts while other teams integrate them into their daily routine.FIG 14Are you using or plan to use advance automation solutions like Low Code/No Code or AI-based automation frameworks?Use of advance autom
176、ation solutionsWQR 2023-Global Results69%19%10%2%Conductinga pilotCurrentlyusingPlanningstage(pre-pilot)On theroadmap in thenext 12 monthsNotconsideringBase:Those Who Selected QA Testing/Testing Manager AND Quality EngineersTOTAL n=315QE Lifecycle Automation27AI,THE FUTURE OF QE“To be very simplisti
177、c,my experience and my view is that AI can do wonders for you,but not before you make yourself mature.You cannot jump in there prematurely because AI wants to take you from the mature level to the next level of excellence,not from the ground level.There are several use cases,real good use cases of A
178、I implementation,but none of them will be helpful if you are not good at your basics”.RAJESH NATARAJANSenior Director,Quality Engineering at Hiscox“We will need skill augmentation,we will need people who can operate bots,who know about the technology.The human workforce will not become obsolete with
179、 AI,instead,the expected skillset for them is going to change”.Senior Vice President of Quality Engineering of a Major Global Bank28World Quality Report I 2023-24 JEBA ABRAHAM Practice Vice President,Quality Engineering&Testing,Sogeti USA CHAITALI LAMBAT AI&Automation Lead,Quality Engineering&Testin
180、g,Sogeti India SAI GRANDHI Associate Vice President,Quality Engineering&Testing,Sogeti USA UDI WEINBERG Director,Product Management,OpenText AYAL COHEN Director,Product Management Functional Testing,OpenTextAMITA KARADKHEDKAR28World Quality Report I 2023-24Artificial Intelligence(AI)is the most exci
181、ting technological advancement today.While the concept of AI is not exactly new,Generative AI is a game changer and has brought in an exciting revolution in how AI is applied across industries and organizations.So much so that one of the biggest pioneers of the technology industry,Bill Gates,in his
182、blog The Age of AI has begun,*he says,“The development of AI is as fundamental as the creation of the microprocessor,the personal computer,the Internet,and the mobile phone.It will change the way people work,learn,travel,get health care,and communicate with each other.Industries across the globe wil
183、l reorient around it.Businesses will distinguish themselves by how well they use it.”So,what does this mean for the future of quality engineering(QE)?Lets dive right in.HOW IS THE SCENE UNRAVELING?We have been experimenting with AI to improve quality outcomes for a while now.However,with the advent
184、of Generative AI,we see for the first time that higher productivity is the primary outcome organizations are using AI for,and we are not really surprised.Its heartening to see data that shows us that 77%of organizations today consistently invest in AI and utilize it to optimize QA processes.We belie
185、ve that this is directly linked to the fact that the trust in training data for AI solutions is very high,which in turn reflects the robust infrastructure and processes that organizations have developed over the years to collect continuous telemetry from all parts of the quality engineering process.
186、TOTAL n=750Base:Those Who Selected VP Applications/Product Owner AND QA Testing/Testing Manager AND Quality Engineers AND Chief Technology Officer/Product Head65%Higher Productivity53%More Velocity41%Improved Customer Experience33%Better Reliability29%Lesser DefectsFIG 15WQR 2023-Global ResultsWhat
187、are the quality outcomes your organization is utilizing AI to drive?Quality outcomes driven by AIAI&the future of QE29Reference:*https:/ n=1750Please indicate how much you agree or disagree with the below statements with regards to the use of AIin your organization.WQR 2023-Global Results74%We activ
188、ely utilize AI to optimize ourtesting processes74%I have confidence that the data I use totrain my AI algorithms/platform is correctand accurate71%My organization is willing to act onintelligence provided by the AI platform68%Business owners trust the intelligenceprovided by the AI platform77%We con
189、sistently invest in AI solutions toimprove quality processes64%I have identified processes/applicationsthat can benefit from AIFIG 16Attitude StatementsWhen we deep dive into what quality engineering use cases those AI investments are driving,we found that the majority of organizations are using AI
190、towards building and improving test scope along with improving performance engineering,as well as the test ecosystem overall.30World Quality Report I 2023-24AI&the future of QEFIG 17How frequently,if at all,has your organization used AI for the following use cases?Frequency of the use of AI in use c
191、asesWQR 2023-Global ResultsAutonomous testing(scope discovery)Performance testing or Test prioritizationPerformance testing/engineeringTest environment provision55%32%13%53%36%11%51%37%12%50%38%11%47%38%15%37%52%12%37%49%14%40%52%9%40%46%14%44%37%19%Total sample size=1750Test case design(automated&m
192、anual)Automated scope selection(impact analysis)Self-healing&test executionCross analysis of test resultsand defect predictionContextual test data provisioning/generationEvent correlation and root causeanalysis of issuesUsed a lotUsed on some occasionsHave not usedTRENDS THAT CAUGHT US BY SURPRISETh
193、at said,there are some big changes in trends we need to delve deeper into.Reliability and fewer defects are at the very end of the list,which came as a big surprise for us.Improving the reliability of tests(be it rooting out stale tests or self-healing automated tests)and reducing the number of defe
194、cts(defect analysis,defect prediction,risky code analysis,etc.)have been focus areas for the application of AI to improve quality engineering for a while now.So,while it wasnt completely a surprise to see productivity be the top outcome organizations were aiming for,we didnt expect to see these two
195、at the very bottom of the list.Whats possibly causing this?WE THINK THERE COULD BE TWO FACTORS AT PLAY HERE:1.With the availability of multiple large language models(LLM),the ability to achieve much higher productivity is possible now without too much effort.Higher productivity naturally leads to hi
196、gher speed.Higher productivity and speed in the quality engineering process makes it possible to ship code more frequently with a high customer experience.Generative AI has drastically upended the cost vs.benefit equation for these use cases,effectively making them low hanging fruits,ripe for harves
197、t.31FIG 18What are the biggest roadblocks/concerns on utilizing Generative AI to improve quality?Biggest roadblocks of Generative AI to improve qualityBase:Those Who Selected VP Applications/Product Owner AND QA Testing/Testing Manager AND Quality Engineers AND Chief Technology Officer/Product HeadT
198、OTAL n=750Lack of privacy35%34%Sustainability concerns17%Lack of contextLack of resilience33%We dont recognise value in adopting AI31%Integrations with existing tools30%Regulations and compliance29%Scalability(hidden cost of use cases)27%Lack of accuracy(hallucinations)24%WQR 2023-Global ResultsBut,
199、interestingly,31%of organizations still dont see the value in adopting AI yet.For this group,we assume that the costs/risks outweigh the benefits and that is a bigger roadblock to overcome.It will probably require sustained investment/effort to break past the barrier.2.With Agile and DevOps practice
200、s being adopted across organizations,there is more continuous testing with multiple philosophies like“fail fast”and“perpetual beta”increasing the tolerance for defects being found,as long as they can be fixed quickly and efficiently.That said,it makes us wonder what(if any)additional changes we woul
201、d be seeing in the survey next year,with the rapid evolution of AI technologies.This is definitely something to watch out for.SOME OLD ROADBLOCKS,SOME NEWThe survey data also suggests that there are multiple challenges for organizations to overcome while driving quality outcomes through AI.The roadb
202、locks range from concerns on privacy and compliance to a lack of resilience and dealing with hallucinations and bias.We feel that the roadblocks to large scale AI adoption in QE havent changed too much over the years.Although organizations have learnt to deal with these roadblocks better,they havent
203、 all been completely overcome yet.With 77%of organizations investing in AI solutions,we feel that those solutions are coming very soon in the near future.32World Quality Report I 2023-24With the amount of investment in AI,there is bound to be a large influx of AI projects across organizations.Succes
204、sfully testing these solutions and deploying them while ensuring high quality will be critical for driving success in the future.AI&the future of QEFIG 1975%Focus on requirements to explainability,transparency,robustness,privacy54%More focus on data science skills as thesystem learns from its mistak
205、es as it goesand it is virtually impossible to reproduce a bug48%Test Data generation to cater to thelarge volumes of test datathat would be required35%Updated test approach(processes,methodology)as the system issophisticated and hard to break23%Additional skillsets in the QA teamWe dont have any pl
206、ans to testAI-based solutionsHow is your organization planning for testing of AI-based solutions?Plans for testing of AI-based solutionsWQR 2023-Global ResultsBase:Those Who Selected VP Applications/Product Owner AND QA Testing/Testing Manager AND Quality Engineers AND Chief Technology Officer/Produ
207、ct HeadTOTAL n=750WHAT WILL IT TAKE TO DRIVE SUCCESS IN THIS NEW AGE?Our respondents overwhelmingly plan to focus on requirements related to transparency,robustness,and privacy of these upcoming AI solutions along with a continued investment in data science skills within their quality teams.The othe
208、r thing we see from the survey is that there is no clear winner when it comes to where organizations want to use AI to improve quality outcomes.Of course,AI can bring benefits to every tenet of the quality engineering lifecycle.We,however,feel that the approach needs to be treated with caution given
209、 the risk of things quickly going too far down the wrong path when it comes to AI solutions,unless they are robust enough to begin with.33FIG 20Please indicate how much you agree or disagree with the following statements regarding your plans forartificial intelligence and testing.Attitude Statements
210、WQR 2023-Global ResultsWe will use AI to speed up our test design78%AI will be used to test data76%We will use AI for more data analysis ofuser feedback and production telemetryfor maximum business impact76%We will leverage AI technologies to buildself-healing test automation74%We intend to use AI f
211、or test resultsanalysis and defect prediction73%AI systems are going to be used to helpus optimize our test scope and increase velocity79%Base:Those Who Selected VP Applications/Product Owner AND QA Testing/Testing Manager AND Quality Engineers AND Chief Technology Officer/Product HeadTOTAL n=750REC
212、OMMENDATIONS AND SUMMARY Higher productivity driven by AI will rapidly increase the development speed and organizations need to continue the investment towards scaling the quality engineering process,skills,and bandwidth accordingly.Implementation of AI use cases in quality engineering will require
213、careful thought and analysis for prioritization along with unambiguous KPIs measuring tangible outputs to drive successful transformation with AI.An iterative approach towards chosen/prioritized use cases along with an MVP strategy is recommended as there is going to be considerable experimentation
214、around AI solutions in the near future with multiple unknowns that will get uncovered during the journey.34World Quality Report I 2023-24 ASHWINI NADIGER Vice President,Portfolio COE Head for Quality Engineering&Testing,Capgemini North America ANISH BEHANAN,Head of Quality Engineering&Testing,Capgem
215、ini UK PADMAJA ALAPATI,Practice Lead,Quality Engineering&Testing,Sogeti India DEEPA TALWARIA Director,QEP Offer Lead,Quality Engineering&Testing,Capgemini USA KYLE ABRAHAM Associate Vice President,Quality Engineering&Testing,Capgemini USA LIOR URBANI,Functional Testing Product,Manager,OpenText AYAL
216、COHEN,Director,Product Management Functional Testing,OpenText35QUALITY ECOSYSTEM“Rapid technological advancements require QE teams to keep pace with testing new platforms,devices,and technologies.QE teams will need to invest in upskilling,and adopting advanced testing tools and methodologies while m
217、aintaining flexibility to adapt to the evolving tech landscape”.VYSHALI REDDYSenior Director,Mobile Quality Engineering(MQE),Charter Communications“Its about testing to make sure that it works.Optimizing the customer journey and making sure that customers have an overall good experience and not just
218、 an accurate experience is important.So,whilst its obviously important that our customers get the right offer,we also want the customer experience to be as easy to navigate as possible”.MARK PAINHead of Product Delivery,Volkswagen Financial Services,UKIn the last few chapters,we pieced together vari
219、ous elements of the quality engineering(QE)landscape.However,to understand the quality ecosystem in depth let us take a step back or two and look at the bigger picture.The quality ecosystem comprises all the capabilities that enable effective testing of applications.This includes,but is not limited
220、to,functions such as test environment management,test data management,etc.Even as organizations continue to test and expand the quality ecosystem capabilities to hyper-focus on customer needs,we look at how emerging industry trends such as cloud testing and mobile testing impact the overall ecosyste
221、m.With site reliability engineering(SRE)increasingly being recognized as a necessary capability,and the eventual movement of workloads to cloud,we dive deeper to further understand the role of cloud testing and SRE in building exceptionally reliable and resilient software for the future.WITNESSING T
222、HE METAMORPHOSIS OF CLOUD TESTINGFIG 2182%18%0%0%Does your organization include cloud and infrastructure testing as partof the development life cycle?Inclusion of cloud and infrastructure testingWQR 2023-Global ResultsYes,for all the projects where applications are hosted oncloud,this is mandatoryNo
223、,this is not followed as a standard practiceThere is no awareness of the requirement of cloud testingand its benefitsThere is no requirement to do this as a separate functionTOTAL n=315Base:Those Who Selected QA Testing/Testing Manager AND Quality EngineersLast year,our World Quality Report(WQR)repo
224、rted that 57%of organizations considered cloud testing as a part of the software development lifecycle but not as a mandatory function.This year,however,82%of the survey respondents highlighted cloud testing as mandatory for applications on cloud.This highlights a positive and decisive shift in the
225、testing strategy that organizations are taking on cloud and infrastructure testing.It also demonstrates how important it is to test cloud-related features for functional and non-functional aspects of applications.This change in thinking is a result of organizations realizing that movement to cloud a
226、lone does not make the system available and reliable.36World Quality Report I 2023-24FIG 22Which,if any,of the following end-to-end tooling plans for cloud testing do you have?End-to-end tooling plans for cloud testingWQR 2023-Global ResultsCloud NativetoolingProject Specific decisionEnterprise-wide
227、 COTS toolingOpen-source tooling preferredIntegrated COTS and Open source toolingNone oftheseTOTAL n=315Base:Those Who Selected QA Testing/Testing Manager AND Quality Engineers00708071%63%50%46%38%1%This years survey also highlighted that 71%of organizations prefer cloud native tooling(wh
228、ile only 50%prefer commercial off-the-shelf tooling)to perform cloud testing,thus,alluding to the ease of using the former when workloads are using a specific cloud platform.FIG 23Cloud platform strategy for provisoning non-production environmentsWQR 2023-Global ResultsWhat is your cloud platform st
229、rategy for provisioning non-production environments?TOTAL n=315Base:Those Who Selected QA Testing/Testing Manager/Quality EngineersHybrid(On-Prem+Single Cloud)EnvironmentsSingle PreferredCloud PlatformMulti-Premise(On-Prem+Multi-Cloud)EnvironmentsMulti CloudEnvironments58%19%12%11%Quality Ecosystem3
230、7The applications hosted on cloud could be impacted by the unexpected turbulence and outages of the cloud services.An hour-long Microsoft Exchange Online outage in March 2023 and a fire that wreaked havoc for Google Cloud users in Europe are a few major examples in recent times.Such incidents have g
231、iven way to focus on site reliability and chaos engineering.Interestingly,a hybrid cloud strategy seems like another upcoming trend since many organizations are using this approach for their non-production environments over single cloud platforms.We think the shift to a hybrid cloud strategy is from
232、 a multi-cloud and a multi-premise model point-of-view and is driven by cost-related decisions.Nonetheless,with almost 58%of organizations adopting a hybrid cloud(single cloud platform along with on-premises environments)strategy this year,we think this trend stands to become the mainstream in the n
233、ext few years,even though 19%of organizations still prefer single cloud platforms to provision non-production workloads.But as more organizations jump onto the cloud wagon,a question looms large what is the point of an immaculate cloud-hosted site,if its unreliable?EMERGENCE OF SITE RELIABILITY AND
234、CHAOS ENGINEERINGSite reliability engineering(SRE)applies engineering to operations,aiding automation,and integration from a quality perspective,and 69%in this years survey emphasized this.While organizations seem to recognize that SRE offers transformative potential in design,scalability,and its fe
235、atures offer enhanced automation and integration to boost app reliability,the lack of a skilled workforce remains a pressing challenge to tackle.FIG 24How are your QA teams embracing the transformation that SRE can bring indesigning performant software and scalable practices?Embracing the transforma
236、tion that SRE can bringWQR 2023-Global ResultsTOTAL n=1040Base:Those Who Selected QA Testing/Testing Manager AND Quality Engineers AND IT Directors AND Chief Information Officer69%23%4%4%Increasing automation and integrationPrioritizing QA efforts to most impactful areas through SLIsIntegrate QA int
237、o engineering,development,and operationsRefocusing tests through SLIs and error budgets38World Quality Report I 2023-24FIG 25How important is it in your organization to test the impact of failure events on applicationsrunning on cloud especially using chaos engineering?Importance of testing impact o
238、f failure eventsWQR 2023-Global ResultsTOTAL n=1040Base:Those Who Selected QA Testing/Testing Manager AND Quality Engineers AND IT Directors AND Chief Information Officer0%24%71%4%Not at allimportantSomewhatimportantVeryimportantEssentialSimultaneously,we see a rise in the importance of testing fail
239、ure in cloud events,with 71%of respondents calling it very important.This brings chaos engineering into the scene.Most Quality Assurance(QA)organizations today are adopting it along with performance engineering to validate if applications hosted on cloud can handle any unplanned infrastructure outag
240、es,service,or configuration changes gracefully without impacting the end-user experience significantly.Hence there is an increasing need for QA organizations to adopt chaos engineering practices and tools to support their customers.Quality Ecosystem39OTHER TRENDS THAT HAVE OUR ATTENTION:ENTERPRISE L
241、EVEL TEST DATA STRATEGY,MOBILE APPLICATION TESTING,AI(ARTIFICIAL INTELLIGENCE)AND MOREFIG 2612%69%18%1%0%0%In your organization,is there an enterprise-wide data provisioning strategyadopted as a standard across all line of business/projects?Adoption of enterprise-wide data provisioning strategyWQR 2
242、023-Global ResultsTOTAL n=1040Base:Those Who Selected QA Testing/Testing Manager AND Quality Engineers AND IT Directors AND Chief Information OfficerYes-a strategy isfully in placeYes-a strategy iscurrently beingadopted/almost readyYes-we have juststarted developing/adopting a strategyNo-we do not h
243、avea strategy in placeyet but we plan to inthe next 12 monthsNo-we do nothave a strategy inplace yet but we planto in the next2-3 yearsNo-we do not havea strategy in placeyet but plan to inthe next 4-5 yearsIt is understood that effective test data management is vital in the test ecosystem.The growi
244、ng demand for end-to-end automation highlights the need for abundant,on-demand test data.This year,69%of organizations adopted or prepared for an enterprise-wide test data provisioning strategy a significant leap from last years 31%.Data privacy remains crucial,yet 61%of organizations overlook integ
245、rating test data provisioning with delivery pipelines.This challenge hinders the provision of suitable test data at the pace of automation.40World Quality Report I 2023-24FIG 2776%41%35%28%20%What steps are being taken by your organization to make mobile application testing more effective?Making mob
246、ile application testing more effectiveWQR 2023-Global ResultsTOTAL n=315Base:Those Who Selected QA Testing/Testing Manager/Quality EngineersAI and Data Science based approachesInclude as part of DevOps toolchainsGeolocation testing for enhanced global performanceIntegration with cloud-based mobile d
247、evice labsHigher mobile test automationAnd talking about data management in an ever-increasing digitized world without mentioning mobile app testing would be unmerited.Its no surprise that 76%of organizations are adopting mobile app testing to optimize testing across countless device variants while
248、employing AI and data science.Cloud-based automated testing with AI/ML techniques has further refined testing efforts for diverse mobile devices worldwide.WHAT SHOULD ORGANIZATIONS FOCUS ON?It is great to see that more organizations are moving non-production environments to cloud even though it is m
249、ore concentrated on a hybrid strategy.Organizations increasingly embracing SRE and leveraging chaos engineering and AI-data science approaches to specialized testing is also a move in the right direction.Our recommendations and summary:Accelerate the inclusion of cloud and infrastructure testing as
250、part of the software development lifecycle,to improve resiliency,security,redundancy,and data recovery.Continue to consider intelligent integrations(including customizing specific features)to make the best use of commercial off-the-shelf and cloud native tooling as a hybrid strategy for non-producti
251、on environment provisioning.Adopt stringent practices to test the impact of failure events on cloud and continue to give extremely high priority to the performance and resiliency of applications on cloud.Invest on workforce transformation in building SRE and chaos engineering capabilities within qua
252、lity engineering organizations to embrace and support the emerging trends in digital and cloud migration strategies of the customers.Quality Ecosystem41 JEBA ABRAHAM,Practice Vice President,Quality Engineering&Testing,Sogeti USA STEFAN GERSTNER,Vice President,Quality Engineering&Testing,Sogeti KHUSR
253、O SIDDIQUI,Director,SAP Business Assurance,Capgemini USA SHASHANK LOKHANDE,Senior Director,Quality Engineering&Testing,Sogeti IndiaIn the last few decades,the world has undergone a transformative journey propelled by an unparalleled force:digitalization.This sweeping phenomenon,characterized by the
254、integration of digital technologies into various aspects of human life and industries,has reshaped our societies,economies,and interactions at a pace that defies historical norms.When the COVID-19 pandemic broke out,much of the world moved online,further accelerating digitalization at a breakneck sp
255、eed*.The accelerated adoption of digital technologies,primarily driven by the necessity to keep functioning during a global pandemic has transformed our world forever.A simple metric is digital payments.In Asia alone,the number of digital payments has increased by 300%between 2017 and 2023*.Needless
256、 to say,the digital core-technology platforms and applications that enable organizations to transform into digital businesses-requires thorough testing to enable success in the digital marketplace of the future.DIGITAL CORE RELIABILITYFIG 28What is most important to your organization when testing di
257、gital core solutions?Most important when testing digital core solutionsWQR 2023-Global ResultsTOTAL n=750Base:Those Who Selected VP Applications/Product Owner AND QA/Testing Manager/Quality Engineers AND Chief Technology Officer/Product Head52%47%46%46%37%37%37%37%CustomerExperienceSystemperformance
258、DataValidationSpeed ofdeploymentCompliancerelated requirementsRoles/Securityand workflowvalidationsGood enoughend-to-endbusinessprocess(no High/Critical defects)AccessibilityandLocalizationUNEARTHING THE CORE OF IT ALL:TESTING,PROCESSES,AUTOMATION,AND MOREWe looked at quality engineering and testing
259、 trends across organizations around the digital core.Some of the findings were expected like the most important focus areas when testing digital solutions are customer experience and system performance(directly impacting the usability of the platform/solution).This reiterates the fact that the digit
260、al core is all about building experiences and solutions,delighting,and retaining customers and driving growth.It also underlines the importance of being able to act in a swift way on market opportunities while ensuring clients digital happiness.References:*https:/www.oecd.org/digital/digital-economy
261、-outlook-covid.pdf *https:/ 42World Quality Report I 2023-2443Digital Core ReliabilityAnother trend that was expected was the focus on end-to-end business processes.Organizations have arranged testing teams for maximum support towards end-to-end(E2E)testing,with 50%of them having dedicated quality a
262、nalysis(QA)teams testing digital core applications.To top it all off,35%of organizations have their business users testing digital core solutions to ensure E2E quality.While the close involvement of business users clearly adds value,further professionalization is required to optimize their involveme
263、nt and gain maximum benefit.FIG 29Who is involved in ensuring the quality of the end-to-end solution?Involvement in ensuring the quality of the end-to-end solutionWQR 2023-Global ResultsTOTAL n=1750Dedicateddigital coreQA teamEnterprisewide TestingCoEProduct/Feature teamsBusinessUsersExternalSystems
264、Implementor49%39%39%35%34%E2E testing is also extremely complex and is one of the top challenges while testing digital core applications,but more on that later.Another trend observed was the pervasive usage of automation across the entire lifecycle of quality.In fact,more than half of the organizati
265、ons use automation across every step of the lifecycle from test scope selection to analysis and reporting.“It is vital for Volkswagen Financial Services(VWFS)that our digital products are robust,as we are representing more than just VWFS.VWFS does have its own market presence,but fundamentally,most
266、consumers will consider the car brand as the brand.So,quality is important,and as you would expect,those car brands are naturally keen to ensure that their customer experience is outstanding,and anything we do as the financial services captive reflects on their brand level too”.MARK PAINHead of Prod
267、uct Delivery,Volkswagen Financial Services,UKFIG 30Where do you use automated solutions for your digital core testing?Where automated solutions are used for digital core testingWQR 2023-Global Results59%59%56%55%55%55%55%54%53%52%50%Configuration ValidationAnalysis&ReportingEnvironments&DeploymentsI
268、mpact Analysis&Test ScopingWorkload modelling for Performance TestingTest PrioritizationTest Case GenerationTest ExecutionAutomated Test Case creationTest Data Provisioning/GenerationTOTAL n=750Base:Those Who Selected VP Applications/Product Owner AND QA/Testing Manager/Quality Engineers AND Chief T
269、echnology Officer/Product Head44World Quality Report I 2023-2445Although automation is pervasive across the lifecycle,we observed that the usage of test automation still has a long way to go.About 90%of organizations have 40%automation coverage of their testing and 26%of them have 20%or less coverag
270、e.This clearly is due to the same top challenges around testing digital core solutions-the complexity of the environment owing to the mix of tools,dependencies related to environment and data availability.When it is hard to test,it is even harder to automate.Theres also another contradiction we need
271、 to address-while 31%of organizations feel the pressure to keep up with the pace of development teams developing digital core solutions,69%of organizations do not feel the pressure.With 40%of automation coverage and digital core solutions becoming more SaaS and less customized,it is a bit of an enig
272、ma to unravel.Why dont organizations feel the pressure to keep up?Is that because they have large QA teams rushing to complete all testing manually?Or are teams stressed by the number and frequency of code drops coming in for testing?BUT HOW MUCH IS AUTOMATION TRULY CONTRIBUTING?FIG 310%/None1-20%21
273、-40%41-60%61-80%More than 80%How much of your digital core testing is automated?Level of automation of digital core testingWQR 2023-Global ResultsTOTAL n=175026%64%9%1%WHATS STOPPING GUARANTEED QUALITY FOR DIGITAL CORE SOLUTIONS?We saw earlier that there is a clear focus on end-to-end testing of bus
274、iness processes.And this is highly relevant for the most important objective of testing digital solutions customer experience.However,we identified numerous challenges to ensuring quality of digital core solutions.Digital Core ReliabilityComplexity around tools,environmental and data dependencies ar
275、e still prevalent.There is also additional complexity in testing business processes end-to-end.This might be due to the dependencies on multiple interconnected systems in complex landscapes,which is quite typical when testing digital core solutions.This could also be due to the number of people/team
276、s involved in ensuring E2E quality through testing process chains and handing off the testing.Determining if identified issues are real issues is another challenge in such environments.ENSURING QUALITY OF DIGITAL CORE SOLUTIONS DEMANDS NEWER STRATEGIES AND APPROACHES What are the key challenges you
277、face while ensuring quality of digital core solutions(ERP,Salesforce etc.)?Key challenges faced while ensuring quality ofdigital core solutionsWQR 2023-Global ResultsTOTAL n=750Base:Those Who Selected VP Applications/Product Owner AND QA/Testing Manager/Quality Engineers AND Chief Technology Officer
278、/Product Head41%38%36%34%33%31%27%Complexitydue to multipletools in the mixComplexity intesting businessprocessesend-to-endEnvironmentalchallengesTest Datacomparable toPRODGaps in DomainexpertisePressure tokeep up withdevelopmentvelocityImpact analysisand Test Scopingfor updates/upgradesFIG 32When i
279、t comes to test automation,replicating what functional testers do manually step by step through a tool might not be the best possible approach.Trying to break the end-to-end tests into more manageable pieces with a level of contract testing could be a more sustainable approach for building automated
280、 tests.The other important success factor is to have the right skills in QA Teams.When we looked at the data,quality assurance skills were rated as the most important skillset over domain or platform skills required for testing digital core solutions.This datapoint feels like bit of an outlier when
281、compared to two other datapoints 35%of organizations still utilize business users in validating digital core solutions and 33%of organizations pointed to gaps in domain expertise as a challenge to overcome when testing digital core solutions.The truth is while solid testing and quality assurance ski
282、lls are sought after due to the nature of digital core solutions,domain expertise remains invaluable.46World Quality Report I 2023-24Digital Core ReliabilityOrganizations need to find a way to integrate quality assurance skills and domain skills into teams.There is more than one way to achieve this
283、of course building and sustaining stable teams over longer durations for functional areas,getting business users to test with QA teams,and upskilling QA teams on domain knowledge are some to be highlighted.RECOMMENDATIONS AND SUMMARYThere is a lot of room to grow automated testing coverage and furth
284、er build customer delight via seamless experiences.These are our recommendations to organizations building and deploying digital core solutions:1.Use newer approaches like test isolation,contract testing,etc.,to drive more segmentation and higher automated test execution this will solve the challeng
285、e around domain skills as well as relieve teams of stress to keep up with the development speed.2.Utilize service virtualization to overcome challenges around multiple applications,environments,and data dependencies.Service virtualization combined with test automation will really move the needle in
286、the future.3.QA teams of the future will need to have a mix of skills strong testing skills will always be needed and will form the core.However,they will need to be augmented with some domain and platform skills to drive efficiencies.FIG 33Base:Those Who Selected VP Applications/Product Owner AND Q
287、A/Testing Manager/Quality Engineers AND Chief Technology Officer/Product HeadWhat is the most important skillset to the testers in your QA organization whentesting digital core solutions?Most important skillset to testers when testingdigital core solutionsWQR 2023-Global ResultsTOTAL n=75064%21%9%7%
288、QualityAssuranceKnowledge oftechnology platformBusiness domainknowledgeCodingskills“You have to test with the right code and the right data,and the biggest shift we recently made is testing with production data.One of the biggest realizations we had was no matter what you test if you are not testing
289、 with prod data,there are going to be big gaps”.RAJESH NATARAJANSenior Director,Quality Engineering at Hiscox47INTELLIGENT PRODUCT TESTING JEAN-BAPTISTE BONNET Offer Leader,Intelligent Testing for Connected Products,Capgemini Engineering,France NARENDRA VAISHAMPAYAN Director,IoT Quality Engineering&
290、Testing,Sogeti India ATUL JADHAV Director ERD,Engineering,Capgemini India VIPUL MOHINDRA Director,Sales Operations-MFG,Auto,Aero,Capgemini North America ERD UDI WEINBERG Director,Product Management,OpenText OREN ZIV Product Manager,OpenText48World Quality Report I 2023-24Point at any car you see on
291、the road,chances are it features a complex infotainment system,maybe additional connected medical devices,wellness devices,or even appliances,which might all be connected to mobile apps.Products today are an extension of us.Over the last decade,the product landscape has transformed in unimaginable w
292、ays.We have long moved from stand-alone products to upgraded globally interconnected products.Naturally,this evolution has introduced a never seen before level of complexity.A lot is at stake.An unsavory user experience can not only jeopardize a product launch but also damage a brand image.And it is
293、 not just about customer experiences-regulatory pressures,changing guidelines,and cyber threats all suggest the need for robust product testing mechanisms.There is even a need to re-invent the product testing approach for this globally interconnected world we now live in.SO,WHAT REALLY GOES INTO CRE
294、ATING THE PERFECT INTELLIGENT PRODUCT TESTING SYSTEM ECOSYSTEM?This years quality survey report highlighted three main activities:1.Ensuring seamless integration with the cloud ecosystem 2.Ensuring the security of the product 3.Ensuring a perfect user experienceWhat was interesting to note in the re
295、sults was that although the trend in the importance of testing activities was within a very narrow range,cloud testing came up as the most prominent factor with 50%of respondents placing high importance on it.In fact,some considered it as important as performance testing itself.This further validate
296、d the predominance of interconnected products.Tending to cybersecurity needs came up as another important activity(47%of respondents)which makes perfect sense considering adherence to future regulations,or deterring threats for connected products is paramount for the overall robustness of the ecosys
297、tem.A few other activities such as functional testing(48%),accessibility testing(46%),and embedded testing(46%),also showed up as relevant testing areas.“We must provide robust solutions for our customers because that is an expectation from a regulatory perspective,but more importantly,we want to en
298、sure a good customer experience”.MARK PAIN Head of Product Delivery,Volkswagen Financial Services,UKIntelligent Product Testing49FROM CHIP TO CLOUDWhile the respondents acknowledged the need for an evolved testing ecosystem,we were surprised to learn that only 36%thought end-to-end(E2E)testing was n
299、ecessary.It has been established that the value of connected products is distributed throughout the entire chain,and its resilience depends on the weakest link in the chain,and yet,the overall value of E2E testing has not been fully recognized.One reason we suspect could be that it requires signific
300、ant investment in test benches,hardware,and cloud infrastructure for E2E testing.An E2E approach is frequently viewed through the abstraction of various layers of the system,such as embedded systems,cloud,and connectivity.However,in our experience,we believe this is going to have a minor impact on i
301、ntelligent product testing in the future.WHAT ARE THE MASSES EXPECTING FROM INTELLIGENT TESTING?One of the significant trends observed in the WQR 2023 survey is the expectation for improving the test definition.The increasing complexity of products,along with hyper-personalization to create a unique
302、 user experience,necessitates passing millions of test cases to achieve the perfect user experience.Realistically speaking,its not possible to determine all possible combinations.According to respondents,the critical aspect of intelligent testing is test-case definition,with test case selection(52%)
303、followed by a prioritization of test cases(50%).It emphasizes that testing approaches need to constantly evolve to prevent systemic failures.Following the two main expected outcomes,the test data and test case generation are aligned(at 45%and 42%respectively),with the required improvement of the tes
304、ting approaches.Another key finding was that 47%of respondents consider root cause analysis to be of high priority.This denotes a shift to the left and a willingness to use test results not only to assess product quality but also to provide R&D with insights throughout the development process.Expect
305、ations regarding intelligent product testing clearly focus on the key elements of the user experience,such as functionality,interoperability,and cybersecurity.Additionally,respondents have the highest expectations of improved test coverage.FIG 34Please rate each of the test activities below in terms
306、 of how important they are to your industry.Importance of test activities to industryWQR 2023-Global ResultsTOTAL n=315CloudTestingPerformanceTestingFunctionalTestingCybersecurityTestingAccessibilityTestingEmbeddedTestingInteroperabilityTestingVisual TestingConnectivity TestingE2E TestingRegulatoryC
307、ompliance TestingAcceptanceTestingBase:Those Who Selected QA/Testing Manager/Quality Engineers50%50%48%47%46%46%45%43%36%36%35%33%50World Quality Report I 2023-24Quality&SustainabilityFIG 35Please rate each of testing use cases below in terms of how important they are to be implementedas part of Int
308、elligent Product Testing?Importance of implementing testing use cases as part ofIntelligent Product TestingWQR 2023-Global ResultsTOTAL n=615Base:Those Who Selected IT Directors AND QA Testing/Testing Manager AND Quality Engineers52%50%47%45%42%39%37%33%15%Test caseselectionTest caseprioritizationRo
309、ot causeanalysisTest datagenerationAutomatedtest casegenerationTest caseminimizationTest caseadequacyCodedefectpredictionTestexecutionanalysis“It is now about automating everything that is a part of the testing life cycle,is taking your time,and needs to be done repeatedly”.RAJ SINGH Vice President,
310、Head of Wholesale IT QA of a Leading Investment Bank and Brokerage Group51AI TAKES THE CENTER STAGE,BUT AT WHAT COST?As previously discussed,there are high expectations for use cases related to test plan definition,especially in terms of prioritization,selection,and automatic generation of test case
311、s.Artificial Intelligence(AI)and Generative Artificial Intelligence(Gen AI)are at the core of all these technologies.The analysis of the WQR 2023 responses indicates that the respondents are well aware of the significant impact that these new technologies will have on their activities,and they need
312、to incorporate these skills into their quality engineering teams.Data is the main area of interest for the respondents.Hyper-automation of testing and expanding test cases to cover the entire product domain results in a significant increase in the volume of data,that needs to be stored and processed
313、.43%of respondents consider integrating data platform technologies like Hadoop or MapReduce to be essential to their core competencies.As expected,AI-related technologies are the most important,with 44%of respondents believing that they will need to use both supervised and unsupervised machine learn
314、ing technologies in their activities.Furthermore,to a lesser extent(42%),natural language processing techniques are also required.For product testers,the latest findings suggest that automation toolchain skills and programming languages are now considered mainstream and no longer considered core com
315、petencies.RECOMMENDATIONS AND SUMMARYThere is a lot at stake in terms of interconnected products or services,as other studies have indicated that a vast majority of consumers abandon the product or service in question if they are disappointed.Our recommendations for organizations building connected
316、products of the future-1.Invest in AI solutions for test prioritization and test case selection to drive maximum value from intelligent testing,against the millions of possible combinations in todays hyperpersonalized connected product environment.2.Focus on an E2E testing approach to ensure seamles
317、s customer experience through effective utilization of abstraction of various architecture layers(embedded software,cloud,and connectivity).3.Give equal priority to the non-functional aspects(performance,security,scalability,usability)of the system as the functional aspects as they equally influence
318、 the end-user experience.FIG 36Please rate each of the items below in terms of how important they are for the skills requiredfor Intelligent Product Testing.Importance for the skills required for IntelligentProduct TestingWQR 2023-Global ResultsTOTAL n=435Base:Those Who Selected VP Applications/Prod
319、uct Owner AND Chief Technology Officer/Product HeadMachine Learning models(Supervised,Unsupervised)Data Platform(Hadoop,MapReduce etc.)Natural Language ProcessingAutomation tool chain(Robot,cucumber etc.)Programming Languages(Python,Java,C#,R etc.)44%43%42%34%31%52World Quality Report I 2023-24 ANDR
320、EW FULLEN Head of Innovation and Technology,Sogeti UK ANTOINE AYMER CTO,Quality Engineering&Testing,Sogeti MAHESHWAR KANITKAR Vice President,Global Quality Engineering&Testing Sales Leader,Sogeti Global RAJARAO TADIMETY Associate Vice President,Quality Engineering&Testing,Capgemini USA JENNA FELDMAN
321、 Director,Corporate Citizenship,OpenTextQUALITY&SUSTAINABILITY“Sustainability is one of our key objectives.Sustainability is extremely important.Really knowing the impact that our actions have is critical”.GANESH PRACTOORDirector-Quality Engineering Group at one of the worlds leading financial organ
322、izations“We look at sustainability as reuse and its not just within our QE practice,but its how we reuse it across our environments.So,were taking certain platforms and saying that this is the baseline for our product,which then becomes the baseline for all of our other parks”.FABIAN RODRIGUEZ Vice
323、President,Application Delivery&Strategy|Digital&Technology,Universal Destinations&Experiences,NBC Universal53FIG 37Very activeActiveA key driverPassiveNot involvedat allWQR 2023-Global ResultsWhich of the following best describes how the Quality Engineering(QE)contributes to the sustainabilityagenda
324、 of an organization?QEs contribution to the sustainability agenda24%67%6%2%TOTAL n=1435Base:Those Who Selected Chief Information Officer AND VP Applications/Product OwnerAND IT Directors AND Chief Marketing Officer AND Chief Digital Officer And Chief Technology Officer/Product Head AND VP OR Directi
325、or Of Research AND Development/EngineeringA constant challenge reported to us over the years by the World Quality Report(WQR)editions revolves around managing various tech environments,and yet there is a far more serious environmental concern-our planet.Thankfully,sustainable IT is increasingly beco
326、ming a boardroom discussion topic highlighting how much of an impact IT really has on the environment.This year,we wanted to deep dive and gauge how far the ideas of sustainability and environmental impact have made their way into quality engineering and testing(QE&T)landscape.In this chapter,we ide
327、ntify some pressing questions and observations around the impact of QE&T on sustainability and understand where we are truly headed as an industry.1.What role does quality engineering play in reducing the environmental impact of IT programs and systems?Especially while reports show that data centers
328、 and cloud contribute more to greenhouse emissions than the aviation sector.2.Understanding the enormity of influence is key to reducing the impact of IT.It may be through being part of an organization-wide program or by helping track emerging metrics.3.What are people currently engaged in?What are
329、their future plans?And what motivates these actions-is it driven by financial considerations or a sense of moral obligation?Let us dive into some of the trends we received from this years WQR survey results.54World Quality Report I 2023-24IS QUALITY ENGINEERING(QE)REALLY CONTRIBUTING TO BUSINESSES S
330、USTAINABILITY AGENDAS?The vast majority(97%)felt QE was active or continually active in driving the sustainability agenda in their organizations.We saw 73%of respondents describing themselves as highly active,or key enablers in driving sustainability best practices in their organizations.Such a high
331、 degree of engagement bodes well for delivering on the promises of a sustainable IT world.FIG 38Itsimportantto ourcustomersand/orinvestorsItsimportantto usand ouremployeesWell lookat itsometimein thefutureWell lookat it inthe next12 monthsIts theright thingto doIts notrelevantto usN/AWQR 2023-Global
332、 ResultsIs sustainability in QE&T a priority for your organization?Priority given to sustainability in QE&TTOTAL n=315Base:Those Who Selected QA Testing/Testing Manager AND Quality Engineers17%38%44%64%59%But our findings were not rid of contradictions-two-thirds said that sustainability was importa
333、nt to their customers and investors,but worryingly,nearly half said they would look at it sometime in the future,with only a third planning to look at it in the next year.Half of those who responded positively called out vendor compliance with green frameworks and regulations as key drivers of focus in this area.We believe this is a trend that will grow as more governments implement green credenti