《Snowflake :2022Q2统一数据管理和分析平台最佳实践报告(英文版)(34页).pdf》由会员分享,可在线阅读,更多相关《Snowflake :2022Q2统一数据管理和分析平台最佳实践报告(英文版)(34页).pdf(34页珍藏版)》请在三个皮匠报告上搜索。
1、By Fern Halper,Ph.D.Modernizing the Organization to Support Data and AnalyticsBEST PRACTICES REPORTQ2 2022Co-sponsored by:tdwi.orgTDWI RESEARCH1BEST PRACTICES REPORTQ2 2022Modernizing the Organization to Support Data and AnalyticsBy Fern Halper,Ph.D.Table of ContentsExecutive Summary .5Introduction:
2、Trends Driving the Need for New Organizational Models .6New Roles .9The New Leader:The CDO/CAO/CDAO in Modern Analytics .9Other Up-and-Coming Roles .11Organizing to Execute .13New Offices .14New Team Structures .15Governance Team Structures .17New Paradigms .17Enabling Technologies .21Building the S
3、kills,Culture,and Talent .24Building the Culture .25Building Data Literacy .26Getting to Value .27Recommendations .30Research Co-Sponsor Snowflake .32 2022 by TDWI,a division of 1105 Media,Inc.All rights reserved.Reproductions in whole or in part are prohibited except by written permission.Email req
4、uests or feedback to infotdwi.org.Product and company names mentioned herein may be trademarks and/or registered trademarks of their respective companies.Inclusion of a vendor,product,or service in TDWI research does not constitute an endorsement by TDWI or its management.Sponsorship of a publicatio
5、n should not be construed as an endorsement of the sponsor organization or validation of its claims.This report is based on independent research and represents TDWIs findings;reader experience may differ.The information contained in this report was obtained from sources believed to be reliable at th
6、e time of publication.Features and specifications can and do change frequently;readers are encouraged to visit vendor websites for updated information.TDWI shall not be liable for any omissions or errors in the information in this report.Modernizing the Organization to Support Data and Analytics tdw
7、i.orgTDWI RESEARCH2About the AuthorFERN HALPER,Ph.D.,is VP and senior director of TDWI Research for advanced analytics.She is well known in the analytics community,having been published hundreds of times on data mining and information technology over the past 20 years.Halper is also co-author of sev
8、eral Dummies books on cloud computing and big data.She focuses on advanced analytics,including predictive analytics,text and social media analysis,machine learning,AI,cognitive computing,and big data analytics approaches.She has been a partner at industry analyst firm Hurwitz&Associates and a lead d
9、ata analyst for Bell Labs.Her Ph.D.is from Texas A&M University.You can reach her by email(fhalpertdwi.org),on Twitter( on LinkedIn( TDWI ResearchTDWI Research provides industry-leading research and advice for data and analytics professionals worldwide.TDWI Research focuses on modern data management
10、,analytics,and data science approaches and teams up with industry thought leaders and practitioners to deliver both broad and deep understanding of business and technical challenges surrounding the deployment and use of data and analytics.TDWI Research offers in-depth research reports,commentary,ass
11、essments,inquiry services,and topical conferences as well as strategic planning services to user and vendor organizations.About the TDWI Best Practices Reports SeriesThis series is designed to educate technical and business professionals about new business intelligence technologies,concepts,or appro
12、aches that address a significant problem or issue.Research for the reports is conducted via interviews with industry experts and leading-edge user companies,and is supplemented by surveys of business intelligence professionals.To support the program,TDWI seeks vendors that collectively wish to evang
13、elize a new approach to solving business intelligence problems or an emerging technology discipline.By banding together,sponsors can validate a new market niche and educate organizations about alternative solutions to critical business intelligence issues.To suggest a topic that meets these requirem
14、ents,please contact TDWI Senior Research Directors David Stodder(dstoddertdwi.org),James Kobielus(jkobielustdwi.org)and Fern Halper(fhalpertdwi.org).AcknowledgmentsTDWI would like to thank many people who contributed to this report.First,we appreciate the many users who responded to our survey,espec
15、ially those who agreed to our requests for phone interviews.Second,our report sponsors,who diligently reviewed outlines,survey questions,and report drafts.Finally,we would like to recognize TDWIs production team:James Powell,Pete Considine,Lindsay Stares,and Michael Boyda.SponsorsAlation,Carto,Datai
16、ku,Denodo,Qlik,SAP,and Snowflake sponsored the research and writing of this report.Modernizing the Organization to Support Data and AnalyticsPosition198 of whom completed the entire survey and met quality standards.This group is used for analysis.Research methods.In addition to the survey,TDWI condu
17、cted telephone interviews with technical users,business sponsors,and analytics experts.TDWI also received briefings from vendors that offer products and services related to these technologies.Survey demographics.Respondents act in a variety of roles.These include corporate exec/VP(27%),data scientis
18、ts/analysts(17%),LOB managers(17%)architects/engineers(13%),business analysts(5%),and others(21%).Respondents came from a range of industries including financial services(11%),education(9%),government(9%),consulting(9%),and manufacturing(8%).Most survey respondents reside in the U.S.(63%),Asia(15%),
19、Canada(8%),or Europe(6%).Respondents come from enterprises of all sizes.Research Methodology and DemographicsReport purpose.Modern analytics represents the evolution of data and analytics.It includes modern techniques such as machine learning and natural language processing.It includes new paradigms
20、 such as the data fabric and the data mesh,as well as new organizational constructs such as the data office,literacy enablement teams,MLOps,and more.It is often cloud-based.Many organizations are excited to use the power of advanced analytics because they understand the value it can provide.However,
21、companies are struggling to move forward with their analytics efforts.Often organizational issues are at the heart of the struggle.This TDWI Best Practices Report addresses organizational structures and enabling technology to support modern analytics.Survey methodology.In February 2022,TDWI sent an
22、invitation via email to the analytics and data professionals in our database,asking them to complete an online survey.The survey collected responses from 224 respondents,Corporate executive/VP27%LOB director/manager 17%Architect/data engineer 13%Business analyst5%Other 21%Data scientist/analyst 17%M
23、odernizing the Organization to Support Data and AnalyticsIndustryGeographyCompany Size by RevenueFinancial Services 11%Government9%Manufacturing(non-computer)8%Healthcare7%Retail6%Education9%Consulting/Professional service9%Construction/Engineering7%Software/Internet6%Other28%United States 63%Mexico
24、,Central/South America 4%Canada 8%Asia 15%Australia/New Zealand 1%Europe 6%Africa 1%Middle East 2%(“Other”consists of multiple industries,each represented by less than 4%of respondents.)Based on 194 respondents.Less than$100 million25%$100$499 million18%$500 million$999 million18%More than$1 billion
25、26%Dont know13%Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH5chief analytics officer or chief data and analytics officer),whose role it is to provide business value to the organization.In the survey for this report,the position was in the early mainstream stage of
26、adoption,although the CDO wasnt always part of the C-suite.CIOs,CTOs,and others still lead many data and analytics efforts.The results of this study,however,indicate that organizations where the CDO is part of the C-suite are more likely to measure the value from their data and analytics efforts.Org
27、anizational models.This Best Practices Report also examines how businesses are organized to extract the greatest benefit from data and analytics.This includes centralized,hub-and-spoke,and decentralized organizational models.The results of the study indicate that no one model is best for modern anal
28、yticsat least not now.The right model for a particular organization will depend on their specific circumstances.The report also explores new paradigms such as the data mesh,which is not yet widely adopted,although many CDOs support the data mesh principles.New roles.Organizations are already impleme
29、nting new roles to help them move ahead on their data and analytics journey.These include data engineers,MLOps engineers,data literacy enablement team members,modern data analysts,and even data product managers.These roles might also have their own models.For instance,some DataOps teams are organize
30、d as pods.These roles are often part of a data office or a center of excellence.Adoption is typically based on where the organization is on its analytics journey,although organizations should be planning for these roles as part of their modernization strategy because these roles are important.Execut
31、ive SummaryAs organizations strive to compete in a dynamic environment,they are trying to modernize their data and analytics environment to help.This modernization includes implementing new technologies such as scalable cloud platforms and unified approaches.It includes utilizing more advanced analy
32、tics such as geospatial analytics and machine learning.It also includes new paradigms such as the data fabric and the data mesh.Moreover,as part of this,modernization may include new organizational constructs such as the data office and new teams such as DataOps,MLOps,and data literacy enablement te
33、ams.This TDWI Best Practices Report examines topics including leadership structures,organizational structures,new roles,and enabling technologies to help with data and analytics modernization.This TDWI Best Practices Report provides best practices research about how successful companies are organizi
34、ng to execute to win with analytics.It focuses on topics including leadership structures,organizational structures,new roles,and new paradigms such as the data mesh.It also examines new technologies and their impact on organizations.Some considerations highlighted in this report include:Leadership m
35、odels.Leadership models for modern data and analytics continue to evolve.One model that is gaining traction is the chief data officer(or Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH6organizational constructs such as the data office,literacy enablement teams,and ML
36、Ops.Enterprises have shown a renewed interest in data governance.Modernizing analytics,then,is the next evolution in analytics that includes new models for a data foundation,more advanced analytics against new data types,and new roles and processes to make it all successful.Modernization includes ne
37、w paradigms such as the data fabric and the data mesh,as well as new organizational constructs such as the data office,literacy enablement teams,and MLOps.Modern analytics is both the reason for and an outgrowth of several trends:The move to the cloud.As organizations collect increasing amounts of d
38、iverse data,they often find that their traditional data warehouse was not built to house newer data types such as text data or machine data.TDWI sees more organizations moving to cloud data warehouses and cloud data lakes to take advantage of the flexibility and scalability the cloud provides.In fac
39、t,in TDWI research,we see the gap in adoption between on-premises data warehouses and cloud data warehouses continues to narrow.Data analysts become modern business analysts.Part of the modernization effort is upgrading employee skills.Many data analysts want to increase their skills and talents to
40、do more sophisticated analytics.At TDWI,we see that organizations are looking to update or expand the skills of these data analysts to become modern business analysts and Enabling technologies.Vendors are offering a range of options to enable modern data and analytics,including cloud platforms and d
41、ata fabrics,to help establish a unified and trusted data foundation,intuitive GUIs,and augmented and automation features to help make analysts more productive.They are also offering tools to help to deploy and manage advanced analytics in production,and tools such as data catalogs to help different
42、personas understand and utilize data for analytics.Although it is still early for many new technologies,roles,offices,and other organizational constructs,TDWI research suggests that they can provide a top-line impact to those companies that use them in modernizing their data and analytics environmen
43、t.Introduction:Trends Driving the Need for New Organizational ModelsModern analytics represents the evolution of data and analytics.Many organizations are excited to make use of the power of advanced analytics because they understand the value it can provide.Organizations are evolving their data man
44、agement infrastructure to support new data types and analytics approaches.They are moving to cloud platforms.They are using more sophisticated analytics such as geospatial analytics,machine learning,natural language processing(NLP),and other advanced techniques.Modernization includes new paradigms s
45、uch as the data fabric and the data mesh,as well as new Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH7perform analytics such as machine learning.At the same time,organizations are trying to free up data analysts by providing tools that help business users derive in
46、sights via self-service or embedded analytics at the point of consumption.New technologies,including automated approaches to the data and analytics life cycle.As the data and analytics life cycle becomes more complex,organizations are looking to automated approaches to help manage the complexity.Acr
47、oss the life cycle,these tools are augmented with advanced analytics such as machine learning or NLP.These technologies are helping identify poor quality data,classify sensitive data,surface insights,provide a natural language query interface to search data,and enable easier analysis of new kinds of
48、 data,such as geospatial data,and much more.New paradigms such as the data fabric and the data mesh.In addition to moving to the cloud,organizations are looking for other new approaches to integrate data that may be in internal and external systems,both on premises and in the cloud.The logical data
49、fabric provides an approach to unify disparate data and bring it together in an intelligent fashion.It stiches data together from across diverse systems;often leveraging data virtualization technology.Organizations are also looking at new frameworks,such as the data mesh,for organizing and executing
50、 against complex data and analytics.All trends are driving the need for new roles.New roles(such as the data engineer,the MLOps team,and the modern analyst)and new organizational constructs(such as the data office)are also becoming more important as organizational data becomes more complex and compa
51、nies want to perform more sophisticated analytics.When we ask data scientists what they need,they often say more data engineers.These new roles are described in more detail later in this report.Modernization is happening.Survey results illustrate the move by organizations to try to modernize.For ins
52、tance:Only a small percentage of respondents arent using the cloud.When asked,“Which statement best describes your organizations data management environment?”only 22%stated that data resides in physical data warehouses,data marts,or data lakes that are on premises(no figure shown).For other responde
53、nts,the data was either in physical data warehouses,data lakes,or data marts in the cloud(22%),both on premises and in the cloud(38%),or spread across on-premises and cloud environments but unified via a logical data virtualization layer(13%).Over half of the respondents to the survey had moved past
54、 structured data to analyze new kinds of data.This includes data such as text data or IoT data(31%),as well as image,audio,or video(21%,no figure shown).This kind of data is important for numerous use cases.For instance,text data can help organizations get to the why behind what happened.IoT data is
55、 often involved in use cases including proactive maintenance.Geospatial data provides insights that can help in retail(e.g.,setting up new locations),insurance(e.g.,risk analysis)and much more.Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH8organization or they haven
56、t even deployed a data and analytics strategy(see Figure 1).We will return to these results later in the report when we compare organizations that have been successful with those that are somewhat or not successful.We will see that there are certain organizational and technology approaches that succ
57、essful organizations implement.This TDWI Best Practices Report examines the organizational and technology structures that can help companies move forward.It will address questions such as who is leading the effort?What are the new roles and responsibilities?What new paradigms should organizations be
58、 aware of?How are companies organized to execute?What are the characteristics of those organizations that are successful?Sixty-five percent of respondents believe that their data and analytics strategy has been somewhat successful,although they havent measured impact.Although enterprises have some s
59、uccess modernizing,there is also room for improvement.Sixty-five percent of respondents believe their data and analytics strategy has been somewhat successful;in other words,they think their efforts are helping the organization although they havent measured them or put metrics in place to measure im
60、pact.Twenty percent have measured a top-line impact or cost savings with their data and analytics strategy.Fifteen percent do not believe that data/analytics has made an impact on their Figure 1How successful is your data and analytics strategy?Based on 198 respondents.SuccessfulWe have measured a t
61、op-line impact or cost savings with our data and analytics strategy(e.g.,revenue per campaign,sales,retention cost per customer,etc.).Somewhat successfulWe think our efforts are helping out the organization,but we havent measured them or have no metrics in place.Not successfulWe havent deployed a da
62、ta or analytics strategy that has made any real impact.Successful20%Not successful 15%Somewhat successful65%Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH9Historically,some of these functions may have been in the CIOs purview,but the reality is the CIO office has en
63、ough on its plate dealing with information technologies.The CDO,of course,will collaborate with the CIO.The CDO also will collaborate with the CFO,COO,and other members of the C-suite.In this survey,about a third of respondents had a CDO/CAO leading BI and analytics efforts.In this survey,about a th
64、ird of the respondents had a CDO/CAO leading the BI and analytics effort in the organization(see Figure 2).However,fewer are leading other efforts,such as data management,data governance,or data literacy.In some cases,the activity is too new to have a leader,such as in the case of data literacy,wher
65、e 27%said the activity did not apply to them.In many cases,the VP/director of IT or the CTO is leading the efforts.Most likely,these organizations have not yet put a CDO/CAO in place.We often see that organizations with CDOs are further along their data and analytics journeys.When the CDO is part of
66、 the C-suite,there is an upside.The majority(70%)of respondents stated that the same group leads both the data and analytics efforts(no figure shown).Only 40%of respondents stated that their data and analytics leaders were part of the C-suite.In most cases,this is not the VP/director of IT but may b
67、e the CDAO,the CIO,or the CTO.However,as we will see when the CDO is part of the C-suite,there is an upside.Anecdotally,we hear that in some cases,the CDO role has high turnover.This is often because the foundational work in data and data literacy New RolesAs part of this effort to modernize,organiz
68、ations often need to implement new roles.This includes new leadership roles as well as new operational roles.The New Leader:The CDO/CAO/CDAO in Modern AnalyticsIn TDWI surveys,respondents often cite analytics leadership as a top area that could bring value to their organization.Over the past few yea
69、rs,the role of the chief data officer(CDO),chief analytics officer(CAO),or chief data/analytics officer(CDAO)has become more popular.The idea is that this person leads a team that drives business value in the organization,whether that be in data(CDO),analytics(CAO),or both(CDAO).Often,this person le
70、ads the data and analytics strategy,including coordinating the governance strategy.Their team may be part of a data office(sometimes called a data hub or a center of excellence).Titles for the position vary,too.For instance,the role may be called a general manager or managing director.For the purpos
71、es of this report,the role will be referred as CDO.The CDO and the data office are responsible for developing and delivering on a business data strategy that meets the companys needs,including leading the effort to develop data policies,standards,governance,and processes to support insights and acti
72、on.Whereas the CIO is responsible for information technology,the CDO and the data office are responsible for the business use of data,deriving value from it,and building a data-driven culture.As such,they are also chief evangelists for data and analytics.Modernizing the Organization to Support Data
73、and Analytics tdwi.orgTDWI RESEARCH10(described below)takes time.Some organizations expect change in 6 to 12 months and often that is simply not possible.In other words,organizations need to have patience with the CDO role and let them do their jobs.Interestingly,organizations are not necessarily ha
74、ppy with the model they have for data and analytics leadership.In fact,in this survey,41%of respondents were on the fence about how their leadership model was working and 20%were dissatisfied.Only 27%were satisfied(no figure shown).As we discuss later in the report,those with CDOs in place tend to b
75、e more satisfied than those with other roles leading data and analytics,although there is room for improvement.Figure 2Who leads the following efforts in your company?Based on 198 responses.Data management 21%26%20%25%8%Business intelligence 15%15%32%26%12%Data science 10%18%30%21%21%Data governance
76、 17%18%23%29%13%Data literacy 10%18%21%25%27%Data-based apps/products 15%21%17%31%15%Chief information officerChief technology officerChief data/analytics officerVP/Director of ITN/AModernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH11Other Up-and-Coming RolesIn addition
77、 to the CDO,CAO,and CDAO,there are other new roles that organizations are adopting to help with modern data and analytics(see Figure 3).These include:Modern analysts.Modern analysts are next-generation analysts empowered to develop strategic insights utilizing new tools,techniques,and methodologies.
78、These modern analysts are a new breed of business or technical analyst who acts as an insights optimizer and an analytics coach.They often have a deep understanding of the business and they can drive action.Data scientist.Although certainly not a new role(TDWI has been following this role for at lea
79、st 10 years),the data scientist is still an up-and-coming role in organizations that are starting to develop more advanced analytics.Data science is an interdisciplin-ary field that combines advanced statistics and computer engineering skills to acquire,prepare,and analyze information across a varie
80、ty of sources.Data scientists have the technical skills to work with and analyze all kinds of data,from structured to unstructured.They build models,develop algorithms,and innovate on data.Data engineers/DataOps team.DataOps is the process by which data is integrated,transformed,and prepared for dep
81、loyment into business intelligence,reporting,and ad hoc analytics applications.It includes a wide range of functional components for handling and managing data throughout its life cycle.These include pipeline services for discovering,acquiring,extracting,transforming,profiling,cleansing,augmenting,s
82、taging,loading,replicating,delivering,indexing,searching,and protecting data.Data engineers are often part of the DataOps team.MLOps team.The MLOps team is responsible for deploying,monitoring,retraining,optimizing,and managing advanced analytics models such as machine learning or deep learning that
83、 power many AI and other intelligent applications.Although the data scientist might build the model,that is only half the process when it comes to putting models into production,which is where the real value lies.MLOps engineers ensure that a model will work in production;they keep it working and up
84、date it when it gets stale.They are often highly technical and work with open source software as well as commercial products.Literacy enablement teams.To move forward successfully with modern analytics,organizations need to become data-literate.Individuals across the organizations need to understand
85、 data,derive insights from it,and communicate results effectively.The level to which they need to do this will vary by role,but data literacy has become a critical skill for an organizations success.The data literacy enablement team is a group of people who build the literacy strategy and carry it o
86、ut.They may be part of the data office.Data product manager.As data is more often viewed as an asset and a product,organizations are starting to fill the role of data product manager.This person is responsible for making sure that data Modernizing the Organization to Support Data and Analytics tdwi.
87、orgTDWI RESEARCH12Figure 3What is the status of the following“up-and-coming”roles for data and analytics in your organization?Based on 198 respondents.consumers are happy with the product as well as developing new products with data.In essence,they may be part data steward and part product manager,e
88、xcept their product is data.Many of these roles are already in place in organizations(Figure 3).For instance,52%of respondents already have data engineers in place and 47%have data scientists.Fifty-three percent have a CDO and/or a CAO in place.Some of the other roles,such as the modern analyst,MLOp
89、s engineer,and data product manager are fewer in number,although there are plans to fill these roles.Much of this will depend on how the organization moves forward with modernizing their analytics efforts.For instance,the modern analyst makes sense if organizations can democratize analytics and open
90、 analytics to business users.In that case,data analysts wont have to spend so much of their time building reports and dashboards and can move forward in their careers and learn more skills.Likewise,if an organization puts more models into production,they will need more MLOps team members.Once the or
91、ganization has a data culture,it may make sense to have a data product manager.Data engineer 52%26%13%9%Data scientist 47%28%19%6%Citizen data scientist/modern analyst 22%38%29%10%MLOps engineer 17%30%35%18%Chief data officer(i.e.,leads the data effort to drive value)31%19%36%14%Chief analytics offi
92、cer(i.e.,leads the analytics effort to drive value)22%23%42%13%Data product manager(i.e.,ensures that data is treated as a product and consumers are happy)30%27%32%12%Currently in placePlanningNo plansDont knowModernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH13Organizi
93、ng to ExecuteAs companies push forward,what organizational models are they using to execute?Typically,when organizational models are discussed,three are mentioned:centralized,decentralized,and hub-and-spoke(see Figure 4).In the centralized model,all data functions,ownership,and activities reside wit
94、hin a central office.That office is responsible for data strategy,the data platform,governance,policies,and standards.Thirty-five percent of respondents are using this model.In the decentralized/distributed model,data activities are distributed into different business units.In other words,the busine
95、ss unit owns its data as well as what it wants to do with the data.The data and analytics functions sit in the business units.This was the most popular model,with 43%of respondents citing it.An office provides support to business units in the hybrid/hub-and-spoke model.The data office provides strat
96、egies,policies,processes,and standards.It supports the architectural design and delivery of an environment that can be used to support analytics,such as machine learning,so the data can be more Figure 4How are the data and analytics functions organized to execute in your company?Based on 198 respond
97、ents.Centralized(35%)The function is centralized under one organization/leader(e.g.,the data office,a center of excellence,or other team)to oversee talent,infrastructure,data literacy,governance,and best practices.Decentralized/distributed(43%)The function is distributed;data and/or analytics profes
98、sionals are in the business units or domains and most data/analytics functions sit there.Hybrid/hub-and-spoke(18%)A data office or center of excellence coordinates across the spokes to build best practices,governance,and data literacy.This team may own the infrastructure or architecture for data and
99、 analytics.Other(4%)Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH14easily used for value-generating activities.The data office may also be responsible for coordinating governance activities.The business units own their data but work closely with the data office to
100、ensure that the data is high quality and available for analytics use.Analytics might be owned and occur in the business units,although the data office may recommend certain tools that work with the infrastructure.In this survey,only 18%of respondents are using this model.Typically,there are three or
101、ganizational models companies use for data and analytics:centralized,decentralized,or hub-and-spoke.Of course,there are pros and cons for each approach.For instance,centralization may work well in smaller organizations with a simpler data infrastructure or companies focused on a single business.Cent
102、ralization may not work well in a very large organization with diverse lines of business,divergent needs and operating models,or entrenched power dynamics.In terms of the decentralized approach,business driving its own programs may be a good thing,as may be having the analytics function sitting in t
103、he business where its needed.However,this can frustrate analysts who want to collaborate across business units.It can also cause separate data standards,which can make data unmanageable and broader analytics efforts more difficult because it is harder to bring siloed data together.There can also be
104、duplication of efforts,which can lead to data inconsistency,untrusted data,and increased cost in decentralized approaches.The hub-and-spoke model can be best for organizations that have varied lines of business that have the resources and skills to perform data and analytics activities.The hub(for i
105、nstance,the data office)provides policies,coordinates governance,and may own the data architecture and may provide some sort of unified data environment,perhaps via a fabric approach.Each business unit has flexibility,but unlike in the decentralized model,this model provides centralized support and
106、policies and the hub has specialized skills.Any of these models can work effectively depending on company requirements and culture.However,there will need to be coordinated governance across both the decentralized and the hub-and-spoke models.Additionally,regardless of the model,it will be important
107、 for data and analytics leaders to have visibility to move analytics forward.For instance,one respondent utilizing a hub-and-spoke model noted,“The leaders of our teams at most companies would be directors.At our company they are technical managers.Theyre below enough hierarchy layers of IT that dat
108、a doesnt have significant enough visibility to executive leadership.”New OfficesAs mentioned,the data office is a relatively new model for providing value for data and analytics.Some organizations may refer to this as a hub.The center of excellence(CoE)construct is also one that includes different t
109、ypes of people,such as data analysts,data architects,data engineers,and data scientists(see Figure 5).About 75%of respondents have either a CoE,a data office,or a hub(not shown).About half report up through IT and the other half through the business(no figure shown).Modernizing the Organization to S
110、upport Data and Analytics tdwi.orgTDWI RESEARCH15The composition of new offices represents the newness of the roles.The composition of these offices reflects the newness of the some of the roles.For just those with a data office/CoE/hub,the most predominant role is the data analyst,followed by data
111、architects,engineers(as part of the DataOps team),and data scientists.Newer roles such as trainers and MLOps engineers are not as prevalent because many organizations are not yet building machine learning models.Even roles such as the modern analyst/citizen data scientist are not yet common because
112、organizations are still working on making data and analytics available to users across the organization in a more self-service manner,opening up the time for data analytics to grow their knowledge and become more sophisticated.New Team StructuresAs illustrated in Figure 5,organizations already have
113、new roles such as DataOps or modern analysts in place.We looked at how some of these teams are organized to execute.DataOps teams are currently either part of IT or are centralized(see Figure 6).An interesting evolving Figure 5Who is part of the CoE/data office/hub?Please select all that apply.Based
114、 on 159 respondents who have an office in place.Data analysts(i.e.,analyzes the data)63%Data architects51%Data engineers(i.e.,builds pipelines for data and analytics)45%Data scientists(i.e.,builds ML and other models)43%Developers33%Data stewards24%Citizen data scientists(i.e.,upskilled data analyst
115、s who build ML and other models)23%Solution managers18%ML operations engineers(i.e.,puts ML models into production,tracks,updates them)17%Trainers10%Other 4%Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH16model is the pod(or small group).In some cases,DataOps teams
116、are organized into pods that may consist of a data engineer,a data scientist,and a data analyst.They are part of a centralized team deployed where needed(18%).Another version of this centralized model is where DataOps is part of a centralized function and deployed where needed(23%).The other option
117、is that DataOps is part of IT(33%).This makes sense given that many organizations still have IT leading the data and analytics function,but companies may evolve this approach.We also asked where modern analysts are found in the organization.This role,where it exists,is scattered across the business.
118、For instance,the majority are in the business unit,where needed(37%,no figure shown).However,others are in a data science group,in a center of excellence that reports to the business,or federated as part of the CoE,but in different business units.USER STORYUsing communities of excellence to bridge t
119、he gap between siloed business unitsAccording to a data and analytics leader at a Fortune 500 insurance company,“We have a big matrix organization and different business units take different data management approaches.For instance,one unit is on premises,another is in the cloud.”Because his team int
120、eracts with both,he sees different challenges.“The cloud-based group has made more progress into machine learning.They also do some natural language processing.This team was able to attract more talent because they are on the cloud and doing more advanced analytics.“However,some of the problems have
121、 been to stay platform agnostic rather than locking into Amazon or Microsoft cloud services.On the other hand,the group that is on premises has been using Figure 6Does your organization have a DataOps team to help develop workflows in which data is continuously integrated,transformed,and prepared fo
122、r deployment into analytics applications?Based on 198 respondents.Yes,we have a DataOps team.They are organized in small pods that may consist of a data engineer,a data scientist,and a data analyst.They are part of a centralized function and deployed where needed.18%Yes,we have DataOps personnel.The
123、y are part of a centralized function and any needed staff are deployed where needed.23%Varies.Different business units may have DataOps teams.14%No,we dont have this team.IT performs this function.33%No,we dont have this team.It isnt needed.9%Other4%Modernizing the Organization to Support Data and A
124、nalytics tdwi.orgTDWI RESEARCH17antiquated tools and people dont know the platforms.However,because they manage their platforms,they dont have to worry about the bandwidth considerations of a multitenant environment or get IT security management comfortable with a new control environment.“From a dat
125、a governance perspective,we are fully siloed with a federated model,complete with different data governance offices”according to this data leader.“None of these organizations has CDOs or reports to the CTO.”Because of the nature of the business units and their specific needs,it hasnt yet been an iss
126、ue.However,since the two groups have developed their own tool sets and skills,they are trying to bridge the gap by providing communities of excellence.These are enterprise-wide teams that have grown from the bottom up to share methodologies,strategy,and philosophy in different areas,such as data gov
127、ernance.The communities hold quarterly meetings to share information and find opportunities.Some offer training and education.The model has worked,according to this data and analytics leader because“each person has a personal vested interest in the success of the community.We have established colleg
128、ial friendships and,as such,the communities have staying power.“Governance Team StructuresData governance is a critical component of a successful data and analytics strategy.In TDWI research,we often see that data governance is a top priority and a top challenge(especially in modern environments,suc
129、h as the cloud).Companies are organizing in different ways for data governance.About a third of respondents still dont have a clear organizational structure for data governance,but some companies have implemented the traditional model of a data governance council,steering committee or working group(
130、28%,no figure shown).We detect a move towards a federated model where a small central hub is responsible for coordinating data governance across the company(36%no figure shown).This model can work well with the hub-and-spoke framework mentioned above.Although the spokes are responsible for their own
131、 data governance,the hub sets the standards and policies for the spokes to follow.In other words,the hub coordinates data governance.Both models can provide value.For example,those organizations that measure value from data and analytics are more likely to use a more traditional model for data gover
132、nance,with a steering committee.That model clearly works for them and that is fine.It may be that the federated model will also provide value.The important thing is to have a model in place.New ParadigmsRecently,the idea of a“data mesh”has become popular.First coined by Zhamak Dehghani in 2019,a dat
133、a mesh supports distributed,domain-specific data consumers and views“data as a product,”with each domain handling its own data pipelines.The tissue connecting these domains and their associated data assets is a universal interoper-ability layer that applies the same syntax and data standards.The dat
134、a mesh encourages distributed groups of teams to manage data as they see fit,albeit with some common governance provisions.11 See Dehghani,Zhamak(2019,May 20).“How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh.”Accessed at https:/ the Organization to Support Data and Analytics tdw
135、i.orgTDWI RESEARCH18There are four pillars to the data mesh framework:Domain-oriented ownership.Business domains(e.g.,customer data or product data)own their own data,so ownership is decentralized.This should scale out data sharing and domains share data across organizational boundaries.Data as a pr
136、oduct.Data is accessible to those who need it and that data is viewed as a product,which means the customers of that product should be satisfied with it and the company should be organized to support a product view of data.Business domains are responsible for data quality,understandability,and inter
137、operability of the data as a product.Self-service.Here,a self-service platform is used to empower teams.Multiple personas can make use of the self-service platform.This will lower costs and enable development of data products such as dashboards,specific products such as network management for teleco
138、mmunications providers,and other data applications.Federated governance.In the data mesh,governance is based on a federated model with team members from different business units part of the effort.The model balances the autonomy of the domains but necessitates compliance,interoperability,and securit
139、y of the mesh.It is still early days for the data mesh framework.It is still early days for the data mesh framework,although some companies are using it.In fact,many of the CDOs TDWI speaks with agree with many of the principles.For instance,they like the idea of data as a product.They are,of course
140、,behind the notion of self-service as this has been something many organizations have been working towards.Many believe that governance should be in individual domains,but there is coordination between the groups.We asked respondents about the pros and cons of the data mesh approach.Some highlights
141、are illustrated in Table 1.We find it interesting that respondents seemed more likely to cite challenges with the data mesh approach than list opportunities.Many of these challenges have to do with education,resources,and cost.Respondents in a centralized model are concerned about how they can move
142、to a distributed model.Some respondents are concerned about the mesh removing jobs;if data and analytics functions move to the domains,what happens to the people currently performing these roles?Others are concerned about cost and allocating resources to various domains.Many just dont understand it.
143、Of course,this may change over time as organizations learn and evolve.Many organizations simply dont understand what a data mesh is all about.Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH19ChallengesNo one really understands itChange management We believe in a cent
144、ral organization with one version of the truth managed by data engineers Getting stakeholders and management to understand the value of mesh over our traditional approach;quantifying the route of mesh For the data mesh to be successful you need people who are data-oriented Removes jobsResources Allo
145、cation of resources and cost to the domains Inefficiency;error-prone;cross-domain analytics There are too many business clients who think they own the data but dont have the resources to manage itInformation not shared among stakeholders provides little valueDecentralized data;no governanceOpportuni
146、ties Simplifying and democratizing data access;easier and instant data delivery for business stakeholderEliminates bias,streamlines more data processed,quicker resultsIt took time to get the business to integrate the extra responsibility of building and delivering its own data as a product to the re
147、st of the organization.An advantage that immediately became apparent was the improvement of data literacy and improved information for business decision making.Have all units participate in data sharing and analyticsWould help reduce the feedback loop,ensuring better productsTable 1.Data mesh commen
148、ts.Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH20In this survey,fewer than 15%of respondents are currently using a data mesh,although about 20%plan to use it in the future.Three in ten respondents dont use a data mesh,although they support its principles.The rest
149、have no plans or dont know(see Figure 7).USER STORYThe Importance of Executive Leadership According to the senior director for data science at a global medical diagnostic provider,“Weve always been a data-driven organization.We started with ERP data in the 1990s,then we moved to store all transactio
150、nal data in a data warehouse in the 2000s.By the early 2000s we were up and running with analytics that was end-user-driven and had good self-service training in place.Weve grown from there to support data lakes for unstructured data,and about five years ago we merged our data warehouse and data lak
151、e into a data lakehouse.”The company has since moved to adopt data mesh principles.According to this senior director,“We treat data as an internal product.That data might be sales,services,or finance.Each domain takes responsibility and ownership for its data.That data is well defined,of high qualit
152、y,and there is information about what it is,where it is,Figure 7Does your organization utilize a data mesh paradigm*?*An organizational and architectural approach to share,access,and manage analytics data in complex and large-scale environments where data ownership is decentralized and each domain o
153、wns its own data,offers it as a product,delivers it in a self-service fashion,and employs federated governance.Based on 198 respondents.Yes (13%)No(30%)But we support many of the data mesh principles.No(20%)But we plan to move to that paradigm.No(25%)And we have no plans to move to that paradigm.Don
154、t know(12%)Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH21and how to get it.”To accomplish this,the organization has data enablement teams that help to document processes as well as architect how the data is evolving and how to help the data talk to each other.How
155、did the organization get to this point?“The CEO put a stake in the ground and said,I want people to analyze data and be more productive.The CEO himself talks with data.The rest of the organization had to figure out how to do that and measure it.Now we operate on measured goals.Youre not penalized if
156、 the goal isnt achieved,but you get kudos if you do.”Enabling Technologies Modernization often involves utilizing completely new technologies and delivering new workloads.Some enabling technologies for modernization were mentioned in the introduction to this report.When modernizing the environment,o
157、rganizations should consider technologies that enable different roles to do more.When modernizing the environment,organizations should consider technologies that enable different roles across the organization to do more.These tools are often automated and augmented.They are easy to use,scalable,flex
158、ible,and can support large amounts of diverse data.They include tools such as automated pipeline tools,cloud platforms,data fabric platforms,tools that automate insights and build ML models,and modern catalogs.We asked respondents what tools they were already using and plan to use in the future(see
159、Figure 8).Automated data pipeline tools are important to organizations.Close to 40%of respondents are already using automated pipeline tools;another 40%are planning to implement them.These tools help organizations move away from manual,error-prone ETL/ELT processes.With the complexity of the data ri
160、sing,organizations(in any configuration)will need these tools to migrate data to the cloud and keep data fresh.Some modern pipelines even provide automated data mapping for metadata.Cloud platforms provide scalability and flexibility.Cloud data warehouses and cloud data lakes are also cited as impor
161、tant for modern data and analytics.In this survey,39%of respondents are already using a cloud data warehouse,although weve seen numbers as high as 55%in other surveys.Another 39%are planning to implement a cloud data warehouse.Cloud data lake use (at 35%)is not far behind,and 42%are planning to use
162、one.These platforms provide the scalability and flexibility organizations need to store and analyze the large amounts of diverse data for modern analytics.Modern tools provide the basis for making data“accessible and easy to use.”Modern data catalogs help organizations understand their data.A data c
163、atalog is a central repository that contains metadata for describing data sets,how they are defined,and where to find them.Data catalogs enables people across the organization to understand their data and Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH22use trusted d
164、ata.Modern catalogs often contain collaboration features so users can rate data sets.This helps to build trust in the data because people can see what others thought of the data sets they are going to use.Data catalogs can contain automation tools to help classify sensitive data;they provide tools f
165、or data cleansing and even metadata discovery.These catalogs help organizations understand their data,regardless of the organizational model deployed.Close to 40%of respondents to this survey were already using a data catalog.Organizations that have measured success with their data and analytics eff
166、orts believe these tools,services,and platforms“provide the basis which makes data accessible and easy to use.”Figure 8What tools and technologies do you currently have in place or are you planning to put in place that can help with modern data and analytics?Based on 198 respondents.39%40%21%Automat
167、ed data pipeline tools39%39%22%Cloud data warehouse38%43%19%Data catalog35%42%23%Cloud data lake30%39%31%Automated data cleansing tools28%33%39%Data virtualization26%47%27%Automated data mapping tools(for metadata)24%49%27%Governance tools that provide compliance monitoring23%47%30%Automated insight
168、s tools20%43%37%Governance tools that provide impact analysis18%41%41%Automated data classification tools18%30%52%Data fabric17%34%49%Coverged platform17%41%42%Automated ML building tools16%38%46%Automated MLOps toolsCurrently in placePlanningNo plans or dont knowModernizing the Organization to Supp
169、ort Data and Analytics tdwi.orgTDWI RESEARCH23A logical data fabric architecture to integrate hybrid environments.Some organizations are taking a logical architectural approach to their data environments rather than a physical one.One example of the logical architecture is the data fabric already me
170、ntioned.The data fabric design can be accomplished with data virtualization,a method that integrates heterogeneous and distributed data across multiple platforms without replicating it.The approach creates a single“virtual”data layer that unifies data and supports multiple applications and users.Dat
171、a virtualization can create logical views in which the data looks consolidated although the data has not been moved or physically altered.A logical data fabric can help organizations knit together disparate data sources in their hybrid universe of data platforms including data warehouses,data lakes,
172、and other data platforms.This logical architectural approach can support newer models,too,such as multi-cloud,hybrid,and edge scenarios.Data virtualization platforms,as part of this logical data fabric,often support analytics,a catalog,self-service,and strategies for optimizing cross-platform perfor
173、mance(such as dynamic query optimization,caching capabilities,summary tables,or in-memory computing),even across multiple cloud providers.The single virtual layer can help multiple personas across the organization make use of data without having to understand the complexity beneath.Close to 30%of re
174、spondents to this survey were already using data virtualization with 33%planning to use it in the future.Automation tools for data governance are becoming part of the equation.In addition to the data catalog,organizations are making use of other automated tools for data governance.For instance,in th
175、is survey,30%are making use of data cleansing tools;26%are making use of automated data mapping tools.These tools and the tools used for analytics(see below)are becoming an important part of an organizations data and analytics toolkits.As one respondent noted,“Automation is freeing up resources for
176、more complex work and reduces time to delivery of our product development.”Another said,“Automation helps to improve productivity and helps us focus on our business goals.”Automated and easy-to-use analytics tools are becoming popular.In TDWI surveys,we often see that demand for advanced analytics,s
177、uch as machine learning and natural language processing,continues to remain strong.Vendors are trying to make their tools easier to use.For instance,some tools automatically surface insights(22%)or build predictive models(18%,no figure shown).Some have natural language query interfaces so users dont
178、 need to know SQL.Other examples include geospatial tools that are simple enough to be used by business users.Easy-to-use tools open up analytics to more users across the organization.Other tools such as MLOps tools are still early in adoption(16%),yet they will become quite important as organizatio
179、ns try to put more models into production.These tools can help MLOps engineers become more productive.For instance,some tools enable automated model monitoring and retraining models based on certain triggers(e.g.,accuracy falls below a certain threshold).Modernizing the Organization to Support Data
180、and Analytics tdwi.orgTDWI RESEARCH24Building the Skills,Culture,and TalentAlthough technology can help an enterprise move data and analytics programs forward,people need to be able to use the technology.To do that,companies need to build a data culture and the skills that embrace using these new to
181、ols.In this TDWI survey,the majority(55%)of respondents stated that the top challenge they have with automated tools are the skills needed to make sure they are working correctly(see Figure 9).No other challenge came close.For example,if a modern analyst uses an AutoML tool,the analyst will still ne
182、ed to be able to defend the output of a model that may have been built using it.Even if there are explainability features as part of the tool,the analyst needs to know how machine learning works.Likewise,a business user who utilizes an analytics tool to surface insights would still need to be able t
183、o think critically about the output in order to use it properly when making decisions.An analyst who builds a pipeline would still need to understand the data they are putting into the pipeline.Building the skills and the culture starts at the top with executives,who set the tone,evangelize the powe
184、r of data and analytics,and walk the walk by using data and analytics in their day-to-day job.An executive speaks the language of analytics.In previous TDWI research,weve seen that organizations that move forward successfully in their data and analytics journey have executive buy-in.Fifty-five perce
185、nt of respondents said that the top challenge they have with automated tools is skills.Building the culture and the skills involves creating a strategy,communicating it,and carrying it out.Figure 9What challenges is your organization facing with automated tools?Please select all that apply.Based on
186、198 respondents.Skills are needed to make sure the tools are working correctly 55%We dont have the staff to train people on new tools 34%The tools are too expensive 29%The tools dont work as advertised.28%We cant get funding to purchase the tools 23%People dont want to use new tools 22%The tools don
187、t integrate well with our environment 21%Performance degrades on data at scale 20%Other 7%Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH25These tasks are discussed in more detail in the next two sections.Building the CultureOrganizational factors can often make or b
188、reak a modernization effort,and culture can be the hardest hurdle to overcome.As mentioned,leadership support is critical;the right leaders and the right funding are needed.As one respondent said,“Our executive leadership needs to catch the vision and start driving change.”The culture should be supp
189、ortive and people should feel they can trust their data and their organization.Organizational factors can often make or break a modernization effort and culture can be the hardest hurdle to overcome.Equally important,people in the organization need to trust their executives.For example,people need t
190、o trust that their organization is considering them as they move to deploy new tooling.A DBA might be concerned that bringing in a cloud platform will cause him or her to be fired.The same concern holds true with the introduction of a data mesh.Any new technology or idea may drive fear of job loss.T
191、he organization will need to let people know how they will fit into a new organizational or technology model.Otherwise,ownership and power dynamics will more likely come into play.On the leaders part,this will involve significant communication of the why and how of the strategy.It may even involve i
192、mplementing formal change management frameworks.As one respondent put it,“well-defined roles and clarity in the process”is critical.It is important to lay out the responsibili-ties of each team clearly.On the organizational front,the enterprise needs to establish a strategy and communicate it to the
193、 business.The strategy should include KPIs to measure achievements so success can be objectively measured.On the data front,we mentioned the need for a solid data foundation and tools such as a data catalog,to help understand the data.Employees need to be able to find the data,understand it,and trus
194、t it.We asked respondents how they are building the culture for new roles and organizational models(see Figure 10).Many of the responses involved communication because it is important to communicate the value of data and analytics,what changes are occurring,and what changes are coming.For instance,a
195、bout a third of respondents are publicizing new processes,roles,and structures.Some are holding company meetings to explain why and how the change is occurring.Communication is happening in more casual settings,too,such as in lunch-and-learn sessions,so people can learn about new roles.Organizations
196、 are also holding meetings with the staff affected by new roles.However,about a third of respondents arent doing anything,and doing nothing does nothing to derive value from data and analytics efforts.A small group of respondents(about 10%)is implementing formal change management frameworks.One exam
197、ple is ADKAR.2 The ADKAR Model is a goal-oriented change management model that guides individual and organizational change.ADKAR is an acronym for awareness,2 Developed by Jeff Hiatt,founder of Prosci.See https:/ the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH26desire,knowledge,
198、ability,and reinforcement.There are other change management frameworks,such as McKinsey 7S(structure,strategy,systems,skills,style,staff,and shared values).3 Organizations should adopt or develop frameworks that work for their particular environment.It is important to remember that building out a cu
199、lture is a continuous process.Models and methods will change through time as ideas and frameworks evolve.Building Data LiteracyAs part of the cultural change,organizations must have a deliberate data literacy practice to move forward in both understanding data and utilizing it properly in analytics
200、and applications.This goes for roles across the organization,including those who will use the output of models(for instance,in customer service).Often times,this function is part of the data office or the center of excellence,under the chief data or analytics officer.Data literacy enablement teams d
201、etermine what roles need what training,devise the strategy,and execute on it.Well-oiled data literacy teams will assess data literacy and then put together programs based on role and experience levels.Sometimes data literacy training is scheduled and mentors are available to help.Other times,it is o
202、nline.In some cases the data literacy team may deliver the training themselves;in other cases,they will outsource the training or send people to vendor-sponsored training(for instance,on a certain cloud platform or a specific analytics product).Critically,the literacy enablement team also monitors a
203、nd measures progress towards its goals.Organizations are taking different routes to data literacy(see Figure 11)and many dont yet have any data literacy teams in place.Slightly less than 15%have literacy teams in a data office or CoE.Figure 10How is your organization building the culture for new rol
204、es and organizational models?Please select all that apply.Based on 198 respondents.We are publicizing new processes,structures,etc.in company newsletters,email,and other forms of communication30%We are holding company-wide meetings,led by C-suite members to discuss changes and why they are happening
205、34%We are holding lunch-and-learns to discuss new roles so people can learn more28%We are implementing formal change management frameworks such as McKinsey 7s or similar10%N/A29%Other5%3 See https:/ the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH27Twenty-two percent of responden
206、ts state that the teams are part of IT.In others,it is up to the business units to determine whether to have a data literacy team.At least half of the organizations surveyed for this report dont have any form of company-wide data literacy team in place.It is still early days for data literacy enable
207、ment teams,and TDWI expects these teams to grow over time.Additionally,as part of the data literacy effort,some companies are looking to upskill certain job functionswhether by helping data management professionals learn cloud platforms or aiding data analysts become data scientists.For those lookin
208、g to improve the skills of data analysts,some enterprises provide training for them in-house or via a third party.Others are self-taught.In addition to using automated tools that produce models,data analysts spend time with data scientists who can mentor them.Companies that have been successful ofte
209、n develop employee skills as they modernize their analytics capabilities.Getting to ValueTo understand how organizations are gaining value from modern analytics and how organizational constructs play a role,we compared three groups of respondents discussed earlier in this report.These include those
210、who were able to measure a top-line impact from modern analytics(31 respondents),those who thought there was value although they havent measured impact(128 respondents),and those who did not measure value(31 respondents).We looked for statistically significant differences between the groups.Although
211、 a relatively small self-identifying sample,there were some interesting findings when the data is examined (see Figure 12).Enterprises that measured value tend to be more mature.Not surprisingly,organizations that measure value tend to be more mature than Figure 11Does your organization have data li
212、teracy enablement teams that plan and execute programs to improve data literacy in your organization?Based on 198 respondents.Yes,they are part of the CoE/data office/hub14%Yes,they are part of our data and analytics practice housed in IT22%Yes,these teams are part of individual business units11%Var
213、ies,up to the individual business units to decide if they are needed and implement them15%We dont have any teams resembling data literacy teams36%Other2%Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH28those that thought they were successful but hadnt measured value
214、and those that stated they are unsuccessful.They are more likely to make use of tools and technologies that are proactive(such as machine learning)and they act on their insights.An organizations culture supports data-driven decisions and the data infrastructure is sound.They are more likely than the
215、 other two groups to make use of a physical data warehouses,data lakes,or data marts that are in the cloud.They are also more likely to use automated tools versus manual tools,as well as a data catalog.Those that have measured success also make use of newer paradigms such as the data mesh.They are a
216、lso more likely to have data literacy enablement teams to help employees learn skills for data and analytics.Figure 12Characteristics of Companies that Measured ImpactOrganizational FactorsTechnology FactorsBased on 190 respondents.Sound data infrastructure 61%25%10%Physical DW/DL on cloud 38%21%6%A
217、utomated pipeline 56%39%19%Automated MLOps 31%13%13%Data catalog in place 46%41%16%CDO part of C-suite 59%40%16%CDO leads BI 49%30%19%CDO leads data science 51%27%16%Satisfied with leadership 64%23%0%Data literacy teams 31%11%6%Successfulmeasured impactSuccessfuldidnt measure impactUnsuccessfulModer
218、nizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH29TDWI has referred to the idea of a virtuous circle in other reports.As organizations begin to reap the benefits of data and analytics,they tend to put more advanced data and analytics in place.The success then builds on its
219、elf.This is most likely what we are seeing here for modern analytics.As organizations become successful,they are in a good position to further innovate and gain value.This value continues to grow and the culture is such that the organization will measure value as part of its processes.For instance,m
220、ature companies are more likely to measure outcomes and think in terms of metrics and KPIs as part of measuring a success strategy.We can learn from them.In this survey,respondents mentioned measuring churn reduction or outage prediction.In this case,they are actually measuring the business value di
221、rectly using before-and-after metrics.In other cases,respondents are using adoption metrics,budget numbers,or reporting against the strategic plan to measure value(which isnt as powerful but can be helpful).The point is,however,that they can measure it.Others would do well to follow their lead becau
222、se this can help move an analytics program forward.The reporting structure is through the C-suite.Those enterprises that measured success are more likely to have a CDO/CAO lead their analytics effortsspecifically in BI and data science initiativesthan the other two groups.The organizational model it
223、self(centralized,hub-and-spoke,or decentralized)doesnt matter as much in terms of driving value.However,organizations that report a measured top-line impact are more likely to have a CDO/CAO/CDAO who is part of the C-suite and the same group leads both data and analytics.The group that measured an i
224、mpact is also more likely to be satisfied with their leadership.Having someone in a data/analytics leadership role and having that person as part of the C-suite gives the role the visibility and power it needs to make an impact.That role,however,needs to coordinate with other data and analytics role
225、s across the company.As one respondent said,“The CDO efforts are good and moving forward,but all other data efforts are basically random.”In other words,the CDO needs to coordinate across the organizations for the strategy to be successful.Those respondents who measured value are more likely to have
226、 a CDO/CAO/CDAO who is part of the C-suite.Good communication is in place.Those who measured a top-line impact are also more likely to have a communications strategy across the organizations than do the other groups.They have a leader evangelizing the power of data and analytics.As mentioned,they ar
227、e more likely to have data literacy teams in place to enhance skills and talents and the team communicates with the organization about training.They are also more likely to use plans such as publicizing changes to the organization via newsletters or other media.However,more can be done in this area(
228、such as or-ganization-wide meetings)as clear communication is vital for the success of a modernization effort.There were no significant differences between companies of different sizes or industries;however,this should be explored in more detail because the sample size for this survey is relatively
229、small.Modernizing the Organization to Support Data and Analytics tdwi.orgTDWI RESEARCH30RecommendationsThis report has detailed many best practices for modernizing data and analytics.In closing,we summarize the report by listing the top best practices for successful modernization,along with comments
230、 about why each is important.Think of the best practices as recommendations that can guide your organization on a successful modernization journey.Remember,however,that it is a journey and roles and processes will evolve.If possible,hire a C-suite leader.The data from this report,as well as other in
231、dustry research,indicates that the CDO should be part of the C-suite.CDOs are especially important in the executive suite in traditional companies that have not historically been data-driven.Becoming a member of the C-suite gives the role the visibility and influence it needs to help the business de
232、velop a strategy for modernization and derive value by executing on(and being responsible for)that strategy.Tie the strategy to key metrics and KPIs.The data office can help.In many organizations,especially those with hub-and-spoke or decentralized organizational models,the data office is important.
233、This office can offer advice,provide policies and standards,coordinate governance,and manage the data infrastructure.It can be a small but mighty.Depending on the status of your companys modernization efforts,consider the data office.Think through your organizational model.Although no single organiz
234、ational model was more likely to generate value in this particular survey,it is important to think through what kind of organizational model makes the most sense for your company.This will depend on company size,needs,resources,and culture.For example,a mature company with the resources and culture
235、may decide that it should implement a decentralized data mesh framework with a small data office.A company that just implemented a centralized cloud platform,in addition to roles for data and analytics,may not determine that it is the right time to move forward with a different organizational struct
236、ure.Consider new roles and organize to execute.Several new organizational rolesincluding data engineers,MLOps engineers,and modern analystswill be important as your organization modernizes.These people can add significant skills and value to the data and analytics life cycle.Start planning for new r
237、oles as you build your strategy.There are tools that can help with insights,modern pipeline development,and MLOps that can make them more productive.Consider these as well.Plan for data literacy enablement teams.Data literacy enablement teams are a relatively new but important role because data lite
238、racy is so critical for success in understanding data and moving forward with self-service and more advanced analytics.Regardless of what the team is called,it is a good idea to have a group involved with training and enablement rather than just hoping it will occur.Modernizing the Organization to S
239、upport Data and Analytics tdwi.orgTDWI RESEARCH31 Put a solid and modern data foundation in place.Modernization will,no doubt,require new data platforms to support large volumes of diverse data and multiple personas.Consider the cloud for scalability and flexibility.If youre in a hybrid environment,
240、think about a data fabric approach using data virtualization to make it easier for accessing and then analyzing the data.In any case,the data foundation must be managed and governed.Implement a data catalog.Many organizations want to implement a data catalog.If that isnt part of your companys plan,i
241、t is something to consider.The data catalog helps organizations better access and understand the data they want to use for analytics.It can help specify the owners of any data asset.As data sources become more complex and diverse,it will be important to have a way to understand and trust that data.M
242、ake sure automated tools are on the road map.As data and analytics environments become larger and more complex,automation will be critical.Whether that automation is used in managing and governing data(such as automating pipelines),identifying sensitive data,or addressing data quality,these tools ca
243、n help the organization become more productive.Additionally,automated and augmented tools in analytics(such as those that surface insights or build machine learning models)can help data scientists,business analysts,and business users all get to insights more quickly.Communicate.Communicate.Communica
244、te.Communication is key to moving any modernization effort forward.It lets those in your organization know what is coming,have time to buy into it,and reduce anxiety of those who fear the changes by helping them better understand the changes and how individuals might be affected personally.Dont be a
245、fraid of a formal change management process.In some cases,cultural change will require that the organization implement formal change management frameworks.Many organizations TDWI speaks to have done this and it may be necessary based on your companys culture.Dont forget about data governance.Weve me
246、ntioned data governance throughout this report,yet it bears mentioning again.Data governancethe rules,policies,and processes that ensure that data is compliant and trustworthyis key to modern analytics.Be patient.Change doesnt happen overnight.It will be important to give the CDO and the data office
247、 time to succeed.tdwi.orgTDWI RESEARCH32Research Sponsors Snowflake delivers the Data Clouda global network where thousands of organizations mobilize data with near-unlimited scale,concurrency,and performance.Inside the Data Cloud,organizations unite their siloed data,easily discover and securely sh
248、are governed data,and execute diverse analytics workloads.Wherever data or users live,Snowflake delivers a single and seamless experience across multiple public clouds.Join Snowflake customers,partners,and data providers already taking their businesses to new frontiers in the Data Cloud.Find out mor
249、e at .TDWI Research provides research and advice for data professionals worldwide.TDWI Research focuses exclusively on data management and analytics issues and teams up with industry thought leaders and practitioners to deliver both broad and deep understanding of the business and technical challeng
250、es surrounding the deployment and use of data management and analytics solutions.TDWI Research offers in-depth research reports,commentary,inquiry services,and topical conferences as well as strategic planning services to user and vendor organizations.A Division of 1105 Media 6300 Canoga Avenue,Suite 1150 Woodland Hills,CA 91367E infotdwi.orgtdwi.org