1、By Philip RussomThe Semantic Layers Critical Roles in Modern Data Architectures CHECKLIST 2023 tdwi.org1 TDWI RESEARCHThe Semantic Layers Critical Roles in Modern Data ArchitecturesBy Philip RussomUnderstanding the semantic layers critical roles:Understand what a semantic layer should be and doConsi

2、der why your data architecture needs a modern semantic layerRecognize where a semantic layer fits in todays data architecturesUnderstand how the semantic layer helps remodel distributed dataAppreciate how a semantic layer contributes to common data architectures12345Many of the most exciting innovat

3、ions and advancements in data management today are occurring within the semantic layer of data architectures and data stacks.For example,we are witnessing new or improved approaches to semantic modeling,data cataloging,and data lineage.Even older forms of managing semanticssuch as metadata and virtu

4、alizationare being infused with new techniques for agile modeling,performance optimization for logical and virtual data environments,and intelligent augmentation(i.e.,tool algorithms driven by machine learning and graph analytics).The innovations of the semantic layer also play a role in improving l

5、arge-scale data and analytics architectures.For example,the new definition of data fabric is not possible without a modern semantic layer,and the semantic layer can be a backbone for unifying new data and analytics architectures in the tdwi.org2 TDWI RESEARCHTDWI CHECKLIST REPORT:THE SEMANTIC LAYERS

6、 CRITICAL ROLES IN MODERN DATA ARCHITECTUREScloud.Furthermore,a well-designed semantic layer allows analytics teams to define business metrics,hierarchies,and dimensions on top of big data while providing a means to centrally govern data access and deliver high-performance interactive queries.This T

7、DWI Checklist educates data and analytics leaders about modern platforms and practices for the semantic layer.It does so by discussing five beneficial characteristics of the modern semantic layer but in the context of the semantic layers critical roles in modern data architectures.NOTE:This report a

8、ssumes the reader is familiar with data architectures.Readers needing more details can read the TDWI Checklist Report:Six Requirements for the Modern Data and Analytics Cloud Stack.1Understand what a semantic layer should be and doThe semantic layer has been around for many years in many guises.The

9、earliest ones originated as functionality embedded and buried within larger tools,typically for reporting,enterprise business intelligence,data integration,and database management.The catch was that such semantic layers were usually optimized for their parent tool and not much else.Today,a truly mod

10、ern semantic layer is a standalone tool type that provides data semantics services for multiple tools within a multitool and multiplatform data architecture.This gives the modern semantic layer the ability to serve many architectural layers,tool types,data platforms,use cases,and business department

11、s without favoring one over another.The data descriptions(or“data about data”)created and managed by a semantic layer may take the form of older techniques(such as metadata management,federation,and virtualization)or newer ones(such as data catalogs,data lineages,dimensional models,or automated gene

12、ration of data descriptions via knowledge graphs).Whether the semantic layer is a unified environment from a vendor or assembled by technical users as a“best of breed”collection of multiple tools from multiple vendors,it should support many semantic tool functions.A semantic layer platform must go b

13、eyond data definitions to provide rich capabilities in semantic modeling and data modeling.In other words,a tool for the semantic layer should actively support the creation of new data structures and data products(whether federated,virtual,logical,or data sets in storage),not just descriptions of so

14、urce data and its characteristics.The semantic layer manages the interactions between data consumers(whether they be humans or AIs)and enterprise data assets(typically stored in the cloud).This interaction hinges on a semantic model that maps the language of raw data(e.g.,field and table names,file

15、and database formats)with business concepts(e.g.,revenue,ship quantity,fiscal quarter).The structure of the semantic model may take different forms,defined using various ways of describing data.A semantic layer must translate data consumer requests to the flavor of SQL preferred by the source data p

16、latform.It must accommodate multiple inbound protocols(not just SQL)because tools themselves support different protocols.For example,tdwi.org3 TDWI RESEARCHTDWI CHECKLIST REPORT:THE SEMANTIC LAYERS CRITICAL ROLES IN MODERN DATA ARCHITECTURESto support Excel,the semantic layer should support MDX.For

17、data science,it should support Python.For application developers,it should support REST.In production,a semantic model solution must deliver“speed of thought,”direct query performance through automated performance optimization.Query performance is imperative because without it,many end users will ma

18、ke redundant and non-governed copies of data in the form of data extracts(TDEs in Tableau)or data imports(Power BI,Qlik,etc.).One of the greatest benefits of a modern semantic layer,extended with query optimization and virtualization,is that the semantic layer serves as an abstraction layer for gove

19、rnance,security,and“single source of truth.”2Consider why your data architecture needs a modern semantic layerThe semantic layer provides a collection of data descriptions(and tools to create and maintain them)to make a single,centralized,and standardized architectural layer for most data semantics.

20、Being centralized,a semantic layer delivers a standardized and consistent way of representing enterprise data to different types of users,tools,and data management processes.This centralized approach simplifies many things,and it delivers important architectural benefits:Provides a process that is a

21、 governed,standardized,and consistent way of representing distributed enterprise data Can be a single point of entry,with single sign-on and other security for systems the semantic layer accesses Friendly descriptions of data that simplify and improve modern data practices(e.g.,self-service,dashboar

22、d customization)Reuse of composable data objects and data products listed in the semantic layer Automation for data governance,monitoring via data,audit,and data observability When done well,can elevate data literacy and democracy Todays data architectures are trending toward centralized data organi

23、zation paradigms(databases,data lakes,data warehouses,data science labs)within both cloud and on-premises architectures.Even when physically consolidated,many data environments are not logically organized to support direct analytics use.A semantic layer helps to unify far-flung data architectures in

24、 that it:Describes data consistently for all layers of the architecture and beyond Reaches all platforms:multiple brands,on multiple clouds or on premises,etc.Can provide data views that incorporate data from many architectural layers and elsewhere Facilitates data access and interfacing for many us

25、ers and tools Can enable virtualization for the logical data warehouse and virtual data lake tdwi.org4 TDWI RESEARCHTDWI CHECKLIST REPORT:THE SEMANTIC LAYERS CRITICAL ROLES IN MODERN DATA ARCHITECTURESBecause it is an abstracted layer,the semantic layer creates a kind of future proofing.By decouplin

26、g the layers of data consumption tools and data storage platforms,a semantic layer provides IT with the freedom to consolidate,move,or transition its data without disrupting end-user data consumption.A semantic layer also provides an open platform for plugging in new or different data consumption to

27、ols as they arise.As you can see,the business and technology benefits of a semantic layer are many,which is why organizations are turning to it.3Recognize where a semantic layer fits in todays data architecturesA modern semantic layer can be deployed to serve as an independent layer near the top of

28、todays data architectures(see Figure 1).It fits just below the data consumption layer(for reporting,analytics,and self-service)and just above the data integration and streaming functions of the data fabric.It represents both data traveling into the data fabric and later being processed by the fabric

29、,as well as data at rest in data storage(both on premises and in the cloud),typically organized as data lakes,warehouses,and other storage options.This is a strategic position for the semantic layer,between the data consumption and data fabric layers.This location enables it to serve multiple functi

30、ons:users and their tools for BI and analytics get a broader enterprise view of data compared to the limited views from their internal semantic layers;the data fabric gets the advanced semantic functionality and automation it needs;but the whole data architecture still has access to standalone and i

31、ndependent semantic layer tooling.This also creates a unification effect,which is beneficial to multiplatform architectures.DATA CONSUMPTIONSemantic LayerReal-Time Data LayerPipelines,ETL,Quality,etc.Data ScienceBusiness IntelligenceData LakeData WarehouseDATA FABRICSelf-ServiceData LakehouseSOURCE:

32、Philip RussomDATA STORAGEFigure 1.The semantic layers strategic position in the data architecture.The modern semantic layer can support multiple styles of data architecture,including those that are centralized,distributed,or a combination of both.For example:A semantic layer provides centralized dat

33、a semantics,which in turn can support the centralized data that many users want in new data architectures(especially in the cloud)An independent semantic layer also supports decentralized and distributed design patterns,including the data mesh The semantic layer can support hybrid combinations of mu

34、ltiple architectural variations,even when in the cloud or on premises tdwi.org5 TDWI RESEARCHTDWI CHECKLIST REPORT:THE SEMANTIC LAYERS CRITICAL ROLES IN MODERN DATA ARCHITECTURESEssentially,a modern,independent semantic layer can support just about any variation of data architecture available today.

35、4Understand how the semantic layer helps remodel distributed dataAs we just saw,a modern semantic layer can be applied to either a physically centralized environment(data lake or warehouse)or to a distributed environment that includes many data sets and platform types(as results from many unplanned

36、or uncontrolled data programs).In these cases,a semantic layer often acts as an abstraction layer that unifies disparate layers of a data architecture and its diverse tools and platforms.It does this by supporting business metadata and other approaches to creating business-friendly views of data and

37、 data products.The resulting business-friendly data models(created at the semantic layer,not physically in storage)hide the complexity of data in the data fabric and storage layers of the architecture,thus making distributed data easier to understand and access for a wide range of users and applicat

38、ions.Ideally,data views may be constructed as semantic models,relational tables,OLAP dimensions,metric hierarchies,or time series.This means that a modern semantic layer tool should be capable of modeling most of the data structures common in data architecture usage,especially for analytics use case

39、s.For example,many organizations are hoping for one data source(or collection of data sources)exposed multiple ways to multiple users,teams,and tools.This is so that users can avail themselves of multiple data consumption styles,unlike the limited approaches typical of embedded semantic layers or tr

40、aditional data warehouses.As another example,data lakes are known for their massive stores of raw data.The semantic layer can make this data far more useful and valuable by creating a variety of views into the lake.This creates a logical data lake that complements and adds value to the actual lake i

41、n storage but without creating redundant copies and without having to maintain data aggregates in storage.5Appreciate how a semantic layer contributes to common data architecturesA semantic layer can add many types of technical and business value to a variety of local data architectures across an en

42、terprise.This is true whether the semantic layer is retrofitted to a mature deployment or utilized from the beginnings of that architecture.In particular,the semantic layer can be beneficial to data warehouses,data lakes,data fabric,and DataOps.THE SEMANTIC LAYER HELPS THE DATA WAREHOUSE(DW)The sema

43、ntic layer modernizes the data semantics that a DW depends on.For example,most DWs suffer from metadata that is sparse,incorrect,and nonstandard;a semantic layer tool helps to correct these problems and give metadata a tdwi.org6 TDWI RESEARCHTDWI CHECKLIST REPORT:THE SEMANTIC LAYERS CRITICAL ROLES I

44、N MODERN DATA ARCHITECTUREShigher quality.Similarly,some DWs operate almost exclusively with technical metadata;a semantic layer helps the DW team embrace business metadata and more advanced forms of semantics,such as the data catalog and data lineage.The business-friendly semantics of the semantic

45、layer enable DWs to participate in more use cases.This is particularly useful when less-technical users want to access the data in a DW for self-service or when users need to personalize their management dashboards with metrics and KPIs from a DW.The semantic layer is a key enabler for the logical D

46、W.A logical DW is inherently multiplatform,hybrid,and distributed.To make this complicated microarchitecture seem more unified and usable,DW professionals use data virtualization and data views.Because a semantic layer is inherently logical,virtual,and view-driven,it can be a natural addition to a D

47、W to make it a true logical DW.THE SEMANTIC LAYER HELPS THE DATA LAKEThe semantic layer helps a data lake avoid becoming a data swamp.A lack of metadata and other data semantics is the leading cause of swamps;a semantic layer provides ample metadata management for this situation.The semantic layer p

48、rovides logical structures for unstructured data lakes.Most lake data is raw,file-based,and multistructured.The semantic layer gives this environment much-needed structure via logical data views without creating redundant data copies and aggregations.In turn,the views make the raw and unimproved dat

49、a of a lake far more accessible for self-service,exploration,and analytics.THE SEMANTIC LAYER HELPS THE DATA FABRICThe most recent definition of the data fabric is“an architecture for unifying and governing multiple data management and data semantics disciplines,from data integration and quality to

50、metadata and data cataloging.”Among other requirements,a data fabric requires sophisticated semantics that are centralized,standardized,and shared for the many tool types found in modern data fabrics.The semantic layer satisfies this requirement.THE SEMANTIC LAYER HELPS DATAOPSAll data-driven develo

51、pment processes benefit from better semantics and DataOps is such a process.In fact,the semantic layer can help DataOps achieve many of its key objectives by providing:Centralized,standardized,and shared data semantics for data engineering but with more features than a metadata repository or catalog

52、 Automation for data semantics to accelerate the delivery of data products Semantic modeling(faster than models based on aggregation)Reduced data prep and design work Quick turnaround for the CI/CD processes of data products tdwi.org7 TDWI RESEARCHTDWI CHECKLIST REPORT:THE SEMANTIC LAYERS CRITICAL R

53、OLES IN MODERN DATA ARCHITECTURESConcluding ThoughtsOrganizations wishing to modernize their data semantics by adopting a semantic layer should heed the following key takeaways and recommendations:Take data semantics more seriously,in general.Make it a priority,from storage to data engineering to da

54、ta consumption.Do more than just create and manage technical metadata.Modernize with business metadata,cataloging,data lineage,and tools optimized for the unified semantic layer.Recognize that todays semantic layer is a standalone architectural layer.It is not buried inside a single tool.It is stand

55、alone,so it can be agnostic and support many tools,users,and use cases.Adopt the semantic layer concept as a best practice and architecture standard.As a best practice,it improves data access,reuse,standards,and engineering.As an architectural layer,it unifies the whole architecture,enables interope

56、rability,and presents architecture-wide observability.Deploy a modern semantic layer for its business and technical benefits.For example,it enables new practices,such as the data fabric and next-level data virtualization.It also modernizes and elevates common data solutions,namely the warehouse,lake

57、,fabric,and DataOps.Look for semantic layer tools with innovative and advanced functions.A comprehensive tool will support multiple forms of data semantics and be open and tool-agnostic.It will also support performance optimization,data virtualization,and multiple approaches to modeling.tdwi.org8 TD

58、WI RESEARCHTDWI CHECKLIST REPORT:THE SEMANTIC LAYERS CRITICAL ROLES IN MODERN DATA ARCHITECTURESAbout our sponsor AtScale enables smarter decision-making by accelerating the flow of data-driven insights.The companys semantic layer solutions simplify,accelerate,and extend business intelligence and da

59、ta science capabilities for enterprise customers across all industries.With AtScale,customers are empowered to democratize data,implement self-service BI,and build a more agile analytics infrastructure for better,more impactful decision-making.For more information,visit .About the authorPhilip Russo

60、m has 25 years of experience as an IT industry analyst researching user best practices,vendor products,and market trends in data management and analytics.This includes data warehousing,data lakes,data integration,data quality,hybrid data architectures,cloud data management,data governance,analytics,

61、and data platforms.He has worked at most of the worlds leading IT analyst firms,namely:Gartner Inc.,Forrester Research,Giga Information Group,TDWI,and Hurwitz Group.In those positions and others,he produced over 650 research reports,magazine articles,speeches,and webinars.Before becoming an industry

62、 analyst,he worked for database software vendors as a product manager,product marketer,and documentation writer.tdwi.org9 TDWI RESEARCHTDWI CHECKLIST REPORT:THE SEMANTIC LAYERS CRITICAL ROLES IN MODERN DATA ARCHITECTURESAbout TDWI ResearchTDWI Research provides industry-leading research and advice f

63、or data and analytics professionals worldwide.TDWI Research focuses on modern data management,analytics,and data science approaches and teams up with industry thought leaders and practitioners to deliver both broad and deep understanding of business and technical challenges surrounding the deploymen

64、t and use of data and analytics.TDWI Research offers in-depth research reports,commentary,assessments,inquiry services,and topical conferences as well as strategic planning services to user and vendor organizations.About TDWI Checklist ReportsTDWI Checklist Reports provide an overview of success fac

65、tors for a specific project in business intelligence,data warehousing,analytics,or a related data management discipline.Companies may use this overview to get organized before beginning a project or to identify goals and areas of improvement for current projects.2023 by TDWI,a division of 1105 Media

66、,Inc.All rights reserved.Reproductions in whole or part are prohibited except by written permission.Email requests or feedback to infotdwi.org.Product and company names mentioned herein may be trademarks and/or registered trademarks of their respective companies.Inclusion of a vendor,product,or serv

67、ice in TDWI research does not constitute an endorsement by TDWI or its management.Sponsorship of a publication should not be construed as an endorsement of the sponsor organization or validation of its claims.A Division of 1105 Media 6300 Canoga Avenue,Suite 1150 Woodland Hills,CA 91367E infotdwi.orgtdwi.org