1、These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Cloud Data Warehousing3rd Snowflake Special Editionby David Ba

2、umThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionPublished byJohn Wiley&Sons,Inc.111 River St.Hoboken,NJ 07030-Copyright 2024 by John Wiley&Sons,Inc.,Hoboken,New Je

3、rseyNo part of this publication may be reproduced,stored in a retrieval system or transmitted in any form or by any means,electronic,mechanical,photocopying,recording,scanning or otherwise,except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act,without the prior written

4、 permission of the Publisher.Requests to the Publisher for permission should be addressed to the Permissions Department,John Wiley&Sons,Inc.,111 River Street,Hoboken,NJ 07030,(201)748-6011,fax(201)748-6008,or online at http:/ Dummies,the Dummies Man logo,The Dummies Way,D,Making Everything Easier,an

5、d related trade dress are trademarks or registered trademarks of John Wiley&Sons,Inc.and/or its affiliates in the United States and other countries,and may not be used without written permission.Snowflake and the Snowflake logo are trademarks or registered trademarks of Snowflake,Inc.All other trade

6、marks are the property of their respective owners.John Wiley&Sons,Inc.,is not associated with any product or vendor mentioned in this book.LIMIT OF LIABILITY/DISCLAIMER OF WARRANTY:WHILE THE PUBLISHER AND AUTHORS HAVE USED THEIR BEST EFFORTS IN PREPARING THIS WORK,THEY MAKE NO REPRESENTATIONS OR WAR

7、RANTIES WITH RESPECT TO THE ACCURACY OR COMPLETENESS OF THE CONTENTS OF THIS WORK AND SPECIFICALLY DISCLAIM ALL WARRANTIES,INCLUDING WITHOUT LIMITATION ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.NO WARRANTY MAY BE CREATED OR EXTENDED BY SALES REPRESENTATIVES,WRITTE

8、N SALES MATERIALS OR PROMOTIONAL STATEMENTS FOR THIS WORK.THE FACT THAT AN ORGANIZATION,WEBSITE,OR PRODUCT IS REFERRED TO IN THIS WORK AS A CITATION AND/OR POTENTIAL SOURCE OF FURTHER INFORMATION DOES NOT MEAN THAT THE PUBLISHER AND AUTHORS ENDORSE THE INFORMATION OR SERVICES THE ORGANIZATION,WEBSIT

9、E,OR PRODUCT MAY PROVIDE OR RECOMMENDATIONS IT MAY MAKE.THIS WORK IS SOLD WITH THE UNDERSTANDING THAT THE PUBLISHER IS NOT ENGAGED IN RENDERING PROFESSIONAL SERVICES.THE ADVICE AND STRATEGIES CONTAINED HEREIN MAY NOT BE SUITABLE FOR YOUR SITUATION.YOU SHOULD CONSULT WITH A SPECIALIST WHERE APPROPRIA

10、TE.FURTHER,READERS SHOULD BE AWARE THAT WEBSITES LISTED IN THIS WORK MAY HAVE CHANGED OR DISAPPEARED BETWEEN WHEN THIS WORK WAS WRITTEN AND WHEN IT IS READ.NEITHER THE PUBLISHER NOR AUTHORS SHALL BE LIABLE FOR ANY LOSS OF PROFIT OR ANY OTHER COMMERCIAL DAMAGES,INCLUDING BUT NOT LIMITED TO SPECIAL,IN

11、CIDENTAL,CONSEQUENTIAL,OR OTHER DAMAGES.For general information on our other products and services,or how to create a custom For Dummies book for your business or organization,please contact our Business Development Department in the U.S.at 877-409-4177,contact infodummies.biz,or visit information a

12、bout licensing the For Dummies brand for products or services,contact BrandedRights&LicensesW.ISBN 978-1-394-21162-3(pbk);ISBN 978-1-394-21163-0(ebk)Publishers AcknowledgmentsSome of the people who helped bring this book to market include the following:Development Editor:Nicole ShollyProject Manager

13、:Jen BinghamAcquisitions Editor:Traci MartinEditorial Manager:Rev MengleSales Manager Molly DaughertyContent Refinement Specialist:Tamilmani VaradharajTable of Contents iiiThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Table of

14、ContentsINTRODUCTION.1About This Book.1Icons Used in This Book.2Beyond the Book.2CHAPTER 1:Introducing Cloud Data Warehousing.3Defining the Data Warehouse.4Defining Data Lakes.4Understanding the Cloud Data Platform.5Tracking the Emergence of Modern Cloud Data Warehousing.6Looking at Data Processing

15、Trends.8Adapting to Data Demands.8CHAPTER 2:Standardizing on a Versatile Data Platform.11Supporting Many Languages.12Working with Many Data Formats.12Utilizing Open Table Formats.14Supporting New Architectural Patterns.14Improving Control with a Data Mesh.15Moving Beyond Data Lakes.16CHAPTER 3:Archi

16、tecting a Cloud Data Platform That JustWorks.17Outlining the Primary Architectural Components.17Spanning Multiple Regions and Clouds.18Consolidating Data for Out-of-the-Box Analytics.20Achieving operational efficiency.21Provisioning and managing resources.22CHAPTER 4:Achieving Exceptional Price and

17、Performance.23Utilizing Consumption-Based Pricing.24Maximizing Efficiency with Columnar Storage.24Calculating and Controlling Costs.25Optimizing Performance and TCO.25iv Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,dis

18、tribution,or unauthorized use is strictly prohibited.CHAPTER 5:Bolstering Data Security and Governance.27Exploring the Fundamentals of Database Security.28Eliminating security silos.28Encrypting data by default.28Verifying vendor participation.29Patching,updates,and network monitoring.29Ensuring dat

19、a protection,retention,and redundancy.30Securing marketplace data.30Controlling user logins.30Applying access controls.31Governing How People View,Access,and Interact with Your Data.31Protecting your data.32Classifying and identifying data.32Demanding attestations and compliance certifications.33Mon

20、itoring data quality.33CHAPTER 6:Enabling Data Sharing.35Confronting Technical Challenges.35Sharing without Copying.36Protecting Sensitive Data.37Monetizing Your Data.37CHAPTER 7:Advancing Analytics.39Considering Geospatial Analytics.40Optimizing Search Functions.40Arming Data Analysts with ML.41Dev

21、eloping AI Applications.41Automating Development,Deployment,and Monetization.42CHAPTER 8:Four Steps for Getting Started with Cloud Data Warehousing.43Step 1:Evaluate Your Needs.43Step 2:Migrate or Start Fresh.44Step 3:Calculate TCO.44Step 4:Set Up a Proof of Concept.44Introduction 1These materials a

22、re 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.IntroductionData is infiltrating all types of business processes and reshaping the way companies operate.Regardless of your industry or market,the ability to manage data easily,securely,and efficien

23、tly has become vital for success.For instance,in the realm of marketing,data is animating cus-tomer segmentation and targeted advertising,allowing busi-nesses to craft personalized marketing campaigns based on the moment-to-moment activities of consumers.In transportation,real-time data enables trav

24、elers to optimize routes,and that same data can be aggregated to reduce traffic congestion and improve roadway efficiency.These examples highlight the immense potential of data and the transformative impact it will continue to have for years to come.Forward-thinking organizations rely on powerful,ea

25、sy-to-use,and out-of-the-box cloud data warehouses to put their data to work.The best cloud data warehouses are built on a cloud data platform a unified,global solution not only for data warehous-ing but also for data lakes,data engineering,AI/ML,and data application development.By concurrently powe

26、ring these and other workloads,a cloud data platform enables everyone in the organization to deliver valuable experiences with their data.Delivered as an affordable,usage-based service,a cloud data platform can help your business users become more efficient and allows your IT team to break free from

27、 mundane data adminis-tration tasks.It provides consistent functionality across multi-ple regions and clouds with instant and near-infinite scalability.Multiple business units can securely share governed data without the complications of duplicating or copying data,as well as extend access to partne

28、rs,customers,and other constituents either directly or through a data marketplace.About This BookWelcome to the third edition of Cloud Data Warehousing For Dummies where you discover how your organization can tap into and transform the power of massive amounts of data into valuable business intellig

29、ence.2 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.In this book,you learn how to create an innovative,cost-effective,and versatile cloud data platform that power

30、s not only your data warehouse but also many other data workloads.Additionally,you learn how to extend an existing data warehouse to take advantage of the latest cloud technologies.Icons Used in This BookThroughout this book,the following icons highlight tips,impor-tant points to remember,and more.T

31、ips alert you to easier ways of performing a task or better ways to use cloud data warehousing in your organization.This icon highlights concepts worth remembering as you immerse yourself in the understanding and application of cloud data warehousing.The jargon beneath the jargon,explained.The case

32、studies in this book reveal how organizations applied cloud data warehousing to save money and significantly improve the speed and performance of their data analytics.Beyond the BookIf you like what you read in this book,visit ,where you can find out more about the companys cloud data platform offer

33、ing,sign up for a free Snowflake trial account.CHAPTER 1 Introducing Cloud Data Warehousing 3These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Chapter1IN THIS CHAPTER Understanding data warehouses,data lakes,and cloud data platform

34、s Diving into the modern cloud data warehouses history Exploring trends in data and analytics Keeping up with the shifting demands ofdataIntroducing Cloud Data WarehousingA traditional data warehouse required purchasing,install-ing,and configuring the necessary hardware,software,and infrastructure t

35、o store and analyze data.Cloud data warehousing emerged as an efficient,cost-effective way for organizations to scale analytics without those upfront costs.And,when a cloud data warehouse lives on a well-architected,modern cloud data platform,it not only enables organizations to acceler-ate analytic

36、s but also broadens data management capabilities to include other architectures,like a data lake,and can securely and efficiently run other workloads.To help you understand data warehouses,data lakes,and the modern cloud data platform,this chapter defines each,and briefly shows how the modern cloud

37、data platform came into being.The chapter wraps up with a quick look at trends in data processing and how those trends require the ability to shift and meet new data demands.4 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any disseminati

38、on,distribution,or unauthorized use is strictly prohibited.Defining the Data WarehouseInitially,data warehouses were simplyrelational databases that stored and queried large volumes of structured data.Today,cloud-built and hybrid cloud data warehouses can also incor-porate semi-structured data,such

39、as JavaScript Object Nota-tion(JSON)weblogs,and unstructured data,such as images and audio conversations.This has allowed modern data warehouses to expand beyond mere analytic repositories for internal business operations and include a burgeoning volume of data from mobile apps,online games,Internet

40、 of Things(IoT)devices,social media networks,generative AI systems,and many other sources.A data warehouse is a computer system dedicated to storing and analyzing data to reveal trends,patterns,and correlations that provide information and insight.Traditionally,organizations have used data warehouse

41、s to capture and integrate data collected from internal sources(usually transactional databases),includ-ing marketing,sales,production,finance,and more.However,unlike transactional databases,data warehouses are designed for analytical work.These software environments serve as federated merged reposi

42、tories,collecting and aggregating data from var-ious operational systems for analysis and generating business insights.Defining Data LakesData lakes arose to supplement traditional data warehouses because the relational model cant accommodate the current diversity of data types and their fast-paced

43、acquisition models.While data warehouses are generally designed and modeled for a particular purpose,such as financial reporting,data lakes dont always have a predetermined use case.Their utility becomes clear later,such as when data scientists conduct data exploration for feature engineering and de

44、veloping predictive models.Data warehouses and data lakes are both widely used to store big data but arent interchangeable.A data lake is a vast pool of raw data that is stored in a highly flexible format for future use.A data CHAPTER 1 Introducing Cloud Data Warehousing 5These materials are 2024 Jo

45、hn Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.warehouse is a repository of filtered data that has been prepro-cessed for a specific purpose.We explore these differences further in Chapter2.Understanding the Cloud Data PlatformA cloud data platform is a s

46、ingle,unified network that enables data analysts,data scientists,data engineers,and more to connect their data,applications,and services that are most critical for their business.It allows for workloads like data warehousing,data lake,data engineering,collaboration,AI/ML,application devel-opment,and

47、 more.It makes it easy to share data with a diverse group of users without requiring the technology team to copy that data or establish a new data silo.It upholds centralized data security,data governance,and regulatory compliance policies to ensure that people obtain complete,consistent,and accurat

48、e data when they issue queries and generate reports without violating data privacy mandates.It also can accommodate new architecture patterns such as a data mesh,and integrate open table formats such as Apache Iceberg tables(for more on this,see Chapter2).Consumption-based pricing allows each user a

49、nd workgroup to allocate costs to specific accounts and cost centers with constant visibility into the compute and storage resources they use.Best of all,a modern cloud data platform operates seamlessly across multiple public clouds via one consistent interface,maximizing flexibility and avoiding th

50、e restrictions of a single cloud provider.Cloud data warehousing,which can live as a workload on a mod-ern cloud data platform,emerged from the convergence of three major trends:1)changes in data sources,volume,and variety;2)increased demand for data access and analytics;and 3)technology improvement

51、s that significantly increased the efficiency of data storage,access,and analytics.6 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Tracking the Emergence of Modern

52、 Cloud Data WarehousingTraditional data platforms are designed to leverage a set of finite computing resources,often within the confines of an on-premises data center.Careful capacity planning is required to size each new data warehouse,data lake,data mart(a subset of a data warehouse that focuses o

53、n specific data for a particular purpose),or other data-driven workload.Because organizations dont always know how popular these workloads will become,they have to overpro-vision them deploying more hardware and software resources than they expect to initially need.MARRIOT SIMPLIFIES ITS DATA PLATFO

54、RM AND ACHIEVES LOWER TOTAL COST OF OWNERSHIPMarriot,a Snowflake customer,comprises 32 global brands across 139 countries,with 8,300 hotels offering 15 million hotel rooms,and 100,000 home and villa properties.Prior to using a unified,single cloud data platform,Marriott used a mix of legacy database

55、 technologies that made their stack complex,costly due to expensive upgrades,and difficult to operate.Data engi-neers spent 20 percent of their time on infrastructure issues such as tuning Spark jobs.Simplifying its data platform on Snowflake has enabled Marriott to achieve transparency and control

56、of its data,faster speed to market,improved collaboration and data sharing,a better user experience,and lower TCO.With Snowflake,Marriott has seen a dramatic improvement in perfor-mance and cost savings in comparison to Spark and Hive-based work-loads.Many users from Marriott have commented on their

57、 improved experience with Snowflake,mentioning queries that used to take five hours or time out on Netezza that now take one hour on Snowflake.Data that previously took 48 hours to one week in Hadoop is now available nearly instantly in Snowflake.CHAPTER 1 Introducing Cloud Data Warehousing 7These m

58、aterials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.As analytic applications,data science applications,data engineer-ing pipelines,and many other types of data applications have grown in popularity and importance,many of these legacy data w

59、arehouse platforms have bowed under the strain.Restricted by a linear architecture,they cant run multiple workloads in paral-lel,leading to long wait times for computing resources and the data-driven insights they impart.Many users complain of slow,inefficient queries,scalability issues,and rising l

60、icensing costs as analytic workloads grow.Complicating matters,many data-driven workloads are charac-terized by occasional bursts of activity,such as when the finance team closes the books at the end of the month or when data sci-entists train ML models.Sizing a data warehouse to accommodate peak lo

61、ads is wasteful because the system needs all that capacity for only a small fraction of the time.These issues stem,in part,from antiquated design principles.Older data warehouses use a“shared nothing”architecture that tightly couples storage,compute,and database resources.This type of architecture m

62、akes it difficult to elastically scale the data-base to respond to the escalating needs of many concurrent users and workgroups,as well as to accommodate occasional bursts in query activity.The steady rise of public cloud services has empowered busi-nesses to provision nearly limitless amounts of co

63、mpute and storage capacity.Theoretically,this has allowed traditional data environments to support a larger number of users and workloads.In practice,however,older data warehouse systems were not structured to take advantage of all this power and capacity.While some of these data environments have b

64、een“lifted and shifted”to the cloud,they have continued to operate under the architec-tural limitations of their legacy,on-premises heritage.In many cases,these information systems have been architected to work with a finite set of resources and to use a single type of data,which has led to data pla

65、tform sprawl a data warehouse for structured data,a data lake for semi-structured data types,and a wide variety of local databases and data marts,some in the cloud and others on-premises with each created to solve a unique set of departmental needs.This sprawl forces IT administrators to contend wit

66、h the problem of data silos,which involves reconcil-ing dissimilar architectures and different types of data stored in many different places.8 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized u

67、se is strictly prohibited.Traditional data platforms dont scale well,and having a fixed set of compute and storage resources limits concurrency(the degree to which users can simultaneously access the same data and computing resources).Today,thanks to the nearly infinite resources available in the cl

68、oud,businesses can easily scale com-pute resources to handle an escalating volume of activity.Looking at Data Processing TrendsHistorically,businesses collected data in a well-defined,highly structured format at a reasonably predictable rate and volume.Even as the speed of older technologies advance

69、d,data access and usage were carefully controlled and limited,given the scar-city of computing resources,to ensure acceptable performance for every user.But now,the business world is experiencing a data deluge,with data arising from sources too numerous and varied to list.The velocity and volume of

70、this data can quickly overwhelm a con-ventional data warehouse.In some cases,this can cause analytics applications to hang or even crash due to an overload of users and the workloads they attempt to run.Adapting to Data DemandsIt may be difficult to predict the amount of computing resources needed t

71、o analyze large and growing data sets,especially when an increasing share of this data originates outside your data center.This makes a cloud data platform the natural location for storing and integrating warehouse data.A modern cloud data platform also enables elasticity to scale all your analytic

72、workloads.Organizations and workgroups can acquire computing power for short periods,making projects eas-ier to execute and allowing even small businesses to reap the ben-efits of a powerful data warehouse.To take full advantage of cloud resources,a new architecture is required that separates but lo

73、gically integrates storage,com-pute,and data warehouse services(such as metadata and user management).Chapter3 explains that because each component CHAPTER 1 Introducing Cloud Data Warehousing 9These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictl

74、y prohibited.is separate,they can be expanded and contracted independently,enabling data warehouses to be more responsive and adaptable.Adapting to the exponential increase of data also requires a fresh perspective(see Figure1-1).The conversation must shift from how big an organizations data warehou

75、se should be to whether it can scale cost effectively,without friction,and in the magnitude necessary to handle massive volumes and varieties of data,arriv-ing at increasing velocity.FIGURE1-1:The modern data warehouse must support many types of data,analytic use cases,and applications.SUMMING UP TH

76、E CHALLENGES OF DATA MANAGEMENTThe modern cloud data warehouse arose in response to several evolving data trends,all of which put a strain on legacy architectures:Variety:Data sources are numerous and varied,resulting in more-diverse data structures that must coexist in a single location to enable e

77、xhaustive and affordable analysis.Resource contention:When data storage and computation are physically tied together,analytics problems typically arise if either resource starts to run low.Velocity:Loading data in batches at specific intervals is still com-mon,but many organizations require continuo

78、us data loading(micro batching)and streaming data(instant loading).(continued)10 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.OVERCOMING SCALABILITY ISSUESAutodes

79、k software solves challenges in architecture,engineering,con-struction,product design,manufacturing,media,and entertainment.Autodesks customer 360 Analytics Data Platform(ADP)supports a variety of BI,data science,and customer-facing use cases.Autodesks data lake architecture was operationally burden

80、some to support and cost-prohibitive to scale.Data ingestion workloads relied on large amounts of homegrown code that led to frequent trouble-shooting and unreliable data.Data-access-control limitations pre-sented data governance challenges.Performance issues inhibited Autodesks product teams and bu

81、siness users from accessing timely insights.Lack of trust in ADP caused teams to consider building their own data environments.Near-zero maintenance reduced administrative work and freed up technical staff to focus on increasing analytics.Adding native SQL support and an extensive network of connect

82、ors,drivers,and programming languages simplified data ingestion and transformation.Autodesks reimagined data architecture allows the data platform team to support even more self-service analytics use cases and gain the following benefits:Significantly reduced administration overhead(by 3x)10 x faste

83、r data ingestion and transformation Increased self-service access to analytics powered ML workloads Elasticity:Scaling up a conventional data warehouse to meet todays increasing storage and workload demands,when possible,is expensive,painful,and slow.Diversity:Proprietary data platforms are often co

84、mplex,requiring specialized skills and lots of tuning and configuration.This wors-ens with the growing number of data sources,users,and queries.Collaboration:Sharing data usually requires building data pipe-lines and copying data around,which takes time and resources and often results in delays and

85、negative downstream impacts.(continued)CHAPTER 2 Standardizing on a Versatile Data Platform 11These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Chapter2IN THIS CHAPTER Supporting many languages Working with many data formats Organi

86、zing data files with open table formats Utilizing new architectural patterns Simplifying data management with a data mesh Taking a modern approach to data lakesStandardizing on a Versatile Data PlatformRegardless of your industry or market,the capability to har-ness your data easily and securely in

87、a multitude of ways has become paramount for success.A modern cloud data platform empowers you to consolidate your data,providing unlimited bandwidth for data analysis,data sharing,data engi-neering,application development,and data science initiatives.As a result,your business users become more effi

88、cient and your IT team can break free from mundane data administration tasks,allowing everyone to focus on delivering valuable experiences.Each role has unique data requirements from developers to data architects to operational workers.As a result,a cloud data warehouse must live on a cloud data pla

89、tform that can work with numerous programming languages,be compatible with prevail-ing architectural patterns,and integrate smoothly with a wide variety of data formats.12 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,d

90、istribution,or unauthorized use is strictly prohibited.Supporting Many LanguagesSQL,Python,Scala,Java,JavaScript developers interact with many languages to access data and build data applications,including non-coding languages,natural languages,and conver-sational interfaces,such as generative AI to

91、ols that use program-ming languages behind the scenes.A cloud data warehouse should live on a cloud data platform that works seamlessly with these languages.In addition,business analysts should be able to use ANSI SQL to manipulate all data,including support for joins across data types and databases

92、.Flexible access via SQL and other popular languages makes it easier to build data pipelines,run exploratory analytics,train ML models,and perform other data-intensive tasks.This is the start-ing point for enabling a broad set of business intelligence(BI),reporting,and analytic use cases.Working wit

93、h Many Data FormatsTraditional data warehouses are optimized for storing relational data in predefined tables.However,todays data warehouses must accommodate many other data types and file formats,including raw and streaming data from weblogs,equipment sensors,social media networks,and other sources

94、 that dont conform to a rigid tabular structure.Web data may be stored as JSON files.Spreadsheets may occupy comma-separated value(CSV)formats or tab-delimited text files.And data interchanged among multiple applications may be defined in extensible markup language(XML),complete with tags and other

95、coding that identify distinct entities within the data.A cloud data platform should natively support popular semi-structured data formats,including the following:JSON,a lightweight,plain-text,data-interchange format based on a subset of the JavaScript Programming Language.JSON data can be produced b

96、y any application.Apache Avro,an open-source data serialization and Remote Procedure Call(RPC)framework originally developed for use with Apache Hadoop.Avro utilizes schemas defined in JSON CHAPTER 2 Standardizing on a Versatile Data Platform 13These materials are 2024 John Wiley&Sons,Inc.Any dissem

97、ination,distribution,or unauthorized use is strictly prohibited.to produce serialized data in a compact binary format.The serialized data can be sent to any destination(that is,application or program)where it can be easily deserialized because the schema is included in the data.Apache ORC(Optimized

98、Row Columnar),a columnar format used to speed up Apache Hive queries.ORC was designed for efficient compression in Hadoop and improved performance of Hive for reading,writing,and processing data.Apache Parquet,a compressed,efficient columnar data representation designed for projects in the Hadoop ec

99、osystem.This file format supports complex nested data structures and uses Dremel record shredding and assembly algorithms.XML,a markup language that defines a set of rules for encoding documents.XML was originally based on standard generalized markup language(SGML),another markup language developed

100、for standardizing the structure and elements that comprise a document.THE THREE BASIC DATA TYPESMost data can be grouped into three basic categories:Structured data(customer names,dates,addresses,order history,product information,and so forth)is generally maintained in a neat,predictable,and orderly

101、 form,such as the tables in a relational database or the rows and columns in a spreadsheet.Semi-structured data(web data,spreadsheet data,XML data)doesnt conform to traditional structured data standards but contains tags or other types of markups that identify distinct entities within the data.Unstr

102、uctured data(audio,video,images,PDFs,and other docu-ments)doesnt conform to a predefined data model or is not organized in a predefined manner.Unstructured information may contain textual information,such as dates,numbers,and facts that are not logically organized into the fields of a database or se

103、mantically tagged document.14 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.A complete cloud data platform can store diverse types of data in their native formats

104、without creating data silos or imposing unique schemas to access data.You dont have to develop or maintain sep-arate storage environments for structured,semi-structured,and unstructured data.It is easy to load,combine,and analyze all data through a single interface while maintaining transactional in

105、tegrity.Utilizing Open Table FormatsIn addition to standardizing on a cloud data platform that sup-ports JSON,Avro,Parquet,and XML file formats,make sure it works with your desired table format,whether proprietary or open source.Apache Iceberg is a widely popular open table for-mat with a large ecos

106、ystem of contributors,vendors,and users,ensuring you dont lock your data into any single vendor.Iceberg adds a SQL-like table structure to the unstructured and semi-structured data stored in files and documents.You can store Iceberg metadata and data files in your object storage and query them in-pl

107、ace.This allows computing engines,such as Spark,Trino,PrestoDB,Apache Flink,Hive,and Snowflake,to easily manage and inspect the data.Open table data formats have tremendous momentum from the commercial and open-source communities.Will your data plat-form support them if needed?Even when most of your

108、 data is maintained in a centralized data warehouse repository,its still possible to accommodate data in external tables(read-only tables that can be used for query and join operations)and materialized views(database objects that contain the precomputed results of a query).This architecture enables

109、seamless,high-performance analytics and governance,even when the data arises from more than one location.Supporting New Architectural PatternsOne reason technology projects fail is because the stakeholders fail to look ahead.Dont just look at your current state;consider how your business may evolve

110、in the future.Historically,companies have invested in special-purpose tech-nologies and data platforms,and its a huge effort to migrate CHAPTER 2 Standardizing on a Versatile Data Platform 15These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly p

111、rohibited.them to more open and versatile formats.Such migrations can become a massive undertaking,sort of like trying to copy a life-times worth of family movies from an analog VHS format to a digital format like MP4.With new types of data,you may encounter new architectural pat-terns that you didn

112、t predict.For instance,you may want a data warehouse to be transformed into a hybrid pattern that merges the strengths of data warehouses and data lakes.Additionally,domain-specific data marts could evolve into a more streamlined and regulated data mesh.A modern data platform supporting the data war

113、ehouse workload must be able to accommodate these patterns and easily adapt to your evolving business needs,as shown in Figure2-1.By making rigid demands about how to structure your data,you may unwittingly determine how to structure your business.The right data platform will allow you to do new thi

114、ngs in familiar ways through a familiar interface.This maximizes flexibility as your business evolves.Improving Control with a Data MeshA data mesh simplifies the process of managing massive data architectures by breaking them down into smaller functional domains,each overseen by a dedicated team.Th

115、ese domain teams are responsible for crucial tasks,such as building and maintain-ing data pipelines,implementing governance policies,upholding FIGURE2-1:A versatile data platform powers a full spectrum of use cases,whether data is stored inside a data warehouse or in external tables.16 Cloud Data Wa

116、rehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.data privacy mandates,and ensuring data quality.Rather than creating silos,a data mesh breaks them down it distributes data respon

117、sibilities across different teams or domains while maintaining data discoverability and accessibility.This architectural pattern confirms that the teams working with the data have in-depth knowledge and expertise,fostering greater ownership and accountability as each data set aligns with the overall

118、 needs of the business.By distributing data responsibili-ties across the organization,a data mesh fosters a culture of data democratization and encourages cross-functional collaboration.When anchored by a modern cloud data platform,a data mesh can incorporate many types of data and file formats and

119、accom-modate external data sources,different workloads,and multiple clouds.Moving Beyond Data LakesData lakes are designed to store huge quantities of raw data in their native formats in a single repository.However,business users often find accessing and securing this vast pool of data difficult,and

120、 many organizations have a hard time finding,recruiting,and retaining the specialized IT experts needed to access the data and prepare it for downstream analytics and data science use cases.Additionally,most of todays data lakes cant effectively organize all of an orga-nizations data,which may origi

121、nate from dozens of data streams and data silos that must be loaded at different frequencies,such as once per day,once per hour,or via a continuous data stream.In response,hybrid platforms have emerged that combine the best attributes of data warehouses and data lakes into a single platform.These so

122、lutions have become the foundation for the modern data lake:a cloud-built repository where structured,semi-structured,and unstructured data can be staged in their raw forms.Anchored by a cloud data platform,these newer data lakes provide a harmonious environment that blends many different data manag

123、ement and data storage options,including a cloud analytics layer,a data warehouse,and a cloud-based object store.With the right software architecture,these data lakes provide nearly unlimited capacity and scalability for the storage and com-puting power you need.They make it easy to derive insights,

124、obtain value from your data,and reveal new business opportunities.CHAPTER 3 Architecting a Cloud Data Platform That Just Works 17These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Chapter3IN THIS CHAPTER Defining essential architect

125、ural attributes Enabling data workloads across regions and clouds Organizing your data for out-of-the-box analyticsArchitecting a Cloud Data Platform That JustWorksCreating an effective cloud data warehouse isnt just a matter of repurposing yesterdays on-premises technologies or moving existing anal

126、ytic applications and databases from your data center to a cloud vendors infrastructure.Properly leveraging the power and scale of the cloud requires a new mindset,a new set of management principles,and new cloud-built capabilities.Outlining the Primary Architectural ComponentsTo best satisfy the re

127、quirements of diverse and ever-escalating data workloads,a modern cloud data platform should be built on a multi-cluster,shared data architecture,in which separate compute,storage,and services can be scaled independently to leverage all the resources of the cloud.18 Cloud Data Warehousing For Dummie

128、s,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.A modern cloud data warehouse includes a central persisted data repository that is accessible from all compute nodes.Like a shared-nothing architectur

129、e,it processes queries using MPP(massively parallel processing)compute clusters.This architecture allows maximum scalability,because each node in the cluster stores a portion of the entire data set locally.A near-limitless number of users can query the same data concurrently without degrading perfor

130、mance,even while other workloads are executing simultaneously,such as running a batch processing pipeline,training a machine learning model,or exploring data with ad hoc queries.A multi-cluster,shared data architecture includes four layers that are logically integrated yet scale inde-pendently from

131、one another:The storage layer holds your data,tables,and query results.This scalable repository should handle structured,semi-structured,and unstructured data and span multiple regions within a single cloud and across major public clouds.The compute layer processes enormous quantities of data with m

132、aximum speed and efficiency.You can easily specify the number of dedicated clusters you want to use for each workload(thus eliminating contention for resources)and have the option to let the service scale automatically.The services layer coordinates transactions across all workloads and enables conc

133、urrent data loading and querying activities,enforcing security,propagating metadata,optimizing queries,and performing other important data management tasks.When each workload has its own dedicated compute resources,operations can run simultane-ously and perform as needed.The cross cloud and global l

134、ayer globally connects data and applications across regions and clouds,securely,through a single,consistent experience,and is described further below.Spanning Multiple Regions and CloudsMany companies store data in multiple clouds and regions,neces-sitating a cohesive cross-cloud strategy that can a

135、ttain business continuity,resilience,and collaboration no matter where data is located.A recent survey,part of Snowflakes Data Trends Report,CHAPTER 3 Architecting a Cloud Data Platform That Just Works 19These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use

136、is strictly prohibited.examined data usage patterns at 7,800 organizationsall Snow-flake customers.According to the survey,the number of organi-zations operating across the three leading public cloud providers(Amazon Web Services,Microsoft Azure,and Google Cloud)grew 207%during the 12 months ending

137、January 2023.These companies need data warehouses that can store and man-age data consistently across many different geographic regions and clouds.However,when working with multiple cloud provid-ers,how do you ensure that the same security configurations,administrative techniques,analytics practices

138、,and data pipelines apply to all your cloud providers?For example,will you have to resolve differences in audit trails and event logs or apply unique tuning and scaling techniques on each cloud?Will your security experts have to deal with varying sets of rules or work with multi-ple key management s

139、ystems to encrypt data?Will data engineers have to create unique pipelines?A cross-cloud data platform enables data administrators to apply consistent policies to all data in all areas.This makes it easier to keep up with changing regulations,apply regional locality con-trols,and take advantage of w

140、hichever public cloud services best match your evolving business strategy.Once you have this type of technology layer in place,it quickly becomes a competitive advantage,allowing you to achieve results faster,comply with data governance procedures more easily,and maintain uninterrupted operations th

141、rough seamless data repli-cation(see Figure3-1).A cross-cloud data warehouse provides a consistent layer of ser-vices across regions of a single public cloud provider and between major cloud providers,with the following emphases:Continuity:The data warehouse must offer inherent resiliency to elimina

142、te disruptions,comply with changing regulations,and simplify data migrations among different vendor clouds.Governance:Your data warehouse should offer flexible policies,tags,and lineage capabilities that follow the data,ensuring consistent enforcement across users,workloads,clouds,and regions.20 Clo

143、ud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Collaboration:A cloud data warehouse should allow workers to instantly discover,access,and share data,services,and appli

144、cations across clouds and regions,without requiring complex integration technology such as file transfer protocol(FTP)or extract,transform,and load(ETL)procedures.Consolidating Data for Out-of-the-Box AnalyticsOne of the fundamental principles of this book is to encour-age all stakeholders in your o

145、rganization including line-of-business managers,data analysts,data engineers,data scientists,application developers,and frontline workers to actively lev-erage the same single source of data.This ensures consistent out-comes and accelerates time to insight by reducing the time spent wrangling data.I

146、n practice,rallying the enterprise around a single source of truth is rarely a seamless process,mainly due to how corporate infor-mation systems have been designed and implemented over the last several decades.Whether on-premises or in the cloud,each FIGURE3-1:A modern cloud data platform should sea

147、mlessly operate across multiple clouds and apply a consistent set of data management services to many types of data workloads.CHAPTER 3 Architecting a Cloud Data Platform That Just Works 21These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly pro

148、hibited.production application creates its own data silo.For example,mar-keting data resides in a marketing automation system,sales data in a customer relationship management(CRM)system,finance data in an enterprise resource planning(ERP)system,and inven-tory data in a warehouse management system,am

149、ong others.These disparities are carried over to the analytic databases derived from these production systems.Operational reporting may be the province of a data warehouse,while departmental analytics relies on data marts and data mining,or exploration requires a data lake.Sharing data among these s

150、ystems may need specialized data pipelines powered by complex ETL procedures.The situa-tion has become even more complex with the rise of thousands of software-as-a-service(SaaS)tools and mobile apps,each with its own unique sources of data.Achieving operational efficiencyAll cloud data warehouse ve

151、ndors offer some degree of automa-tion,but its crucial to delve deeper to determine the level of auto-mation they truly provide.Ideally,your data warehouse platform should be seamlessly managed,updated,secured,governed,and administered without requiring extra effort from your IT team.When it comes t

152、o software updates,you should automatically receive the latest functionality without enduring a lengthy,manual upgrade process.You shouldnt have to worry about planning for updates,experiencing downtime,or making modi-fications to your installation.The cloud data warehouse provider handles all admin

153、istrative tasks related to storage,encryption,table structure,query optimization,and metadata management behind the scenes,effectively eliminating the need for manual administration.To determine how much work will be necessary,ask your cloud data warehouse vendor these questions:Do we have to optimi

154、ze resource usage or manually scale the system,such as requesting an additional cluster when more compute power is required?Does the provider automatically apply software updates,such as security patches,as soon as those updates are available?Or does it merely manage the underlying infrastructure an

155、d require us to keep the software platform up to date?22 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Does the service automatically encrypt all our data at rest

156、and in motion with industry-standard encryption,or do we have to set up and apply encryption to the data manually?Does the encryption system hinder query performance?Does the service scale up and out instantaneously and elastically and then release extra compute or storage resources when theyre no l

157、onger in use?Or do we have to handle these tasks manually?Does the cloud provider automatically replicate your data to ensure business continuity across regions?After cross-regional replication is established,do we have to set up change data capture(CDC)procedures to keep multiple databases in sync,

158、or does the vendor handle that for us?Do we need to partition data,tune SQL queries,and optimize performance,or does the platform handle this automatically?Provisioning and managing resourcesYour cloud data warehouse should allow you to right-size the computing infrastructure to match the resource n

159、eeds of each workload.For example,if youre running a data pipeline with low compute requirements,you can match a small cluster to that workload rather than incur the cost of an overprovisioned cluster.If you need to test new machine learning modules or run advanced analytics,you can utilize a large

160、cluster.The best cloud data platforms have an elastic performance engine that permits variable concurrency without resource contention,tuning,or the need to manage the system.The data platform sup-ports any number of users,quantity of jobs,or volume data with reliable multi-cluster resource isolatio

161、n.This gives you fine-grained scalability for each workload while minimizing usage costs.With some cloud data platforms,IT is responsible for provision-ing and managing new resources.In other platforms,all the infrastructure is provisioned and managed behind the scenes;you simply run your queries or

162、 processing jobs and the cloud data platform does the rest,abstracting technical complexities and automating system management activities in the background.CHAPTER 4 Achieving Exceptional Price and Performance 23These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthori

163、zed use is strictly prohibited.Chapter4IN THIS CHAPTER Ensuring value through consumption-based pricing Using columnar storage to maximize efficiency Looking at the right metrics to keep costs down Improving performance and total cost of ownership(TCO)by fine-tuning compute resourcesAchieving Except

164、ional Price and PerformanceFast analytical performance is crucial for data-informed decision-making.However,the more data you ingest and process in your data warehouse,the more cloud resources you consume,which can have a direct impact on costs.There are three essential aspects to cost optimization

165、in a cloud data warehouse:Visibility:Users can fully understand their spending and attribute it accurately to designated cost centers.Control:Administrators can set limits and take corrective actions to govern resource use.Optimization:Companies can identify inefficient spending and reallocate funds

166、 for more impact.This chapter dives into these aspects and describes how to achieve cutting-edge performance while simultaneously monitoring data warehouse costs and optimizing resource use.24 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,In

167、c.Any dissemination,distribution,or unauthorized use is strictly prohibited.Utilizing Consumption-Based PricingMake sure that the pricing model for your cloud data warehouse matches the value you obtain from it.Paying for a set amount of storage and computing power,commonly known as subscription-bas

168、ed pricing,can incur significant yearly costs and typically requires regular management.To ensure that you dont pay for more capac-ity than you need,your cloud data platform should offer usage-based pricing.Usage-based pricing allows you to choose how data users at your organization consume resource

169、s.Some cloud data platforms allow you to pay for usage per second with a one-minute minimum,increasing control over costs.Maximizing Efficiency with Columnar StorageData uploaded into the data warehouse should be reorganized into a compressed columnar format.Because columnar databases use less memor

170、y to output data,more data can be stored,speeding up queries.Examine the terms of your usage agreement:Expect to pay only for storage you use,not for excess or reserved storage capacity.You also shouldnt pay to clone databases within your data ware-house for development and testing activities.You wa

171、nt to be able to reference not copy your data multiple times and therefore not have to pay extra for storage.Chapter6 covers data sharing and collaboration in detail.Compute resources are more expensive than storage resources,so your data warehouse service should allow you to scale each resource ind

172、ependently and make it easy to spin up exactly the compute resources you need under a usage-based pricing model.The vendor should bill you only for the resources you use down to the second and automatically suspend compute resources when you stop using them.Its useful to receive those charges in an

173、all-inclusive bill with no hidden costs or fees.CHAPTER 4 Achieving Exceptional Price and Performance 25These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Calculating and Controlling CostsAs enterprises migrate IT workloads to the c

174、loud,theyre transi-tioning from a world of scarcity to a world of abundance marked by nearly limitless data storage resources and nonstop data pro-cessing capacity.Its important to control costs and rein in exces-sive consumption.The cost of using a cloud data warehouse is typically based on three i

175、nterrelated metrics:data transfer volume,data storage consumption,and compute resources.A cloud data platform sep-arates these three services to give administrators complete con-trol over data warehouse usage.Your data platform must make it easy to track the consumption of all cloud services.This in

176、cludes built-in resource monitoring and management features that provide transparency into usage and billing,ideally with granular chargeback capabilities to tie usage to individual budgets,departments,and workgroups.Data warehouse administrators can set guardrails to ensure that no individual or wo

177、rkgroup spends more than expected.For example,they can set time-out periods for each type of workload along with auto suspend and auto resume features to automatically start and stop resource accounting when the platform isnt processing data.They may also set limits at a granular level,such as deter

178、mining how long a query can run before its terminated,which helps to avoid unexpected costs associated with runaway queries.Optimizing Performance and TCOFine-tuning the compute resources provided by a cloud data warehouse can improve the performance of a query or set of que-ries.Administrators can

179、resize the environment whenever neces-sary,even while running production workloads in tandem.They can also start or stop the entire data warehouse at any time to optimize overall price and performance.26 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wi

180、ley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Look for a cloud data warehouse solution that automatically optimizes performance and eliminates administrative effort to incorporate new resources.Whether its search optimization(SO)capabilities,more efficient st

181、orage compression techniques,or reduced compilation time for SQL queries,you shouldnt have to do anything to gain access to new features or the latest capabilities.Thats the beauty of subscribing to cloud services from a reputa-ble data platform provider:New functionality appears instantly,without t

182、edious upgrade cycles.Regularly released platform opti-mizations and software updates continuously improve perfor-mance,often while simultaneously lowering costs.AUTOMATION DRIVES INNOVATIONVeradigm,a Snowflake customer,is a technology company that deliv-ers care and financial solutions to healthcar

183、e providers.To provide stakeholders with actionable data and insights,the company ingests and analyzes large amounts of data on electronic health records,disease registry data,and claims data.Unfortunately,with Veradigms legacy data warehouse environment,onboarding new data sources took up to nine m

184、onths.Furthermore,data processing limitations made it difficult to join tables that con-tained medication,laboratory,and other healthcare data.Realizing the need for a more modern data environment,Veradigm subscribed to a cloud data platform with a multi-cluster shared data architecture.The platform

185、 automatically scales storage and compute resources,eliminating performance issues,lowering costs,and offering more granular control.For example,one group at Veradigm reduced its resource consumption from$40,000 per month to less than$4,000 per month,even though team members were processing twice as

186、 much data.With a fully managed infrastructure and near-zero maintenance,Veradigms cloud data platform has enabled the company to support additional data use cases such as a data lake without increasing head-count and easily meets its service level agreements(SLAs)for each workload.All data resides

187、in one multipurpose repository,which is much simpler than wrangling multiple disparate data sets.CHAPTER 5 Bolstering Data Security and Governance 27These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Chapter5IN THIS CHAPTER Securing

188、 data through encryption,user login controls,access controls,and more Applying governance policies to protect data and maintain the quality of yourdataBolstering Data Security and GovernanceIn recent years,there has been a spike in the proliferation of data generated and collected by organizations.W

189、ith data from third-party sources becoming more common such as data from SaaS apps,popular application clouds,data marketplaces,data exchanges,and more data security,data privacy,data governance,and regulatory compliance have become much more complicated.Organizations need to understand the source o

190、f common threats and take a hard look at who might be trying to misuse,breach,or attack their database management systems.For example,trade secrets may be valuable to industry competi-tors,while energy grid information is a target for political saboteurs.Understanding these realities is the starting

191、 point for setting up comprehensive security,governance,and compliance policies that can be consistently enforced across your entire data estate.28 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthori

192、zed use is strictly prohibited.Exploring the Fundamentals of Database SecuritySecuring your data and complying with pertinent regulations is fundamental to the architecture,implementation,and opera-tion of a cloud data warehouse service.All aspects of the service must be centered on protecting your

193、data as part of a multilayered strategy that considers both current and evolving security threats.Your security strategy should address external interfaces,access control,data storage,and physical infrastructure in conjunction with comprehensive network monitoring,alerts,and verifiable cybersecurity

194、 practices.Eliminating security silosSome organizations enforce security and governance policies by creating unique data silos and then limiting access to each silo based on account,region,role,and other variables.This approach complicates data governance.Rather than creating unique data silos with

195、unique data protection policies,establish universal,application-level controls that apply to one centralized repository.Just as it is important to eliminate data silos,a good security strategy seeks to eliminate identity silos as well.Encrypting data by defaultEncrypting data means applying an encry

196、ption algorithm to translate the clear text into cipher text.All warehouse data should be encrypted by default using the latest security standards and best practices.Encrypt data from the time it leaves your premises,through the internet,and into the warehouse:when its stored on disk,moved into a st

197、aging location,placed within a database object,and cached within a virtual data warehouse.Query results should also be encrypted.The vendor must protect the decryption keys that decode your data.The best service providers employ AES 256-bit encryption with a hierarchical key model.This method encryp

198、ts the encryp-tion keys and instigates key rotation that limits the time during which any single key can be used.CHAPTER 5 Bolstering Data Security and Governance 29These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Data encryption

199、and key management must be always on and entirely transparent.Having the option to supply your own encryption keys is important so that you can disconnect the cloud provider from your data if necessary.Verifying vendor participationSome cloud data warehouse vendors automate only rudimentary security

200、 capabilities,leaving many aspects of data encryption,access control,and security monitoring to the customer.Other vendors handle these tasks for you.Before standardizing on a cloud data platform for your data warehouse deployment,ask the vendor these questions:Does the service enforce essential sec

201、urity attributes by default,such as encryption,threat detection,and incident response?Does it follow Center for Internet Security(CIS)Benchmarks for configuring IT systems,software,networks,and cloud infrastructure?Are security controls global,comprehensive,and easy to configure?Does the vendor subs

202、cribe to a shared responsibility model,and is it clear whos responsible for which aspects of security?Can we bring our own identity and establish SSO(single sign-on)?Can our data administrators set granular access controls(such as column-and row-level restrictions),along with role-based access to da

203、tabase tables?Is security applied not only to the central data repository but to external tables as well?Does the vendor regularly perform compliance audits and have the necessary security attestations to show?Patching,updates,and network monitoringSoftware patches and security updates must be insta

204、lled on all pertinent software components as soon as those updates are available.The vendor should deploy periodic security testing 30 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is st

205、rictly prohibited.(also known as penetration testing)by an independent security firm to proactively check for vulnerabilities.As an added protection,file integrity monitoring(FIM)tools ensure that critical system files arent tampered with,and IP address allowed lists enable you to restrict access to

206、 the data warehouse to only trusted networks.Security“events,”generated by cybersecurity monitoring systems that watch over the network,need to be automatically logged in a tamper-resistant security information and event management(SIEM)system.Automatic alerts should be sent to security per-sonnel w

207、hen suspicious activity is detected.Ensuring data protection,retention,and redundancyIn case of a mishap,you should be able to instantly restore or query previous versions of your data in a table or database within a specified retention period,as governed by your service-level agreement(SLA)with the

208、 cloud data warehouse provider.A com-plete data-retention strategy goes beyond duplicating data within the same cloud region or zone;it replicates that data among mul-tiple availability zones for geographic redundancy.Optionally,automatic failover to these other zones can ensure continuous business

209、operations.Securing marketplace dataA growing number of organizations leverage a data warehouse to develop data applications not only for internal use but also for external use via a data marketplace.Sharing data through mar-ketplace apps necessitates another level of security.Data pro-viders must b

210、e able to guard,monitor,and review application submissions to vet potential users.In some cases,data providers create data clean rooms that enforce designated governance policies.These sanitized data sets can be confidently shared with partners and other external constituents without exposing sensit

211、ive information.Controlling user loginsFor maximum convenience and security,a cloud data ware-house will allow you to apply your chosen SSO and identity access CHAPTER 5 Bolstering Data Security and Governance 31These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthori

212、zed use is strictly prohibited.management(IAM)procedures.The data warehouse should also permit you to apply multifactor authentication(MFA)at the account level.This permits you to require some or all users to pass through a secondary level of verification such as entering a one-time security code se

213、nt to the users mobile phone.SSO procedures and federated authentication make it easier for people to log in to the data warehouse service directly from other sanctioned applications.Federated authentication centralizes iden-tity management and access control procedures,making it easier for data war

214、ehouse stakeholders to manage user access privileges.Applying access controlsTo protect sensitive data,a cloud data warehouse service must authorize users,authenticate credentials,and grant people access only to the data theyre authorized to see.Role-based access control(RBAC)policies need to be app

215、lied to all database objects,includ-ing tables,schemas,and virtual extensions to the data warehouse.Ideally,data administrators can apply granular access controls down to the rows and columns of database tables.For example,this type of control could be used to permit users to see basic employee data

216、 but not Social Security numbers,salaries,and other sensitive information.Governing How People View,Access,and Interact with Your DataGovernance policies establish rules and procedures to control the ownership and accessibility of your data.Applying global,univer-sal data governance policies allows

217、you to scale your data estate with confidence.For example,interaction controls,like secure views,secure joins,and secure user-defined functions(UDFs),are applied as people interact with the data:Secure views give data custodians control over data access,preventing security breaches.For instance,cust

218、omers can view specific rows of data from a table that excludes rows pertaining to other customers.32 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Secure joins es

219、tablish linkages without revealing personally identifiable information(PII).It allows discreet connections to people,devices,cookies,or other identifiers.Secure UDFs let users analyze fine-grained data while protecting raw data from being viewed or exported by other parties.Protecting your dataOrgan

220、izations concerned about safeguarding sensitive data can control access at a more granular level.Common data protection methods include the following:Row access policies allow users to see only information relevant to them.For example,sales reps may only access customer data for their own accounts w

221、hile regional managers can access all customer data within their regions.Dynamic data masking selectively conceals data during queries.This technology allows you to store PII and still perform robust analytics on the data without exposing it to unauthorized users.External tokenization transforms dat

222、a into an unrecogniz-able string of characters with no meaningful value in case of a system breach.The data can be dynamically detokenized at query runtime.Classifying and identifying dataClassification and identification policies help you avoid data pri-vacy leaks and compliance breaches by trackin

223、g the types of data in use,its lineage,and how it changes.For example,you can use object tagging to control access to confidential and sensitive infor-mation such as salary amounts and Social Security numbers.Traceability tools let users track data wherever it resides,ensuring continuous protections

224、 and enabling data deletion when neces-sary(including the“right to be forgotten”).Data lineage tools,whether embedded in the cloud data platform or provided as additional services,help you understand how data flows through your data-processing systems.This knowledge CHAPTER 5 Bolstering Data Securit

225、y and Governance 33These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.assists compliance officers in tracing the usage of sensitive data,including its sources,destinations,and any transformations along the way.Demanding attestations

226、 and compliance certificationsCompliance isnt just about robust cybersecurity practices.Its also about ensuring that your data warehouse provider can prove it has the required security procedures in place.Industry-stan-dard attestation reports that verify cloud vendors use appropri-ate security cont

227、rols.For example,a cloud data warehouse vendor needs to demonstrate that it adequately monitors and responds to threats and security incidents and has established sufficient inci-dent response procedures.In addition to industry-standard technology certifications,such as ISO/IEC 27001 and SOC 1/SOC 2

228、 Type II,youll want to verify that your data warehouse provider complies with all applicable government and industry regulations.Depending on your busi-ness,this could include the following:Payment Card Industry Data Security Standards(PCI-DSS)GxP data integrity requirements HIPAA/HITRUST privacy co

229、ntrols ISO/IEC 27001 security management provisions International Traffic in Arms Regulations(ITAR)FedRAMP certificationsAsk your providers to supply complete attestation reports for each pertinent standard.Monitoring data qualityData governance requires rigorous oversight to maintain the quality of

230、 the data your company uses internally and shares with external constituents.Bad data can lead to missed or poor business decisions,loss of revenue,and increased costs.Data stewards charged with overseeing data quality must be empowered to proactively uncover anomalies in the data,such as when data

231、is corrupt,inaccurate,or not being refreshed often enough to be 34 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.relevant.The best data platforms include out-of-th

232、e-box sys-tem metrics for the most common types of data quality issues,and make it easy to define,measure,and monitor data quality via integrated,cloud-native facilities(see Figure5-1).Establishing comprehensive security and governance policies is not only about reducing risk but also about increasi

233、ng productiv-ity.If your data platform lacks an integrated set of applications for data custodians,data stewards,compliance officers,and other experts,youll have to cobble together these capabilities from third-party tools.At best,this scattered approach will make it difficult to enforce organizatio

234、n-wide policies.At worst,it will introduce delays or even cause users to mistrust the data,lead-ing to poor decision-making,a lack of a data-driven culture,and inefficiency.As you provide access to your users,pay attention to these tenets of data governance:Know your data:Classify data,tag sensitive

235、 data,and audit data usage Protect your data:Secure sensitive and regulated data with granular access policies Connect your ecosystem:Seamlessly extend your data governance policies as you share data,internally and externally,across regions and clouds.FIGURE5-1:A complete cloud data platform empower

236、s data stewards to enforce data quality via cloud-native management facilities.CHAPTER 6 Enabling Data Sharing 35These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Chapter6IN THIS CHAPTER Recognizing and overcoming technology limita

237、tions Sharing data without copying or duplication Extending security and governance policies to shared data Monetizing data and data services via a data marketplaceEnabling Data SharingData sharing is the act of providing access to data both within an enterprise and between enterprises.The organizat

238、ion that makes its data available,or shares its data,is a data provider.The organization that wants to use the shared data is a data consumer.Any organization can be a data provider,a data consumer,or both.Theres an abundance of potential value to unlock from the worlds burgeoning data sources.Until

239、 recently,however,no technology existed for sharing data without a significant amount of risk,cost,headache,and delay.Confronting Technical ChallengesTraditional data-sharing methods,such as File Transfer Protocol(FTP),application programming interfaces(APIs),and email,require you to make a copy of

240、the shared data and send it to your data consumers.These cumbersome,costly,and risky methods produce static data that quickly becomes dated and must be refreshed with more current versions,requiring constant data movement and management via data pipelines,and causing a loss 36 Cloud Data Warehousing

241、 For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.of data version control.These complexities,coupled with data-base inconsistencies,authenticity headaches,and the difficulty of sharing larg

242、e volumes of data add up to frustrating,expensive,and time-consuming data exchange processes.Look for a cloud data platform that allows you to accomplish the following:Share data easily and securely across clouds,companies,teams,departments,functions,and business units Easily set up security and gov

243、ernance with built-in permis-sions and roles for ease of administration Share data,views,and dashboards to permit collaborative decision-making through a single,consistent user interface Deliver direct access to live,ready-to-query data across clouds and regions with on-demand fulfillment and no pro

244、grammatic APIs,FTP transfers,or ETL procedures Safely share highly sensitive or regulated data without exposing it to unauthorized users by applying privacy-enhancing technologies and cross-cloud data clean roomsSharing without CopyingA cloud data platform is ideal for a data-sharing service because

245、 it enables authorized members of a cloud ecosystem to tap into live,read-only versions of the data.Organizations can easily share and receive slices of data in a secure and governed way.This method doesnt require data movement,extract,transform,load(ETL)technology,or constant updates to keep data c

246、urrent.Theres no need to transfer data via FTP or to configure APIs to link applica-tions.Because data is shared rather than copied,no additional cloud storage is required.With this superior architecture,data providers can easily and securely publish data for instant dis-covery,query,and enrichment

247、by data consumers,as shown in Figure6-1.CHAPTER 6 Enabling Data Sharing 37These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Protecting Sensitive DataIn some cases,portions of a data warehouse are subject to strict security and conf

248、identiality policies.Before you can share these parts of the data set,you may need to mask or anonymize certain fields,rows,or columns.This allows people to analyze the data without seeing the sensitive data elements.Choose a cloud data platform that allows data providers to easily control access to

249、 individual database tables with granular protec-tions policies and privacy-enhancing technologies.All the per-tinent data security and governance capabilities should apply to your data-sharing architecture(for more on this,see Chapter5).For example,controlling who can view and analyze sensitive or

250、regulated data should be easy.Furthermore,you need to be able to share tables without exposing designated elements,either through privacy-enhancing technologies,such as aggregation and projection constraints,or data clean rooms.Monetizing Your DataModern data-sharing technology sets the stage for co

251、llaborat-ing and monetizing data via marketplaces online communities that facilitate the purchase and sale of data and data services.For example,a telecommunications company can sell location data FIGURE6-1:Identifying the attributes of modern data sharing.38 Cloud Data Warehousing For Dummies,3rd S

252、nowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.to help retailers target consumers with ads.Consumer packaged goods companies can share purchasing data with online advertis-ers or directly with customers.I

253、n addition to monetizing data,a marketplace allows you to monetize business logic,such as user-defined functions(UDFs),as well as applications.If sharing data and applications through a marketplace is impor-tant to you,opt for a cloud data platform that has a thriving marketplace associated with it.

254、Some platforms make it easy to discover third-party data,data services,and applications from hundreds,or even thousands,of providers,and can market and deliver your data products and services(see Figure6-2).Marketplace customers can use cloud credits and budgets to purchase data and data services.Su

255、ch platforms may also offer built-in facilities to meter application usage and handle the asso-ciated billing.These capabilities allow data providers to focus on supplying value-added data services rather than getting caught up in administrative chores.FIGURE6-2:A cloud data platform enables you to

256、securely leverage your data warehouse to share and collaborate with your data,for every scenario.CHAPTER 7 Advancing Analytics 39These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Chapter7IN THIS CHAPTER Accommodating geospatial ana

257、lytics Optimizing search activities Exploring the benefits of ML-powered functions Developing AI applications Understanding the importance of automationAdvancing AnalyticsBusiness intelligence(BI)is no longer merely the domain of executives,professional analysts,and data scientists.An effective clou

258、d data platform that supports data warehouse workloads establishes not only a common repository for all types of data and analytics but also empowers diverse teams to collab-orate and easily manage data.Popular analytic methods include the following:Ad hoc analytics allow business users to answer di

259、screte business questions iteratively,such as tracking monthly sales or reviewing on-hand product inventory.Dynamic elasticity and dedicated resources for each workload power these queries without slowing down other workloads.Event-driven analytics constantly incorporate new data to update reports a

260、nd dashboards so managers can monitor the business in real time or near-real time.Ingesting and processing streaming data requires an elastic data ware-house to handle variations and spikes in data flow.Embedded analytics operate as separate and distinct business processes within applications.The cl

261、oud facilitates data transfers from cloud-based applications to a cloud data warehouse where inherent scalability and elasticity can better support fluctuations in users and workloads.40 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any

262、dissemination,distribution,or unauthorized use is strictly prohibited.The data warehouse workload in your cloud data platform should support a broad ecosystem of third-party BI solutions,as well as offer native tools for specific types of analysis.Some of the pri-mary capabilities are summarized bel

263、ow.Considering Geospatial AnalyticsMost companies use geospatial data due to its capability to offer insights into location-based trends and patterns.For example,retailers collect geospatial data about sales territories,store loca-tions,and customer addresses to design better supply chains.Healthcar

264、e companies collect geospatial data to track the pene-tration of viruses and diseases.Telecommunications firms use it to monitor subscriber usage and optimize their communications networks.Logistics companies collect it to plan routes and opti-mize shipping activities.In some cases,this data is stor

265、ed as sim-ple numeric coordinates.In other cases,it resides in specialty data types such as spherical(geography)or flat surface(geometry).Collecting and analyzing spatial data involves new methods of data integration,analysis,governance,and interpretation.Tradi-tional data warehouse systems cant han

266、dle location data at scale because they have limited processing power,lack robust spatial analysis capabilities,and are difficult to integrate with geographic information systems(GIS).Select a data platform that can store and process any type of spatial vector object and perform complex geospatial t

267、ransfor-mations,such as converting geographic coordinates to street addresses.The processing engine must be able to handle location data at scale and seamlessly integrate with leading GIS tools.Optimizing Search FunctionsSearch optimization features can significantly improve the perfor-mance of cert

268、ain types of queries on tables such as the following:Queries that use selected geospatial functions with geogra-phy values Selective point lookup queries on tablesCHAPTER 7 Advancing Analytics 41These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strict

269、ly prohibited.Substring and regular expression searches Queries on fields in columns that use certain types of predicatesYour cloud data platform should offer optimized search capabili-ties that allow analysts to efficiently explore and query large vol-umes of data for point lookup queries,log analy

270、tics,star joins,substring searches,and geospatial searches.These capabilities are especially useful for needle in the haystack searches(such as a customer lookup)along with cybersecurity and log search use cases(such as when an analyst seeks to find the logs for a partic-ular IP address).Arming Data

271、 Analysts with MLMany data analysts want to take advantage of the benefits of machine learning(ML)but are daunted by the complexity of ML frameworks.In response,some cloud data platform vendors have created SQL functions that use ML to detect patterns in data.When backed by a robust data processing

272、engine,these ML func-tions make it easy to scale from one to millions of dimension-value combinations.In addition,data engineers can integrate calls to these functions into their data pipelines just as any other SQL function.Some examples of SQL functions with ML under-the-hood include the following

273、:Forecasting functions allow data scientists to construct accurate time series forecasts with automated handling of seasonality,scaling,and other variables.Anomaly detection functions empower analysts to find outlier events that should be investigated for suspicious activity,along with unlikely situ

274、ations that should be excluded from future analysis.Developing AI ApplicationsYou may start out using a cloud data platform for a traditional warehousing workload.As your volume of data grows,as your data analysts advance,and as you hire data scientists to join your team,you can start using the clou

275、d data platform to store and 42 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.process artificial intelligence(AI)/ML workflows,train predictive models,and then put

276、 those models into production.ML algorithms learn from data;the more data you provide,the more capable they become.A cloud data platform gives you one place to instantly access all relevant data for AI and ML workflows without complex data pipelines.It enables data science teams to store and process

277、 nearly limitless volumes of data at a progres-sively lower cost via powerful arrays of computers that can be scaled up and down at will.It unifies data security and data gov-ernance activities,fosters collaboration,and provides elastic scal-ability for data science and related analytic endeavors.Th

278、e most advanced cloud data platforms allow developers to deploy containerized data apps on accelerated computing infrastructure such as leading graphical processing units(GPUs),expanding the processing power that can be applied to these resource-intensive workloads.One popular application for these

279、advanced process-ing scenarios is the ability to natively run large language models(LLMs)within the platform and access them through an associ-ated marketplace.This arrangement allows cloud data platform customers to utilize these applications within their own accounts.Although your data platform sh

280、ould be able to securely deploy and process non-SQL code including Python,Java,and Scala SQL remains the industry standard for querying data.As such,your cloud data platforms data warehousing workload should include innovative SQL tools for data management,data transformation,data integration,visual

281、ization,BI,and all types of analysis.Automating Development,Deployment,and MonetizationAs AI becomes a more important aspect of many of todays soft-ware development projects,a cloud data platform gives advanced data analysts and data scientists native tools to facilitate ML application development s

282、uch as turning data and ML models into interactive applications.These platforms should work readily with popular open-source frameworks,tools,and languages,and include native libraries and functions that automate the data sci-ence life cycle.Some platforms even have out-of-the-box capa-bilities to t

283、urn Python scripts into web apps with no front-end development required.CHAPTER 8 Four Steps for Getting Started with Cloud Data Warehousing 43These materials are 2024 John Wiley&Sons,Inc.Any dissemination,distribution,or unauthorized use is strictly prohibited.Chapter8Four Steps for Getting Started

284、 with Cloud Data WarehousingT his chapter guides you through four key steps to choosing a cloud data warehouse for your organization.Step 1:Evaluate Your NeedsConsider the nature of your data,the skills and tools already in place,your usage needs,your plans,and how a cloud data platform can take you

285、r business in new directions.Think beyond data warehousing(storing and analyzing data).Ideally,you want one integrated platform that enables many workloads,including data warehouses for analytics;data lakes for data exploration;data engineering for data ingestion and transformation;data science for

286、developing predictive applications and machine learn-ing(ML)models;data application development and operation;and data sharing for easily and securely sharing data among authorized users.44 Cloud Data Warehousing For Dummies,3rd Snowflake Special EditionThese materials are 2024 John Wiley&Sons,Inc.A

287、ny dissemination,distribution,or unauthorized use is strictly prohibited.Step 2:Migrate or Start FreshAssess how much of your existing environment should migrate to the new data platform and what should be built from scratch.Defining strategy and goals,taking account of budget and resources to migra

288、te,and understanding your data volume can help you make this decision.To better understand which approach is best for your organization,talk to the professional services team of the data platform youre considering.Your BI solutions,data visualization tools,data science libraries,and other software d

289、evelopment tools must easily adapt to the new architecture.Step 3:Calculate TCOSelect a vendor that allows you to pay for actual usage in per-second increments.Consumption-based pricing eliminates soft-ware license fees,reduces infrastructure costs,and minimizes maintenance so you can reallocate tec

290、hnology resources to higher-value business priorities.Plus,when it comes to minimiz-ing TCO,dont overlook the value of productivity.Dont overlook the savings possible with features such as scaling up and down dynamically in response to changing demand.Step 4:Set Up a Proof of ConceptRequest a POC fr

291、om a prospective vendor with the general under-standing that if the solution performs satisfactorily,youll sub-scribe to the service.A proof of concept(POC)tests a solution to determine how well it serves your needs and meets your success criteria.Request a POC from a prospective vendor with the general understanding that if the solution performs satisfactorily,youll subscribe to the service.Obtaining first-hand experience via a POC will set you up for success with future data warehouse endeavors.WILEY END USER LICENSE AGREEMENTGo to to access Wileys ebook EULA.