《Lakehouse Architecture to Advance Security Analytics at the Department of State.pdf》由会员分享，可在线阅读，更多相关《Lakehouse Architecture to Advance Security Analytics at the Department of State.pdf（18页珍藏版）》请在三个皮匠报告上搜索。

1、Lakehouse Architecture to Advance Security Analytics at the Department of State Tim AhrensBrendan BarsnessDatabricks2023Agenda1What is the Department of States Center for Analytics(CfA)2The Challenge3OrganizationalThe Response and Recommendations4TechnicalThe Response and RecommendationsUNCLASSIFIED

2、UNCLASSIFIEDCenter for AnalyticsWho We SupportWe empower employees across every bureau and over 200 missions,from working-level to the Secretary.Who We AreCfA is the Department of States enterprise data management and analytics capability.Led by the Chief Data Officer,we transform data into bold ins

3、ights that help make better management and foreign policy decisions.3UNCLASSIFIEDUNCLASSIFIEDWhat is State Department M/SS CfA?ANALYTICSDATA CULTUREDATA MANAGEMENTDATA GOVERNANCEAccelerate Decisions through AnalyticsCultivate a Data CultureEstablish Mission-Driven Data ManagementEnhance Enterprise D

4、ata GovernanceAnalyticsEnterprise Engagement&CommunicationsEnterprise Data ManagementTechnology Empower the Departments global workforce to utilize data by providing easy access to the Departments data assets,modern analytics tools,and customer service to enable their use.Recruit,train,and incentivi

5、ze a workforce andworkplace where data is routinely sought,valued,andfluently utilized for decision-making at all levelsand geographies.Implement technology solutions to effectively create,collect,store,protect,and share data across the Department,the interagency,and with the public.Enable oversight

6、 and coordination of Department data through effective stewardship,policies,process controls,and investment decisions that appropriately value data.UNCLASSIFIEDUNCLASSIFIED1_DAIS_Title_SlideThe Challenge*M-21-31 Improving the Federal Governments Investigative and Remediation Capabilities Related to

7、Cybersecurity IncidentsOMB 21-31*RequirementsM-21-31 defines event logging requirements to support the detection,investigation,and remediation of cyber incidents on federal information systems.RequirementM-21-31 requirements are substantial and prescriptiveUNCLASSIFIEDRetain a prescriptive list of r

8、equired log types,fields,formats for systems and other IP addressable assetsSignificant retention duration(Up to 2.5 years for many log types)Central visibility for Security Operations Center(SOC)123UNCLASSIFIEDThe ChallengeUNCLASSIFIEDDATA C E N T E RSystems must meet all logging requirements for b

9、readth or duration of retentionTB/day-scale data is expensive to store,especially On Prem or in SIEM tools,which can lead to a massive duplication of stored data.Expensive to exfil between cloudsUNCLASSIFIEDThe ChallengeUNCLASSIFIEDDATA C E N T E RDATA C E N T E RSystems must meet all logging requir

10、ements for breadth or duration of retentionTB/day-scale data is expensive to store,especially On Prem or in SIEM tools,which can lead to a massive duplication of stored data.Expensive to exfil between cloudsUNCLASSIFIEDThe ChallengeUNCLASSIFIEDDATA C E N T E RDATA C E N T E RA Z U R E C LO U DSystem

11、s must meet all logging requirements for breadth or duration of retentionTB/day-scale data is expensive to store,especially On Prem or in SIEM tools,which can lead to a massive duplication of stored data.Expensive to exfil between cloudsUNCLASSIFIEDThe ChallengeUNCLASSIFIEDDATA C E N T E RDATA C E N

12、 T E RA Z U R E C LO U DAW S C LO U DSystems must meet all logging requirements for breadth or duration of retentionTB/day-scale data is expensive to store,especially On Prem or in SIEM tools,which can lead to a massive duplication of stored data.Expensive to exfil between cloudsUNCLASSIFIEDThe Chal

13、lengeUNCLASSIFIEDDATA C E N T E RDATA C E N T E RA Z U R E C LO U DAW S C LO U DG O O G L E C LO U DSystems must meet all logging requirements for breadth or duration of retentionTB/day-scale data is expensive to store,especially On Prem or in SIEM tools,which can lead to a massive duplication of st

14、ored data.Expensive to exfil between cloudsUNCLASSIFIED1_DAIS_Title_SlideTechnicalThe Response:TechnicalEnterprise Lakehouse effort will help system owners and the cybersecurity community respondWhat is the Lakehouse?Distributing data processing to each cloud or data“node”allows the Department to cr

15、eate direct connections to a centralized analytics cloud without migrating raw data across boundaries,enabling a cloud-agnostic storage solution.Centralized Analytics Platform with Distributed ComputingProtocols for securely sharing data from individual nodes allow a centralized analytics platform t

16、o compile,search,visualize,and perform advanced analytics on distributed data from nodes,with compute handled locally.BenefitsSupports multi-cloud&multi-regionSupports long-term data retentionSupports petabyte scaleBuilt in AI/ML to support batch processing and near real-time advanced analyticsReduc

17、e egress transfer costsReduce data duplicationImprove visibility across the enterpriseLeverage low-cost cloud storageCentral data governance modelLakehouse HubService ProviderUsersNodeOwner:Dept/OfficeNodeOwner:Dept/OfficeNodeOwner:Dept/OfficeNodeOwner:Dept/OfficeNodeOwner:Dept/OfficeUNCLASSIFIEDUNC

18、LASSIFIEDUnity Catalog in Azure GovtImplement dataset/domain taggingLeverage RBAC/ABACEstablish multi-layer securityEnsure maintainabilityTechnologyDevelop fungible/reusable parsersExercise Databricks AutoloaderAd-hoc queriesScheduled queriesContinuous queriesDelta optimizationsLimit Silver and Gold

19、 tablesDevelop intelligent queriesImplement query guardrailsBalance agency and flexibility while supporting“citizen”operatorsIntegrate BI toolsBuild common pre-canned queriesData Access ManagementData ProcessingFederated QueriesRecommendationsUNCLASSIFIEDUNCLASSIFIED1_DAIS_Title_SlideOrganizationalT

20、he Response:OrganizationalLakehouse is supported by stakeholder governance and policy to ensure participation,implementation,and securityLakehouse Oversight GroupReviews and authorizes new use cases,changes to access control policy,and new data connectionsPolicy enablement to support participationTh

21、ou shalt share CIO Action Memo;How To policyCyber Ops to reduce barriers to participatingTracking and reporting to IT leadershipLive reporting will show systems and nodes connectedIntegrated with tracking of M-21-31 progress by system,by data elementSystem owner incentives once fully operationalComp

22、liant systems receive incident response common controls and support moving to Continuous AuthorizationUNCLASSIFIEDUNCLASSIFIEDRecommendationsEstablish clear and delineated responsibilitiesEnsure executive support(political buy-in)to succeedExecutive oversight group,aka the LOGWorking group to suppor

23、t collective developmentOrganizationalUpdate policies as necessary to support federal requirementsEstablish guidance for system owners/data providersFind willing participantsQuick winsPrioritize systems/orgs based on requirementsSufficient and compelling use casesEngage your stakeholdersGovernancePo

24、licyOutreachUNCLASSIFIEDUNCLASSIFIED18Thank YouThis publication contains general information only,and none of the member firms of Deloitte Touche Tohmatsu Limited,its member firms,or their related entities(collective,the“Deloitte Network”)is,by means of this publication,rendering professional advice

25、 or services.Before making any decision or taking any action that may affect your business,you should consult a qualified professional adviser.No entity in the Deloitte Network shall be responsible for any loss whatsoever sustained by any person who relies on this publication.As used in this documen

26、t,“Deloitte”means Deloitte Consulting LLP,a subsidiary of Deloitte LLP.Please see for a detailed description of the legal structure of Deloitte USA LLP,Deloitte LLP and their respective subsidiaries.Certain services may not be available to attest clients under the rules and regulations of public accounting.Copyright 2023 Deloitte Development LLC.All rights reserved.Member of Deloitte Touche Tohmatsu Limited.