《D-Lite:将基于Dolly的轻量级ChatGPT类模型集成到组织工作流程中.pdf》由会员分享,可在线阅读,更多相关《D-Lite:将基于Dolly的轻量级ChatGPT类模型集成到组织工作流程中.pdf(31页珍藏版)》请在三个皮匠报告上搜索。
1、Whats new in Databricks WorkflowsBilal Aslam,Sr.Director of Product Management,DatabricksProduct safe harbor statementThis information is provided to outline Databricks general product direction and is for informational purposes only.Customers who purchase Databricks services should make their purch
2、ase decisions relying solely upon services,features,and functions that are currently available.Unreleased features or functionality described in forward-looking statements are subject to change at Databricks discretion and may not be delivered as planned or at all.2023 Databricks Inc.All rights rese
3、rvedConfidential and Proprietary The Databricks Workflows Story Recent innovations Looking ahead DemoAgenda2023 Databricks Inc.All rights reservedConfidential and Proprietary 2015 Cron-based jobs 2016 Notebook workflows 2020 Multi-task jobs Reliability Monitoring 2022 The best lakehouse orchestrator
4、 Unified with the lakehouse Streaming Cluster reuse Simplicity 2023+2023 Databricks Inc.All rights reservedConfidential and ProprietaryModern data engineering requires modern data orchestration2023 Databricks Inc.All rights reservedConfidential and ProprietaryModern data engineering requires modern
5、data orchestrationMultiple data sources&triggersMultiple use casesMultiple dependenciesComplex,multi-stage data flowsExtract,transform,loadMachine learning Model trainingBI dashboard refresh“Data pipelines are growing in size,volume,and complexity,with multistage processing and dependencies between
6、various data assets.”*Gartner Data Engineering Essentials,Patterns and Best Practices,September 2022Orchestrating processes across all data,analytics and AI use cases is business critical Real-time applications2023 Databricks Inc.All rights reservedConfidential and ProprietaryBut organizations strug
7、glewith so many toolsof organizations are using 10+data engineering and intelligence toolsSource:IDC DataOps Survey,202065%2023 Databricks Inc.All rights reservedConfidential and ProprietarySessionsClicksJoinFeaturizeAggregateAnalyzeTrainOrdersBI&DataWarehousingDataStreamingDataScience&MLData Engine
8、eringLakehouse Platform?Many ways to orchestrate your Lakehouse2023 Databricks Inc.All rights reservedConfidential and ProprietaryData teams are less productiveHard to use for many practitionersBad data lowers value of downstream applicationsHigher cost of ownership and lower reliabilityDifficult to
9、 understand root cause when issues occurComplex architecture to manage and maintainThese tools are not unified with your LakehouseExternal orchestrators create challenges 2023 Databricks Inc.All rights reservedConfidential and ProprietaryUnity CatalogDelta LakeBI&DataWarehousingDataStreamingDataScie
10、nce&MLData EngineeringDatabricks WorkflowsUnified orchestration for data,analytics,and AI on the Lakehouse PlatformLakehouse Platform Simple authoring Actionable insights Proven reliabilityWorkflowsSessionsClicksJoinFeaturizeAggregateAnalyzeTrainOrders2023 Databricks Inc.All rights reservedConfident
11、ial and ProprietaryTop 3 reasons why customers love Databricks WorkflowsSimple authoring for all data practitionersAny data practitioner can accelerate their development by easily orchestrating Workflows from inside their Databricks workspace in just a few clicks.Advanced users can use their favorit
12、e IDEs with full support for CI/CD.Actionable insightsfrom real-time monitoring Full visibility into every task in every workflow.See the health of all your production workloads in real-time with detailed metrics and analytics to identify,troubleshoot,and fix issues fast.Proven reliabilityfor produc
13、tion workloadsA fully managed service with serverless data processing and years of 99.95%uptime.Workflows is trusted by thousands of Databricks customers running millions of production workloads.2023 Databricks Inc.All rights reservedConfidential and ProprietaryImproved collaboration80-90%faster pro
14、cessing7k customers|10 million VMs/day|99.95%uptimeADF Workflows4.5x faster deployment50%cost reductionAirflow Workflows60%cost reduction90%faster processing2023 Databricks Inc.All rights reservedConfidential and ProprietaryWe have built the best orchestrator for the lakehouse.And we are not done.20
15、23 Databricks Inc.All rights reservedConfidential and Proprietary40 new features shipped this year This is the highlight reel.2023 Databricks Inc.All rights reservedConfidential and ProprietaryOrchestrate the entire Lakehouse Databricks Workflows is fully integrated across the Lakehouse,including pa
16、rtners.Tool consolidation Deep insights into each task Cost efficiencyDatabricks SQL taskDelta Live TablesAuto Loadernon-Sparktask2023 Databricks Inc.All rights reservedConfidential and ProprietaryModel big,complex real-world workflows Job parameters Conditional Task(if/then)Queuing(great for backfi
17、lls!)2023 Databricks Inc.All rights reservedConfidential and ProprietaryManage complex data dependencies across organizational boundaries through data driven orchestration Trigger on file arrival Trigger child jobs Trigger on table changes Orchestrate across teams SOON2023 Databricks Inc.All rights
18、reservedConfidential and ProprietaryReliable,observable streaming Schedule and monitor Structured Streaming and Delta Live Tables Continuous execution Stream backlog monitoringSOON2023 Databricks Inc.All rights reservedConfidential and ProprietaryAlways know the health of your Workflows and data to
19、fix issues quickly Visual monitoring Webhook integrations Duration alertsMonitoring at your fingertips SOON2023 Databricks Inc.All rights reservedConfidential and ProprietaryJobs data in System Tables Customize your own operational dashboards using system tables and Databricks SQL Template queries+d
20、ashboards Historic cost analysis Failure pattern detectionPreviewDAISSessionsThr 11:30am:Lakehouse ObservabilityThr 3:30pm:Cost Management2023 Databricks Inc.All rights reservedConfidential and ProprietaryWe have built the best orchestrator for the lakehouse.And we are not done.2023 Databricks Inc.A
21、ll rights reservedConfidential and ProprietaryEasy CI/CD and version control CLI$cd src$IDEdata worker“Alice”gitWorkspace UIusers can see and test changes in devToolsusers commit changes to GitCI/CDpull requests&integration testsare deployed to stagingcode is deployed to prod after tests&approvalsde
22、v workspaceqa workspaceprod workspace bricks bundle deploy-e“dev”bricks bundle run pipeline refresh-all-e“dev”bricks bundle deploy-e“qa”bricks bundle run pipeline refresh-all-e“qa”bricks bundle deploy-e“production”bricks bundle run pipeline refresh-all-e“production”New end-to-end CI/CD flow Databric
23、ks Asset Bundles git integration Review changes in git or in Workflows Simple UIPreviewDAISSessionsThr 11:30am:Bridging production gapWed 4:30pm:Databricks asset bundles2023 Databricks Inc.All rights reservedConfidential and ProprietaryWorkflows Authoring Toolkit Python SDK Terraform supportWorkflow
24、s Authoring Toolkit:Easily develop Workflows in your IDE as Python code Compare changes Collaborate with UI-only usersCode review your WorkflowsSOON2023 Databricks Inc.All rights reservedConfidential and ProprietaryOrchestrate anything anywhere Easily integrate withLakehouse partnersand beyond Conne
25、ctors withone-time config Customizable tasks All in one placeNEWWorkflows2023 Databricks Inc.All rights reservedConfidential and ProprietaryServerless compute+Workflows Hands-off,auto-optimizing computeBenefit from Databricks scale:High efficiency:No idle,auto-optimizedReliability:Shielded fromcloud
26、 disruptionsElasticity:Fast scale upand downSimplicity:Every user canset up serverlessPreview2022 Databricks Inc.All rights reservedDBUsInfrastructure costOperational costDBUs(Single bill incl.infra+ops cost)ClassicServerlessValueFully managed service-operationally simpler,more reliableFast compute,
27、auto-scaling-better user experience,lower costAutomatic optimization for performance and costTCO savings Serverless reduces TCO 2023 Databricks Inc.All rights reservedConfidential and ProprietaryEliminate compute infra management DB SQL ML Model servingWorkflows&DLT&NotebooksZero management|Instant,
28、elastic compute|Lower TCO Serverless computeProd readyProd readyPreview2023 Databricks Inc.All rights reservedConfidential and ProprietaryDemo2022 Databricks Inc.All rights reservedExperience Databricks and Workflows Workflows yourself without accounthttps:/128.pl/awOGyDAISSessionsWed 3:30pm:5 thing
29、s about WorkflowsLearn more at the summit!We kindly request your valuable feedback on this session.Please take a moment to rate and share your thoughts about it.You can conveniently provide your feedback and rating through the Mobile App.Tells us what you thinkWhat to do next?Visit the Learning Hub
30、at the Databricks Zone!Take complimentary certification at the event;come by the Certified LoungeVisit our Databricks Learning website for more training,courses and workshops! trained and certifiedDiscover more related sessions in the mobile app!Visit the Demo Booth:Experience innovation firsthand!More Activities:Engage and connect further at the Databricks Zone!Databricks Events App2023 Databricks Inc.All rights reservedConfidential and Proprietary Thank you