1、云原生开源数据湖最佳实践吕侣小红书大数据负责人Cloud native open source data lake best practices年轻用户在这里通过短视频、图文等形式记录生活点滴分享生活方式，并基于兴趣形成互动这里有超过70%的90后年轻人是年轻人不可替代的生活方式平台和消费决策入口小红书介绍1 亿月活跃年轻用户100.125.145.160.180.195.220.225.235.240.260.290.330.360.400.100.200.300.400.500.07/3108/3109/3010/3111/3012/3101/3102/2803/3104/3005/310

2、6/3007/3108/3109/30数据量小红书数据量增长近14个月数据量增长4倍核心扩展性仓端分隔拥抱云考量点HDFSS3OSSAlluxioAgentFSHiveMetaParquetORCAvroIcebergHudiDynamicSchemaK8sYarnPavoTezSparkFlinkClickhouseTiDBPrestoDorisKylin联邦查询函数治理WaypointVelaGravityTableauGuanyuanDorRugalPokaGalaxyVirgo实验平台小红书大数据架构图应用层计算资源层计算引擎层数据层存储层研发工具产品运营赋能分析报表离线引擎实时引擎即

3、席分析物化引擎引擎治理存储层，以对象存储为核心标准型低频型归档型冷归档型统一的生命周期管理强一致数据模型，应用架构更简单多存储类型，全面覆盖各种访问模式数据成本海量数据场景下成本能匹配数据价值性能满足多种不同数据场景的性能诉求可靠性数据高可用，安全合规生态支持多种数据快速接入，多种数据引擎共享数据存储层-AgentFSAgentFS计算引擎MetaServerS3/OSSHiveMeta-StorePorterDBPrometheus创建、查询、删除修改文件路径等查询表文件路径删除文件根据路径读写文件数据归位查询表或者分区的地址添加分区信息查询分区信息修改分区地址查询迁移表信息监控指标批量删除

4、文件数据层-Dynamic Schemacreate external table dw_user_profile_day(user_token string,gender struct,location_info struct,app_info struct,user_app_list struct user_channel_info_list struct)PARTITIONED BY(dtm string)ROW FORMAT SERDEorg.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDeSTORED AS INPUTFO

5、RMATorg.apache.hadoop.hive.ql.io.parquet.serde.MapredParquetInputFormatOUTPUTFORMATorg.apache.hadoop.hive.ql.io.parquet.serde.MapredParquetOutputFormatdw_user_profile_day表结构定义user_profile_location_infoCLocationInfouser_profile_userapplistCUserAppListinsert overwrite table reddw.dw_user_profile_basic

6、_info_day partition(dtm=20210816)select a.user_token,a.s.col2.profile_gender as profile_gender,a.s.col2.profile_latest_country as profile_latest_country,a.s.col2.profile_usually_city as profile_usually_cityfrom(select user_token,max(struct(dtm,named_struct(profile_gender,default.pb_deserializer(gend

7、er).enum_gendertype,profile_latest_country,default.pb_deserializer(location_info).latestlocation.country,profile_usually_city,default.pb_deserializer(location_info).usuallylocation.city)s from reddw.dw_user_profile_daywhere dtm=f_getdate(20210816,-2)group by user_token)a自定义udf pb_deserializer查找调用相关类

8、PB deserializer注册中心实时引擎BIOLAPREDClickhouseFlinkHive on TezSpark SQL计算引擎层TiDBShare NothingClickhouse1Shard1MergeTreeMetaMergeTreePartClickhouse2Shard2MergeTreeMetaMergeTreePartClickhouse3Shard3MergeTreeMetaMergeTreePart服务调度计算存储ServerMetaStoreAdminKV StoreWorker ClusterWorker本地存储Worker ClusterWorker本地

9、存储SSDJuiceFS对象存储计算引擎层-Clickhouseoa apiyarn apirbacfe apiwebhook apiaws java sdkqcloud sdkaliyun sdkaws cloudwatchpushgatewayopenfaasdispathcher apiscenario metric apiscenario apilog apicluster apiaws ig apicluster manager apiig expand shrink apicluster pool apilog apibvao servercluster Discovery-Ser

10、viceovap server计算资源层-Pavo产品用户端/显示端核心系统/产品功能外部服务/组件Pavo前端AirflowDorVenus数据应用-Vela报表系统TiDBkafkaVela任务Vela任务Vela任务上游依赖上游依赖上游依赖upstream data readySQL交互AST解析血缘解析任务构建对象存储数据湖核心数据应用-Galaxy数据应用-实验平台探索中的应用Iceberg点查数据根据uid shuffle排序后写入Iceberg表Kafka集群OSSS3对象存储Flink集群Spark集群定周期读取数据排序写入后单分区数据分布变化date=2021-09-01mi

11、nute15=1minute15=Nbucket=0bucket=99minute15=1 minute15=Nminute15=1 minute15=Nfille0 fille1 filleN fille0 fille1 filleN fille0 fille1 filleN fille0 fille1 filleNdate=2021-09-01minute15=1minute15=Nbucket=0bucket=99minute15=1 minute15=Nminute15=1 minute15=Nfille0 fille1 filleN fille0 fille1 filleN fill

12、e0 fille1 filleN fille0 fille1 filleNselect*from ubt where id=xx1 and date=xxxid=xx1ubt表简化结构CREATE TABLEhive_prod.iceberg_db.ubt(id STRING,content BINARY,date STRING,minute15 STRING)USING iceberg PARTITIONED BY(last_date,bucket(100,id),minute15)TBLPROPERTIES(write.metadata.metrics.default=truncate(32),write.format.default=parquet,write.target-file-size-bytes=536870912,read.split.target-size=536870912);THANKS