1、Efficient tool for Brain ResearchWhole-Brain Direct Visualization based onNVIDIA GPU魏家飞脑科学与智能技术卓越创新中心中国科学院ChineseAcademyofSciences#page#OutlinecCenter and Research Background Introduction Solution for Whole-Brain Direct VisualizationCases Study and Perormance AnalysisSummary and Prospects#page#Cente
2、r for Excellence in Brain Science and Intelligence Technology (CEBSIT)脑科学与智能技术卓越创新中心(中科院脑智卓越中心)One Body Two Wings1999TO一体两翼()IONBrain DiseasesBasic NeuroscienceBrain-inspired inteligence类脑人工智能脑疾病机理脑认知神经基础神经科学研究所2014()CEBSITCenter for Excellence in BrainDawalopeffectiveapproachasin earlyUndarstandneu
3、ralbasiofcogntive functonstechnologiesScience and Inteligence Technology脑科学与智能技术卓越创新中心#page#Background Introduction- The Beautiful Brain#page#Background Introduction - Brain ImagingBigData脊椎类Brain Imaging脑成像哺乳类灵长类Brain 3D Image Data大脑三维图像数据Two-photon microscopyMagnetic Resonance Imaging双光子显微镜磁共振成像#p
4、age#Background Introduction- Big DataResolutionVolumeMouse Brain500 mm3TBmnbrafish Brain0.2mm3GETime#page#Background Introduction- Problem and ChallengeChallengesHardware Requirements Disk StorageVisalize CPU GPUMemory Software Requirements Scalables ParallelRemote High Rendering Performance#page#So
5、lution for Whole-Brain Direct VisualizationHardwares NVIDIA DGX ClusterDGX-1,DGX-2,DGXA100DVIDIAs Lenovo Distributed Storage (DSS-G)GPU-Accelerated Cluster SoftwareNVIDIAINDEXFORNVIDIA IndeXPARAVIEW PLUG-INInteractive Volume Visualization for Faster Disc ParaView + NVIDIA IndeX PluginSlurm08ParaView
6、IBM Spectrum ScaleslurmLenovo Distributed Storage SolutionIBMSpectrumScale#page#Solution Architecture.HPC Cluster (Shanghai)slurmanViePC(Beijing)ServeDGXA100DGX-1DGXHanaViDGX partitiondisplaysbatch#page#Sbatch for Running ParaView Server on Single Node#/bin/bash#SBATCH-Jrun_nvidia index paraview plu
7、gin#SBATCH-0OUL%#SBATCH-eerr_%申请集群资源#SBATCH-pdgx#SBATCH-N1#SBATCH-n8#SBATCH-gres gpu8申明GPU资源export CUDA_VISIBLE DEMICES-0,12,3,4.5,6.7指定NVIDIAIndexLicense位置export NMINDEX PVPLUGIN HOME-Spath to nvidlia index paraview plugin configure_file载入ParaView模块module load Paraview/5.8.1启动ParaView服务器端mpiexec-m8
8、pvserverforce-offscreen-rendering-=log.txt,9#page#Local Port Forwardingssh -t-t -p sport number Suser nameslogin node address-L Sport number :localhost Sport number ssh Sdgx listening node-L Sport number:localhost: Sport numberCompute NodesFront EndWorkStationInternetParaViewClientParaViewServer#pag
9、e#Connect ParaView Server from Client8cslhos苏#page#Setup NVIDIA Index for ParaView PluginConfigure NVIDIA IndeX License3methodstoconfigurelicensekey1.DefineenvironmentvariablesLoad plugin at both server and client sideexpNINDEXVENDORKEY=Slicense_keyexpoNVINDEXSECRET_KEY=slicensekesConfigure fle in s
10、peciic pathexponNVINDEXPVPLUGINHOME=Sicence_JocasoDisable ParaView Parallel Render IceTSNVINDEXPVPLUGINHOME/nvindex_config.xmlConfgfilein default path3.SHOME/.conig/Paraiewnvindex_configl#page#Configure ParaView Remote Parallel RenderingInteractiveRenderingalleRemote/ParallelRenderingsttil1Client/Se
11、rverthe elienlRenderingeSquirt/124ait#page#Study Visualization PerformanceParview5.8.1File Edit VewGUITime Log dialogmpiexec-n 8pvserver-force-offscreen-rendering-l=log.txt,9口-log/-l=SlogFile,Sverbosity_level口-verbosity=level口-2 is used for ERRORCommand Line口-1 for WARNINGTime Log File口OforINFO(defa
12、ult)口.口9forTRACE#page#Cases Study and Performance AnalysisCase 1: Zebrafish Brain VisualizationCase 2: Mouse Brain Visualization Case 3: Fish Brain Activity Real Time Visualiizatiion#page#Cases Data Source Fish Brain DataFacility of Mapping Brain-wide Mesoscale Connectome, CEBSITThanks to head of th
13、e Faciity Du Xufel Mouse Brain Data Mesoscopic mapping project of the mouse brain connectome, CEBSIT Thanks to head of the Lab Sun Yangang and teacher Wan Danying Fish Brain Activity Data Lab of Sensorimotor Transformation, CEBSITThanks to head of the Lab Mu Yu and teacher Gao Hao#page#Case 1:Zebraf
14、ish Brain VisualizationData(*.nrrd)Still RenderingResourcesnteractive Renderin#page#Case 1: Performance AnalysisCPU/GPUNVIDIASpeed UpIndex vs.Rendering ProcessParaViewSpeed Up Single GPU vsNumberIndex(s)Volume(s)Multiple GPUs(index)ParaView Volume4.382.4NAFirst Time Rendering10.511CPU 1GPUCase 1.10.
15、369.8227.3Still Renderingb801.193.5Interactive RenderingNA3.16NOT SUPPOT1First Time Rendering1CPU 1GPU0.501Still RenderingCase1.210.47Interactive RenderingNA2.72NOT SUPPOT1.16First Time Rendering2CPU2GPU0.153.33Still RenderingCase 1.2336Interactive Rendering0.141816.093.361.72First Time Rendering4CP
16、U4GPU0.155.1234.13semStill RenderingCase1.20.160.845.252.94Interactive RenderingUsing Paraview defaultLODlevel=1#page#Case 1: Performance AnalysisPerformance Comparison: NVIDIA Index vs.ParaView Volume4Speedup1050.ParaViewVolume(CPU)NVIDIA Index(GPU)#page#Case 1: Performance AnalysisPerfo
17、rmance Comparison: Single GPU vS. Multiple GPU8.335580.5Case1.2FirstTime RenderingCase 12:Stil RenderigCase 12:nteractive RenderingD1GPU 2GPU 4GPU#page#Case 2: Mouse Brain VisualizatiorData(*.tiff)Rend租Resources#page#Case 2: Performance Analysis NVIDIA Index has high performance in still rendering a
18、nd interactive rendering1CPU +1GPU(s)4CPU+4GPU(s)8CPU+8GPU(s)Data Read57.42168.770434.84484.264First Time Rendering267.04657.879116.5250.2880.245Still RenderingInteractive Rendering114.3120.3100.275oupport sucnaige uensionuata8 CPU NOT Support 16 CPU NOT Support40 CPU NOT Support#page#Case 3: Fish B
19、rain Activity Real Time VisualizationData(*.vti)Resources#page#Case 3: Performance AnalysisData size2405 file*500 M=1.2T Performance for total update0.4fps:2.5s/fileTotal ttime= 2405*2.5=6000s= 100 min#page#Case 3: Performance Analysis1 CPU +1 GPU4 CPU+4GPUAverage Time:1.492 secondsAverage Time:1.33
20、3 seconds24.3%30.0%0.02%0.7%Hon0.0169.3%75.5%#page#Summary and ProspectsSolution of“DGX GPUS+ NVIDIAIndex+ ParaView is scale of datawhich can handle unlimited size ofvolume datavisualization theoreticallyNVIDAIndexhas greatcapabiitiestovisualize largescaleand highqualitydatawith high perormancecompa
21、reto ParaViewdefautvolumerenderingand providemuch better userexperience in interactionvisualizationParaView isa powerfultoolthat cansupport multiplefleformatsrealtime renderingandfexiblecolortransferfuncton andcolor map nativelyueauousuasupue aupueslapun uue ossaodauaeaeue uoezensaduealoM#page#Summa
22、ry and ProspectsFuture of Biology: Direct Visualization“Itsour beliefwewill neverunderstand complexliving systems by breakingthem into parts.Only optical microscopes can look at living systems andgather information we need to truly understand the dynamics of life,themobility of cellsand tissues,how
23、cancer cells migrate.These are things wecan now directly observe.“The future of biology is Direct Visualization of living things rather thanpiecing together information gleaned by very indirect means”Eric Betzig埃里里克白兹格#page#AcknowledgementThanks to directors of CEBSIT: Du Jiulin and Sun Yangang supp
24、ort!Thanks to leaders of lab departments approval and help on cases datalDu Xufei, Head of Facility of Mapping Brain-wide Mesoscale Connectome for zebrafishbrain datalSun Yangang, Head of Facility of Mouse Brain Connectome and teacher Wang Danying for mouse brain data! Mu Yu, Head of Lab of Sensorim
25、otor Transformation and teacher Gao Hao for zebrafish brain activity data and data process!Thanks to team members: Hu Zhonghua, Li Wanlan and Wu Youming on datapreparation and data process#page#Thanks#page#Referenceshttps:/ returnuRL=https%3As0999680dz%uoasqnbuuz%howall93Dtruehttps:/en-us/data-centerindexparaviewplugihts:/pcligov/sofwareisualizatonsofwareparaviewumingparaviewcientServer-modehttps:/paravieworg/ik/Setting_p_aParaViewServerhtps:/docsparavieworg/enlatesReferenceManuaparalllDatVisualizaton.hlhtps:/