《SNIA-SDC23-French-Advancing-Accessing-Remote-Files-SMB3_2.pdf》由会员分享,可在线阅读,更多相关《SNIA-SDC23-French-Advancing-Accessing-Remote-Files-SMB3_2.pdf(62页珍藏版)》请在三个皮匠报告上搜索。
1、1|2023 SNIA.All Rights Reserved.Virtual ConferenceSeptember 28-29,2021Advancing Access to Remote Files:Exploring Recent Enhancements to the Linux SMB3.1.1 ClientPresented by Steve FrenchPrincipal Software EngineerMicrosoft Azure Storage(and Samba team member)2|2023 SNIA.All Rights Reserved.This work
2、 represents the views of the author(s)and does not necessarily reflect the views of Microsoft CorporationLinux is a registered trademark of Linus Torvalds.Other company,product,and service names may be trademarks or service marks of others3|2023 SNIA.All Rights Reserved.Who am I?Steve French Author
3、and maintainer of Linux cifs vfs(for accessing Samba,Azure,Windows and various SMB3/CIFS based NAS appliances)Co-maintainer of the kernel server(ksmbd)Member of the Samba team(co-creator of the“net”utility)coauthor of SNIA CIFS Technical Reference,former SNIA CIFS Working Group chairPrincipal Softwa
4、re Engineer,Azure Storage:Microsoft4|2023 SNIA.All Rights Reserved.OutlineOverview of Linux FS activityRecent ksmbd(server)improvementsRecent client improvementsComing soon what to look forward toTesting5|2023 SNIA.All Rights Reserved.Linux Kernel:A year ago and now.Now:6.6-rc2(“Hurr durr Ima ninja
5、sloth”)then:6.0-rc485,695 changesets!44,062 files changed3,670,278 insertions1,763,533 deletionsLSF/MM/eBPF summit is back in person too Pictures from 2023 summit in May Many excellent Linux storage developers working together7|2023 SNIA.All Rights Reserved.Some Linux FS topics of interest discussed
6、 recentlyTesting testing and more automated testing (e.g.kdevops)Folios,netfs,iov_iter,variable size pages,and the redesign of page cache and offline(fscache),io_uring(async i/o improvements)Idmapped mounts,fine grained timestampsLeveraging eBPF(not just dynamic tracing)Extending in kernel encryptio
7、n:TLS handshake(for NFS)and QUIC(SMB3.1.1 and other)Shift to cloudBetter support for faster storage(NVME)and net(RDMA/smbdirect)Last week Linus partial fix for“Why is glibcs fstat so slow?”Linux Filesystems Activity over past year(since 6.0-rc4)5207 filesystems changesets(6.1%of total kernel changes
8、ets,one of the most watched parts of the kernel,and FS activity is up slightly)Linux kernel fs are 1.08 million lines of code(measured this week)While total only up slightly in lines of code from last year(about 1%)note that some duplicate code removed and lots of improvementsAnd ReiserFS deprecated
9、Most Active Linux Filesystems over the past year VFS(mapping layer)461 changesets(activity up)VFS layer and small subset of the many filesystems dominate activityMost active are BTRFS 1195,ext4 454(up a lot),XFS 405(down),F2FS(350)SMB3.1.1(cifs.ko)340(flat)Then NFSD(server)224 and NFS(client)199(dow
10、n)cifs.ko had more than 3x the lines changed.It has been a VERY active year for cifs.koOtherGfs2(181),ksmbd(153,up),erofs(144),ntfs3(141),ceph(105)SMB3.1.1 Activity was strong this year cifs.ko activity was strong,340 changesets-cifs is 61KLOC kernel code(not counting user space utilities)ksmbd acti
11、vity up-25KLOC kernel code,379 changesets since its introduction in 5.15 kernelSamba server(userspace)is over 3.5 million lines of code(orders of magnitude bigger than the kernel smbd server or any of the NFS servers)and is even more activeRepeating our Goals for SMB3.1.1 on LinuxBe the fastest,most
12、 secure general-purpose way to access file data,whether in the cloud or on premises or virtualizedImprove directory lease supportKeep improving compounding,multichannelSupport more Linux/POSIX features so apps dont know they run on SMB3 mounts(vs.local)SMB3.1.1 POSIX extensions,new FSCTLsUse xfstest
13、s to locate new features to emulateAs Linux evolves,quickly add features to Linux kernel client and Samba and ksmbdMore test automation and keep adding more testsOne of the strengths of SMB3.1.1 is broad interop testingIn-person plugfests are back!SMB3.1.1 plugfests restarted,colocated with SDC last
14、 fallAnd again this year!Many exciting things being testedGreat discussions on future changes too13|2021 Storage Developer Conference.Insert Company Name Here.All Rights Reserved.Progess update for Linux Kernel Server(ksmbd)Thank you to Namjae Jeon(linkinjeonkernel.org)for providing much of this inf
15、ormation14|2023 SNIA.All Rights Reserved.KSMBD is no longer“experimental”Two years of testing and review in mainline Many security fixes and improvementsMany bugs found(and killed!)Ready for more production usePlease continue to report any bugs or problems found(we found,and fixed,more this week)15|
16、2023 SNIA.All Rights Reserved.Ksmbd is getting more security analysis which is helpingZDI disclosures helped,and recently another set of tooling announced:16|2023 SNIA.All Rights Reserved.Recent focusMuch of the focus over the last year has been on improving stability and addressing vulnerabilities
17、as ksmbd code is used and tested and analyzed more broadlyKsmbd now more broadly supports compounding(even including open/read/close and read/read/close)for all common use scenariosKsmbd even supports most of the SMB3.1.1 POSIX Extensions17|2023 SNIA.All Rights Reserved.How is ksmbd different?GPLv2(
18、as is most of the Linux kernel)not GPLv3(as Samba is e.g.)Supports RDMA(smbdirect)Great performance for multichannelIn kernel server rather than userspace so can more directly access some kernel fs features,and has shorter path lengthsMuch smaller code base(under 30KLOC in kernel and another 15K in
19、ksmbd specific userspace,not counting Samba code in userspace).Samba is 80 x bigger 18|2023 SNIA.All Rights Reserved.Next stepsEnable leases by default for files(currently defaults to simpler oplocks for caching)When testing looks good,proceed to enabling directory leases(for improved client caching
20、 of directory entries and long path revalidation)QUIC integration:As developers like Xin Long make progress on a kernel driver,enabling QUIC support for SMB3.1.1 can improve performance,work around the“port 445 problem”and help with some commonly requested security configurations Improve io_uring in
21、tegration(even better async i/o)“separate message processing from IO processing,ensuring that both can achieve maximum throughpu”-If interested in contributing there are lots of cool features to work on,as well as improved integration with Samba(e.g.user space upcalls for additional features).The SM
22、B3.1.1 family of protocols is huge!-Roles:Namjae(the maintainer)has done a lot,but additional features or subcomponents could be delegated.I am managing the git merges,ensuring additional functional testing is done regularly,and reviewing patches as requested by Namjae(my focus is largely on the cli
23、ent)-Namjae would welcome additional help with code reviews,security auditing,testing and new features-Very exciting time!Linux Kernel Server,KSMBD(continued)20|2021 Storage Developer Conference.Insert Company Name Here.All Rights Reserved.Recent improvements in the kernel client(cifs.ko)21|2023 SNI
24、A.All Rights Reserved.Multichannel improvementsMultiple Reconnect and Perf improvements including improved channel allocation for SMB3.1.1 requests(thank you Shyam Prasad)Soon will be enabled by default(when server supports multiple interfaces or RSS)22|2023 SNIA.All Rights Reserved.DFS ie“The Globa
25、l Namespace”-improvementsDFS Interlink support(DFS link that points to another namespace)Sharing of DFS connections between mountsReparse mount points-as cross fs boundaries on server-createsubmounts with noserverinoReconnection improved for DFS use casesNow handles cases where nested links have som
26、e bad targets(and can handle somescenarios which other clients cant)Linux File SS API still growing(9 recently)Syscall nameKernel Version introducedio_uring_setup,setup,io_uring_registertruncate64,ftruncate64,pread64,pwrite64,sync_file_rangefchmodat25.175.196.6-rc1e.g.:very important feature:“folios
27、”added&changes to internal APIs(netfs,fscache)and io_uring continues to improveCase Sensitivity without extensions can open the wrong file and owner and mode bits usually set to defaults(not real owner)With POSIX extensions more accurateServer is current Samba on this slide and next Without POSIX ex
28、t.can be confusingBut still work to do -here is local fs view of filesBut still work to do(ksmbd example):Mode bits and owner ok but symlinks(and FIFOs)not shownSamba POSIX improved this week.Server side symlinks fixed(thx Volker),FIFOs not fixed yetWireshark trace to Volkers latest Samba update5.19
29、 kernel(July 31st,2022)Important performance optimization for directory searches,now we cache root directory content(to the many servers which support dir leases)reducing amount of network traffic for queries of root directoryMultichannel reconnect improvements(e.g.when address or interfaces change)
30、Conversion begins for cifs.ko to use new mm layer design read_folio/release_folioRDMA(smbdirect)improvementsNew mount parm“nosparse”to optionally disable use of sparse files6.0 kernel(October 2nd,2022)(cifs module version 2.39)Fallocate improvements(insert and collapse range)Module size shrunk signi
31、ficantly when SMB1/CIFS(insecure legacy)disabledNew mount parm“closetimeo”allows extending deferred closes(handle leases)longer or even disabling the feature(and default increased to 5 seconds from 1 sec)Important deferred close fixMultichannel perf(locking)improvements6.1 kernel(Dec 11th,2022)(cifs
32、 module ver:2.40)Performance improvement for path revalidation(metadata ops perf better)by using cached dentry for subdirectory if lease held on itExpanding cached directories to include subdirectories(thanks Ronnie!)New ioctl for change notify added that returns the name(s)of any changed files in t
33、he directory(not just that the directory has changed)e.g.so app can do their own offline caching of files and sync with serverImprove symlink handling(avoid an extra roundtrip when symlink detected via STOPPED_ON_SYMLINK message)RDMA(smbdirect)improvements(thanks Tom Talpey and Metze!)6.2 kernel(Feb
34、ruary 19th,2023)(cifs module 2.41)Important SMB3.1.1 POSIX extensions improvement(parse owner and group SIDs to improve stat output)DFS performance improvements(reducing roundtrips)and DFS fixesMultichannel and reconnect and DFS improvementsIntegration with the new kernel page caching infrastructure
35、,folios(e.g.migrate_folios support),iov_iter and memory management layering cleanup(e.g.deprecated writepage/writepages API removed)6.3 kernel(April 23rd,2023)(cifs module 2.42)very active release!Kernel idmapping improvementsImprovements to use folios(better mm integration and cached writes)RDMA(sm
36、bdirect)improvements(thanks Metze and David)Many multichannel improvements(including using least loaded channel for sending I/O,and improvements for reconnect).Thanks Shyam!Various DFS fixesLower default deferred close timeout6.4 kernel(June 2024)(cifs.ko version:2.43)Important deferred close(lease
37、break corner case)fixesReconnect and DFS fixesImportant crediting improvements expectedCompounding improvements expected6.5 kernel(August 2024)(cifs.ko version:2.44)Deferred close perf improvement(avoid unneeded lease break acks)Crediting(flow control)improvements to avoid low credit perf issue Reco
38、nnect and DFS fixesFix null auth(sec=none)regressionAllow dumping decryption keys(eg for reading network traces)via directory name,it just file.Display client GUID and network namespace in/proc/fs/cifs/DebugData6.6-rc kernel(expected Nov.2024)(cifs.ko version:2.45)DFS(global namespace)fixesImproveme
39、nts handling reparse points.Perf improvement for querying reparse point symlinksReconnect improvement(write retry with channel sequence number)Add new mount parm max_cached_dirs to control how many directories are cached when server supports directory leases.Add new module parm/sys/module/cifs/param
40、eters/dir_cache_timeoutto control length of time a directory is cached with directory leasesDebugging improvementsNew dynamic tracepointsNew debugging scripts leveraging eBPF40|2021 Storage Developer Conference.Insert Company Name Here.All Rights Reserved.Features we are working on for the coming re
41、leasesSection Subtitle41|2023 SNIA.All Rights Reserved.What features can you expect in next few releases?Analyze cases where use of directory leases could better optimize network traffic while caching safely(caching dirents and dir metadata)Add use of compounding in more cases being tested in plugfe
42、st now(e.g.open/querydir/querydir instead of open/querydir),and better use existing file leases for compound reqs which include SMB3 openContinued focus on multichannel performance improvementsAutomatically using multichannel,picking optimal channels for special casesSMB3.1.1 compression support(all
43、ow compressing network traffic based on the SMB3.1.1 compress mount parm)42|2023 SNIA.All Rights Reserved.Optimizing common cases:improve creating symlinksCurrent behavior with“mfsymlinks”43|2023 SNIA.All Rights Reserved.Optimizing common cases:improve“ls”when symlinksCurrent behavior with“mfsymlink
44、s”44|2023 SNIA.All Rights Reserved.Optimizing common cases:improve“ls-l”when symlinksCurrent behavior with“mfsymlinks”(noterepeated read e.g.)45|2023 SNIA.All Rights Reserved.What features can you expect in next few releases?Improved packet signing(faster GMAC)Support for new auth mechanisms(e.g.loc
45、al KDC being investigated)Reenabling support for swapfile over SMB3.1.1 mountsSupport for creating with O_TMPFILE(being tested at plugfest now)Improvements to enable fanotify/inotify over SMB3.1.1 mounts(currently requires a private SMB3.1.1 specific ioctl)Prototype of SMB3.1.1 over QUIC(new encrypt
46、ed network transport)More testing of the SMB3.1.1 POSIX with new Samba server support46|2023 SNIA.All Rights Reserved.What POSIX features to expect soon?Allowing creating special files like FIFOs and CHR and BLK devices via reparse points(currently requires“sfu”mount option and xattrs)chmod fixes47|
47、2023 SNIA.All Rights Reserved.Other fixes being discussed this weekReconnect,retry improvements(missing retry flag when channel sequence number updated)Byte range lock sequence numbers for reconnectEnabling multichannel by default if srv support and how many channelsBetter use of parent lease keyRed
48、ucing number of unneeded lease breaks by checking for lease key in more cases(e.g.some compounding scenarios)How to hold root directory open longer even when no dir lease supportI/O compression in SMB 3.1.1Brief overviewSEPTEMBER 15,2023Enzo Matsumiya SummaryI/O compression in SMB 3.1.1Message compr
49、ession was introduced in SMB 3.1.1 with the goal to reduce the SMB message size on large reads/writes,thus decreasing network load and increasing bandwidth and throughputMS-SMB2 specifies the usage of the Xpress Compression Algorithm to achieve this,which is a set of compression algorithms specified
50、 inMS-XCAThis overview is about its current implementation(early-alpha-quality code)for Linux cifs.koGoalsI/O compression in SMB 3.1.1Keep it simple-Implement a simple API,generic enough that could,maybe,turn into a standalone lib-the compressor object has only the algo,compress mode(*),and the comp
51、ress/decompress function pointers-API design makes it plug&play for any of the MS-XCA algos-currently only Pattern_v1 and LZ77+Huffman are implemented(others are LZNT1 and LZ77)-Minimally intrusive in the rest of the code-Compression:trade-off between CPU/memory usage and compressed size-low trade-o
52、ff ratio efficient-To achieve this:compress only write requests larger than 4096 bytes(most recent Windows Servers does the same,as far as I could confirm)-This implementation has both“data”and“full”compression modes(MS-SMB2 3.1.4.4,item 2-3):-“data”mode compresses only the write data(SMB2 header st
53、ays uncompressed)-“full”mode compresses everything*further benchmarks are still needed,but the difference in payload size(see Current issues though)and resource usage are negligibleGoalsI/O compression in SMB 3.1.1EfficientGoalsI/O compression in SMB 3.1.1Performance and of course,the end goal is to
54、 improve I/O performance on cifs.ko mounts-Hard to give exact numbers,since uncompressed data layout has a huge impact-Microsofts demo(*)(Windows client and server)shows:20GB file transferUncompressedCompressedTime2 min 43 sec28 secThroughput123 Mbps730 MbpsBandwidth1 Gbps1.5 MbpsCPU usage5-10%40-50
55、%*https:/ SUSE 202353GoalsI/O compression in SMB 3.1.1Performance(cont.)-So,compressed SMB2 I/O is“about 6x faster”-No reason similar results could not be achieved on Linux-unless Windows does extra tricks to achieve thoseCopyright SUSE 202354Current issuesI/O compression in SMB 3.1.1-LZ77+Huffman u
56、ses about 100k bytes(for hashtables and to build the Huffman tree)to compress each block.In this very early stage,to make it work,Im still kvmallocing them,but I already have a PoC using rbtrees that could cut that in half(most of that memory is used for sorting/ordered access,and that comes for fre
57、e-ish with rbtrees)-It also has a fixed 260 bytes overhead for each compressed block,and it requires the input to be at least that big.But sometimes,with an input just a bit larger than that,its possible that the final compressed data is actually bigger than the input.Thats why compressing the whole
58、 messages doesnt make much sense,IMHO,as it adds unnecessary overhead compressing the SMB2 header,with little final benefit-Decompression(large reads)doesnt work yet-Lots of bugs to fix still,and a lot of improvement to make,but an RFC should appear on CIFS mailing list soon Thank you.Feel free to e
59、mail me any questions or feedback.ematsumiyasuse.de56|2023 SNIA.All Rights Reserved.SMBDIRECT Transport Improvements:“RDMA for the World”As discussed last year SMBDIRECT is an abstraction layer(“framing layer”)for making RDMA useable more broadly.Has no SMB dependencies(SMB3 was just the first consu
60、mer of this generic transport layer,but it applies more broadly)Longer term plan is to:Bring common from cifs.ko and ksmbd for RDMA into smbdirect.koEnable user space access to RDMA through smbdirect.ko so user space applications can benefit from the performance gains of RDMAImprovements to this com
61、mon module will benefit both client and server(and userspace)57|2023 SNIA.All Rights Reserved.SMBDIRECT Transport Improvements:“RDMA for the World”smbdirect.ko will providePF_SMBDIRECT socketsSend message and receive message will get MSG_OOB messages for read and write offload,greatly improving perf
62、ormance and reducing CPU overhead(SMB independent)“echo server client”smbdirect tests under development to improve regression tests without requiring SMBThanks to Metze for this work.Feedback welcome58|2023 SNIA.All Rights Reserved.Some SMBDIRECT recommendations from Metze and Tom Talpey from last y
63、ear are still relevantReduce SGE usage,and decrease maximum fragment sizeNeedless memory usage,High SGE usage impacts performanceFix RDMA“responder resources”,which do NOT apply to RDMA WritesSignificant performance limiter for bulk readsFix sends to not wait for completion before returning.Stalls p
64、ipeline,context switchingUse RDMA post-multiple to improve compound send efficiencyEnsure packet kmem cache optimal packing(3x1364=4092)Review protocol parsing and state validationE.g.ksmbd allows renegotiate(?),reassembles oversize segments(?)Merge the two implementations:fs/cifs/smbdirect.ch and f
65、s/ksmbd/transport_rdma.chEither refactor and merge,or consider metzes alternative“smbdirect socket”driverTesting ImprovementsTest test test.60|2023 SNIA.All Rights Reserved.Additional tests are encouraged(generic or smb specific)Xfstests are the standard Linux filesystem functional tests“Buildbot”mi
66、grated to new host,and is being modified(down this week)Over last 18 months added 21 to the main“cifs-testing”regression testing group(up to 245 tests run on every checkin from this group)Various server specific groups have added even moreAzure SMB3.1.1 multichannel:up 25%more tests,now includes 133
67、 testsKsmbd(Linux kernel server target)up 15%,now includes 144 testsDetailed wiki pages on wiki.samba.org go through how to setup xfstests with cifs.ko,and what features need to be added to enable more tests(tests that currently skip or fail so arent run in the buildbot)Thank you for your timeFuture
68、 is very bright!SMB3+Additional Resources to Explore for SMB3 and Linuxhttps:/ particular MS-SMB2.pdf at https:/ Linux CIFS client https:/wiki.samba.org/index.php/LinuxCIFSSamba-technical mailing list and IRC channelAnd various presentations at http:/www.sambaxp.org and Microsoft channel 9 and of co
69、urse SNIA http:/www.snia.org/events/storage-developerAnd the code:https:/git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/fs/smbFor pending changes,soon to go into upstream kernel see:https:/git.samba.org/?p=sfrench/cifs-2.6.git;a=shortlog;h=refs/heads/for-next Kernel server code:https:/git.samba.org/ksmbd.git/?p=ksmbd.git(ksmbd-for-next branch)