1、ISSCC 2024Forum 4Intelligent Sensing 2024 IEEE International Solid-State Circuits ConferenceForum F4:Intelligent SensingInternational Solid-State Circuits ConferenceFebruary 22nd,2024Start of presentations at 8:00amISSCC 20241 of 4 2024 IEEE International Solid-State Circuits ConferenceHosted by the

2、 IMMD,DAS,ANA,and Security Subcommittees Organizers:Kea-Tiong Tang,National Tsing Hua University,Hsinchu,TaiwanMahsa Shoaran,EPFL,Lausanne,SwitzerlandCommittee:Rangharajan Venkatesan,NVIDIA,Santa Clara,CAMarco Berkhout,Goodix Technology,Nijmegen,NetherlandsTakeshi Sugawara,The University of Electro-

3、Communications,Tokyo,JapanChampions:Bruce Rae,STMicroelectronics,Edinburgh,United KingdomMakoto Ikeda,University of Tokyo,Tokyo,JapanOrganizing CommitteeISSCC 20242 of 4 2024 IEEE International Solid-State Circuits Conference8 talksEach talk is 40-45 minutes followed by a Q&A2 coffee breaks and one

4、lunch breakElectronic copies of the slides are available for downloadPlease switch off or mute your mobile phonesRemember to fill out the speaker evaluation form using the ISSCC appNo panel session at the end of the ForumGeneral Information ISSCC 20243 of 4 2024 IEEE International Solid-State Circui

5、ts ConferenceAgendaStartTitleSpeakerAffiliation8:15IntroductionKea-Tiong TangMahsa ShoaranNational Tsing Hua Univ.EPFL8:25Hope for the Best,but Plan for the Worst:Considering Risk and its Mitigation when Designing Intelligent Sensing SystemsTimothy DenisonUniversity of Oxford9:15Intelligent Vision U

6、sing Smart Imager with Processing-in-Sensor TechniquesChih-Cheng HsiehNational Tsing Hua Univ.10:05Break10:20Activity-Driven Perception for Intelligent Edge Sensory SystemsShih-Chii LiuUniversity of Zurich and ETH Zurich11:10Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal

7、in-Memory Computing for Distributed Adaptive Intelligence at the EdgeGert CauwenberghsUC San Diego12:00Lunch13:20Cross-Layer Innovations for Enabling Real-Time and Efficient Eye Tracking in VR/ARYingyan LinGeorgia Institute of Tech.14:10Aggressive Design Reuse for Ubiquitous Security From Design-Tim

8、e to Run-Time Intelligent Attack Detection and CounteractionMassimo AliotoNat U Singapore15:00Break 15:15Implementing on-Sensor Machine Learning for Ultralow Power,Always-on Inferencing at the Extreme EdgeMahesh ChowdharySTMicroelectronics16:05Benefits of System Architecture re-Design for 3D Chiplet

9、 Integration TechnologiesDragomir MilojevicIMEC16:55Closing remarksKea-Tiong TangMahsa ShoaranNational Tsing Hua Univ.EPFL 2024 IEEE International Solid-State Circuits ConferenceHope for the best,but plan for the worst:considering risk and its mitigation when designing intelligent sensing systemsTim

10、 Denison,Ph.D.ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems1 of 47 2024 IEEE International Solid-State Circuits ConferenceBeing smart about intelligent sensingCore Design Principles that Cross Disciplines:Automated Flight Control Self-Driving CarsISSCC 2024-Forum 4.1:Risk in Intelligent S

11、ensing Systems2 2024 IEEE International Solid-State Circuits ConferenceBioelectronic Therapies in 2024:A Didactic Model for Sensor-enabled SystemsStandard practice:clinician-set,continuous therapyConstant stimulation can lead to side effectsBalance between symptom control and side effect profile on

12、averageState of the art:Closed loop,adaptive/reactive therapy based on biomarkerMinimise side effects Configured in the clinic for 24/7 operation?Challenges:Keeping Patients SafeSystem Engineering for Fault-ToleranceISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems3 of 47 2024 IEEE Internatio

13、nal Solid-State Circuits ConferenceFuture:Devices that restore normative physiologyPredict changes in patient needs(e.g.circadian rhythms)Biomarker patternsAnticipate sleep/wakeRespond to behaviour and biomarker changes(sensors)Design for Safe Autonomy Structured approach to designRegulatory standar

14、ds evolving ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems4 of 47 2024 IEEE International Solid-State Circuits ConferenceSummary of Learning ObjectivesProvide a framework and common language using examples from physiologic closed-loop controllers(PCLCs)that can be extended to other systems

15、Use examples from multiple sensing-based domains to provide intuition for PCLC operation,and the problems they help solve,as template for any system using intelligent sensingReinforce key terms and concepts for safe sensing-based systems from FDA guidance documents and PCLC standards(e.g.risk,60601-

16、1-10)for robust design and use design templateEmpower you to analyse“closed-loop”sensor-enabled systems with a clearer mental model of their operation and safetyISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems5 of 47 2024 IEEE International Solid-State Circuits ConferencePrinciples and Terms

17、Ozer,Ahmet.(2011).Exact boundary controllability and feedback stabilization for a multi-layer Rao-Nakra beam.ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems6 of 47 2024 IEEE International Solid-State Circuits ConferenceStandards and Common Language:Definition of a“Physiological Closed Loop

18、Controller”Physiological closed-loop controlled device:a medical device that automatically adjusts or maintains a physiologic variable(s)through delivery or removal of energy or article14(e.g.,drugs,or liquid or gas regulated as a medical device)using feedback from a physiologic-measuring sensor(s)F

19、DA Guidance(1500085):Technical Considerations for Medical Devices with Physiologic Closed-Loop Control TechnologyYour sensor goes here,but how does it fit in system?How can that system“misbehave?”ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems7 of 47 2024 IEEE International Solid-State Circ

20、uits Conference“Feedback”Control:Terms Measure a variable and feedback to adjust actuator target goal PCLC focuses on a physiologic variable Algorithm controls the adjustments Sensors record relevant variable Actuator manipulates the system(patient)Safety features limit risks(robust)User interface f

21、or configurationISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems8 of 47 2024 IEEE International Solid-State Circuits Conference“Physiological Closed Loop Controller”ExamplesJuan Albino Mendez,et al.,Improving the anesthetic process by a fuzzy rule based medical decision system,Artificial Int

22、elligence in Medicine,2018Blauw,Helga&Keith-Hynes,Patrick&Koops,Robin&DeVries,J.(2016).A Review of Safety and Design Requirements of the Artificial Pancreas.Annals of biomedical engineering.44.10.1007/s10439-016-1679-2.Closed-loop Anaesthesia Systems“Artificial Pancreas”ISSCC 2024-Forum 4.1:Risk in

23、Intelligent Sensing Systems9 of 47 2024 IEEE International Solid-State Circuits ConferenceThe Complexity of the Closed-Loop:DiabetesImage Courtesy of Lane DesboroughISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems10 of 47 2024 IEEE International Solid-State Circuits ConferenceBrain-Computer-

24、Interface-PCLC?Key point:sensing systems can benefit from methods even when not a formal PCLCOpri et al,Chronic embedded cortico-thalamic closed-loop deep brain stimulation for the treatment of essential tremor,Science Translational Medicine,2020ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Syste

25、ms11 of 47 2024 IEEE International Solid-State Circuits ConferenceIntelligent Sensing ExamplesConcepts and ApplicationsISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems12 of 47 2024 IEEE International Solid-State Circuits ConferenceControl Systems*often involves combinations of methods*and ph

26、ysiology!Houk JC.Control strategies in physiological systems.FASEB J.1988 Feb;2(2):97-107.doi:10.1096/fasebj.2.2.3277888.PMID:3277888ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems13 of 47 2024 IEEE International Solid-State Circuits Conference“Feedforward”Control:Informed AdjustmentsImage

27、courtesy of Diabetes UKMake adjustments!Device usually has a“basal”setting A disturbance arises which perturbs the system If we can observe/predict the disturbance,use this information to adjust actuator to counteract Example:insulin injection prior to a meal Example:adjust neurostimulator for med b

28、olus Can be very effective control mechanism for responding to the patients environment!ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems14 of 47 2024 IEEE International Solid-State Circuits ConferenceSCS:Posture Responsive StimulationPosture Change Adjust Stim AmplitudeCardiac Pacing:Motion

29、Adaptive Rate AdjustMore Motion Faster Pacing(estimate)“Feedforward”in Sensing-Enabled Implantable DevicesISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems15 of 47 2024 IEEE International Solid-State Circuits ConferenceControl Systems often involves combinations of methodsISSCC 2024-Forum 4.1

30、:Risk in Intelligent Sensing Systems16 of 47 2024 IEEE International Solid-State Circuits ConferenceCorrecting for predictable disturbancesGenerally well-behaved,stable defined actionsControl Systems often involves combinations of methodsISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems16 of

31、47 2024 IEEE International Solid-State Circuits ConferenceActuator(Electrical Stim)Algorithm(Threshold On/Off)Sensor(Local Field Potential)Physiologic Signal:“Mental Model”of OperationVerification andValidation StrategyClosed-Loop Control of Brain Rhythms for ParkinsonsISSCC 2024-Forum 4.1:Risk in I

32、ntelligent Sensing Systems17 of 47 2024 IEEE International Solid-State Circuits ConferenceDetails of the Implementation-Technology StackBouthour,W.,Mgevand,P.,Donoghue,J.et al.Biomarkers for closed-loop deep brain stimulation in Parkinson disease and beyond.Nat Rev Neurol 15,343352(2019).ISSCC 2024-

33、Forum 4.1:Risk in Intelligent Sensing Systems18 of 47 2024 IEEE International Solid-State Circuits ConferenceStack:Sensing Interface(Chopper stabilized)VinVoutRfChopped auxiliary pathwith pre-charge assistDC servo loop for high-pass filterCr=6pFLarge-signal CM cancellationCin/AcmCin/AcmCaCbCaCcRcCcR

34、cCin=1pFCin=1pFCf=50fFCf=50fFCDSL=100fFCDSL=100fFRfCint=12pFCint=12pFRintRintRBRBRL=2M CL=6pFRLRB2Cr=6pFCa=Cb=1pFAcm=2fcfcfcfcChopping freq fc=23.44kHzgm1gm2RLCLMulti-rate duty-cycled resistorHariprasad Chandrakumar,ISSCC 2017Electrode Polarization(input source DC)Avoid loading electrodes(Capacitive

35、 interface)Stimulation Artifacts(106 ptp relative to brain signal)ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems19 of 47 2024 IEEE International Solid-State Circuits ConferenceStack:Classification StageExtract physiological signals of interest(e.g.spectral power)Custom DSP block improves p

36、ower consumption Addition of machine learning subsystem for unsupervised learningChoice of feature selection can be criticalMuller,VLSI,2021,A 1.5nJ/cls Unsupervised Online Learning Classifier for Seizure Detection ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems20 of 47 2024 IEEE Internatio

37、nal Solid-State Circuits ConferenceStack:Algorithms used in PCLCs(Now,Future)Algorithms are often partitioned into two sub-components Classifier:Provide an estimate of the system state based on sensor data Control Policy:Determine next action based on current state,desired state and constraints Evol

38、ving algorithm complexity based on neuroscience and technology“Classical control”(proportional-integral-derivative)used in many PCLCs Model Predictive Control used for diabetes,anaesthesia*explored for brain But.trade-offs in complexity versus performance,constraints such as power(for now)ISSCC 2024

39、-Forum 4.1:Risk in Intelligent Sensing Systems21 of 47 2024 IEEE International Solid-State Circuits ConferenceFeedback Control “Minor Loops”Can also close feedback loops on sub-systemsCan improve the performance of a therapy without requiring“full”closed-loopBasis for many existing therapy technolog

40、ies!ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems22 of 47 2024 IEEE International Solid-State Circuits ConferenceAdjust stimulation to maintain a consistent evoked potentialCounteracts“disturbance”of the tissue electrode interface,broadlyHelps stabilize therapyClosed loop for“pain”?NoMino

41、r Loop Example of PCLCCorrecting for Actuator Variation Image courtesy Levy 2019,Evoke Study https:/doi.org/10.1111/ner.12932ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems23 of 47 2024 IEEE International Solid-State Circuits ConferenceControl Systems often involves combinations of methodsI

42、SSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems24 of 47 2024 IEEE International Solid-State Circuits ConferenceCan maintain a physiologic variable around a setpointNeed to understand variable,how it behaves,clinical correlationControl Systems often involves combinations of methodsISSCC 2024-

43、Forum 4.1:Risk in Intelligent Sensing Systems24 of 47 2024 IEEE International Solid-State Circuits Conference“Feedback”Control+Adaptive MethodsSystems are dynamic,algorithm mechanisms static?Rhythms:e.g.circadian,same therapy awake and sleeping?Therapy optimization(AI):explore the parameter space?IS

44、SCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems25 of 47 2024 IEEE International Solid-State Circuits ConferenceApply PCLC mindsetThe physiologic signal seems to disappear at nightBut,experience suggests benefit of stimulation at night!How to reconcile?Circadian Rhythms in Brain Dynamic System

45、sISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems26 of 47 2024 IEEE International Solid-State Circuits ConferenceApply PCLC mindset:Signal disappears at night?Make an adaptive controller that is“sleep aware”Time(24 hours)Patient TrialsConsider Slowly Varying Algorithms:“Gain Switching”ISSCC

46、2024-Forum 4.1:Risk in Intelligent Sensing Systems27 of 47 2024 IEEE International Solid-State Circuits ConferenceOptimization:Exploration“vs”ExploitationAvailable parameter spaceTension between exploiting“known”(140Hz,90us,xmA)and exploring what might be betterSystematic methods for managing this t

47、ension for example,Bayesian Optimization(graph below)PCLC infrastructure helps to manage these algorithms through adaptive algorithm controlFor example:Louie,K.H.et al.Semi-automated approaches to optimize deep brain stimulation parameters in Parkinsons disease.J NeuroEngineering Rehabil 18,83(2021)

48、.https:/doi.org/10.1186/s12984-021-00873-9“hidden reality”“estimate”“find area with high potential reward”Gather data,reassess next sample pointISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems28 of 47 2024 IEEE International Solid-State Circuits ConferenceMethods of Testing and VerificationM

49、achine Learning requires data for training as does exploration of new therapy conceptsCautionary tales about reliance on acute testing of closed-loops aka diurnal/circadian variation(beware!)Sampling bias?Motivates chronic algorithm-enabled PCLC research tools,combined with mathematical modelsGregg

50、N.Thalamic deep brain stimulation modulates cycles of seizure risk,in press Sci.Reports 2021ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems29 of 47 2024 IEEE International Solid-State Circuits ConferenceControl Systems often involves combinations of methodsISSCC 2024-Forum 4.1:Risk in Intel

51、ligent Sensing Systems30 of 47 2024 IEEE International Solid-State Circuits ConferenceControl Systems often involves combinations of methodsISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems30 of 47Compensate/reinforce for the natural rhythms of the bodyCoordinate how we explore the therapy sp

52、ace as community(scale!)2024 IEEE International Solid-State Circuits ConferenceCautionary Tales!Motivate the Need forRisk ManagementISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems31 of 47 2024 IEEE International Solid-State Circuits ConferenceArtifacts 1.0:Contamination of Biomarkers Albert

53、o Priori,Guglielmo Foffani,Lorenzo Rossi,Sara Marceglia,Adaptive deep brain stimulation(aDBS)controlled by local field potential oscillations,Experimental Neurology,2013(From Thakor NV:From Holter monitors to automatic defibrillators:Developments in ambulatory arrhythmia monitoring.IEEE Trans Biomed

54、 Eng 31:770,1984.)ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems32 of 47 2024 IEEE International Solid-State Circuits ConferenceArtifacts 2.0:Actuator Impacts Sensor or SystemPhilippe Ryvlin,Neuromodulation in epilepsy:state-of-the-art approved therapies,The Lancet Neurology,Volume 20,Issu

55、e 12,2021What about physiology?VNS for epilepsyDevice related?Anso,JNE,2022ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems33 of 47 2024 IEEE International Solid-State Circuits ConferenceDesign Considerations for Risk ManagementISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems34 of 47

56、 2024 IEEE International Solid-State Circuits ConferenceSafety!Risk Management for PCLCs(60601-1-10)ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems35 of 47 2024 IEEE International Solid-State Circuits ConferenceReview Bi-directional Brain Interface from Risk Management PerspectiveOpri et al

57、,Chronic embedded cortico-thalamic closed-loop deep brain stimulation for the treatment of essential tremor,Science Translational Medicine,2020ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems36 of 47 2024 IEEE International Solid-State Circuits ConferenceIn-depth Example:Research SystemIllus

58、trating Key Principles of Risk Management ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems37 of 47 2024 IEEE International Solid-State Circuits ConferenceFunctional Blocks:Enablement of the PCLC and risk mitg.ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems38 of 47 2024 IEEE Internat

59、ional Solid-State Circuits ConferenceFunctional implementation and system partitioningISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems39 of 47Investigational use only 2024 IEEE International Solid-State Circuits ConferencePractical PCLCs:Balancing Flexibility,Usability,SafetyNeurosurgeonNeur

60、ologistNeuralEngineer*Sample“Checklist”for Safe Design:Fallback Mode/Actuation LimitsClear Entrance/Exit CriteriaStim Constraints(Bounds)Data Logging of EventsAlarms and Alerts(Transparent)Safe Infrastructure is Enabler for“AI”Clinician Researcher System Set-upPatient PCLC Alerts&EngagementISSCC 202

61、4-Forum 4.1:Risk in Intelligent Sensing Systems40 of 47 2024 IEEE International Solid-State Circuits ConferenceComplementary Topics:Automated Flight ControlAutomated pitch control update;awareness of algorithm(e.g.mental model);single sensor mode;fallback modeAutomation Bias:in algorithms we trustCo

62、mplacency:less monitoring of technology because of belief in accuracyEntrance Criteria:what do we need to load into system before entering?Exit Criteria:information required when we exit?Loss of Situational Awareness:loss of state due to automationSkill Degradation:reduction of decision making abili

63、ty and skill decay if automation used most of the timeISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems41 of 47 2024 IEEE International Solid-State Circuits ConferenceComplementary Topics:Autonomous VehiclesStuck in the street;awareness of algorithm(e.g.mental model);single sensor mode;fallba

64、ck modeAutomation Bias:in algorithms we trustComplacency:less monitoring of technology because of belief in accuracyEntrance Criteria:what do we need to load into system before entering?Exit Criteria:information required when we exit?Loss of Situational Awareness:loss of state due to automationSkill

65、 Degradation:reduction of decision making ability and skill decay if automation used most of the timeISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems42 of 47 2024 IEEE International Solid-State Circuits ConferenceRisk Management Summary for Intelligent SensorsISSCC 2024-Forum 4.1:Risk in Int

66、elligent Sensing Systems43 of 47 2024 IEEE International Solid-State Circuits ConferenceA(General)Intelligent Sensing Safety FrameworkSensing-based systems require a complete system design mindsetAlgorithm how are adjustments determined?FF/FB/Adapt,combo?Sensors what are the relevant signals(above)?

67、Actuator how do you effect change in the target system?Control policySafety features how do we collectively limit risks?User interface intuitive but flexible+System verification and validation what is the required environment to confirm your algorithm is safe and effective?ISSCC 2024-Forum 4.1:Risk

68、in Intelligent Sensing Systems44 of 47 2024 IEEE International Solid-State Circuits ConferenceA(General)Safety Framework and ChecklistMental model of operation an intuitive explanation of principlesAlerts and logging how to convey states and log dataFallback modes what is state if algorithm goes awr

69、y;how to get there?Actuation limits how do set safe operating zone in natural environmentsSystem variation what are ranges of intra-and inter-patient variations,and impact?ISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems45 of 47 2024 IEEE International Solid-State Circuits ConferenceReview L

70、earning ObjectivesProvide a framework and common language drawn from physiologic closed-loop controllers(PCLCs)that can be extended to other domainsUse examples from multiple domains to provide intuition for PCLC operation,and Reinforce key concepts from FDA guidance documents and PCLC standards(606

71、01-1-10)for robust design and useNow:Empower you to analyse intelligent sensor-based systems with a clearer mental model of operation you can now assess core operation and identify risk mitigations and design your own system using intelligent(and now wise)sensing architecturesISSCC 2024-Forum 4.1:Ri

72、sk in Intelligent Sensing Systems46 of 47 2024 IEEE International Solid-State Circuits ConferenceReferencesISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems47 of 47Denison T,Litt B.Advancing neuromodulation through control systems:a general framework and case study in posture-responsive stimu

73、lation.Neuromodulation.2014 Jun;17 Suppl 1:48-57.doi:10.1111/ner.12170.PMID:24974775*Excellent summary:M.J.Khodaei,N.Candelino,A.Mehrvarz and N.Jalili,Physiological Closed-Loop Control(PCLC)Systems:Review of a Modern Frontier in Automation,in IEEE Access,vol.8,pp.23965-24005,2020,doi:10.1109/ACCESS.

74、2020.2968440.Gunduz et al,Adding Wisdom to Smart Medical Systems,Bioelectronics in Medicine 2019 2024 IEEE International Solid-State Circuits ConferenceISSCC 2024-Forum 4.1:Risk in Intelligent Sensing Systems48 of 16Please Scan to Rate Please Scan to Rate This PaperThis Paper 2024 IEEE International

75、 Solid-State Circuits ConferenceIntelligent Imager withProcessing-in-Sensor TechniquesISSCC Forum 4.2Chih-Cheng HsiehNational Tsing Hua University,Taiwan 2024 IEEE International Solid-State Circuits Conference 2024 IEEE International Solid-State Circuits Conference Introduction Spatial Domain Featur

76、e Extraction Temporal Domain Feature Extraction Challenges&Future Prospects ConclusionOutlineChih-Cheng HsiehISSCC Forum 4.22 2024 IEEE International Solid-State Circuits Conference Introduction Spatial Domain Feature Extraction Temporal Domain Feature Extraction Challenges&Future Prospects Conclusi

77、onOutlineChih-Cheng HsiehISSCC Forum 4.23 2024 IEEE International Solid-State Circuits ConferenceCIS market regain in the futureMobileAutomotiveCMOS Image Sensor Market PredictionChih-Cheng HsiehISSCC Forum 4.2Beyond cameras,diversified applications are blooming 4 2024 IEEE International Solid-State

78、 Circuits ConferenceFrom Centralized Cloud to Edge DevicesCentralized cloudHigh-level processing/complex taskTransmission cost(power&latency)User privacy concernsNetwork requirementEdge devicesLocal/real-time decision-makingLow/Mid-level processingPreserve data privacyPower-constrained environmentsC

79、hih-Cheng HsiehISSCC Forum 4.21 I.Kang,ISSCC 2022*UE:User Equipmentuplinkdown linkUE5G New Radio5G CoreEdge ServerPublic Cloud1ms100ms5 2024 IEEE International Solid-State Circuits ConferenceWhat is really meaningful?What?/Where?/Who?/When?More than the image itselfFeature extractionMachine visionSm

80、art Imager Exceed Human Eyes CapabilitiesChih-Cheng HsiehISSCC Forum 4.2PIS6 2024 IEEE International Solid-State Circuits ConferenceAdvantages of Processing-in-Sensor(PIS)Chih-Cheng HsiehISSCC 2024 Forum 4.27CMOS Image Sensor(CIS)PixelArray ADCControllerDigital ProcessorImage Transfer fpsImage Frame

81、 1Memory BankComplex ALU/MAC1243Image Frame 2Image Frame N4132Proposed CIS with PIS*TechnologyPixelArray ADC PIS*CircuitImage Transfer(Feature/ROI)*Processing-in-Sensor(PIS)CircuitControllerCNN ProcessorMemory BankALU/MACConventional imagerHigh-resolution ADCData transfer bottleneckHigh-complexity c

82、omputationHigh-capacity memoryIntelligent imager using PISPIS circuit for feature extractionRelieve ADC resolutionTransfer feature/ROI onlyRelieve memory capacity 2024 IEEE International Solid-State Circuits ConferenceArchitectures of Processing-in-Sensor(PIS)Near-sensor processingCapability&flexibi

83、lity Suffered latency/efficiencye.g.,classificationIn-pixel processingHigh parallelismSuffered FFe.g.,pre-processing Chih-Cheng HsiehIn-column processingIn-between solutionMedium performancee.g.,attention2 T.-H.Hsu,IEDM 2019ISSCC Forum 4.28 2024 IEEE International Solid-State Circuits Conference Int

84、roduction Spatial Domain Feature Extraction Temporal Domain Feature Extraction Challenges&Future Prospects ConclusionOutlineChih-Cheng HsiehISSCC Forum 4.29 2024 IEEE International Solid-State Circuits ConferenceFeature Extraction Spatial Domain InformationConceptStatic texture filteringCoarseness,c

85、ontrast,directionality,regularityPurposeExtract key characteristicsRemove redundant dataMethodologiesLocal binary pattern(LBP)Histogram of Oriented Gradients(HOG)Neural network(NN)ApplicationsFingerprint,retina,and face for forgery detectionBiomedical diagnosisEnvironment recognition,autonomous vehi

86、cleDefect detection,smart agricultureChih-Cheng HsiehISSCC Forum 4.2CapturedImageImageCollectionFeatureExtractionFeatureExtractionSimilarityMatchingShowResults10 2024 IEEE International Solid-State Circuits ConferenceConceptEncode the relationship between the central pixel and surrounding pixelsAppl

87、icationsTexture classificationObject recognitionProsComputational efficiencyImmune to illumination and rotationConsLimited global feature descriptionNoise sensitiveLocal Binary Pattern(LBP)Chih-Cheng HsiehISSCC Forum 4.236564LBP imageRaw dataOriginal imageLBP data101

88、11+=00024022212023Decimal dataRaw data174865873Decimal Value#of pixelsFeature histogram11 2024 IEEE International Solid-State Circuits ConferenceEnergy-efficient omnidirectional LBP extractionIn-column processing relaxed complexityDecentralized 4-pixel group computation reduced the computation load

89、Local Binary Pattern(LBP)Chih-Cheng HsiehISSCC Forum 4.2Raw ImageLBP ImageEdge Image3 X.Zhong,VLSI-C 201812 2024 IEEE International Solid-State Circuits ConferenceConceptCompute the distribution of gradient orientationsApplicationsPedestrian detectionHuman pose estimationProsImmune to illuminationCo

90、nsSensitivity to rotationHistogram of Oriented Gradients(HOG)Chih-Cheng HsiehISSCC Forum 4.2Detection Window8 cells16 cellsBlockCell2x2 cells8x8 pixels0 180 3456789Decimal ValuemagnitudeFeature histogram13 2024 IEEE International Solid-State Circuits ConferenceEnergy-efficient gradient ex

91、tractionIn-column processing relaxed complexityLog-gradient operation eliminates illumination-related informationMulti-scale object detectionHistogram of Oriented Gradients(HOG)Chih-Cheng HsiehISSCC Forum 4.2Raw Image2.75b Log-gradientHOG Image4 C.Young,JSSC 201914 2024 IEEE International Solid-Stat

92、e Circuits ConferenceConvolutional Neural Network(CNN)154FeatureExtractionFeatureInput Image Content of Image Color Texture Shape PositionConvolutionNeural Network(Filter descriptor Feature)2024 IEEE International Solid-State Circuits ConferenceConvolution:multiply-accumulate(MAC)operationMultiplica

93、tion:I=V*(1/R),Q=I*TAccumulation:Wired sum(I)Integration(Q)Rectifier linear activation(ReLU)Polarity judgement:ComparatorMaximum Pooling(MaxPool)Max finding:Winner-take-allSpatial Domain Image Processing16ReLUMACMaxPool 2024 IEEE International Solid-State Circuits ConferenceTernary-weight feature ex

94、tractionLogarithmic sensor Iph-V-I conversionMultiplication:ternary weight with current switchingAccumulation:wired sum Two-stage Viola-Jones cascade classifier for face region-of-interest detectionCIS with MAC OperationChih-Cheng HsiehISSCC Forum 4.25 M.Lefebvre,ISSCC 202117 2024 IEEE International

95、 Solid-State Circuits ConferenceProgrammable weight feature extractionPWM pixel Iph to pulse widthIDAC 3-bit weight to currentMAC:Low-power switch-current integrationConvolutional CMOS Image Sensor(C2IS)Chih-Cheng HsiehISSCC Forum 4.2Memory BankALU/MAC UnitsProposed CIS3-Row Rolling Access3333333Reg

96、fileMAC Input RegMAC ProcessingAccumulation RegData AccessEdgeSharpenCNN Processor-1-1-1-1-1-1-1 8-10-1 00-1 0-1 5-1Programmable KernelImage Transfer(Feature/ROI)Feature Match?Yes：awake CNN Processor No：standby Pixel Array128x128Row Control BiasRamp PWM Ramp GeneratorAnalog ConvolutionCircuitSS ADCC

97、olumn Control2.0 mm2.46 mmVerticalHorizontalEdge6 T.-H.Hsu,JSSC 202118 2024 IEEE International Solid-State Circuits ConferenceSystem Architecture of C2IS3 3 Sub-pixel Array(PWM Pixel)9 MultiplicationAccumulationAD Conversion9 Weighted CurrentChih-Cheng HsiehISSCC Forum 4.219 2024 IEEE International

98、Solid-State Circuits ConferencePWM pixel:convert voltage signal to pulse widthAnalog convolution:switch-current-integration(SCI)Switch-Current Integration(SCI)WeightOperationPositive(+)ADD enabledNegative(-)SUB enabledZero(0)ADD,SUB disableVPDVPWMCPMCNMRDPWPWPWVRRSELPIX_RSTRSELBPWPWPWCOLphWeight Rep

99、resentation(ZERO/SIGN)Chih-Cheng HsiehISSCC Forum 4.220 2024 IEEE International Solid-State Circuits ConferenceSCI design SPEC:8bitVSWING 0.45VVLSB=1.75mVTINT=5s(Max.TPW)Capacitor CP/NCurrent sourceRatio and Inoise,SCIComparatorIRN Vnoise,CMPSwitch-Current Integration(SCI)Chih-Cheng HsiehISSCC Forum

100、 4.221 2024 IEEE International Solid-State Circuits ConferenceCapacitor CP/N=1pF Over design for KTC noise requirement(64V)Capacitor:for better unit current designUnit current design(W*Iunit)(N*TPW)=1pF 0.45VWavg=4,Navg=5,TPW,avg=2.5sDesign Iunit=10nARatio error of weighted-currentRatio error:95%App

101、lication:Hand Gesture RecognitionFiveOneRockTwoZeroOthersChih-Cheng HsiehISSCC Forum 4.224 2024 IEEE International Solid-State Circuits ConferenceProgrammable weight feature extractionIn-column processing relaxed complexityCustomized tiny convolutional neural network to achieve face detection taskCI

102、S+Tiny CNN7 T.-H.Hsu,JSSC 2023Chih-Cheng HsiehISSCC Forum 4.225 2024 IEEE International Solid-State Circuits Conference3x3 Conv.+ReLUInput raw image:8-bitWeight:3-bit,(-8 +8)Output:8-bit2x2 MP Input:8-bitOutput:8-bit1x1 FC layerInput:8-bitWeight:1-bit,(-1,0,+1)Output 1bit(with applied threshold)Netw

103、ork ImplementationChih-Cheng HsiehISSCC Forum 4.226 2024 IEEE International Solid-State Circuits Conference128x128 arrayPWM PixelPIS circuitsConvolution+ReLUMaxPooling(MP)ADCFully-connected(FC)Peripheral Row/column controlConv.weight storageFC weight storage System ArchitectureChih-Cheng HsiehISSCC

104、Forum 4.227 2024 IEEE International Solid-State Circuits ConferenceReadout modeRaw imagesize:126x126Feature map(FM)size:21x21Classification size:1Data Processing FlowChih-Cheng HsiehISSCC Forum 4.228 2024 IEEE International Solid-State Circuits ConferenceData Volume Reductionvs.Pure CIS+CNN Processo

105、rReadout modeRaw imagesize:126x126Feature map(FM)size:21x21Classification size:1Chih-Cheng HsiehISSCC Forum 4.229 2024 IEEE International Solid-State Circuits ConferenceTSMC 0.18m 1P6M standard processChip size 2.18mm2.46mmPixel pitch 7.6mChip Micrograph&Image Mode Captured Raw ImageChih-Cheng Hsieh

106、ISSCC Forum 4.230 2024 IEEE International Solid-State Circuits ConferenceExperimental setupOff-chip model training using 10k images(face/non-face)Load trained weighting parametersValidation 5.4k(2670 face/2791 non-face)images auto playing on screenStatistical Analysis of Face DetectionChih-Cheng Hsi

107、ehISSCC Forum 4.231 2024 IEEE International Solid-State Circuits ConferenceMeasurement ResultChih-Cheng HsiehISSCC Forum 4.232 2024 IEEE International Solid-State Circuits ConferencePerformance ComparisonChih-Cheng HsiehISSCC Forum 4.233 2024 IEEE International Solid-State Circuits Conference Introd

108、uction Spatial Domain Feature Extraction Temporal Domain Feature Extraction Challenges&Future Prospects ConclusionOutlineChih-Cheng HsiehISSCC Forum 4.234 2024 IEEE International Solid-State Circuits ConferenceFeature Extraction Temporal Domain InformationConceptDetect temporal changes in each pixel

109、Report the level-difference image or locations of the triggered pixelsPurposeMotion detection between consecutive imagesRemove redundant dataMethodologiesEvent-based reporting:dynamic vision sensor(DVS)Frame-based reporting:frame differencing sensor(FDS)ApplicationsMotion detection 8,9Motion directi

110、on 10Saliency detection 11Dynamic depth sensing 9Temporal derivative 1213Chih-Cheng HsiehISSCC Forum 4.235 2024 IEEE International Solid-State Circuits ConferenceConceptReport event location by thresholding temporal changes per pixelArchitectureSensor:real-time logarithmic Iph-V conversionReadout:as

111、ynchronous x-y location reportApplicationsHigh-speed/high-dynamic-range Image deblurringEye-trackingObstacle avoidanceEvent Reporting(ER)UZH Robotics and Perception Group14 P.Lichtsteiner,JSSC 2008Chih-Cheng HsiehISSCC Forum 4.236 2024 IEEE International Solid-State Circuits ConferenceBuilding Block

112、s of ER SensortIPHtVSFtVCHANGEttONOFFPhotoreceptorConvert the photocurrent into voltageLogarithmic response Change amplifier(CC-PGA)Performs signal differencing and amplificationReset after event detectionComparatorsEvent thresholdingAER logicAddress-Event-Representation(AER):Asynchronous-event-repo

113、rting control and readoutChih-Cheng HsiehISSCC Forum 4.237In-Pixel 2024 IEEE International Solid-State Circuits ConferencePixel pitch reduction via 3D stackingRaw image+event reportingBuilt-in event signal processorAnti-flickerAuto-thresholdEvent frame readoutBetter compatibility with DNN enginesRec

114、ent ER Works 15 K.Kodama,ISSCC 202316 M.Guo,ISSCC 202317 A.Niwa,ISSCC 2023Chih-Cheng HsiehISSCC Forum 4.238 2024 IEEE International Solid-State Circuits ConferenceConceptReport temporal level difference or thresholding event of two consecutive framesArchitectureSensor:integrating Iph-V conversionRea

115、dout:Synchronous frame reportApplicationsSaliency detectionMotion detectionMotion direction detectionDynamic depth sensingFrame Differencing(FD)Chih-Cheng HsiehISSCC Forum 4.239Frame 2IncreasedIntensity?ON?DecreasedIntensity?OFF?Frame DifferenceFrame 2 Frame 1No Intensity ChangeFrame 1In-pixel/Colum

116、nThresholding 2024 IEEE International Solid-State Circuits ConferenceColumn-Level FDColumn-wise processing for FD and background subtraction(BS)Need pixel reconfiguration(22)for FD/BS operationsLarge and complicated column PEsNeed in-pixel caps(memory)for FD and off-chip SRAM for BSColumn PE8 X.Zhon

117、g,JSSC 2020Chih-Cheng HsiehISSCC Forum 4.240 2024 IEEE International Solid-State Circuits ConferenceColumn-Level FD+Temporal DerivativeColumn-wise frame difference processing and event detectionUsing overlapping exposure of even-odd pixels without in-pixel storage capacitorTemporal derivative output

118、 using event-reset discharging operation Pixel patch(3x3)FD operation complicated&reduced resolution&mismatch12 D.W.Jee,ESSCIRC 2023Chih-Cheng HsiehISSCC Forum 4.241 2024 IEEE International Solid-State Circuits ConferencePixel-Level FDIn-pixel consecutive frame differencingPing-pong pixel to elimina

119、te parasitic mismatch effectSaliency detection using grouping event summationNeed 10T+2C(in-pixel storage capacitors)11 T.-H.Hsu,JSSC 2021Chih-Cheng HsiehISSCC Forum 4.242 2024 IEEE International Solid-State Circuits ConferenceImproved Pixel-Level FD Improved in-pixel consecutive frame differencingO

120、nly 6T+1C with improved pixel pitch,fill factor,conversion gainSupport on-chip LBP&ROI reporting Complicated pixel control,requires exposure compensation scheme(ECS)9 M.-Y.Chiu,JSSC 2023Chih-Cheng HsiehISSCC Forum 4.243 2024 IEEE International Solid-State Circuits ConferenceSensing ER:real-time loga

121、rithmic Iph-V conversion with non-linear HDR response,need in-pixel amplification FD:integrating Iph-V conversion with a linear response,inherent gain from exposure.ReadoutER:Asynchronous x-y event location reporting of thresholding temporal differenceFD:Synchronous frame reporting of temporal level

122、 difference or thresholding event of two consecutive frames Specifications to be considered:Event sensitivity,Dynamic range,Speed,Compatibility with processorsEvent Reporting(ER)vs.Frame Differencing(FD)DAVIS 240C Event Output from 18Accumulation Time=38.8 msSimulated Frame DifferenceChih-Cheng Hsie

123、hISSCC Forum 4.244 2024 IEEE International Solid-State Circuits ConferenceEvent Detection:High-Illumination SceneInput VideoSimulated DVS Event FrameTACC=33.3msSimulated Frame DifferenceLowContrastRegion(High Illumination)Event Reporting:Event detection with current ratioFrame Differencing:Event det

124、ection with current differenceFD shows a better event sensitivity in high-illumination low-contrast scene,Chih-Cheng HsiehISSCC Forum 4.245 Chih-Cheng HsiehISSCC Forum 4.246 2024 IEEE International Solid-State Circuits ConferenceHigh-illumination scene(H)ER:Log-response=ER_HFD:Low-gain=LG_HFD:High-g

125、ain=HG_H(ER_H)(HG_H)Low-illumination scene(L)ER:Log-response=ER_LFD:Low-gain=LG_LFD:High-gain=HG_L(ER_L)(LG_L)(LG_L)VTHS27 of 47 2024 IEEE International Solid-State Circuits ConferenceDesigns use mixed-signal or digital circuitsDesigns need to handle irregular memory access&data flowSNNs have been p

126、aired with sensor front endsSNN Accelerators(1)Shih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory SystemsS.Esser et al.,PNAS,2016S.Furber et al.,JNE,201628 of 47 2024 IEEE International Solid-State Circuits ConferenceSNN Accelerators(2)Shih-Chii LiuISSCC 2024 F

127、orum 4.3:Activity-Driven Perception for Intelligent Edge Sensory SystemsD.Wang et al.,ASSCC,2020D.Wang et al.,ISSCC,2021Asynchronous digital SNN acceleratorActivity-driven clock generation with fine-grained clock-&power-gatingBinary synaptic weight”multiplication”using SRAMSub-W end-to-end KWS when

128、paired with an activity-driven sensor29 of 47 2024 IEEE International Solid-State Circuits ConferenceSNN Accelerators(3)Shih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory SystemsY.Liu et al.,ISSCC,2022Asynchronous mixed-signal SNN acceleratorCapacitor as silico

129、n analogy of biological neuron membraneMulti-bit synaptic”multiplication”using in-memory computing(IMC)techniques30 of 47 2024 IEEE International Solid-State Circuits ConferenceTrain ANNs for sparse delta activationsBenefits:Uses common ANN training techniquesOnly“active”neurons propagate updatesTem

130、poral sparsity leads to energy reductionActivity-Driven Temporal Sparsity NetworksShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory SystemsTimeFrame-driven ANNEvent-driven ANNTimeInspiredD.Neil et al.,ICML,201731 of 47 2024 IEEE International Solid-State Circu

131、its ConferenceTemporal Sparsity Networks Delta NetworksShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory SystemsS-C.Liu et al.,IEDM,2022C.Gao et al.,TNNLS,June 20221L-1024 H LSTM RNN trained on TIMITImplementation on Zynq-7100Efficiency from Batch-1 Inference3

132、2 of 47 2024 IEEE International Solid-State Circuits ConferenceGating on helps to produce sparse activation in ANNs(1)Gating the activation changes can increase sparsity(2)Activity Sparsity in Delta NetworksShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory Sys

133、tems+1+11133 of 47(1)(2)2024 IEEE International Solid-State Circuits ConferenceSpatial+temporal sparsityCBTD-based pruning1.1 TOPS/WDelta Network AcceleratorsShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory SystemsC.Gao et al.,TNNLS,June 202234 of 47Temporal

134、sparsity1.2 TOPS effective throughput218 GOPS/WC.Gao et al.,FPGA,2018 2024 IEEE International Solid-State Circuits ConferenceMore Possibilities of Activity-Driven ComputingShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory Systems1.5 W integrated KWS SoC with A

135、FE+FE+NNAdaptive frame-skipping using Skip RNNExploits frame-level activity sparsityJ.-H.Seol et al.,ISSCC,202335 of 47 2024 IEEE International Solid-State Circuits ConferenceOutline Introduction&motivation Activity-driven sensing Activity-driven computing Hierarchical task-gating Conclusions&outloo

136、kShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory Systems36 of 47 2024 IEEE International Solid-State Circuits ConferenceEstimated using the power efficiency of DNPU(1.1 TOPS/W)System power not includedExternal memory access ignoredPower Gating for Complex Ta

137、sksShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory SystemsD.Shin et al.,ISSCC 2017S-C.Liu et al.,IEDM,202237 of 47 2024 IEEE International Solid-State Circuits ConferenceSD KWS SVCoarse-grained gatingHierarchical Task-GatingShih-Chii LiuISSCC 2024 Forum 4.3:

138、Activity-Driven Perception for Intelligent Edge Sensory SystemsJ.Giraldo et al.,JSSC,Apr.2020Z.Wang et al.,ISSCC,2021General-purpose wake-upFine-grained event-driven compute38 of 47 2024 IEEE International Solid-State Circuits ConferenceOutline Introduction&motivation Activity-driven sensing Activit

139、y-driven computing Hierarchical task-gating Conclusions&questionsShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory Systems39 of 47 2024 IEEE International Solid-State Circuits ConferenceDynamic activity sparsity in sensory intelligent systems provides many opp

140、ortunities for improving energy efficiency beyond static model compressionActivity-driven perception exploits dynamic activity sparsity at all abstraction levels;from low-level sensor output to high-level task relevanceUnlocking full potential of activity-driven perception requires hardware and soft

141、ware innovations in activity-driven sensing,activity-driven computing and hierarchical task-gatingActivity-driven perception will benefit from task-gatingConclusionsShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory Systems40 of 47 2024 IEEE International Solid

142、-State Circuits ConferenceActivity-driven readout can be applied to other high-resolution sensor systemsSonar,EEG,etc.Heterogenous computing blocks combining gated synchronous and asynchronous domainsIn-memory computing(IMC)architectures for local processingTask-gating architectures for perceptive s

143、ystemsOutlookShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory Systems41 of 47 2024 IEEE International Solid-State Circuits ConferenceWhat is the hardware-software co-design needed for dynamic activity processing?What is the overhead cost for hierarchical task

144、-gating systems?How can IMC support activity-driven computing?Questions for ForumShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory SystemsJ.Giraldo et al.,JSSC,Apr.2020Z.Wang et al.,ISSCC,202142 of 47 2024 IEEE International Solid-State Circuits ConferenceTobi

145、 Delbruck,Sheng Zhou,Kwantae Kim,Chang Gao(Sensors Group,UZH)AcknowledgementsShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory Systems43 of 47 2024 IEEE International Solid-State Circuits ConferenceP.Lichtsteiner et al.,A 128128 120dB 15s Latency Asynchronous

146、Temporal Contrast Vision Sensor,JSSC,Feb.2008,pp.566-576.C.Brandli et al.,A 240180 130dB 3s Latency Global Shutter Spatiotemporal Vision Sensor,JSSC,Oct.2014,pp.2333-2341.C.Li et al.,Design of an RGBW Color VGA Rolling and Global Shutter Dynamic and Active-Pixel Vision Sensor,ISCAS,2015.B.Son et al.

147、,A 640480 Dynamic Vision Sensor with a 9m Pixel and 300Meps Address-Event Representation,ISSCC 2017,paper 4.1.T.Finateu et al.,A 1280720 Back-Illuminated Stacked Temporal Contrast Event-Based Vision Sensor with 4.86m Pixels,1.066GEPS Readout,Programmable Event-Rate Controller and Compressive Data-Fo

148、rmatting Pipeline,ISSCC 2020,paper 5.10.M.Guo et al.,A 3-Wafer-Stacked Hybrid 15MPixel CIS+1MPixel EVS with 4.6GEvent/s Readout,In-Pixel TDC and On-Chip ISP and ESP Function,ISSCC 2023,paper 5.1.K.Kodama et al.,1.22m 35.6Mpixel RGB Hybrid Event-Based Vision Sensor with 4.88m-Pitch Event Pixels and u

149、p to 10K Event Frame Rate by Adaptive Control on Event Sparsity,ISSCC 2023,paper 5.2.A.Niwa et al.,A 2.97m-Pitch Event-Based Vision Sensor with Shared Pixel Front-End Circuitry and Low-Noise Intensity Readout Mode,ISSCC 2023,paper 5.3.S.-C.Liu et al.,Asynchronous Binaural Spatial Audition Sensor Wit

150、h 2644 Channel Output,”Transactions on Biomedical Circuits and Systems,Aug.2014,pp.453-464.M.Yang et al,A 0.5V 55W 64x2-Channel Binaural Silicon Cochlea for Event-Driven Stereo-Audio Sensing,ISSCC 2016,paper 22.5.Expanded version in JSSC,Sep.2016,pp.25542569.M.Yang et al.,Nanowatt Acoustic Inference

151、 Sensing Exploiting Nonlinear Analog Feature Extraction,JSSC,Oct.2021,pp.3123-3133.W.-W.Lee et al.,”A Neuro-Inspired Artificial Peripheral Nervous System for Scalable Electronic Skins,”Sci.Robot.,July 2019.References Activity-Driven SensorsShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception

152、 for Intelligent Edge Sensory Systems44 of 47 2024 IEEE International Solid-State Circuits ConferenceS.-C.Liu et al.,Energy-Efficient Activity-Driven Computing Architectures for Edge Intelligence,IEDM 2022.M.Yang et al.,A 1W Voice Activity Detector using Analog Feature Extraction and Digital Deep Ne

153、ural Network,ISSCC 2018,paper 21.2.Expanded version in JSSC,June 2019,pp.1764-1777.D.Wang et al.,A Background-Noise and Process-Variation-Tolerant 109nW Acoustic Feature Extractor Based on Spike-Domain Divisive-Energy Normalization for an Always-On Keyword Spotting Device,ISSCC 2021,paper 9.9.K.Kim

154、et al.,A 23W Solar-Powered Keyword-Spotting ASIC with Ring-Oscillator-Based Time-Domain Feature Extraction,ISSCC 2022,paper 22.6.Expanded version in JSSC,Nov.2022,pp.3298-3311.J.-H.Seol et al.,A 1.5W End-to-End Keyword Spotting SoC with Content-Adaptive Frame Sub-Sampling and Fast-Settling Analog Fr

155、ontend,ISSCC 2023,paper 29.6.M.Cho et al.,A 142nW Voice and Acoustic Activity Detection Chip for mm-Scale Sensor Nodes Using Time-Interleaved Mixer-Based Frequency Scanning,ISSCC 2019,paper 17.2.Expanded version in JSSC,Nov.2019,pp.3005-3016.S.Jeong et al.,A 12nW Always-on Acoustic Sensing and Objec

156、t Recognition Microsystem Using Frequency-Domain Feature Extraction and SVM Classification,ISSCC 2017,paper 21.6.Expanded version in JSSC,Jan.2018,pp.261-274.W.Shan et al.,AAD-KWS:A Sub-W Keyword Spotting Chip with an Acoustic Activity Detector Embedded in MFCC and a Tunable Detection Window in 28nm

157、 CMOS,JSSC,Mar.2023,pp.867-876.K.Badami et al.,Context-Aware Hierarchical Information-Sensing in a 6W 90nm CMOS Voice Activity Detector,ISSCC 2015,paper 24.2.Expanded version in JSSC,Jan.2016,pp.291-302.J.Giraldo et al.,Vocell:A 65nm Speech-Triggered Wake-Up SoC for 10W Keyword Spotting and Speaker

158、Verification,JSSC,Apr.2020,pp.868-878.Piotr Dudek et al.,Sensor-Level Computer Vision With Pixel Processor Arrays for Agile Robots,Sci.Robot.June 2022.References Intelligent Sensory SystemsShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory Systems45 of 47 2024

159、IEEE International Solid-State Circuits ConferenceT.-H.Hsu et al.,AI Edge Devices Using Computing-In-Memory and Processing-In-Sensor:From System to Device,IEDM 2019.T.-H.Hsu et al.,A 0.8V Intelligent Vision Sensor with Tiny Convolutional Neural Network and Programmable Weights Using Mixed-Mode Proce

160、ssing-in-Sensor Technique for Image Classification,ISSCC 2022,paper 15.9.Expanded version in JSSC,Nov.2023,pp.3266-3274.Z.Wang et al.,A 148nW General-Purpose Event-Driven Intelligent Wake-Up Chip for AIoT Devices Using Asynchronous Spike-Based Feature Extractor and Convolutional Neural Network,ISSCC

161、 2021,paper 12.1.Expanded version in JSSC,Nov.2021,pp.3274-3288.References Intelligent Sensory SystemsShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory Systems46 of 47 2024 IEEE International Solid-State Circuits ConferenceS.-C.Liu et al.,Event-Driven Sensing

162、for Efficient Perception:Vision and Audition Algorithms,Signal Processing Magazine,Nov.2019,pp.29-37.V.Sze et al.,Efficient Processing of Deep Neural Networks:A Tutorial and Survey,Proceedings of the IEEE,Dec.2017,pp.2295-2329.Y.Liu et al.,An 82nW 0.53pJ/SOP Clock-Free Spiking Neural Network with 40

163、s Latency for AloT Wake-Up Functions Using Ultimate-Event-Driven Bionic Architecture and Computing-in-Memory Technique,ISSCC 2022,paper 22.7.Expanded version in TCAS I,Aug.2023,pp.3075-3088.C.Frenkel and G.Indiveri,ReckOn:A 28nm Sub-mm2Task-Agnostic Spiking Recurrent Neural Network Processor Enablin

164、g On-Chip Learning over Second-Long Timescales,ISSCC 2022.G.Bellec et al.,”A solution to the learning dilemma for recurrent networks of spiking neurons,”Nature Communications,July 2020.D.Wang et al.,Always-On,Sub-300-nW,Event-Driven Spiking Neural Network based on Spike-Driven Clock-Generation and C

165、lock-and Power-Gating for an Ultra-Low-Power Intelligent Device,ASSCC 2020.D.Neil et al.,Delta networks for optimized recurrent network computation.ICML 2017.C.Gao et al.,DeltaRNN:A Power-Efficient Recurrent Neural Network Accelerator,FPGA 2018.C.Gao et al.,EdgeDRNN:Enabling Low-latency Recurrent Ne

166、ural Network Edge Inference,AICAS 2020.C.Gao et al.,Spartus:A 9.4 TOp/s FPGA-Based LSTM Accelerator Exploiting Spatio-Temporal Sparsity,TNNLS 2022.A.Aimar et al.,NullHop:A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps,TNNLS,Mar.2019,pp.644-656.J.-S

167、.Park et al.,A 6K-MAC Feature-Map-Sparsity-Aware Neural Processing Unit in 5nm Flagship Mobile SoC,ISSCC 2021,paper 9.5.D.Shin et al.,DNPU:An 8.1TOPS/W reconfigurable CNN-RNN processor for general-purpose deep neural networks,ISSCC 2017,paper 14.2.D.Shin et al.,The Heterogeneous Deep Neural Network

168、Processor With a Non-von Neumann Architecture,Proceedings of the IEEE,Aug.2020,pp.1245-1260.References Algorithms&AcceleratorsShih-Chii LiuISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory Systems47 of 47 2024 IEEE International Solid-State Circuits ConferenceShih-Chii Liu

169、ISSCC 2024 Forum 4.3:Activity-Driven Perception for Intelligent Edge Sensory Systems48 of 47Please Scan to Rate Please Scan to Rate This PaperThis Paper 2024 IEEE International Solid-State Circuits ConferenceEvent-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computi

170、ng for Distributed Adaptive Intelligence at the EdgeGert CauwenberghsUC San DiegoForum 4:Intelligent SensingIEEE ISSCC 2024Gert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the

171、 Edge1 of 39 2024 IEEE International Solid-State Circuits ConferenceOutlineIntroduction and motivationPushing the analog-digital boundary in sensing and computingNeuromorphic bioinspired designEvent-driven sensory analog processingProcessing gain for overall improvements in performance and efficienc

172、yAnalog-to-information conversion at the sensory interfaceMassively parallel mixed-signal in-memory computingLearning and inference in memoryDistributed adaptive intelligence at the edgeConcluding remarksTake-away lessons in analog and mixed-signal designGert CauwenberghsISSCC 2024-Forum 4.4:Event-D

173、riven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge2 of 39 2024 IEEE International Solid-State Circuits ConferenceOutlineIntroduction and motivationPushing the analog-digital boundary in sensing and computingNeurom

174、orphic bioinspired designEvent-driven sensory analog processingProcessing gain for overall improvements in performance and efficiencyAnalog-to-information conversion at the sensory interfaceMassively parallel mixed-signal in-memory computingLearning and inference in memoryDistributed adaptive intell

175、igence at the edgeConcluding remarksTake-away lessons in analog and mixed-signal designGert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge3 of 39 2024 IEEE International

176、 Solid-State Circuits ConferenceConventional Digital Sensory ProcessingThe prevailing DSP mantra:“Convert the analog signal to digital at the earliest opportunity!”Pros:General-purpose programmability,fully software reconfigurable.Potentially high precision,limited by analog front-end and A/D resolu

177、tion.Cons:High power!Digital compute is cheap,but A/D at full resolution and Nyquist bandwidth is extremely expensive.A/DDSPSensorsAnalogDigitalGert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptiv

178、e Intelligence at the Edge4 of 39 2024 IEEE International Solid-State Circuits ConferenceAnalog and Mixed-Signal Sensory ProcessingAnalog Sensory Processing(ASP)paradigm:“Process the analog signal at full depth to extract and convert only those low-dimensional,low-bandwidth features of interest down

179、 the digital processing pipeline.”“Smart”A/D:direct analog-to-information conversion of the signal of interest.Pros:Processing gain:high-rate analog data compression at minimum information loss.Low power:substantial reduction in bandwidth and resolution of A/D.Cons:Limited hardware configurabilityAS

180、PA/DSensorsDigitalAnalogGert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge5 of 39 2024 IEEE International Solid-State Circuits ConferenceEvent-Driven Reconfigurable Sen

181、sory ProcessingData drivenAER:AddressEvent RepresentationCommunication bandwidth adjusts to information bandwidth in the signalAsynchronousNo quantization(binning)of timeNo power-hungry clocks and synchronization across network nodesHighly energy efficientSignificant energy savings over Nyquist samp

182、ling for signals of sparse activity and medium amplitude resolutionRobust to additive noise in the signalASPA/DSensorsAsynchronous DigitalContinuous-Time AnalogADSPGert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for

183、 Distributed Adaptive Intelligence at the Edge6 of 39 2024 IEEE International Solid-State Circuits ConferenceOutlineIntroduction and motivationPushing the analog-digital boundary in sensing and computingNeuromorphic bioinspired designEvent-driven sensory analog processingProcessing gain for overall

184、improvements in performance and efficiencyAnalog-to-information conversion at the sensory interfaceMassively parallel mixed-signal in-memory computingLearning and inference in memoryDistributed adaptive intelligence at the edgeConcluding remarksTake-away lessons in analog and mixed-signal designGert

185、 CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge7 of 39 2024 IEEE International Solid-State Circuits ConferenceEvent-Driven Reconfigurable Sensory ProcessingAsynchronous

186、routing of sensory address eventsExpandable integration of sensory modalitiesReconfigurable adaptive general-purpose processing and identificationSpatiotemporal sensory event stream encoder Time-frequency sensory event stream encoder Event stream transceiver and decoderTDMAAsynchronous BusMic,Ultras

187、onic transducer,Strain gauge,Imager,Touch screen,AER time eventsAER time eventsAER time eventsClassification eventse.g.anomaly detection,wake-up receivere.g.filterbank zero-crossingse.g.spatial/temporal gradient threshold detectionSensor fusion,feature binding,and adaptive pattern recognitionASPA/DS

188、ensorAsynchronous DigitalContinuous-Time AnalogADSPGert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge8 of 39 2024 IEEE International Solid-State Circuits ConferenceEven

189、t-Driven Reconfigurable Sensory ProcessingAsynchronous routing of sensory address eventsExpandable integration of sensory modalitiesReconfigurable adaptive general-purpose processing and identificationSpatiotemporal sensory event stream encoder Time-frequency sensory event stream encoder Event strea

190、m transceiver and decoderTDMAAsynchronous BusMic,Ultrasonic transducer,Strain gauge,Imager,Touch screen,AER time eventsAER time eventsAER time eventsClassification eventse.g.anomaly detection,wake-up receivere.g.filterbank zero-crossingse.g.spatial/temporal gradient threshold detectionSensor fusion,

191、feature binding,and adaptive pattern recognitionASPA/DSensorAsynchronous DigitalContinuous-Time AnalogADSPGert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge9 of 39 2024

192、 IEEE International Solid-State Circuits ConferenceEvent-Coding Silicon RetinaModels coding and communication of visual events in the mammalian retina and optic nerve Integrated photosensors(rods)On and off transient and sustained ganglia cell outputsSpatiotemporal compressed coding and communicatio

193、n in optic nerveAddress-event coding of spikesZaghloul and Boahen,2006Gert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge10 of 39 2024 IEEE International Solid-State Cir

194、cuits ConferenceDynamic Vision SensingDynamic Vision Sensors(DVS)aka Event CamerasVisual temporal change event detection on the focal plane.High dynamic range,high temporal resolution.Event rate and power scale with temporal activity.Lichtsteiner,Poch,and Delbruck,2008Gert CauwenberghsISSCC 2024-For

195、um 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge11 of 39 2024 IEEE International Solid-State Circuits ConferenceActive Pixel vs Dynamic Vision SensorsAsynchronous event-driven dynamic vision sensor

196、s offer greatest event bandwidth,at the expense of silicon area and always-on power.Synchronous query-driven DVS alternatives strike a balance between density,efficiency,and event bandwidth.VPHPREAMPVONVOFFALWAYS ON LOCAL BIASESAER HANDSHAKEVINON EVENTOFF EVENTCOLUMN ARBITERROW ARBITERRREQRACKCREQCA

197、CKRESETCSTAPS FIXEDt.tt0t1t2t3.(x0,y0)(x1,y1)(x2,y2)(x3,y3)TeDVS VARIABLERSRESETSENSEVDDVPHBUFFERVDDADCCOLUMN SHAREDGLOBAL BIASRSBIASVREFRESETSENSEVDDVPHVINCPH CREF ATIACOLUMN SHAREDGLOBAL BIASACTIVE PIXEL SENSOR(APS)EVENT DRIVEN DYNAMIC VISION SENSOR(eDVS)QUERY DRIVEN DYNAMIC VISION SENSOR(qDVS)IMA

198、GE ACQUISITION METHODPIXEL DESIGNTqDVS FIXEDt.Gert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge12 of 39 2024 IEEE International Solid-State Circuits ConferenceActive P

199、ixel vs Dynamic Vision SensorsAsynchronous event-driven dynamic vision sensors offer greatest event bandwidth,at the expense of silicon area and always-on power.Synchronous query-driven DVS alternatives strike a balance between density,efficiency,and event bandwidth.PARAMETERAPSeDVSqDVSLATENCYHIGHES

200、T(upto 100fps 10ms)LOWEST(frameless 10s)MEDIUM AND VARIABLE(upto 1000fps 1ms)BANDWIDTHUTILIZATIONHIGHEST Activity independent(High resolution,high speed ADC)MEDIUMActivity dependent(Pixel address+2 bit event+REQ/ACK)LOWESTActivity dependent(2 bit event)DENSITYHIGHEST(3T-5T/pixel)LOWEST(10T/pixel)HIG

201、H(7T-10T/pixel)POWERHIGH DYNAMICMEDIUM STATIC(100-500 nW per pixel)MEDIUM DYNAMICHIGH STATIC(80-500 nW per pixel)LOW DYNAMICLOW STATIC(1,000 TMACS/W efficiencyEMAC=lHCC V21fJ 10%10fF 1VJoshi et al,2017,2019Jiang et al,2020Hur et al,2022CijCij+1Ci+1jCi+1j+1Ci+1j-1Cij-1Ci-1j-1Ci-1jCi-1j+1Gert Cauwenbe

202、rghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge30 of 39 2024 IEEE International Solid-State Circuits ConferenceCharge-Based Compute-in-Memory1 fJ/MAC in 0.25 m CMOSAdiabatic reso

203、nant clocking conserves charge energyEnergy efficiency on par with human brain(1015SynOP/S at 15W)100 aJ/MAC in 22 nm FDSOI CMOS10 super-humanKarakiewicz et al,2007,2012Gert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computin

204、g for Distributed Adaptive Intelligence at the Edge31 of 39 2024 IEEE International Solid-State Circuits ConferenceAdiabatic Energy RecoveryDynamic energy dissipationC V2irreversibly lost every cycleHot clock recycles C V2 energyLC tank resonant clockReversible computationEnergy recovery logic(ERL)M

205、oon,1996CMOS logicGert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge32 of 39 2024 IEEE International Solid-State Circuits ConferenceResonant Adiabatic Energy RecoveryCh

206、arge-domain processing-in-memory computing at 1 fJ of energy per multiply-accumulate.Energy recovery logic(ERL)adiabatic line drivers recover 98%of the CV2electrostatic energy in the charge-mode array.resonance(capacitive load)resonancecapacitive loadKarakiewicz et al,2007,2012Gert CauwenberghsISSCC

207、 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge33 of 39 2024 IEEE International Solid-State Circuits ConferenceResonant Adiabatic Energy RecoveryResonant adiabatic energy recovery for wir

208、eless power and telemetry.Cyclic On-Off Keying(COOK)modulation offers full bandwidth efficiency,allowing to transmit one bit of data every carrier cycle while simultaneously receiving RF power over the same high-Q inductive link.Ha et al,2016Gert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory

209、 Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge34 of 39 2024 IEEE International Solid-State Circuits ConferenceOutlineIntroduction and motivationPushing the analog-digital boundary in sensing and computingNeuromorphic bioin

210、spired designEvent-driven sensory analog processingProcessing gain for overall improvements in performance and efficiencyAnalog-to-information conversion at the sensory interfaceMassively parallel mixed-signal in-memory computingLearning and inference in memoryDistributed adaptive intelligence at th

211、e edgeConcluding remarksTake-away lessons in analog and mixed-signal designGert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge35 of 39 2024 IEEE International Solid-Stat

212、e Circuits ConferenceOutlineIntroduction and motivationPushing the analog-digital boundary in sensing and computingNeuromorphic bioinspired designEvent-driven sensory analog processingProcessing gain for overall improvements in performance and efficiencyAnalog-to-information conversion at the sensor

213、y interfaceMassively parallel mixed-signal in-memory computingLearning and inference in memoryDistributed adaptive intelligence at the edgeConcluding remarksTake-away lessons in analog and mixed-signal designNever collect data that youve already got!Never close a switch across non-zero voltage/charg

214、e!Mismatch and noise arent your enemies when youre adaptive!Gert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge36 of 39 2024 IEEE International Solid-State Circuits Conf

215、erenceReferences(1/3)S.Bongki et al.,“A 640480 Dynamic Vision Sensor with a 9m Pixel and 300Meps Address-Event Representation,”IEEE ISSCC,2017.S.Chen et al.,“Live demonstration:CeleX-V:a 1M Pixel Multi-Mode Event-Based Sensor,”IEEE CVPR Workshop,2019.Y.M.Chi,U.Mallik,M.Clapp,E.Choi,G.Cauwenberghs an

216、d R.Etienne-Cummings,“CMOS Camera with In-Pixel Temporal Change Detection and ADC,”IEEE J.Solid-State Circuits,vol.42(10),pp.2187-2196,2007.G.Detorakis,S.Sheik,C.Augustine,S.Paul,B.U.Pedroni,N.Dutt,J.Krichmar,G.Cauwenberghs,and E.Neftci,“Neural and Synaptic Array Transceiver:A Brain-Inspired Computi

217、ng Framework for Embedded Learning,”Frontiers in Neuroscience,vol.12,pp.583:1-19,2018.T.Finateu et al.,“A 1280 720 Back-Illuminated Stacked Temporal Contrast Event-Based Vision Sensor with 4.86 m Pixels,1.066 GEPS Readout,Programmable Event-Rate Controller and Compressive Data-Formatting Pipeline,”I

218、EEE ISSCC,2020.S.Ha,C.Kim,J.Park,S.Joshi and G.Cauwenberghs,“Energy Recycling Telemetry IC with Simultaneous 11.5 mW Power and 6.78 Mb/s Backward Data Delivery Over a Single 13.56 MHz Inductive Link,”IEEE J.Solid-State Circuits,vol.51(11),pp.2664-2678,2016.J.Hur,Y.Luo,A.Lu,T.Wang,S.Li,A.I.Khan and S

219、.Yu,“Nonvolatile Capacitive Crossbar Array for In-Memory Computing,”Advanced Intelligent Systems,vol.4(2100258),2022.Z.Jiang,S.Yin,J.-S.Seo and M.Seok,“C3SRAM:An In-Memory-Computing SRAM Macro Based on Robust Capacitive Coupling Computing Mechanism,”IEEE J.Solid-State Circuits,vol.55(7),pp.1888-1897

220、,2020.Gert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge37 of 39 2024 IEEE International Solid-State Circuits ConferenceReferences(2/3)S.Joshi,C.Kim,S.Ha,Y.M.Chi and G.

221、Cauwenberghs,“21.7 2pJ/MAC 14b 88 linear transform mixed-signal spatial filter in 65nm CMOS with 84dB interference suppression,”IEEE ISSCC 2017.S.Joshi,C.Kim,C.M.Thomas and G.Cauwenberghs,“Digitally Adaptive High-Fidelity Analog Array Signal Processing Resilient to Capacitive Multiplying DAC Inter-S

222、tage Gain Error,”IEEE T.Circuits and Systems I:Regular Papers,vol.66,(11),pp.4095-4107,2019.R.Karakiewicz,R.Genov,and G.Cauwenberghs,“480-GMACS/mW Resonant Adiabatic Mixed-Signal Processor Array for Charge-Based Pattern Recognition,”IEEE J.Solid-State Circuits,vol.42(11),pp.2573-2584,2007.R.Karakiew

223、icz,R.Genov,and G.Cauwenberghs,“1.1 TMACS/mW Fine-Grained Stochastic Resonant Charge-Recycling Array Processor,”IEEE Sensors Journal,vol.12(4),pp.785-792,2012.R.Kubendran,W.Wan,S.Joshi,H.S.P.Wong,and G.Cauwenberghs,“A 1.52 pJ/Spike Reconfigurable Multimodal Integrate-and-Fire Neuron Array Transceive

224、r,”IEEE Int.Conf.Neuromorphic Systems(ICONS2020),Nov.2020.R.Kubendran,A.Paul and G.Cauwenberghs,“A 256x256 6.3pJ/pixel-event Query-driven Dynamic Vision Sensor with Energy-conserving Row-Parallel Event Scanning,”IEEE CICC 2021.D.Kuzum,R.G.Jeyasingh,B.Lee,and H.S.P.Wong,“Nanoelectronic Programmable S

225、ynapses Based on Phase Change Materials for Brain-Inspired Computing,”ACS Nano Letters,vol.12(5),pp.2179-2186,2011.P.Lichtsteiner,C.Posch,and T.Delbruck,“A 128128 120 dB 15s Latency Asynchronous Temporal Contrast Vision Sensor,”IEEE J.Solid-State Circuits,vol.43(2),pp.566576,2008.Gert CauwenberghsIS

226、SCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge38 of 39 2024 IEEE International Solid-State Circuits ConferenceReferences(3/3)Y.Moon and D.-K.Jeong,“An Efficient Charge Recovery Logic

227、Circuit,”IEEE J.Solid-State Circuits,vol.31(4),pp.514-522,1996.J.Park,T.Yu,S.Joshi,C.Maier,and G.Cauwenberghs,“Hierarchical Address Event Routing for Reconfigurable Large-Scale Neuromorphic Systems,”IEEE Trans.Neural Networks and Learning Systems,vol.28(10),pp.2408-2422,2017.W.Wan,R.Kubendran,S.B.Er

228、yilmaz,W.Zhang,Y.Liao,D.Wu,S.Deiss,B.Gao,P.Raina,S.Joshi,H.Wu,G.Cauwenberghs,H-S.P.Wong,“33.1:A 74 TMACS/W CMOS-RRAM Neurosynaptic Core with Dynamically Reconfigurable Dataflow and In-situ Transposable Weights for Probabilistic Graphical Models,”IEEE ISSCC,2020.W.Wan,R.Kubendran,C.Schaefer,S.B.Eryil

229、maz,W.Zhang,D.Wu,S.Deiss,P.Raina,H.Qian,B.Gao,S.Joshi,H.Wu,H.-S.P.Wong,and G.Cauwenberghs,“A Compute-in-Memory Chip Based on Resistive Random-Access Memory,”Nature,vol.608,pp.504512,2022.M.Yang et al.,“A Dynamic Vision Sensor With 1%Temporal Contrast Sensitivity and In-Pixel Asynchronous Delta Modul

230、ator for Event Encoding,”IEEE J.Solid-State Circuits,vol.50(9),pp.2149-2159,2015.K.A.Zaghloul and K.Boahen,“A Silicon Retina that Reproduces Signals in the Optic Nerve,”Neural Engineering,vol.3(4),pp.257-267,2006.J.Zhang,Z.Wang and N.Verma,“In-Memory Computation of a Machine-Learning Classifier in a

231、 Standard 6T SRAM Array,”IEEE J.Solid-State Circuits,vol.52(4),pp.915-924,2017.Gert CauwenberghsISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge39 of 39 2024 IEEE International Solid-

232、State Circuits ConferenceISSCC 2024-Forum 4.4:Event-Driven Sensory Analog Processing and Massively Parallel Mixed-Signal In-Memory Computing for Distributed Adaptive Intelligence at the Edge40 of 39Please Scan to Rate Please Scan to Rate This PaperThis Paper 2024 IEEE International Solid-State Circu

233、its ConferenceCross-Layer Innovations for Enabling Real-Time and Efficient Eye Tracking in VR/AR Yingyan(Celine)LinAssociate ProfessorISSCC 2024-Forum F4:1 of 66 2024 IEEE International Solid-State Circuits ConferenceOur Research Overview 2 of 66https:/eiclab.scs.gatech.edu/Efficient AI AlgorithmsEf

234、ficient AI HardwareAutomated ToolsTowardsUbiquitousAI-Powered Intelligent Devices and Green AIEfficient Machine Learning through cross-layer innovationstowards ubiquitous on-device intelligence and green AIISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceOur Research Overvie

235、w 3 of 66https:/eiclab.scs.gatech.edu/Efficient AI AlgorithmsEfficient AI HardwareAutomated ToolsTowardsUbiquitousAI-Powered Intelligent Devices and Green AIEfficient Machine Learning through cross-layer innovationstowards ubiquitous on-device intelligence and green AIISSCC 2024-Forum 4:2024 IEEE In

236、ternational Solid-State Circuits ConferenceOur Enabling and Integrated Systems:Example 1-FlatCam:A 253 FPS,91.49 J/Frame Ultra-Compact Intelligent Lensless Camera4 of 66Published in VLSI 2022ISSCC 2024-Forum 4:Won 1st place at DAC 2022 University Demo 2024 IEEE International Solid-State Circuits Con

237、ferenceOur Enabling and Integrated Systems:Example 2Gen-NeRF(FPGA):Real-time and Efficient Generalizable Rendering and Semantic Segmentation with Interactive View Control5 of 66Published in ISCA 2023ISSCC 2024-Forum 4:Won 2nd place at DAC 2023 University Demo 2024 IEEE International Solid-State Circ

238、uits ConferenceBackground-Eye Tracking in AR/VREye tracking has become an essential human-machine interface modality in AR/VR Liu,IDEM 20216 of 66e.g.,High-resolution rendering in a zone gazed by eyes vs.low-resolution rendering outside the zoneISSCC 2024-Forum 4:2024 IEEE International Solid-State

239、Circuits ConferenceBackground and MotivationEye tracking requirements in AR/VRLiu,IDEM 2021 240 FPS(real-time requirement for eye tracking)Low power consumption in mWSmall form factor7 of 66Existing works for AR/VR Feng,VR 2022;Bong,VLSI 2015An order of magnitude slower Large form factor due to the

240、lens-based cameraISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceOur Proposed -FlatCam System Overview-FlatCam system features:A lensless camera called FlatCamAsif,TCI 2017Small form factor:5 10 thinner8 of 66vs.ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuit

241、s ConferenceOur Proposed -FlatCam System-FlatCam system features:A lenFlatCamsless camera called FlatCamAsif,TCI 2017Small form factor:5 10 thinnerAn AI acceleration(Accel.)chip:240FPS(real-time requirement for eye tracking in VR/AR)mW power consumption9 of 66ISSCC 2024-Forum 4:2024 IEEE Internation

242、al Solid-State Circuits ConferenceOur Proposed -FlatCam Demonstration-FlatCam demonstration setting:Won 1st place in University Demo Best Demonstration at Design Automation Conference(DAC Demo)202210 of 66Published in VLSI2022ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference

243、FlatCam:A Lensless CameraA Lensless camera called FlatCamAsif,TCI 2017Small form factor:5 10 thinner11 of 66FlatCam:a binary coded maskLens-based camera mm FlatCam 2mm 5 10 thinnerSensor measurementReconstructedeye imageReconstruction the readable images by matrix-matrix multiplication computationsI

244、SSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCam Integrates Cross-Layer Optimization12 of 66Algorithm-Hardware Co-DesignLeveraging Application-Level OpportunitiesApplication-level:Predict-then-focus pipelineAlgorithm-level:SmartExchangeHardware-level:co-designed alg.

245、and acceleration chipISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCams Application-Level OptimizationBaseline:Directly Use Input Images for Gaze Estimation13 of 66Baseline1.81.61.41.21.00.80.60.40.20Mega OPs(100)ISSCC 2024-Forum 4:2024 IEEE International Solid-State

246、 Circuits Conference-FlatCams Application-Level OptimizationApplication level:We propose a predict-then-focus pipeline14 of 66Baseline1.81.61.41.21.00.80.60.40.20Mega OPs(100)ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCams Application-Level OptimizationApplication

247、 level:We propose a predict-then-focus pipelineMotivation:Leverage opportunities in eye-tracking to complexity15 of 66Baseline1.81.61.41.21.00.80.60.40.20Mega OPs(100)ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCams Application-Level OptimizationOpportunities in th

248、e eye-tracking application:Spatial redundancy Only region-of-interest(ROI)is useful16 of 66Baseline1.81.61.41.21.00.80.60.40.20Mega OPs(100)ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCams Application-Level OptimizationOpportunities in the eye-tracking application:

249、Spatial redundancy Only region-of-interest(ROI)is usefulTemporal redundancy Don not need to update ROI for every frame17 of 66Baseline1.81.61.41.21.00.80.60.40.20Mega OPs(100)ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCams Application-Level OptimizationEye detecti

250、on+Gaze estimation:18 of 66ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCams Application-Level OptimizationEye detection+Gaze estimation:Extract region-of-interest(ROI)19 of 66ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCams Applic

251、ation-Level OptimizationEye detection+Gaze estimation:Extract region-of-interest(ROI)Then only focus on the extracted ROI20 of 66ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCams Application-Level OptimizationEye detection+Gaze estimation:Extract region-of-interest(

252、ROI)Then only focus on the extracted ROI Reduce 62%Ops21 of 66Baseline+ROI1.81.61.41.21.00.80.60.40.2062%Mega OPs(100)ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCams Application-Level OptimizationEye detection+Gaze estimation:Extracted ROI Reduce 62%Ops22 of 66Bas

253、eline+ROI1.81.61.41.21.00.80.60.40.2062%+ROI+Temp.20%46 Mega OPsMega OPs(100)Utilize temporal redundancy across frames:Update ROI on only 5%of frames Further reduce 20%OPsISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCams Application-Level OptimizationOur predict-the

254、n-focus pipeline matches the accuracy of the winner in an industry-led challenge OpenEDS20Crisiina,Sensors 202123 of 66Baseline+ROI1.81.61.41.21.00.80.60.40.2062%+ROI+Temp.20%46 Mega OPsMega OPs(100)9.68ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference24 of 66-FlatCams Algor

255、ithm-Level OptimizationISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference25 of 66Can we trade the higher-cost memory access for the lower-cost computation?Yes!SmartExchange algorithm-FlatCams Algorithm-Level OptimizationSmartExchange AlgorithmSmartExchange Accelerator Availab

256、ility of Big Data Explosion of computing powerSmartExchange Availability of Big Data Explosion of computing powerPublished in ISCA2020ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference26 of 66Efficient Processing via SmartExchange:Motivation Data movement cost Computation cos

257、tWell-known observation:ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference27 of 66Can we trade the higher-cost memory access for the lower-cost computation?Data movement cost Computation costWell-known observation:ISSCC 2024-Forum 4:Efficient Processing via SmartExchange:Moti

258、vation 2024 IEEE International Solid-State Circuits Conference28 of 66Can we trade the higher-cost memory access for the lower-cost computation?Yes!SmartExchange:A High-level ViewA well-known observation:Data movement cost Computation costISSCC 2024-Forum 4:2024 IEEE International Solid-State Circui

259、ts Conference29 of 66Can we trade the higher-cost memory access for the lower-cost computation?Yes!SmartExchange:A High-level ViewData movement cost Computation costVanilla SituationData movementComputation200 +1ISSCC 2024-Forum 4:A well-known observation:2024 IEEE International Solid-State Circuits

260、 Conference30 of 66Can we trade the higher-cost memory access for the lower-cost computation?Yes!SmartExchange:A High-level ViewData movement cost Computation costVanilla SituationData movementComputation200 +1SmartExchangeDesired SituationData movementComputatin200+1ISSCC 2024-Forum 4:A well-known

261、observation:2024 IEEE International Solid-State Circuits ConferenceSmartExchange:Also Effective for TrainingFurther developed SmartExchange for efficient on-device training:31 of 6610.6 and 4.5 reduction in the storage and the training energy cost compared with state-of-the-art training baselinesPub

262、lished in IEEE Transactions on Neural Networks and Learning Systems 2021Image source:“enabling-device-learning-scale”,Qualcomm ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCams Acceleration Chip-Leve Optimization32 of 66ISSCC 2024-Forum 4:2024 IEEE International Sol

263、id-State Circuits Conference-FlatCams Acceleration Chip-Leve Optimization33 of 66ThroughputEnergy-EfficiencyAlg.AccuracyArea-EfficiencyControl-EfficiencyISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCams Acceleration Chip-Leve Optimization34 of 66ISSCC 2024-Forum 4:2

264、024 IEEE International Solid-State Circuits Conference-FlaatCams Acceleration Chip-Leve Optimization35 of 66Design ChallengesAI Accel.Chips HighlightsSmartExchanges Parameters(i.e.,Basis and Coefficient Matrices)Fully Utilize SmartExchange toMinimize Weight Storage&Access EnergySmartExchangesStructu

265、re SparsityEfficient Support for Structure Sparsity with High Area-EfficiencyChip Control CostCustomized Instruction Set Arch.(ISA)to Minimize Control OverheadChip control costs 10%of the overall energy Chen,JSSC 2016ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceAI Accel.

266、Chips Micro-Architecture36 of 66Output BufferInput Buffer50KB Act.GB 150KB Act.GB 0BufferISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceAI Accel.Chips Micro-Architecture37 of 66Output BufferInput Buffer50KB Act.GB 150KB Act.GB 0Buffer64*PE(Compute Element)LinesEach PE_line

267、 has 8*8-bit multipliersISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceAI Accel.Chips Micro-Architecture38 of 66Output BufferInput Buffer50KB Act.GB 150KB Act.GB 0Buffer64*PE(Compute Element)LinesEach PE_line has 8*8-bit multipliersInput/Output Memory HierarchyActivation(A

268、ct.)Global Buffer(GB)and Input/Output BufferISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceAI Accel.Chips Micro-ArchitectureOutput BufferInput Buffer50KB Act.GB 150KB Act.GB 0Buffer64*PE(Compute Element)LinesEach PE_line has 8*8-bit multipliersInput/Output Memory Hierarchy

269、Activation(Act.)Global Buffer(GB)and Input/Output BufferWeight(or Parameter)Memory HierarchyWeight GB and Weight Buffers39 of 66ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceAI Accel.Chips Micro-ArchitectureOutput BufferInput Buffer50KB Act.GB 150KB Act.GB 0Buffer64*PE(Co

270、mpute Element)LinesEach PE_line has 8*8-bit multipliersInput/Output Memory HierarchyActivation(Act.)Global Buffer(GB)and Input/Output BufferWeight(or Parameter)Memory HierarchyWeight GB and Weight BuffersCompression-aware Modules for SmartExchange Algorithm40 of 66ISSCC 2024-Forum 4:2024 IEEE Intern

271、ational Solid-State Circuits ConferenceAI Accel.Chips Micro-ArchitectureOutput BufferInput Buffer50KB Act.GB 150KB Act.GB 0Buffer64*PE(Compute Element)LinesEach PE_line has 8*8-bit multipliersInput/Output Memory HierarchyActivation(Act.)Global Buffer(GB)and Input/Output BufferWeight(or Parameter)Mem

272、ory HierarchyWeight GB and Weight BuffersCompression-aware Modules for SmartExchange AlgorithmOn-chip Controller with A Customized ISA41 of 66ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceAI Accel.Chips Highlights42 of 66Design ChallengesAI Accel.Chips HighlightsSmartExch

273、anges Parameters(i.e.,Basis and Coefficient Matrices)Fully Utilize SmartExchange toMinimize Weight Storage&Access EnergySmartExchangesStructure SparsityEfficient Support for Structure Sparsity with High Area-EfficiencyChip Control CostCustomized Instruction Set Arch.(ISA)to Minimize Control Overhead

274、Chip control costs 10%of the overall energy Chen,JSSC 2016ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceSmartExchange:Minimize Storage&Energy 43 of 66Minimize storage/access energy for weights:Store and Basis Matrix()Coefficient Matrix()Weight()Restore the original weight

275、 when neededISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceMinimize storage/access energy for weights:Store and SmartExchange:Minimize Storage&Energy 44 of 66Basis Matrix()Coefficient Matrix()Weight()ISSCC 2024-Forum 4:Restore the original weight when needed 2024 IEEE Inte

276、rnational Solid-State Circuits ConferenceSmartExchange:Minimize Storage&Energy 45 of 66Minimize storage/access energy for weights:Store and Restore the original weight when neededISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceSmartExchange:Minimize Storage&Energy 46 of 66M

277、inimize storage/access energy for weights:Store and Restore the original weight when neededBasis Matrix()ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceSmartExchange:Minimize Storage&Energy 47 of 66Minimize storage/access energy for weights:Store and Restore the original w

278、eight when neededStream Only stream in needed for local weight restoreCoefficient matrix()ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceAI Accel.Chips Highlights48 of 66Design ChallengesAI Accel.Chips HighlightsSmartExchanges Parameters(i.e.,Basis and Coefficient Matrices

279、)Fully Utilize SmartExchange toMinimize Weight Storage&Access EnergySmartExchangesStructure SparsityEfficient Support for Structure Sparsity with High Area-EfficiencyChip Control CostCustomized Instruction Set Arch.(ISA)to Minimize Control OverheadChip control costs 10%of the overall energy Chen,JSS

280、C 2016ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceSmartExchange:Structure Sparsity49 of 66Row-/channel-wise sparsity for CONV/PWSkip the unnecessary computationsInput Activation Feature MapOutput Activation Feature MapRow-wise Sparsity for CONVsWeight()Channel-wise Spar

281、sity for PWs Weight()ISSCC 2024-Forum 4:Require higher activation(Act.)global buffer(GB)bandwidth 2024 IEEE International Solid-State Circuits ConferenceSmartExchange:Structure Sparsity50 of 66Row-/channel-wise sparsity for CONV/PWSkip the unnecessary computations Require higher activation(Act.)glob

282、al buffer(GB)bandwidthInput Activation Feature MapRow-wise Sparsity for CONVsWeight()Output Activation Feature MapISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceSmartExchange:Structure Sparsity51 of 66Row-/channel-wise sparsity for CONV/PWSkip the unnecessary computations

283、Require higher activation(Act.)global buffer(GB)bandwidthInput Activation Feature MapRow-wise Sparsity for CONVsWeight()ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceSmartExchange:Higher Act.GB Bandwidth52 of 66Row-/channel-wise sparsity for CONV/PWSkip the unnecessary co

284、mputations Require higher activation(Act.)global buffer(GB)bandwidthISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceSmartExchange:Higher Act.GB Bandwidth53 of 66Row-/channel-wise sparsity for CONV/PW Require higher activation(Act.)global buffer(GB)bandwidthProposed Sequenti

285、al-write-parallel-read(SWPR)input buffer for 2 Act.bandwidth,w/0.58%area overheadParallel-readSequential-writeISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceAI Accel.Chips Highlights54 of 66Design ChallengesAI Accel.Chips HighlightsSmartExchanges Parameters(i.e.,Basis and

286、Coefficient Matrices)Fully Utilize SmartExchange toMinimize Weight Storage&Access EnergySmartExchangesStructure SparsityEfficient Support for Structure Sparsity with High Area-EfficiencyChip Control CostCustomized Instruction Set Arch.(ISA)to Minimize Control OverheadChip control costs 10%of the ove

287、rall energy Chen,JSSC 2016ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceAI Accel.Chip:ISAs Features55 of 66Proposed ISAs FeaturesISA ChallengesFrequent Instruction Fetching from Instr.BufferFrequent Instruction DecodingLarge Instr.Buffer SizeISSCC 2024-Forum 4:2024 IEEE I

288、nternational Solid-State Circuits ConferenceAI Accel.Chip:ISAs Features56 of 66Proposed ISAs FeaturesISA ChallengesFrequent Instruction Fetching from Instr.BufferFrequent Instruction DecodingLarge Instr.Buffer SizeSingle Instruction Multiple PE_line ProcessingISSCC 2024-Forum 4:2024 IEEE Internation

289、al Solid-State Circuits ConferenceAI Accel.Chip:ISAs Features57 of 66Proposed ISAs FeaturesISA ChallengesFrequent Instruction Fetching from Instr.BufferFrequent Instruction DecodingLarge Instr.Buffer SizeSingle Instruction Multiple PE_line Processing“1”InstructionTensHundreds of Processing cyclesISS

290、CC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceAI Accel.Chip:ISAs Features58 of 66Proposed ISAs FeaturesISA ChallengesFrequent Instruction Fetching from Instr.BufferFrequent Instruction DecodingLarge Instr.Buffer SizeSingle Instruction Multiple PE_line Processing“1”Instructio

291、nTensHundreds of Processing cycles“Jump”Instructions to Save Instr.Buffer SizeISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceAI Accel.Chip:ISAs Features59 of 66The customized ISA is general:Repetitive compute patterns in AI Alg.Parallel processing of the acceleratorISSCC 2

292、024-Forum 4:Proposed ISAs FeaturesISA ChallengesSingle Instruction Multiple PE_line Processing“Jump”Instructions to Save Instr.Buffer Size“1”InstructionTensHundreds of Processing cyclesFrequent Instruction Fetching from Instr.BufferFrequent Instruction DecodingLarge Instr.Buffer Size 2024 IEEE Inter

293、national Solid-State Circuits Conference-FlatCams Full System Setup&AI Accel.Chip-FlatCams full system measurement setup:60 of 66AI Accel.chip implementation:ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conference-FlatCams Full System Measurement Result-FlatCams full system measur

294、ement setup:61 of 66Measurement results on an industry standard dataset(OpenEDS20)Crisiina,Sensor 2021Real-time FPS(i.e.,253 FPS (0.55V,0.6V),115MHz)91.49 J/Frame,one order of magnitude smaller energy/frame than the previous work Bong,VLSI 2015Published in VLSI 2022ISSCC 2024-Forum 4:2024 IEEE Inter

295、national Solid-State Circuits Conference-FlatCam:Integration and Demonstration-FlatCam integrates application-,camera-,algorithm-and hardware-level optimization:Won 1st place at DAC 2022 University Demo62 of 66Published in VLSI 2022ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits Conf

296、erence-FlatCam EyeCoD:MotivationEye segmentation-based ROI prediction63 of 66AI Accel.chip implementation:EyeCoDs Pipeline-FlatCams PipelineGeneral eye tracking for AR/VREye SegmentationGaze EstimationDirect Gaze Interaction.Meta,OpenEDS 2020ISSCC 2024-Forum 4:2024 IEEE International Solid-State Cir

297、cuits Conference64 of 66EyeCoDs Pipeline-FlatCams PipelinePrediction ResolutionROIPredictionGaze Estimation1281285656Unet(segmentation)998 Mega OPs per frameMobileNetV1(detection)1.03 Mega OPs per frameFBNet-C100140 Mega OPs per frameMobileNetV123.7 Mega OPs per frame186.75.95.2How to Design the Acc

298、elerator w/o Significantly Increasing the Chip Area and Memory Bandwidth?ISSCC 2024-Forum 4:EyeCoDs Pipeline vs.-FlatCams Pipeline 2024 IEEE International Solid-State Circuits ConferenceEyeCoDs Pipeline vs.-FlatCams Pipeline65 of 66EyeCoD,a first-of-its-kind ultra-compact intelligent lensless camera

299、 system for a general eye tracking function integrating a segmentation-involved pipeline EyeCoD achieves real-time FPS(i.e.,240 FPS),8.1 higher energy efficiency,and 5 10 thinner camera form factorPaving the way for eye-tracking in next-generation AR/VR devicesPublished in ISCA 2022Selected as IEEE

300、Micro Top Picks of 2023ISSCC 2024-Forum 4:2024 IEEE International Solid-State Circuits ConferenceAcknowledgment66 of 66Students:Sponsors:ISSCC 2024-Forum F4:2024 IEEE International Solid-State Circuits ConferenceISSCC 2024-Forum X.Y:67 of 66Please Scan to Rate Please Scan to Rate This PaperThis Pape

301、r 2024 IEEE International Solid-State Circuits ConferenceAggressive Design Reuse for Ubiquitous Security-From Design-Time to Run-Time Intelligent Attack Detection and CounteractionMassimo Aliotomaliotoieee.orgNational University of SingaporeFeb 22,2023ISSCC 2024-Forum on Intelligent Sensing F4.6 202

302、4 IEEE International Solid-State Circuits ConferenceOutlineHW security challenges towards trillions of connected devicesFrom root of trust to system security:zero-trust networksFoundations of trust:on-chip root of trustTrends on root of trustImmersed-in-logic and in-memory root of trustPhysical cont

303、ext awareness:on-chip sensing for physical attack detectionPower analysisLaser Voltage ProbingMachine learning-based side-channel attack counteraction and“HW patching”Protection of encryption circuitsPrevention of neural network reverse engineering(intelligent sensors)Conclusions2 of 60ISSCC 2024-Fo

304、rum on Intelligent Sensing F4.6Prof.Massimo Alioto 2024 IEEE International Solid-State Circuits ConferenceOutlineHW security challenges towards trillions of connected devicesFrom root of trust to system security:zero-trust networksFoundations of trust:on-chip root of trustTrends on root of trustImme

305、rsed-in-logic and in-memory root of trustPhysical context awareness:on-chip sensing for physical attack detectionPower analysisLaser Voltage ProbingMachine learning-based side-channel attack counteraction and“HW patching”Protection of encryption circuitsPrevention of neural network reverse engineeri

306、ng(intelligent sensors)Conclusions3 of 60ISSCC 2024-Forum on Intelligent Sensing F4.6Prof.Massimo Alioto 2024 IEEE International Solid-State Circuits ConferenceHW Security Challenges towards the Trillions(1/3)Connected sensors as interface btwn physical world and cloud4 of 6010 Billions10 cm10+$1-10

307、 W1 Trillion1-10 mm1$0.1-1,000 WGATEWAYS/CONCENTRATORSIoT NODES#DEVICES WORLDWIDE(100X)size(100X)cost/item(100X)INTERNETOF THINGSPHYSICAL WORLDUSERS100 Millionsmeters1,000-10,000$1,000 WCLOUDcompute-ability(100X)100-1,000 MFLOPS1-100 MFLOPS100-1,000 GFLOPsavailableenergy/day(1,000X)100kJ0.1J100 MJed

308、geM.Alioto,“Trends in Hardware Security:from Basics to ASICs,IEEE Solid-State Circuits Magazine(invited),Aug.2019P.Sparks,The Route to a Trillion Devices-The Outlook for IoT Investment to 2035,White Paper,June 2017 OnlineISSCC 2024-Forum on Intelligent Sensing F4.6Security limited by weakest link ze

309、ro-trust networks(end-to-end)05000200400600800020232026202920322035#devices added per yearyearconnectede devices worldwidecumulative connected device count1 trillionsimilar to Moores law(attack surface)Prof.Massimo Alioto 2024 IEEE International Solid-State Circuits ConferenceL

310、ong-lived device vulnerabilities temporal sustainability(lifespan),“HW patching”5 of 60M.Alioto,“Aggressive Design Reuse for Ubiquitous Zero-Trust Edge Security From Physical Design to Machine Learning-Based Hardware Patching,”IEEE Open Journal of the Solid-State Circuits Society(invited),vol.3,pp.1

311、-16,2023ISSCC 2024-Forum on Intelligent Sensing F4.6HW Security Challenges towards the Trillions(2/3)Exponentials generally hard to sustain(same for security)Inexpensive/low power for ubiquitous securityArea/design reuseSECURITY PROPERTIES/GOALSTECHNOLOGY DIRECTIONSDESIGN ENABLERSHW patchingmachine

312、learning-based run-time counteraction on-chip sensorization for attack detectionattack sensors with fully-digital fully-automated designPHYSICAL AWARENESSsensors with low design/integration effortSUSTAINABILITY ACROSS DEVICE LIFESPANarchitectures upgradeable over timetimedesign and resource reusein-

313、memory,immersed-in-logicECONOMIC SUSTAINABILITYlow design/integration effort,areaCOST=NRE+RE Pubiquitous securityzero-trust securityZero-trust networksHW-backed device authentication on-chip root of trust for end-to-end encryption physical context awareness(sensors for attack detection)Prof.Massimo

314、Alioto 2024 IEEE International Solid-State Circuits ConferenceSecurity requirements6 of 60ISSCC 2024-Forum on Intelligent Sensing F4.6HW Security Challenges towards the Trillions(3/3)BobAliceeavesdroppingEveCraigTrudypassword crackerintruder(send/delete/modify msgs)M.Alioto,Hardware security from Ba

315、sics to ASICs,Tutorial at ISSCC 2019,San Francisco(USA),Feb 17,2019transistorgateRTLArchitectureOScompilerlibraryprogramming languageapplicationHARDWARESOFTWARE(firewalls,protocols)Secret key(on-chip root of trust)+crypto chain of trustconfidentialityintegrityauthenticitynon-repudiationsignatureno e

316、avesdroppingno alterationverify senderno other sendercertifyProf.Massimo Alioto 2024 IEEE International Solid-State Circuits ConferenceOutlineHW security challenges towards trillions of connected devicesFrom root of trust to system security:zero-trust networksFoundations of trust:on-chip root of tru

317、stTrends on root of trustImmersed-in-logic and in-memory root of trustPhysical context awareness:on-chip sensing for physical attack detectionPower analysisLaser Voltage ProbingMachine learning-based side-channel attack counteraction and“HW patching”Protection of encryption circuitsPrevention of neu

318、ral network reverse engineering(intelligent sensors)Conclusions7 of 60ISSCC 2024-Forum on Intelligent Sensing F4.6Prof.Massimo Alioto 2024 IEEE International Solid-State Circuits ConferenceOn-Chip Intrinsic Root of Trust:FoundationsEntropy generated endogenously at run time(no storage)Low cost,low p

319、ower(e.g.,no Flash memory)Immune to most attacks to memories(but prone to others)8 of 60ISSCC 2024-Forum on Intelligent Sensing F4.6True Random Number Generators(TRNGs)Random over time(0%repeatable)dynamic entropyFresh session keys,nonces,initialization vectors,padding valuesTRNG die#1random numbers

320、1011010010M.Alioto,“Trends in Hardware Security:from Basics to ASICs,IEEE Solid-State Circuits Magazine(invited),Aug.2019Physically Unclonable Function(PUF)Si fingerprint(100%repeatable)static entropyWeak(few CRPs),strong PUF(10 xCRPs)Chip ID,device authentication,private key exchange(no asymmetric

321、crypto),PUF die#1challenge01101response1011010010challenge-response pair CRPProf.Massimo Alioto 2024 IEEE International Solid-State Circuits ConferenceStrong PUFs:relaxed stability constraint,soft matchingOn-Chip Intrinsic Root of Trust:Challenges9 of 60ISSCC 2024-Forum on Intelligent Sensing F4.6PU

322、FsNot stable/repeatable,expensive design margin(PVT+noise)0%1%2%3%4%5%0040005000%of bits#of repeated PUF evaluationsUnstable bits 0.8VWorst BER 0.8VTRNGsConsistent entropy quality across P,V,T,supply noise(0.99997)00.10.20.30.40.50.60.70.80.930.940.950.960.970.980.9910.60.70.80.911.11.2P-

323、value(NIST 800-22 tests)entropyVDD(V)all p-values min.NIST target(0.01)all points min.NIST target(0.991)Shannon entropymin-entropyP-value NIST 800-22 tests(average all tests)00.10.20.30.40.50.60.70.80.930.940.950.960.970.980.00temperature(C)all p-values min.NIST target(0.01)all points

324、min.NIST target(0.991)P-value(NIST 800-22 tests)entropyShannon entropymin-entropyP-value NIST 800-22 tests(average all tests)Weak PUFs:expensive post-processing/ECC(100-1,000X)DESIGN TIMETESTING TIMECHIP BOOT TIMEpost processing(spatial/temporal majority vote,darkbits.)PUF circuit design(topology,or

325、ganization.)testing time fixes(OTP fuses,testingPVT corners,masking,hardening)Error Correcting Code circuits(BCH)stable PUF for HW securityBoot time fixes(dark bit masking)Inherent robustness+monitoring(energy/area hungry,design effort)TRNG(noise-based)refine statistical propertiespost-processinggen

326、erates raw random sequence from random process(noise)on-line testingalarmanti-fault injection sensors(voltage,temperature)continuous entropy checkrandom output0110.Prof.Massimo Alioto 2024 IEEE International Solid-State Circuits ConferenceSome memory PUFs have sufficient native stabilityECC-less som

327、etimes feasible(100X area and energy saving)1.E-101.E-091.E-081.E-071.E-061.E-051.E-041.E-031.E-021.E-011.E+001.E+011998 2000 2002 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022 2024BER after stability enhancement(%)yearanalogdelaymemory(volatile)memory(non-volatile)metastabilitymonostablehybrido

328、thersweak PUF post-stability enhancement BER vs yearECC-lessStability enhancement helps suppressing ECC1.E-101.E-091.E-081.E-071.E-061.E-051.E-041.E-031.E-021.E-011.E+001.E+011.E+021998 2000 2002 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022 2024native BER before stability enhancement(%)yearweak

329、 PUF native BER vs yearanalogdelaymemory(volatile)memory(non-volatile)metastabilitymonostablehybridothersECC-lessTrends in On-Chip Root of Trust:Weak PUF10 of 60ISSCC 2024-Forum on Intelligent Sensing F4.6M.Alioto,“HW security primitives Green IC database,”Online.Available:https:/www.green-ic.org/ha

330、rdware-security-database-hwsecdb/Prof.Massimo Alioto 2024 IEEE International Solid-State Circuits Conference1.E+001.E+011.E+021.E+031.E+041.E+051.E+061.E+071998 2000 2002 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022 2024normalized area/bit(F2)yearweak PUF area per bit vs yearanalogdelaymemory(v

331、olatile)memory(non-volatile)metastabilitymonostablehybridothersTrends in On-Chip Root of Trust:Weak PUF11 of 60ISSCC 2024-Forum on Intelligent Sensing F4.6PUF area scaling follows technology scalingBest density(cost)in memory PUFs1.E-051.E-041.E-031.E-021.E-011.E+001.E+011.E+021.E+031.E+041998 2000

332、2002 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022 2024energy/bit(pJ)yearweak PUF energy per bit vs yearanalogdelaymemory(volatile)memory(non-volatile)metastabilitymonostablehybridothersCircuit innovation reduces energy/bit much faster than technology scalingMonostable particularly efficient(fJ/

333、bit and below)Prof.Massimo AliotoM.Alioto,“HW security primitives Green IC database,”Online.Available:https:/www.green-ic.org/hardware-security-database-hwsecdb/2024 IEEE International Solid-State Circuits ConferenceTrends in On-Chip Root of Trust:Strong PUF12 of 60ISSCC 2024-Forum on Intelligent Sensing F4.6Area scaling much faster than technology scaling in memory strong PUFsMemory PUFs also hav