1、Dont Get Ghosted:Designing and Implementing an Enrollment Prediction ModelAndrew LaVeniaColumbia University School of Professional StudiesMarialejandra Chuy SilvaColumbia University School of Professional StudiesMichelle VakmanColumbia University School of Professional StudiesDavid DysartPitzer Coll

2、egeDont Get Ghosted:Designing and Implementing an Enrollment Prediction ModelAndrew LaVenia,Associate DirectorMarialejandra Chuy Silva,Assistant DirectorMichelle Vakman,DirectorThe problem But first,a story!Ghosting:when someone cuts off all communication without explanation.“Most of us think about

3、it in the context of digital departure(.)but it happens across all social circumstances(.).A ghost is a specter,something we think is there but really isnt.Weve all probably acted like this if were honest.Weve all probably been ghosted,too(.).These are supernatural times.”-Adam Popescu.Columbia SPS:

4、Who We Are One of 17 schools at Columbia University 18 Masters programs 25 visiting,certificate,and non-degree programs Multiple Pre-College and High School programs 10 American Language Program offerings Over 11,000 students Fall,Spring,and Summer intakes!Slate users since 2019Exploring tangible ac

5、tions Other teams concerns and needs Identifying common threads With X number of admitted students,and Y number of students ghosting us:what steps could we take to better report on the likelihood of enrollment?The solution?The“Ghost”field!Buthow to track this?Butwhere will this live?Buthow will this

6、 impact reporting?Butwill anyone know what were talking about?!?!How can we actually track likelihood to enroll?Step 1:Clean dataStep 2:Post-decision statusStep 3:Identify enrollment indicatorsStep 4:Build it outPost decision statusesThe result:Applicant Engagement tabThe result:“Viability”table on

7、pipeline reportsHow it worked!We had to find a way to navigate Internal and External Source Data From:Login History Actions by User Commitments Payments Information from Affiliate Offices ISSO AdvisingGhost Field RulesGhost field rules(“applicant engagement”created to track applicant behavior)Applic

8、ation Status(Post-Decision)RulesAdded additional application scoped field to track status following admission separate from the decision tableStatistical AnalysisWe analyzed our preliminary results using the variables we wanted to know about are that appeared to be guiding factors:Days.Between.Creat

9、ed.and.Submitted Days.from.Submission.to.Decision Average.Time.to.ReplyStatistical AnalysisFor the logistic regression:addresses_US Yes MS_Y_N Yes While ideally a p-value 0.05,95%confidence interval is regarded as standard,if we widen the confidence interval to p-value 0.15,85%,whether an applicant

10、was applying to a Masters program compared to a CPA,this variable becomes significant.Guiding FactorsGuiding FactorsWhat this reveals While it is true that the analysis found that having an address in the US seems to be associated with a higher likelihood of declining the offer,its important to unde

11、rstand that the statistical coefficients that were calculated are just a way of measuring the relative contribution of each variable to the final decision.In this model,having an address in the US and the average time to reply both seem to be significant factors in the decision to decline the offer.

12、Next Steps While this analysis was done to understand what factors might influence the decision to decline an offer,its not being used to actually make predictions in this case.Instead,its just providing some useful information about how a prediction model would weigh the different factors.Assess ad

13、ditional terms to continue analysis,focusing on indicatorsContact us:Andrew LaVenia-al4422columbia.eduMarialejandra Chuy Silva-mc4685columbia.eduMichelle Vakman-mv2776columbia.eduLeveraging Slate to Calculate Demonstrated Interest and Likelihood to Apply and EnrollThis session focuses on how Pitzer

14、is leveraging Slate to identify behaviors and model interest and future actionsPitzer CollegeSmall Liberal Arts CollegeClaremont CollegesLocated in Southern California1,000 undergraduate students On Slate since 2016Small Liberal Arts CollegeClaremont CollegesLocated in Southern California1,000 under

15、graduate students On Slate since 2016What You Will NeedQuery(Configurable Joins)Source FormatsRules&FormulasSPSS(or similar statistical software)Knowledge of Statistical Analysis(particularly Binary Logistic Regression)Previous dataA lot of trial and errorIntroductionIntroductionWhat You Will NeedQu

16、ery(Configurable Joins)Source FormatsRules&FormulasSPSS(or similar statistical software)Knowledge of Statistical Analysis(particularly Binary Logistic Regression)Previous dataA lot of trial and errorFocusFocusExploration processCollaborationQueries,Source Formats,and Rules usedExplain some of the st

17、atistical principles and processes usedExploration processCollaborationQueries,Source Formats,and Rules usedExplain some of the statistical principles and processes usedDemonstrated InterestPitzer-specific scores of behaviors already captured in SlateCalculate automatically and early in the processD

18、emonstrated InterestSlate has allowed us to track and catalogue Demonstrated InterestI wanted to be able to capture and calculate this early in the processTo help inform:Strategic RecruitmentApplication reviewReader TrainingDemonstrated Interest-ExplorationI came up with an initial draft of behavior

19、s and point scoresStart the collaboration with my Admission Officers.Sent a Slate Form to collect their thoughtsThey suggested new behaviors and scoringI rebuilt my Query and scoring rubricI rebalanced and validated the score to the internal score weighting Demonstrated Interest-ExplorationOur Admis

20、sion Officers have been looking at Demonstrated Interest in the Application Review ProcessI came up with an initial draft of behaviors and point scores to start the collaboration.Sent a Slate Form to collect their thoughtsThey suggested new behaviors and scoringI rebuilt my Query and scoring rubricI

21、 rebalanced and validated the score to the internal score weighting Demonstrated Interest-ExplorationDemonstrated Interest-Exploration12 Behaviors3 Bonuses to replace 3 Behaviors that some records may not performBonuses take Total and reduce it to appropriate“weight”100 pt Score for context Demonstr

22、ated Interest Slate TranslationAverage DI will increase through cycleIt would also be hard to score 100Demonstrated Interest Slate TranslationTo further contextualize Used TranslationsConvert 100pt to 5ptReflects our Staff evaluationsBuilt into Person Dashboard and Report for Admission Officers Quer

23、yxxxxxUploading the Data and Calculating ScoresCreate fields for each Sub ScoreBuild Source Formats to process the scheduled exports of SFTP serverFiles run over the weekend during slow times Uploading the Data and Calculating Scores3 Rules create the score2 Bonuses1 Final Score rule10-pt Bonus exam

24、pleSum other behaviorsDivide by 9Add to TotalVisualizing the Validationx Variable 33 possible scores:0,5,10Evaluations correlate with variableWeight:.5 Manual,4.5 Calculated Variable 71 Point per behaviorEvaluations correlate with variable Weight:.9 Manual,9 Calculated but more varianceDemonstrated

25、Interest-ResultsPromising ResultsManual and Calculated Scores highly correlated Both on data used to build score and to predict FA 23 scoresNext StepsIncorporate FA 23 data and emerging ways to demonstrate interestFinetune scoring Hold Feedback Sessions with Admission OfficersCreate similar scores t

26、o aid in holistic admissionDashboard(WIP)xxxxxLikelihood to ApplyUse Slate data in a statistical modelPredict how likely an Inquiry is to apply Automatically apply model to all Slate recordsLikelihood to ApplyNext Step in this was to build a statistical model to predict Application LikelihoodTurned

27、to SPSS Statistical Analysis Software(STATA,R,etc.)Exploration,Trial,and Error to find the best variables and best coding Run a Binary Logistic Regression.Create Formula is Slate to transform results into a%likelihood to applyBuild the Queries and Source FormatsCreate new Fields to capture the snaps

28、hot dataViola Likelihood to Apply There are several uses you can leverage this for Planning your cycle Application evaluation time/staffing,Strategic recruitmentInterventions and application pushesVisualizing the ValidationxCount:The number of behaviorsFlag:If they have the behavior at any level%App

29、ly:What%of the FA 20-22 Population Applied Variable 1Linear relationship between the Count and%Applying.Not very consistent,but a better fit than the Flag Variable 3Linear relationship between the Count and%Applying.Very inconsistent.Flag a better fit than the CountsGetting Snapshot DataConfigurable

30、 JoinsDecide on Counts versus Existence(Yes/No)FlagsIMPORTANT Only pull behaviors before they applyGetting Snapshot DataUse Comparison subquery filtersCompare behaviors date to Date Applied/Application Date for non-appliesPulling data for multiple historical years is a more complicated formulaWatch

31、for re-applies and other falloutsData PointsRemoved ProspectsRemoved Stealth ApplicantsMissing Data ModelData PointsRemoved ProspectsRemoved Stealth ApplicantsMissing Data PredictiveChecking ResultsI had 3 promising models1.Counts with Outliers2.Counts excluding Outliers3.FlagsLikelihood to ApplyUse

32、 Unstandardized Regression Weight(B)in your Slate Query Likelihood to ApplyExport Formula:Likelihood to Apply LogitExport Formula:Likelihood to Apply%ScoreLikelihood to Apply-ResultsFor Fall 2023Removing Stealth apps,Prospect to App,no ScorePredicted 1179 appsReceived 1351appsUnderpredicted low beha

33、vior records=10%,overpredictedPredicted 803 appsReceived 669 appsLikelihood to ApplyLikelihood to Apply LogitLikelihood to Apply ScoreLikelihood to Apply There are several uses you can leverage this for Planning your cycle Application evaluation time/staffing,Strategic recruitmentInterventions and a

34、pplication pushesLikelihood to EnrollUse Slate data in a statistical modelPredict how likely an Applicant is to enroll Automatically apply model to all Slate recordsLikelihood to Enroll(with 2nd Models)Use Slate data in a statistical modelPredict how likely an Applicant is to enroll Automatically ap

35、ply model to all Slate recordsLikelihood to EnrollChanging Base Person to ApplicationsModel based on Admitted Regular Decision ApplicantsSnapshot of behaviors at time of Admit Decision ModelPredictiveChecking ResultsOnly 1 modelBetter FitLess Variance explainedLikelihood to Apply LogitLikelihood to

36、EnrollLikelihood to Apply ScoreFA 22 Financial Aid YesLikelihood to Enroll(bonus models)(with 2nd Models)FA 22 Financial Aid NoLikelihood to Enroll-ResultsAs of 5/1/23 for Fall 2023Underpredicted by 7 studentsModel is based on Census,so summer melt may get closerFina Yes Overpredicted by 8 studentsF

37、ina No Underpredicted by 15 students Underpredicted Enrollment rateBands:10-20%,60-80%likely to enrollOverpredicted Enrollment rateBands:50-60%likely to enrollLikelihood to Enroll Results(with 2nd Models)As of 5/1/23 for Fall 2023“Textbook”FA 20-22 model Fina Yes Overpredicted by 8 studentsFina No U

38、nderpredicted by 15 students FA 22 Fina Yes and FINA No models Fina Yes Underpredicted by 6 students Fina No Overpredicted by 24 studentsDifferencesLarge differences in individual-level predictions between models(up to50-60%differences)Next Steps-Model Collaborate with Institutional Research to get

39、their inputExplore multi-models(FINA,International,etc.)Explore if there are groups Im systematically miss-predicting Model Retention behaviorsFinetune variablesBehaviors like emails and logins can warp modelsUse of non-Pandemic era dataNext Steps-UsePlug into Net Tuition Revenue modelBetter underst

40、and funding and headcount Work with OSA on supportProvide proactive outreach or supportAugment yield effortsMailings/Calls/Programming to manage yieldMeasure effectiveness of useAnalyze the results of those receiving special yield outreach in moving yield likelihood LimitationsIts hard right now wit

41、h the pandemic and changing higher education landscape to treat multiple cycles of prospective students the same way.That will make generalizing your historical data to upcoming cycles harder.Also,every student is different!Applying a one-size-fits-all model to tens of thousands of people will creat

42、e outliers.Some people apply as their first documentable action.Some people visit campus,submit documents,email 20 times,and still dont applyTipsReverse coding!If you have a Behavior or score where 1 is better than 5,It might be easier recoding it in the Query so the highest value=the best score.Tha

43、t makes interpreting your results and formulas more intuitive to work withOverfitting your modelDont overly predict your historical data that it cant predict future behaviorsLooking more times at more variables=more likely to find random significant effectsTipsChanging Policies(especially pandemic-r

44、elated changes in model data)Be mindful if you started or stopped offering policies or ways to engage with your schoolCode/Group your variables to best suit your model(with caution)Recoding your data may better predict behaviors,but there are a lot of considerations to be taken to adhere to good mod

45、elingData SnapshotsDont model based on behaviors after your modeling!Applying students go to more events after applying,Enrolling students send more messages after theyve committedSummaryWhat You Will NeedQuery(Configurable Joins)Source FormatsRules&FormulasSPSS(or similar statistical software)Knowledge of Statistical Analysis(particularly Binary Logistic Regression)Previous dataA lot of trial and errorBriefcases and Resourceshttps:/www.pitzer.edu/admission/pitzer_resource_modeling/Questions?David DysartSr.Associate DirectorAdmission OperationsDavid_DysartPitzer.edu#SlateSummit