1、SIMULATION UNCERTAINTYHow Reliable are Our Tail Statistics?June 2023Dean Marcus,FCAS CERA|SVP Actuary|New York1.Background and Motivation2.Spoiler Alert:Key Takeaways3.Whats the Deal with Second-Order Uncertainty?4.Whats the Deal with Third-Order Uncertainty?5.Cat,Quota Share and PPR Case Studies:Ho

2、w Reliable are Our Percentiles?6.Practical Implications and Recommendations7.Appendices:Proof Outline and Further DetailsAgenda41.Background and Motivation5 5Background and MotivationTwo Questions1.What is the statistical nature of our simulated percentiles?What can I say about the error bands aroun

3、d the percentiles,and are the error bands themselves a bit stretchy/blurry with uncertainty?2.How many realizations should we run?The answer will depend on the context,but our decisions should be well-informed by empirical evidence and solid theory,including the answer to question 1Addressing these

4、two questions will help guide practical decisions,and ensure sound advice to brokers and clients6 6Background and MotivationTwo Questions1.What is the statistical nature of our simulated percentiles?What can I say about the error bands around the percentiles,and are the error bands themselves a bit

5、stretchy/blurry with uncertainty?2.How many realizations should we run?The answer will depend on the context,but our decisions should be well-informed by empirical evidence and solid theory,including the answer to question 1Addressing these two questions will help guide practical decisions,and ensur

6、e sound advice to brokers and clients7 7Background and MotivationTwo Questions1.What is the statistical nature of our simulated percentiles?What can I say about the error bands around the percentiles,and are the error bands themselves a bit stretchy/blurry with uncertainty?2.How many realizations sh

7、ould we run?The answer will depend on the context,but our decisions should be well-informed by empirical evidence and solid theory,including the answer to question 1Addressing these two questions will help guide practical decisions,and ensure sound advice to brokers and clients8 8Background and Moti

8、vationTwo Questions1.What is the statistical nature of our simulated percentiles?What can I say about the error bands around the percentiles,and are the error bands themselves a bit stretchy/blurry with uncertainty?2.How many realizations should we run?The answer will depend on the context,but our d

9、ecisions should be well-informed by empirical evidence and solid theory,including the answer to question 1Addressing these two questions will help guide practical decisions,and ensure sound advice to brokers and clients9Background and MotivationSample Mean and CVIf we re-simulate any given model/met

10、ric using a different random seed,the key results will change but they converge with high samples:Sample mean is well-understood and close to the theoretical mean its CV is#,so,e.g.,re-simulating 1m realizations on a distribution that has CV 250%will typically produce a new mean within 0.25%of the p

11、revious oneSample standard deviation is well-understood but converges relatively slowly to the theoretical standard deviationLognormal example below:10k samples produces reliable estimates for the mean,but unreliable estimates for the CV,whereas 500k samples converges well10Background and Motivation

12、Sample Mean and CVIf we re-simulate any given model/metric using a different random seed,the key results will change but they converge with high samples:Sample mean is well-understood and close to the theoretical mean its CV is#,so,e.g.,re-simulating 1m realizations on a distribution that has CV 250

13、%will typically produce a new mean within 0.25%of the previous oneSample standard deviation is well-understood but converges relatively slowly to the theoretical standard deviationLognormal example below:10k samples produces reliable estimates for the mean,but unreliable estimates for the CV,whereas

14、 500k samples converges well11Background and MotivationSample Mean and CVIf we re-simulate any given model/metric using a different random seed,the key results will change but they converge with high samples:Sample mean is well-understood and close to the theoretical mean its CV is#,so,e.g.,re-simul

15、ating 1m realizations on a distribution that has CV 250%will typically produce a new mean within 0.25%of the previous oneSample standard deviation is well-understood but converges relatively slowly to the theoretical standard deviationLognormal example below:10k samples produces reliable estimates f

16、or the mean,but unreliable estimates for the CV,whereas 500k samples converges well12Background and MotivationSample PercentilesSample percentiles arent discussed as much and can vary wildly!This will be our focus,with very interesting findingsLognormal example below:10k samples produces unreliable

17、estimates for the tail stats,whereas 500k samples converges well13Background and MotivationSample PercentilesSample percentiles arent discussed as much and can vary wildly!This will be our focus,with very interesting findingsLognormal example below:10k samples produces unreliable estimates for the t

18、ail stats,whereas 500k samples converges well14Background and MotivationSample PercentilesSample percentiles arent discussed as much and can vary wildly!This will be our focus,with very interesting findingsLognormal example below:10k samples produces unreliable estimates for the tail stats,whereas 5

19、00k samples converges well15Background and MotivationUncertainty FrameworksSo we can think of simulation uncertainty as having two levels:1.First-Order Uncertainty:Any simulated metric(random variable)like gross losses,net losses,ceded to a contract,etc.has percentiles i.e.,a CDF2.Second-Order Uncer

20、tainty:The simulated percentiles themselves have inherent uncertainty and will change if we re-simulate i.e.,our attempt at Error Bars using the imperfect information from our simulation16Background and MotivationUncertainty FrameworksSo we can think of simulation uncertainty as having two levels:1.

21、First-Order Uncertainty:Any simulated metric(random variable)like gross losses,net losses,ceded to a contract,etc.has percentiles i.e.,a CDF2.Second-Order Uncertainty:The simulated percentiles themselves have inherent uncertainty and will change if we re-simulate i.e.,our attempt at Error Bars using

22、 the imperfect information from our simulation172.Spoiler Alert:Key Takeaways18Key TakeawaysThere is a well-known formula to estimate the CV upon re-simulation,but it uses the simulated gradient of the CDF around the percentileThe estimated CV of any simulated percentile itself varies on re-simulati

23、on with CV 1 because the simulated gradients are noisy Stay tuned for!Percentiles for RMS Cat XOL models have very high simulation uncertainty,whereas Quota Shares and Per Risk XOLs seem to converge quickerIf your cat reinsurance decision is driven by RMS tail statistics and isnt diversified by regi

24、on/peril then you will often need 2m realizations!19Key TakeawaysPercentiles for RMS Cat XOL models have very high simulation uncertainty,whereas Quota Shares and Per Risk XOLs seem to converge quickerThere is a well-known formula to estimate the CV upon re-simulation,but it uses the simulated gradi

25、ent of the CDF around the percentileIf your cat reinsurance decision is driven by RMS tail statistics and isnt diversified by region/peril then you will often need 2m realizations!The estimated CV of any simulated percentile itself varies on re-simulation with CV 1 because the simulated gradients ar

26、e noisy Stay tuned for!20Key TakeawaysPercentiles for RMS Cat XOL models have very high simulation uncertainty,whereas Quota Shares and Per Risk XOLs seem to converge quickerThere is a well-known formula to estimate the CV upon re-simulation,but it uses the simulated gradient of the CDF around the p

27、ercentileIf your cat reinsurance decision is driven by RMS tail statistics and isnt diversified by region/peril then you will often need 2m realizations!The estimated CV of any simulated percentile itself varies on re-simulation with CV 1 because the simulated gradients are noisy Stay tuned for!21Ke

28、y TakeawaysPercentiles for RMS Cat XOL models have very high simulation uncertainty,whereas Quota Shares and Per Risk XOLs seem to converge quickerThere is a well-known formula to estimate the CV upon re-simulation,but it uses the simulated gradient of the CDF around the percentileIf your cat reinsu

29、rance decision is driven by RMS tail statistics and isnt diversified by region/peril then you will often need 2m realizations!The estimated CV of any simulated percentile itself varies on re-simulation with CV 1 because the simulated gradients are noisy Stay tuned for!223.Whats the Deal with Second-

30、Order Uncertainty?23Second-Order UncertaintyVolatility of Percentile Estimates:Known DistributionsThere is a well-known asymptotic formula:11,where F(x)is the CDF of the underlying distribution,so F-1 is the graph below and F-1 is just the gradient of the graph below at each percentileFor high perce

31、ntiles,our estimates are more reliable as the samples increase or as the gradient decreasesLognormal example below:Theory matches our experiment on Slide 13 above!24Second-Order UncertaintyVolatility of Percentile Estimates:Known DistributionsThere is a well-known asymptotic formula:11,where F(x)is

32、the CDF of the underlying distribution,so F-1 is the graph below and F-1 is just the gradient of the graph below at each percentileFor high percentiles,our estimates are more reliable as the samples increase or as the gradient decreasesLognormal example below:Theory matches our experiment on Slide 1

33、3 above!25Second-Order UncertaintyVolatility of Percentile Estimates:Known DistributionsThere is a well-known asymptotic formula:11,where F(x)is the CDF of the underlying distribution,so F-1 is the graph below and F-1 is just the gradient of the graph below at each percentileFor high percentiles,our

34、 estimates are more reliable as the samples increase or as the gradient decreasesLognormal example below:Theory matches our experiment on Slide 13 above!26Second-Order UncertaintyVolatility of Percentile Estimates:Unknown DistributionsBut our simulations are crazy they dont have closed-form analytic

35、al distributions for 1!So we use one simulation to estimate the gradient at each percentile,focusing on its small neighborhoodLognormal example below to illustrate27Second-Order UncertaintyVolatility of Percentile Estimates:Unknown DistributionsBut our simulations are crazy they dont have closed-for

36、m analytical distributions for 1!So we use one simulation to estimate the gradient at each percentile,focusing on its small neighborhoodLognormal example below to illustrate28Second-Order UncertaintyVolatility of Percentile Estimates:Unknown DistributionsBut our simulations are crazy they dont have

37、closed-form analytical distributions for 1!So we use one simulation to estimate the gradient at each percentile,focusing on its small neighborhoodLognormal example below to illustrate the estimation at the 99.8th percentile29Second-Order UncertaintyVolatility of Percentile Estimates:Unknown Distribu

38、tionsInstead of aiming for,well aim for the =using the neighborhood of O observations around the percentile:1 =%The CVAs N grows,the Multiplier grows but Uncertainty%typically shrinks quicker due to the zoom-in effectLognormal example below:Although the Multiplier increased with more samples,the Unc

39、ertainty%decreased by far more;so the 500k 99.8th percentile is more reliable than the 10k 99.8th,as expected!30Second-Order UncertaintyVolatility of Percentile Estimates:Unknown DistributionsInstead of aiming for,well aim for the =using the neighborhood of O observations around the percentile:1 =%T

40、he CVAs N grows,the Multiplier grows but Uncertainty%typically shrinks quicker due to the zoom-in effectLognormal example below:Although the Multiplier increased with more samples,the Uncertainty%decreased by far more;so the 500k 99.8th percentile is more reliable than the 10k 99.8th,as expected!31S

41、econd-Order UncertaintyA New FrameworkInstead of aiming for,well aim for the =using the neighborhood of O observations around the percentile:1 =%So we can think of simulation uncertainty as having three levels:1.First-Order Uncertainty:Any simulated metric(random variable)like gross losses,net losse

42、s,ceded to a contract,etc.has percentiles i.e.,a CDF2.Second-Order Uncertainty:The simulated percentiles themselves have inherent uncertainty and will change if we re-simulate i.e.,our attempt at Error Bars using the imperfect information from our simulation3.Third-Order Uncertainty:The attempt at S

43、econd-Order Uncertainty(well focus on the CV)will vary by simulation,depending on the Uncertainty%in the simulated neighborhood of observations around the percentile i.e.,the Stretchiness/Blurriness of the Error Bars32Second-Order UncertaintyA New FrameworkInstead of aiming for,well aim for the =usi

44、ng the neighborhood of O observations around the percentile:1 =%So we can think of simulation uncertainty as having three levels:1.First-Order Uncertainty:Any simulated metric(random variable)like gross losses,net losses,ceded to a contract,etc.has percentiles i.e.,a CDF2.Second-Order Uncertainty:Th

45、e simulated percentiles themselves have inherent uncertainty and will change if we re-simulate i.e.,our attempt at Error Bars using the imperfect information from our simulation3.Third-Order Uncertainty:The attempt at Second-Order Uncertainty(well focus on the CV)will vary by simulation,depending on

46、 the Uncertainty%in the simulated neighborhood of observations around the percentile i.e.,the Stretchiness/Blurriness of the Error Bars33Whats the Deal with Third-Order Uncertainty?34Third-Order UncertaintyRecap and New DirectionWhen simulating a random variable we realized that the resulting percen

47、tiles(First-Order Uncertainty:CDF)themselves have uncertainty in the sense that they will change on re-simulation i.e.,they have Second-Order Uncertainty(Error Bars)Luckily,though,we have a nice formula to calculate the CV of any simulated percentile:1 =%But even that is uncertain,as it will change

48、on re-simulation depending on the Uncertainty%in the neighborhood.So the natural question to ask is How reliable is our?That is,how much should we expect this to change each time we re-simulate?Or How stretchy/blurry are our error bars?We will find the CV of at any given percentile!This will be our

49、measure of Third-Order Uncertainty,and the first place to look is in some simulated results on well-understood distributions35Third-Order UncertaintyRecap and New DirectionWhen simulating a random variable we realized that the resulting percentiles(First-Order Uncertainty:CDF)themselves have uncerta

50、inty in the sense that they will change on re-simulation i.e.,they have Second-Order Uncertainty(Error Bars)Luckily,though,we have a nice formula to calculate the CV of any simulated percentile:1 =%But even that is uncertain,as it will change on re-simulation depending on the Uncertainty%in the neig

51、hborhood.So the natural question to ask is How reliable is our?That is,how much should we expect this to change each time we re-simulate?Or How stretchy/blurry are our error bars?We will find the CV of at any given percentile!This will be our measure of Third-Order Uncertainty,and the first place to

52、 look is in some simulated results on well-understood distributions36Third-Order UncertaintyRecap and New DirectionWhen simulating a random variable we realized that the resulting percentiles(First-Order Uncertainty:CDF)themselves have uncertainty in the sense that they will change on re-simulation

53、i.e.,they have Second-Order Uncertainty(Error Bars)Luckily,though,we have a nice formula to calculate the CV of any simulated percentile:1 =%But even that is uncertain,as it will change on re-simulation depending on the Uncertainty%in the neighborhood.So the natural question to ask is How reliable i

54、s our?That is,how much should we expect this to change each time we re-simulate?Or How stretchy/blurry are our error bars?We will find the CV of at any given percentile!This will be our measure of Third-Order Uncertainty,and the first place to look is in some simulated results on well-understood dis

55、tributions37Third-Order UncertaintyRecap and New DirectionWhen simulating a random variable we realized that the resulting percentiles(First-Order Uncertainty:CDF)themselves have uncertainty in the sense that they will change on re-simulation i.e.,they have Second-Order Uncertainty(Error Bars)Luckil

56、y,though,we have a nice formula to calculate the CV of any simulated percentile:1 =%But even that is uncertain,as it will change on re-simulation depending on the Uncertainty%in the neighborhood.So the natural question to ask is How reliable is our?That is,how much should we expect this to change ea

57、ch time we re-simulate?Or How stretchy/blurry are our error bars?We will find the CV of at any given percentile!This will be our measure of Third-Order Uncertainty,and the first place to look is in some simulated results on well-understood distributions38Third-Order UncertaintyEmpirical Behavior:10

58、Sims of a Lognormal Percentiles CVs39Third-Order UncertaintyEmpirical Behavior:10 Sims of a Lognormal Percentiles CVsHow much do the CV estimates vary by simulation?What is the CV of the CVs at each percentile?40Third-Order UncertaintyEmpirical Behavior:10 Sims of a Lognormal Percentiles CVs41Third-

59、Order UncertaintyEmpirical Behavior:10 Sims of a Lognormal Percentiles CVs42Third-Order UncertaintyEmpirical Behavior:10 Sims of a Pareto Percentiles CVs43Third-Order UncertaintyEmpirical Behavior:10 Sims of a Pareto Percentiles CVsHow much do the CV estimates vary by simulation?What is the CV of th

60、e CVs at each percentile?44Third-Order UncertaintyEmpirical Behavior:10 Sims of a Pareto Percentiles CVs45Third-Order UncertaintyEmpirical Behavior:10 Sims of a Pareto Percentiles CVs4646Third-Order UncertaintyKey Results1.The 1=%2.The percentiles s themselves vary across simulations due to the rand

61、omness in%:regardless of the percentile or the distribution(Proof in Appendix),so is only reliable with many realizations and a big Observation Region on a well-behaved distributionAs an example,the error bands on the right might stretch up/down around 20%if I run a new simulation with O=30!4747Thir

62、d-Order UncertaintyKey Results1.The 1=%2.The percentiles s themselves vary across simulations due to the randomness in%:regardless of the percentile or the distribution(Proof in Appendix),so is only reliable with many realizations and a big Observation Region on a well-behaved distributionAs an exam

63、ple,the error bands on the right might stretch up/down around 20%if I run a new simulation with O=30!4848Third-Order UncertaintyKey Results1.The 1=%2.The percentiles s themselves vary across simulations due to the randomness in%:regardless of the percentile or the distribution(Proof in Appendix),so

64、is only reliable with many realizations and a big Observation Region on a well-behaved distributionAs an example,the error bands on the right might stretch up/down around 20%if I run a new simulation with O=30!495.Cat,Quota Share and PPR Case Studies:How Reliable are Our Percentiles?50How Reliable a

65、re Our Percentiles?Two Questions in PracticeAll results so far are asymptotic for well-behaved distributions its unclear whether the results hold at all for our simulations in the sub-10m realization regime.1.If the CV is small enough then it doesnt really matter how reliable it is!That is,if the as

66、ymptotic Second-Order Uncertainty is 0.5%,then Im happy with my simulated results and dont really care about my empirical or theoretical Second or Third-Order Uncertainty estimate.2.If the CV isnt small,then is it at least reliably close to?In particular,if we re-simulate with a new seed:i.Are the p

67、ercentiles actually within around 2 of the initially estimated CV?ii.Are the new estimated CVs wildly different to the initial ones i.e.,way more than off?51How Reliable are Our Percentiles?Two Questions in PracticeAll results so far are asymptotic for well-behaved distributions its unclear whether

68、the results hold at all for our simulations in the sub-10m realization regime.1.If the CV is small enough then it doesnt really matter how reliable it is!That is,if the asymptotic Second-Order Uncertainty is 0.5%,then Im happy with my simulated results and dont really care about my empirical or theo

69、retical Second or Third-Order Uncertainty estimate.2.If the CV isnt small,then is it at least reliably close to?In particular,if we re-simulate with a new seed:i.Are the percentiles actually within around 2 of the initially estimated CV?ii.Are the new estimated CVs wildly different to the initial on

70、es i.e.,way more than off?52How Reliable are Our Percentiles?Two Questions in PracticeAll results so far are asymptotic for well-behaved distributions its unclear whether the results hold at all for our simulations in the sub-10m realization regime.1.If the CV is small enough then it doesnt really m

71、atter how reliable it is!That is,if the asymptotic Second-Order Uncertainty is 0.5%,then Im happy with my simulated results and dont really care about my empirical or theoretical Second or Third-Order Uncertainty estimate.2.If the CV isnt small,then is it at least reliably close to?In particular,if

72、we re-simulate with a new seed:i.Are the percentiles actually within around 2 of the initially estimated CV?ii.Are the new estimated CVs wildly different to the initial ones i.e.,way more than off?53How Reliable are Our Percentiles?Two Questions in PracticeAll results so far are asymptotic for well-

73、behaved distributions its unclear whether the results hold at all for our simulations in the sub-10m realization regime.1.If the CV is small enough then it doesnt really matter how reliable it is!That is,if the asymptotic Second-Order Uncertainty is 0.5%,then Im happy with my simulated results and d

74、ont really care about my empirical or theoretical Second or Third-Order Uncertainty estimate.2.If the CV isnt small,then is it at least reliably close to?In particular,if we re-simulate with a new seed:i.Are the percentiles actually within around 2 of the initially estimated CV?ii.Are the new estima

75、ted CVs wildly different to the initial ones i.e.,way more than off?54How Reliable are Our Percentiles?Quota Share and Prop Per Risk Using RMS Seem Good!55How Reliable are Our Percentiles?Workers Comp XOL incl.Cat(RMS)Might be Good!56How Reliable are Our Percentiles?High Excess Cat Using RMS is Prob

76、lematic:Client 157How Reliable are Our Percentiles?High Excess Cat Using RMS is Problematic:Client 258How Reliable are Our Percentiles?High Excess Cat Using RMS is Problematic:Clients 3 and 4596.Practical Implications and Recommendations60Practical Implications and RecommendationsIdeally we would ju

77、st simulate enough realizations so virtually all of our tail statistics more-or-less converge,but this seems impossible for our cat models in the sub-10m simulation regimeThe estimated CV of any simulated percentile itself varies on re-simulation with CV 1 ,so we should aim for 100If your cat reinsu

78、rance decision is driven by RMS tail statistics and isnt diversified by region/peril then it is worthwhile simulating 2m realizations61Practical Implications and RecommendationsIdeally we would just simulate enough realizations so virtually all of our tail statistics more-or-less converge,but this s

79、eems impossible for our cat models in the sub-10m simulation regimeThe estimated CV of any simulated percentile itself varies on re-simulation with CV 1 ,so we should aim for 100If your cat reinsurance decision is driven by RMS tail statistics and isnt diversified by region/peril then it is worthwhi

80、le simulating 2m realizations62Practical Implications and RecommendationsIdeally we would just simulate enough realizations so virtually all of our tail statistics more-or-less converge,but this seems impossible for our cat models in the sub-10m simulation regimeThe estimated CV of any simulated per

81、centile itself varies on re-simulation with CV 1 ,so we should aim for 100If your cat reinsurance decision is driven by RMS tail statistics and isnt diversified by region/peril then it is worthwhile simulating 2m realizations63Open ProblemsAreas for Further Investigation and ImprovementRecommended R

82、ealizations:More robust justification/s;including understanding the impact of book size,retention/limit,etc.Irreducibly Empirical?Is it even possible to make theory-grounded recommendations given the asymptotic results,our convoluted distributions,highly varied contracts&exposures,etc.?Maybe a deep

83、theoretical understanding of RMSs end-to-end modeling could be used to infer something?AIR Implications:Better understanding of the implications for AIR-based models that include non-cat but for which we run 10k realizations to perfectly exhaust the catalog e.g.,does the a priori 0 simulation error

84、for the cat component produce low overall simulation error?New Generation RMS:Will upcoming RMS releases change our recommendations e.g.,if they shift to a kind of catalog like AIROther Distributions and Contracts:More work and experiments on other theoretical distributions,and on casualty cat/cyber

85、647.Appendices:Proof Outline and Further Details65Appendices:Proof Outline and Further Details1.Proof Outline and Thought Process()using the neighborhood of O observations in the neighborhood around the percentile:1 12 1Proof of(1)is simple using the asymptotic formula for the standard deviation,app

86、lying the inverse function derivative rule,and estimating the gradient as the gradient of the line segment connecting the left-most point of the neighborhood to the right-most pointProof of(2)relies on a handful of theorems,properties and techniques,but to give some intuition:Simulations can be thou

87、ght of as coming from U(0,1)and then just taking the corresponding x when mapped onto the CDF of the distribution of interest :because the Multiplier is a constant and 1 is asymptotically independent of the gaps(per section 3.2 of 2017)and approaches a constant using the Delta Method or Slutskys The

88、orem So we can just focus on the CV of RangeFor the Uniform,it is easily shown that :=+1 (+2),which clearly approaches 1.The proof follows straightforwardly from the fact that the Range is Beta where X is UniformFor other distributions,here are two possible intuitions before giving the rigorous proo

89、f:1.The uncertainty(CV)in Range for any distribution is explained fully by the uncertainty(CV)in Range for the Uniform because of the first bullet in this section2.In the neighborhood of the pth percentile,the gap until the next observation can be thought of as n samples of your RV all constantly tr

90、ying at the same time to hit a value around the neighborhood,each having()probability,and us waiting until one hits-this is essentially an Exponential distribution with parameter 1()by the scaling property.E.g.if n is 4 and f(x)is a half then its a wait time with expected value 2.Given that the gaps

91、 are asymptotically independent,Range is just the sum of these Exponentials,which has CV 1The rigorous proof of(2)takes Theorem 1 from 2017 which runs roughly like my 2 above,and then the sum of these Exponentials is an Erlang that produces the CV of 1.After writing all this,I found out that 2020 S2

92、.1 notes historical framings that are similar to mine but stop at the rather than the CV66Appendices:Proof Outline and Further Details2.Second-Order Uncertainty:Volatility of Percentile Estimates for Unknown DistributionsInstead of aiming for,well aim for the =using the neighborhood of O observation

93、s around the percentile:1 =%The MultiplierIncreases as N increases,because zooming in shrinks the domain over which the gradient is calculatedDecreases as the number of Observations in the neighborhood increases,reflecting more credible data/smoothingIs highest in the middle of the distribution beca

94、use#samples pth percentile is Bi(N,p)The Uncertainty%Increases if the neighborhood is more volatile,which is partially caused by adding Observations to it(all else equal)Decreases if the neighborhood is less volatile,which is partially caused by increasing the number of realizations in the simulatio

95、n so the neighborhood range clusters tighter with less uncertaintyIs highest at sparse extreme regions and very low in the middle of the distribution(usually)67Appendices:Proof Outline and Further Details3i.Second-Order Uncertainty:Volatility of Percentile Estimates for some known DistributionsInste

96、ad of aiming for,well aim for the =using the neighborhood of O observations around the percentile:1 =%68Appendices:Proof Outline and Further Details3ii.Second-Order Uncertainty:Volatility of Percentile Estimates for some known DistributionsInstead of aiming for,well aim for the =using the neighborho

97、od of O observations around the percentile:1 =%0%5%10%15%20%25%30%35%0%20%40%60%80%100%PercentileTheoretical Uncertainty%in Obs Region(O=30)Pareto N=500k and O=30Pareto N=1mand O=30Lognormal N=500k and O=30Lognormal N=1m and O=3099.99th Theoretical Uncertainty:17%99.99th Theoretical Uncertainty:8%99

98、.99th Theoretical Uncertainty:14%99.99th Theoretical Uncertainty:29%0%5%10%15%20%25%30%35%0%20%40%60%80%100%PercentileTheoretical Uncertainty%in Obs Region(O=15)Pareto N=500k and O=15Pareto N=1mand O=15Lognormal N=500k and O=15Lognormal N=1m and O=1599.99th Theoretical Uncertainty:8%99.99th Theoreti

99、cal Uncertainty:4%99.99th Theoretical Uncertainty:7%99.99th Theoretical Uncertainty:14%69Appendices:Proof Outline and Further Details3iii.Second-Order Uncertainty:Volatility of Percentile Estimates for some known DistributionsInstead of aiming for,well aim for the =using the neighborhood of O observ

100、ations around the percentile:1 =%0%2%4%6%8%10%12%0%20%40%60%80%100%PercentilePercentile CV EstimatorN=500k and O=302%Uncert(all Buckets)ParetoLognormalUncertainty in Obs Region99.99th CV:4.0%99.99th CV:6.8%0%2%4%6%8%10%12%0%20%40%60%80%100%PercentilePercentile CV EstimatorN=1m and O=301.5%Uncert(all

101、 Buckets)ParetoLognormalUncertainty in Obs Region99.99th CV:2.8%99.99th CV:4.6%70Appendices:Proof Outline and Further Details4i.Uncertainty Frameworks:Contextualizing Simulation ErrorSo we can think of simulation uncertainty as having two levels:1.First-Order Uncertainty:Any simulated metric(random

102、variable)like gross losses,net losses,ceded to a contract,etc.has percentiles i.e.,a CDF2.Second-Order Uncertainty:The simulated percentiles themselves have inherent uncertainty and will change if we re-simulate i.e.,our attempt at Error Bars using the imperfect information from our simulationAnd th

103、is can be contextualized within the following useful Error/Uncertainty decomposition:Process/Stochastic Error:The best we can do is model probabilities we almost never predict metrics with certainty,even if we have the best data,model,and parameters;and we use simulations to represent the randomness

104、Parameter Error:Even if our model structure and logic are perfect,our parameter selections are limited by the quality,quantity and projectibility of historical data&our parameter selection procedure/sModel Error:Even if weve perfectly selected parameters,the model inputs,structure and logic might no

105、t perfectly represent the dynamics of the thing being modeledData and Assumed Constants Error:There may be errors in the data provided,or in the assumed constants(like inflation)that we use71Appendices:Proof Outline and Further Details4ii.Uncertainty Frameworks:Broader Considerations and Value-AddTh

106、e present focus on Simulation Uncertainty ignores whether our data and model are any good to begin with!Minimizing Simulation Uncertainty is like carefully checking the spelling and grammar on an emailIt cant fix the underlying reasoning,content,structure,messaging,style,etc.It occasionally reveals

107、substantive issues,like when a typo or grammatical error changes the meaning of a sentenceIt shouldnt be the case that one of the drivers of results/YOYs is the random seed that we happened to useThink carefully about the other sources of uncertaintyData Error:Extensive data validation tests,YOY com

108、parisons,outlier detections,system upgrades/reviews,etc.Parameter Error:Sensitivity tests on key parameters,bootstrapping,parameter uncertainty in vendor models,etc.Model Error:Alternative selection methodologies,model structures&types,levels of granularity,etc.GCs value prop is essentially to help

109、clients navigate uncertaintyProvide sensitivity tests,uncertainty bands,appropriate caveats/limitation,and rounded resultsConsider Cat Model adjustments speak to the client and GC Model EvaluationUse brokers and GCs market knowledge to inform uncertainty on execution risk,market capacity,pricing,etc.A business of Marsh McLennan