Article
How to maximize the value of a CECL model validation
Nov 07, 2023 · Authored by Ivan Cilik, Sean Statz
Current Expected Credit Losses (CECL) models generally represent high risk compared to other models that lending institutions use. Not only does the CECL estimate go directly into an entity’s financial statement, but it is likely to be subject to audit and regulatory review. A model validation can help ensure you’ve completed all CECL requirements, while also preparing you for future regulatory requirements. It provides valuable insights related to sensitive inputs and assumptions in addition to testing model logic and algorithms. Even if you have already had a validation completed on your CECL model, it is still vital to ensure you are getting the most out of your validation and tie up the loose ends. Double check assumptions, focus on stress testing and back testing, while also preparing for regulatory exams.
What makes a model validation effective?
Financial institutions are often hyper-focused on external risks, however internal risks are just as important to catch and mitigate. The use of models invariably produces model risk, which is the potential for adverse consequences from decisions based on incorrect or misused model outputs and reports. While models themselves can be valuable management tools, they can pose risk if their output incorrectly informs important decisions that your financial institution makes which can lead to financial loss and reputation damage.
An independent model validation should be performed for all models developed in-house or by a third party. By completing an evaluation of your model’s conceptual soundness, you will be able to assess the quality of your model’s design and construction. This step in the model validation process should ensure that any decisions made during the process of model design and construction are well informed and consistent. Financial institutions should employ sensitivity analysis in model development and validation to check the impact of small changes in input and parameter values on model outputs to make sure they fall within an expected range. There are four important steps in the evaluation of your model’s conceptual soundness:
- Assessment: Assess the quality of model design and construction
- Comparison: Compare the model to alternative theories and approaches
- Limitations: Inspect for potential limitations
- Relevance: Check the relevance of the data used to build the model
Ongoing monitoring confirms that the model is appropriately implemented, is being used properly and is performing as intended. This step in the validation process is essential because it will allow you to evaluate whether changes in market conditions, clients or activity necessitate adjustment, redevelopment or replacement of the model. It will also allow you to verify that any extension of the model beyond its original scope is valid. With process verification, all components will be tested to ensure they are functioning as designed. Along with process verification, benchmarking can be used to compare a given model’s inputs and outputs to estimates from alternative internal or external data or models.
Outcomes analysis involves the comparison of model outputs to corresponding actual outcomes. Back testing is one form of outcomes analysis. Specifically, it involves the comparison of actual outcomes with model forecasts during a sample time period not used in model development and at an observation frequency that matches the forecast horizon or performance window of the model.
Key components of an effective model validation
Developing a proper framework for conducting model validations is key. Our framework development solution includes data flow mapping, establishing an ongoing monitoring plan, identifying information gaps, model testing and streamlining processes and resources. For most organizations, validation will be recommended at least annually.
There are four phases of a comprehensive model validation framework, including:
Developing and maintaining strong governance, policies and controls over the model risk management framework is fundamentally important to its effectiveness. All financial institutions that rely on models should implement an appropriate governance program that includes:
- Board and management oversight
- Updated policies and procedures
- Defined roles and responsibilities
- Ongoing assessment of model performance
- Identified model documentation and validation standards
Clear and comprehensive model documentation is critical to providing internal and external parties with an understanding of the final model. This includes thoroughly outlining all model assumptions and limitations, including potential impacts. Key focus areas within model documentation include:
- Internal controls including access controls, process controls and change management
- Procedures documentation including roles and responsibilities
- Qualitative factor framework documentation
Model data inputs verify the accuracy and cleanliness of the data going into the model. It is important to understand how the data your organization is using is mapped from your core system into the model, and also how the data is being segmented into pools. Depending on the CECL method selected, the data input can vary significantly from loan contractual data (payment type, amount, interest rate, etc.
Model assumptions and back testing stresses the importance of incorporating forward-looking data into the calculation of expected losses over the life of a loan. Forecasted economic conditions and voluntary and involuntary (default) prepayment rates are the main drivers of an institution’s model assumption set.
As for the approach to model assumptions, we suggest validating for primary areas:
- Economic forecast assumptions
- Default rates
- Prepayment speeds
- Discounting methodology
Model methodology and testing is, in our perspective, the most crucial part of a valuable validation implementation. There are two types of validations – standard, which is a basic testing of instruments and possibly a few shadow calculations; and replication, which requires obtaining all data sets and assumptions used by the institution using those data sets and assumptions to independently model a CECL estimate and then comparing the two results, category by category.
The basic approach to model methodology and testing entails:
- Review of model framework
- Full replication of the model
- Category-level reviews
- Assumption testing
- Scenario and stress testing analysis
- Back testing
Lessons learned and best practices
As our CECL and model risk management specialists have conducted model validations, here are some key best practices and lessons they have learned.
Internal data accuracy and understanding the loan level data that is going into the model each month/quarter is essential. It is also vital to understand the historical data that was included in the model, how much data the model is using to develop assumptions and whether peer data was involved in supplementing gaps in loss data.
If you are using a cash-flow based method, you need to understand each data point that is needed in the model and develop internal controls to ensure contractual data is complete and accurate each month.
All aspects of model risk management should also be covered by suitable policies, including:
- Model and model risk definitions
- The assessment of model risk
- Acceptable practices for model development, implementation and use
- Appropriate model validation activities
- Governance and controls over the model risk management process
The prioritization, scope and frequency of validation activities should be addressed in these policies.
For internally developed CECL models, model validation is key to ensure that the method follows CECL guidelines as well as the policies your organization has in place. A validation of the loss history and assumptions applied will ensure that the excel-based formulas are working properly. Also, pay attention to whether the process of updating the model each month/quarter is efficient and if the controls are ensuring the data is complete and accurate each time.
The benefits of completing a model validation include:
- Replication approach gives you full assurance on “black box” calculations
- Validation includes all parts of the CECL process from start to finish
- Data quality control
- You can learn more about how assumptions and methodology impact your profile
- You are able to obtain recommendations on recommended policy, documentation and governance
Understanding q-factors in a model validation
Qualitative factors (q-factors) are another key assumption that will impact an institution’s CECL estimate outside of the modeling. Typically, q-factors are included after the modeling is complete and the organization adjusts its historical estimate based on a variety of situations. Some of the major factors consist of changes in lending policies and procedures, changes in economic and business conditions, changes in past-due loans and changes in the value of underlying collateral to name a few. Since these factors are typically applied outside of the CECL modeling, many validations overlook this area. However, these factors will have varying impacts on an institution’s final CECL estimate based on their region, organization structure and historical loss experience.
For more information on best practices for financial institutions when implementing q-factors, check out our insight on the subject.
Model validation is a crucial aspect of model risk management. Refer to our webpage for more information. If you have any questions regarding CECL model validations, schedule a 30-minute meeting with one of our banking industry specialists.
Register now for our 2023 annual year-end banking webinar, taking place on Dec. 7 from 12-2 p.m. ET.
