WEBVTT NOTE This file was generated by Descript 00:00:00.460 --> 00:00:02.370 Samantha: Hello, this is Samantha Shares. 00:00:02.990 --> 00:00:06.560 This episode provides a comprehensive summary with extensive direct 00:00:06.560 --> 00:00:10.100 quotes from the OCC's Comptroller's Handbook booklet "Model Risk 00:00:10.100 --> 00:00:12.680 Management" released as Version 1.0, 00:00:12.880 --> 00:00:14.070 August 2021. 00:00:14.710 --> 00:00:18.380 Complex credit unions can improve operations via the principles 00:00:18.380 --> 00:00:19.630 discussed in the handbook. 00:00:20.389 --> 00:00:23.619 This podcast is educational and is not legal advice. 00:00:24.129 --> 00:00:28.149 We are sponsored by Credit Union Exam Solutions Incorporated, whose 00:00:28.149 --> 00:00:31.199 team has over two hundred and Forty years of National Credit 00:00:31.229 --> 00:00:33.149 Union Administration experience. 00:00:33.619 --> 00:00:37.209 We assist our clients with N C U A so they save time and money. 00:00:37.629 --> 00:00:41.579 If you are worried about a recent, upcoming or in process N C U A 00:00:41.579 --> 00:00:46.019 examination, reach out to learn how they can assist at Mark Treichel DOT COM. 00:00:46.449 --> 00:00:50.799 Also check out our other podcast called With Flying Colors where we provide tips 00:00:50.799 --> 00:00:53.369 on how to achieve success with N C U A. 00:00:54.099 --> 00:00:56.829 And now the comprehensive summary with direct quotes. 00:00:57.493 --> 00:00:58.903 Introduction and Purpose 00:00:59.584 --> 00:01:03.604 The Office of the Comptroller of the Currency's (O C C) Comptroller's Handbook 00:01:03.604 --> 00:01:08.664 booklet, "Model Risk Management," is "prepared for use by O C C examiners 00:01:08.664 --> 00:01:12.314 in connection with their examination and supervision of national banks, 00:01:12.514 --> 00:01:16.264 federal savings associations, and federal branches and agencies of 00:01:16.264 --> 00:01:18.054 foreign banking organizations." 00:01:18.834 --> 00:01:21.954 The booklet aligns with the principles laid out in the "Supervisory 00:01:21.954 --> 00:01:27.354 Guidance on Model Risk Management" conveyed by O C C Bulletin 2011-12. 00:01:28.191 --> 00:01:30.371 This booklet serves multiple purposes. 00:01:30.851 --> 00:01:35.221 It is "designed to guide examiners in performing consistent, high-quality 00:01:35.221 --> 00:01:39.511 model risk management examinations," it "presents the concepts and general 00:01:39.511 --> 00:01:44.481 principles of model risk management," and "informs and educates examiners about 00:01:44.481 --> 00:01:48.871 sound model risk management practices that should be assessed during an examination." 00:01:49.691 --> 00:01:53.181 Additionally, it "provides information needed to plan and coordinate 00:01:53.181 --> 00:01:57.731 examinations on model risk management, identify deficient practices, and 00:01:57.731 --> 00:01:59.421 conduct appropriate follow-up." 00:02:00.289 --> 00:02:02.249 Definition and Background of Models 00:02:02.926 --> 00:02:07.676 The O C C provides a clear definition of what constitutes a model: "For the 00:02:07.676 --> 00:02:11.976 purposes of this document, the term model refers to a quantitative method, 00:02:12.086 --> 00:02:16.446 system, or approach that applies statistical, economic, financial, or 00:02:16.446 --> 00:02:20.336 mathematical theories, techniques, and assumptions to process input 00:02:20.336 --> 00:02:22.286 data into quantitative estimates." 00:02:23.108 --> 00:02:25.318 A model consists of three components: 00:02:26.089 --> 00:02:26.419 1. 00:02:27.449 --> 00:02:30.919 "An information input component, which delivers assumptions 00:02:30.919 --> 00:02:32.119 and data to the model" 00:02:32.867 --> 00:02:33.257 2. 00:02:34.237 --> 00:02:38.127 "A processing component, which transforms inputs into estimates" 00:02:38.883 --> 00:02:39.283 3. 00:02:40.243 --> 00:02:43.393 "A reporting component, which translates the estimates into 00:02:43.393 --> 00:02:45.053 useful business information" 00:02:45.802 --> 00:02:49.882 The guidance acknowledges that "models are simplified representations of 00:02:49.882 --> 00:02:54.392 real-world relationships among observed characteristics, values, and events" 00:02:54.792 --> 00:02:59.122 and that "simplification is inevitable, due to the inherent complexity of those 00:02:59.122 --> 00:03:03.232 relationships, but also intentional, to focus attention on particular 00:03:03.232 --> 00:03:07.182 aspects considered to be most important for a given model application." 00:03:07.902 --> 00:03:11.772 It's important to understand that "models are never perfect" and their 00:03:11.772 --> 00:03:16.222 quality can be measured in many ways, including "precision, accuracy, 00:03:16.302 --> 00:03:20.332 discriminatory power, robustness, stability, and reliability." 00:03:21.172 --> 00:03:24.972 The handbook emphasizes that "in all situations, it is important to 00:03:24.972 --> 00:03:28.122 understand a model's capabilities and limitations given its 00:03:28.122 --> 00:03:29.982 simplifications and assumptions." 00:03:30.746 --> 00:03:35.216 In contrast to models, the guidance notes that "a quantitative tool not meeting 00:03:35.216 --> 00:03:39.806 the definition of a model described in the M R M Supervisory Guidance may apply 00:03:39.806 --> 00:03:44.306 deterministic rules or algorithms to process information and produce outcomes 00:03:44.306 --> 00:03:46.296 defined by the deterministic rules." 00:03:47.086 --> 00:03:50.756 The determination of "whether a quantitative tool is considered a model 00:03:50.756 --> 00:03:54.826 is bank-specific, and a conclusion regarding the tool's categorization 00:03:54.826 --> 00:03:58.136 should be based on a consideration of all relevant information." 00:03:58.925 --> 00:04:01.225 Expanded Use and Importance of Models 00:04:01.950 --> 00:04:05.890 Banks now "rely heavily on quantitative analysis and models in 00:04:05.890 --> 00:04:10.180 most aspects of financial decision making" and "routinely use models 00:04:10.180 --> 00:04:12.020 for a broad range of activities." 00:04:12.710 --> 00:04:14.690 Examples of model uses include: 00:04:15.516 --> 00:04:17.386 "Underwriting and managing credits" 00:04:18.010 --> 00:04:19.750 "Valuing trading exposures" 00:04:20.518 --> 00:04:21.158 "Pricing" 00:04:21.851 --> 00:04:22.721 "Risk hedging" 00:04:23.420 --> 00:04:25.030 "Managing client assets" 00:04:25.697 --> 00:04:28.747 "Measuring compliance with internally established limits" 00:04:29.378 --> 00:04:32.928 "Measuring compliance with laws and regulations (including consumer 00:04:32.928 --> 00:04:35.468 protection-related laws and regulations)" 00:04:36.195 --> 00:04:40.535 "Estimating the allowance for credit losses (A C L) and capital adequacy" 00:04:41.232 --> 00:04:42.942 "Issuing public disclosures" 00:04:43.673 --> 00:04:46.473 "Preventing and detecting fraud and money laundering" 00:04:47.209 --> 00:04:51.259 The expanded use of models reflects their benefits: "Models can help 00:04:51.299 --> 00:04:55.679 increase automation, transparency, and consistency of bank activities." 00:04:56.489 --> 00:05:00.619 However, the handbook notes that with these benefits come costs: "There is 00:05:00.619 --> 00:05:05.049 the direct cost of devoting resources to develop and implement models properly. 00:05:05.709 --> 00:05:09.839 There are also the potential indirect costs of relying on models, such as the 00:05:09.839 --> 00:05:14.299 possible adverse consequences (including financial loss) of decisions based on 00:05:14.299 --> 00:05:16.599 models that are incorrect or misused." 00:05:17.405 --> 00:05:21.385 The handbook points out that "the expanded use of models combined with 00:05:21.385 --> 00:05:25.805 their increasing complexity and value in decision making underscore the importance 00:05:25.805 --> 00:05:27.495 of sound model risk management." 00:05:28.353 --> 00:05:30.943 Technological Advances and Model Complexity 00:05:31.651 --> 00:05:34.931 The guidance acknowledges that "technological and analytical 00:05:34.931 --> 00:05:38.581 advances are contributing to increased model complexity and use." 00:05:39.351 --> 00:05:43.531 It specifically mentions artificial intelligence (A I), which is "broadly 00:05:43.531 --> 00:05:47.401 defined as the application of computational tools to address tasks 00:05:47.401 --> 00:05:49.801 traditionally requiring human analysis." 00:05:50.530 --> 00:05:54.830 Examples of A I uses in banks include "fraud detection and prevention, 00:05:54.980 --> 00:05:58.830 marketing, chatbots, credit underwriting, credit and fair lending 00:05:58.830 --> 00:06:01.360 risk management, robo-advising (i.e., 00:06:01.590 --> 00:06:05.730 an automated digital investment advisory service), trading algorithms and 00:06:05.730 --> 00:06:11.810 automation, financial marketing analysis, cybersecurity, Bank Secrecy Act/anti-money 00:06:11.810 --> 00:06:16.070 laundering (B S A/A M L) suspicious activity monitoring and customer due 00:06:16.070 --> 00:06:21.120 diligence, robotic process automation, and audit and independent risk management." 00:06:21.955 --> 00:06:25.775 The handbook notes that "some A I may meet the definition of a model" 00:06:26.025 --> 00:06:29.705 while other A I outputs "are not always quantitative in nature." 00:06:30.365 --> 00:06:34.675 Regardless of how A I is classified, "the associated risk management should 00:06:34.675 --> 00:06:38.415 be commensurate with the level of risk of the function that the A I supports." 00:06:39.176 --> 00:06:41.436 Risks Associated with the Use of Models 00:06:42.146 --> 00:06:45.856 The handbook identifies eight categories of risk for bank supervision 00:06:45.856 --> 00:06:50.676 purposes: "credit, interest rate, liquidity, price, operational, 00:06:50.756 --> 00:06:53.236 compliance, strategic, and reputation." 00:06:53.966 --> 00:06:57.706 It notes that "model use can affect risk in all eight categories of 00:06:57.706 --> 00:07:02.166 risk" and that "the use of models can increase or decrease risk in each 00:07:02.166 --> 00:07:06.386 risk category depending on the models' purpose, use, and the effectiveness of 00:07:06.386 --> 00:07:08.366 any relevant model risk management." 00:07:09.196 --> 00:07:13.876 Model risk is defined as "the potential for adverse consequences from decisions 00:07:13.876 --> 00:07:18.166 based on incorrect or misused model outputs and reports" which "can lead 00:07:18.166 --> 00:07:22.436 to financial loss, poor business and strategic decision making, or 00:07:22.436 --> 00:07:24.286 damage to a bank's reputation." 00:07:25.035 --> 00:07:29.135 The handbook explains that model risk occurs primarily for two reasons: 00:07:29.910 --> 00:07:30.250 1. 00:07:31.290 --> 00:07:34.600 "The model may have fundamental errors and may produce inaccurate 00:07:34.630 --> 00:07:39.000 outputs when viewed against the design objective and intended business uses." 00:07:39.807 --> 00:07:40.077 2. 00:07:41.127 --> 00:07:44.027 "The model may be used incorrectly or inappropriately. 00:07:44.637 --> 00:07:48.477 Even a fundamentally sound model producing accurate outputs consistent 00:07:48.477 --> 00:07:51.947 with the design objective of the model may exhibit high model risk 00:07:52.237 --> 00:07:54.247 if it is misapplied or misused." 00:07:55.045 --> 00:07:59.105 The guidance emphasizes that "banks should identify the sources of risk 00:07:59.135 --> 00:08:03.355 and assess the magnitude" and that "model risk increases with greater 00:08:03.355 --> 00:08:07.785 model complexity, higher uncertainty about inputs and assumptions, broader 00:08:07.785 --> 00:08:10.065 use, and larger potential impact." 00:08:10.828 --> 00:08:11.868 Strategic Risk 00:08:12.581 --> 00:08:17.461 Strategic risk is defined as "the risk to current or projected financial condition 00:08:17.461 --> 00:08:21.851 and resilience arising from adverse business decisions, poor implementation 00:08:21.851 --> 00:08:25.811 of business decisions, or lack of responsiveness to changes in the banking 00:08:25.811 --> 00:08:27.731 industry and operating environment." 00:08:28.587 --> 00:08:32.057 The handbook notes that "the board of directors and senior management 00:08:32.127 --> 00:08:35.367 are the key decision makers that drive the strategic direction of 00:08:35.367 --> 00:08:38.817 the bank and establish a governance framework for using models." 00:08:39.567 --> 00:08:44.137 Strategic risk increases when "models and associated risk management do not 00:08:44.137 --> 00:08:48.747 keep pace with strategic changes, the capability of employees, the operating 00:08:48.747 --> 00:08:51.157 environment, and regulatory requirements." 00:08:51.976 --> 00:08:53.066 Operational Risk 00:08:53.812 --> 00:08:58.282 Operational risk is described as "the risk to current or projected financial 00:08:58.282 --> 00:09:02.032 condition and resilience arising from inadequate or failed internal 00:09:02.032 --> 00:09:07.482 processes or systems, human errors or misconduct, or adverse external events." 00:09:08.258 --> 00:09:12.518 The guidance states that "operational risk is the primary risk associated 00:09:12.518 --> 00:09:16.858 with the use of models" and can result from "fundamental errors in a model when 00:09:16.858 --> 00:09:20.968 viewed against the design objective and intended business uses without sufficient 00:09:20.968 --> 00:09:25.228 use of model overlays and adjustments when model limitations become apparent." 00:09:26.068 --> 00:09:30.358 Operational risk can also increase when "personnel who do not have sufficient 00:09:30.358 --> 00:09:35.088 skills and training to develop, implement, use, and validate the bank's models." 00:09:35.878 --> 00:09:36.908 Reputation Risk 00:09:37.654 --> 00:09:41.264 Reputation risk is "the risk to current or projected financial 00:09:41.264 --> 00:09:44.614 condition and resilience arising from negative public opinion." 00:09:45.484 --> 00:09:49.604 The handbook notes that "inadequate policies and processes, operational 00:09:49.604 --> 00:09:53.894 breakdowns, or other weaknesses in any aspect of model risk management or 00:09:53.894 --> 00:09:56.084 governance can increase reputation risk." 00:09:56.884 --> 00:10:00.654 Specifically, "a bank could incur reputation risk from biased data 00:10:00.694 --> 00:10:05.954 outcomes, data losses, noncompliance with regulations, fraud, downtime, and 00:10:05.954 --> 00:10:07.864 insufficient consumer protections." 00:10:08.648 --> 00:10:09.648 Compliance Risk 00:10:10.414 --> 00:10:14.794 Compliance risk is described as "the risk to current or projected financial 00:10:14.794 --> 00:10:19.554 condition and resilience arising from violations of laws or regulations, or 00:10:19.554 --> 00:10:24.034 from nonconformance with prescribed practices, internal bank policies and 00:10:24.034 --> 00:10:26.254 procedures, or ethical standards." 00:10:27.010 --> 00:10:30.910 The handbook states that "compliance risk is elevated when banks do not 00:10:30.910 --> 00:10:35.670 comply with model-related laws and regulations" and when "models result in 00:10:35.670 --> 00:10:40.170 potential discrimination on a prohibited basis or other violations of consumer 00:10:40.170 --> 00:10:42.720 protection-related laws and regulations." 00:10:43.441 --> 00:10:47.581 It specifically mentions that "a bank's fair lending compliance risk could 00:10:47.581 --> 00:10:52.031 increase when a bank's credit decisioning models include algorithms, variables, 00:10:52.141 --> 00:10:56.001 or other processes that result in disparate impact on credit applicants or 00:10:56.001 --> 00:11:01.061 customers based on prohibited factors, such as race, ethnicity, or sex." 00:11:01.867 --> 00:11:02.587 Credit Risk 00:11:03.343 --> 00:11:07.403 Credit risk is defined as "the risk to current or projected financial 00:11:07.403 --> 00:11:11.003 condition and resilience arising from an obligor's failure to meet 00:11:11.003 --> 00:11:14.993 the terms of any contract with the bank or otherwise perform as agreed." 00:11:15.797 --> 00:11:20.407 The handbook notes that "banks use models to increase efficiency in all stages 00:11:20.407 --> 00:11:24.867 of lending" and that "if credit risk models do not incorporate underwriting 00:11:24.867 --> 00:11:29.357 changes in a timely manner, flawed and costly business decisions could occur." 00:11:30.077 --> 00:11:33.937 However, "models that are well-designed and effectively managed can help 00:11:33.937 --> 00:11:38.157 management make prudent risk selection and monitor and manage credit risk." 00:11:38.955 --> 00:11:39.925 Liquidity Risk 00:11:40.650 --> 00:11:45.150 Liquidity risk is "the risk to current or projected financial condition and 00:11:45.150 --> 00:11:49.210 resilience arising from an inability to meet obligations when they come due." 00:11:50.120 --> 00:11:54.060 The handbook states that "liquidity risk can increase because of inaccurate 00:11:54.100 --> 00:11:58.450 or untimely inputs, assumptions, model adjustments, and outputs." 00:11:59.182 --> 00:12:04.022 Examples of common sources of liquidity risk in modeling include "unsupported 00:12:04.082 --> 00:12:08.202 or unreasonable contingent funding assumptions; stress scenarios that 00:12:08.202 --> 00:12:13.032 do not consider all relevant legal or regulatory constraints; and inaccurate 00:12:13.112 --> 00:12:15.222 or unsupported behavioral assumptions." 00:12:15.993 --> 00:12:17.023 Interest Rate Risk 00:12:17.768 --> 00:12:21.518 Interest rate risk is "the risk to earnings or capital arising 00:12:21.518 --> 00:12:23.168 from movements in interest rates." 00:12:24.018 --> 00:12:27.988 The handbook notes that "interest rate risk models depend on assumptions to 00:12:27.988 --> 00:12:33.358 accurately project cash flows from assets, liabilities, and off-balance-sheet items." 00:12:34.189 --> 00:12:38.469 Common interest rate modeling issues include "failing to assess potential 00:12:38.469 --> 00:12:42.979 exposures over a sufficiently wide range of interest rate movements," "failing 00:12:42.979 --> 00:12:47.199 to modify or vary assumptions for products with embedded options to reflect 00:12:47.199 --> 00:12:52.309 individual rate scenarios," and "failing to periodically assess the reasonableness 00:12:52.339 --> 00:12:54.009 and accuracy of assumptions." 00:12:54.817 --> 00:12:55.567 Price Risk 00:12:56.339 --> 00:13:00.899 Price risk is defined as "the risk to current or projected financial condition 00:13:00.899 --> 00:13:05.189 and resilience arising from changes in the value of either trading portfolios 00:13:05.289 --> 00:13:09.019 or other obligations that are entered into as part of distributing risk." 00:13:09.827 --> 00:13:14.027 The handbook states that "a bank incurs heightened price risk when trading 00:13:14.027 --> 00:13:17.907 instruments with prices that are hard to model," such as "instruments that 00:13:17.907 --> 00:13:22.267 are illiquid or trade infrequently," "newer instruments," and "instruments 00:13:22.267 --> 00:13:26.017 with fair value that depends on accurately modeling human behavior." 00:13:26.799 --> 00:13:27.719 Risk Management 00:13:28.437 --> 00:13:33.457 The handbook emphasizes that "each bank should identify, measure, monitor, and 00:13:33.457 --> 00:13:37.787 control risk by implementing an effective risk management system appropriate for the 00:13:37.787 --> 00:13:40.187 size and complexity of its operations." 00:13:40.867 --> 00:13:44.687 It notes that "model risk should be managed like other types of risk" 00:13:45.007 --> 00:13:48.677 and that "developing and maintaining strong governance, policies, 00:13:48.737 --> 00:13:52.287 and controls over the model risk management framework is fundamentally 00:13:52.287 --> 00:13:53.877 important to its effectiveness." 00:13:54.641 --> 00:13:59.011 The guidance advises that "the extent and sophistication of a bank's governance 00:13:59.011 --> 00:14:03.591 function is expected to align with the extent and sophistication of model usage" 00:14:04.071 --> 00:14:08.341 and that "materiality is an important consideration in model risk management." 00:14:09.167 --> 00:14:13.557 The handbook states that "even with skilled modeling and robust validation, 00:14:13.667 --> 00:14:17.157 model risk cannot be eliminated, so other tools should be used to 00:14:17.157 --> 00:14:19.107 manage model risk effectively." 00:14:19.827 --> 00:14:24.097 These tools include "establishing limits on model use, monitoring model 00:14:24.097 --> 00:14:29.457 performance, adjusting or revising models over time, and supplementing model results 00:14:29.457 --> 00:14:31.507 with other analysis and information." 00:14:32.341 --> 00:14:32.941 Governance 00:14:33.563 --> 00:14:37.573 Sound governance includes "board and management oversight, policies 00:14:37.573 --> 00:14:42.213 and procedures, a system of internal controls, internal audit, a model 00:14:42.213 --> 00:14:44.143 inventory, and documentation." 00:14:44.983 --> 00:14:48.943 The handbook introduces the concept of three lines of defense: "(1) 00:14:48.943 --> 00:14:53.283 frontline units, business units, or functions that create risk; (2) 00:14:53.283 --> 00:14:58.043 independent risk management (I R M), loan review, compliance officer, and 00:14:58.043 --> 00:15:01.813 chief credit officer to assess risk independent of the units that create 00:15:01.813 --> 00:15:06.203 risk; and (3) internal audit, which provides independent assurance." 00:15:06.960 --> 00:15:10.760 The guidance emphasizes the importance of "effective challenge" described 00:15:10.760 --> 00:15:15.010 as "critical analysis by objective, informed parties who can identify 00:15:15.010 --> 00:15:18.620 model limitations and assumptions and produce appropriate changes." 00:15:19.340 --> 00:15:22.650 Effective challenge "depends on a combination of incentives, 00:15:22.780 --> 00:15:24.350 competence, and influence." 00:15:25.129 --> 00:15:26.689 Board and Management Oversight 00:15:27.486 --> 00:15:31.386 The handbook states that "model risk governance is provided at the highest 00:15:31.386 --> 00:15:35.746 level by the board of directors and senior management when they establish a bank-wide 00:15:35.746 --> 00:15:37.686 approach to model risk management." 00:15:38.426 --> 00:15:42.296 The board is "responsible for setting the tone at the top and overseeing 00:15:42.296 --> 00:15:46.126 management's role in fostering and maintaining a sound corporate culture." 00:15:46.858 --> 00:15:50.438 Senior management is "responsible for day-to-day implementation 00:15:50.438 --> 00:15:52.208 of sound model risk management." 00:15:53.068 --> 00:15:57.098 Their duties include "establishing adequate policies and procedures and 00:15:57.098 --> 00:16:01.798 ensuring compliance, assigning competent staff, overseeing model development 00:16:01.828 --> 00:16:06.498 and implementation, evaluating model results, ensuring effective challenge, 00:16:06.628 --> 00:16:10.598 reviewing validation and internal audit findings, and taking prompt 00:16:10.598 --> 00:16:12.478 remedial action when necessary." 00:16:13.274 --> 00:16:17.294 The handbook notes that "sound governance encourages all key stakeholders to 00:16:17.294 --> 00:16:21.424 have effective communication, be transparent, and actively participate 00:16:21.424 --> 00:16:25.674 in model risk management" and that "appropriate change management before 00:16:25.704 --> 00:16:29.964 implementing or when changing models and related technologies is an important 00:16:29.964 --> 00:16:31.674 component of model governance." 00:16:32.482 --> 00:16:32.982 Personnel 00:16:33.768 --> 00:16:37.678 The guidance emphasizes that "the bank should have competent and qualified 00:16:37.678 --> 00:16:41.868 personnel to execute and oversee model risk management" and that "the 00:16:41.868 --> 00:16:45.128 skills and expertise of management and other personnel should be 00:16:45.128 --> 00:16:49.238 commensurate with the nature, extent, and complexity of the use of models." 00:16:50.022 --> 00:16:53.602 The handbook suggests that "well-thought-out personnel development, 00:16:53.702 --> 00:16:58.132 recruiting, succession planning, and compensation processes promote successful 00:16:58.132 --> 00:17:02.182 hiring and retention of individuals with highly technical skills for model 00:17:02.182 --> 00:17:06.622 development and for model risk management across the three lines of defense." 00:17:07.345 --> 00:17:08.305 Model Owners 00:17:08.981 --> 00:17:13.351 Model owners are described as having "ultimate accountability for model use 00:17:13.351 --> 00:17:17.311 and performance within the framework set by bank policies and procedures." 00:17:18.051 --> 00:17:21.761 They should "ensure that models are properly developed, implemented, 00:17:21.891 --> 00:17:25.611 and used" and "that models in use have undergone appropriate 00:17:25.611 --> 00:17:27.791 validation and approval processes." 00:17:28.568 --> 00:17:32.998 Model owners are generally responsible for implementing policies and standards, 00:17:33.158 --> 00:17:37.458 establishing and maintaining processes for identifying and measuring risks, 00:17:37.718 --> 00:17:41.888 maintaining internal controls, and maintaining documentation standards. 00:17:42.567 --> 00:17:44.447 Independent Risk Management Staff 00:17:45.189 --> 00:17:49.949 As the second line of defense, independent risk management (I R M) "typically 00:17:49.989 --> 00:17:54.109 oversees business unit risk-taking and risk management activities" and 00:17:54.239 --> 00:17:57.969 "validates and challenges business unit testing and other first-line 00:17:57.969 --> 00:17:59.979 model risk management processes." 00:18:00.761 --> 00:18:05.001 I R M's responsibilities include implementing policies and standards, 00:18:05.111 --> 00:18:09.661 establishing processes for identifying and measuring risks, validating model 00:18:09.691 --> 00:18:14.711 inputs and outputs, confirming models are performing as intended, assessing issues 00:18:14.711 --> 00:18:18.621 for themes or patterns, and reporting to senior management and the board. 00:18:19.387 --> 00:18:20.277 Internal Audit 00:18:21.075 --> 00:18:25.375 As the third line of defense, internal audit "reviews model governance 00:18:25.375 --> 00:18:28.965 and risk management and provide independent assurance to the board on 00:18:28.965 --> 00:18:32.985 the effectiveness of governance, risk management, and internal controls." 00:18:33.780 --> 00:18:37.550 The handbook states that internal audit "should assess the overall 00:18:37.550 --> 00:18:41.120 effectiveness of the model risk management framework, including the 00:18:41.120 --> 00:18:45.490 framework's ability to address both types of model risk for individual models 00:18:45.630 --> 00:18:49.450 and in the aggregate" and "evaluate whether model risk management is 00:18:49.450 --> 00:18:51.930 comprehensive, rigorous, and effective." 00:18:52.706 --> 00:18:57.216 Internal audit is typically responsible for verifying acceptable policies are 00:18:57.216 --> 00:19:01.436 in place, documenting and reporting findings, verifying records of model 00:19:01.436 --> 00:19:05.556 use and validation, assessing the accuracy and completeness of the model 00:19:05.556 --> 00:19:10.016 inventory, evaluating processes for establishing and monitoring limits, 00:19:10.266 --> 00:19:14.226 and evaluating the objectivity and competence of validation participants. 00:19:14.942 --> 00:19:16.422 Policies and Procedures 00:19:17.132 --> 00:19:20.642 The guidance states that "consistent with good business practices and 00:19:20.642 --> 00:19:24.862 existing supervisory expectations, banks should formalize model risk 00:19:24.862 --> 00:19:28.672 management activities with policies and the procedures to implement them." 00:19:29.422 --> 00:19:33.272 These policies should be "commensurate with the bank's relative complexity, 00:19:33.422 --> 00:19:37.852 business activities, corporate culture, and overall organizational structure" 00:19:38.262 --> 00:19:42.012 and should be approved by the board or its delegates and reviewed annually. 00:19:42.805 --> 00:19:46.495 Policies and procedures regarding model risk management may include 00:19:46.495 --> 00:19:50.055 descriptions of governance and controls, definitions of models and 00:19:50.055 --> 00:19:54.625 model risk, acceptable practices for model development and implementation, 00:19:54.735 --> 00:19:59.295 roles and responsibilities, standards for model inventory, fair lending 00:19:59.295 --> 00:20:02.835 considerations, and controls for model development and use. 00:20:03.513 --> 00:20:04.383 Risk Assessment 00:20:05.164 --> 00:20:09.274 The handbook explains that "assessing risk includes identifying and 00:20:09.274 --> 00:20:13.274 measuring the sources and magnitude of risks associated with model use." 00:20:14.034 --> 00:20:17.554 This is "particularly important as a bank increases in size and 00:20:17.554 --> 00:20:22.124 complexity, the use of models becomes more widespread, or model results 00:20:22.124 --> 00:20:24.374 significantly influence decision making." 00:20:25.209 --> 00:20:29.589 A sound model risk assessment process generally "identifies risk both 00:20:29.589 --> 00:20:33.449 from individual models and models in the aggregate," "identifies 00:20:33.449 --> 00:20:37.429 the model's capabilities and limitations," and "measures the risks 00:20:37.429 --> 00:20:41.379 associated with model activities accurately and in a timely manner." 00:20:42.215 --> 00:20:42.675 Planning 00:20:43.384 --> 00:20:47.024 The guidance states that "effective governance for modeling begins with 00:20:47.024 --> 00:20:51.244 appropriate planning" and that "a clear statement of purpose is typically 00:20:51.244 --> 00:20:54.694 the first step to developing models aligned with the intended use." 00:20:55.466 --> 00:20:59.536 Key planning considerations include identifying stakeholders, performing 00:20:59.536 --> 00:21:03.706 risk assessments, making informed decisions about implementing new models, 00:21:03.936 --> 00:21:08.296 understanding the purpose and limitations of models, integrating new technology 00:21:08.296 --> 00:21:12.256 with legacy systems, and ensuring appropriate controls for monitoring 00:21:12.256 --> 00:21:14.236 outputs that may be discriminatory. 00:21:14.954 --> 00:21:16.044 Model Inventory 00:21:16.761 --> 00:21:20.551 The handbook advises that "banks should maintain a comprehensive set 00:21:20.551 --> 00:21:24.301 of information for models implemented for use, under development for 00:21:24.301 --> 00:21:28.831 implementation, or recently retired" and that "a specific party should 00:21:28.831 --> 00:21:32.901 also be charged with maintaining a firm-wide inventory of all models." 00:21:33.663 --> 00:21:36.893 A comprehensive model inventory may include information such 00:21:36.893 --> 00:21:41.513 as model identifier, version, ownership, status, purpose, inputs 00:21:41.513 --> 00:21:45.903 and outputs, risks, validation activities, issues or limitations, 00:21:45.993 --> 00:21:47.703 and expected validity timeframe. 00:21:48.502 --> 00:21:49.402 Documentation 00:21:50.105 --> 00:21:53.775 The guidance emphasizes that "without adequate documentation, 00:21:53.915 --> 00:21:56.915 model risk assessment and management will be ineffective." 00:21:57.735 --> 00:22:01.875 Documentation should be "sufficiently detailed so that parties unfamiliar 00:22:01.875 --> 00:22:05.815 with a model can understand how the model operates, its limitations, 00:22:05.905 --> 00:22:07.165 and its key assumptions." 00:22:07.916 --> 00:22:12.326 Documentation benefits developers, users, and risk management personnel, 00:22:12.636 --> 00:22:16.596 and should include information supporting decisions related to model selection, 00:22:16.956 --> 00:22:21.886 testing, governance, development, internal controls, and third-party risk management. 00:22:22.669 --> 00:22:23.569 Data Management 00:22:24.250 --> 00:22:28.610 The handbook states that "to measure risk effectively, the data inputs for 00:22:28.610 --> 00:22:32.860 models should be reliable" and that there should be "rigorous assessment 00:22:32.860 --> 00:22:36.370 of data quality and relevance, and appropriate documentation." 00:22:37.131 --> 00:22:41.121 Sound data management includes "providing reasonable and reconcilable 00:22:41.121 --> 00:22:45.401 support for qualitative factors," regularly analyzing "the integrity 00:22:45.401 --> 00:22:49.351 and applicability of internal and external information sources and related 00:22:49.351 --> 00:22:53.921 controls," and appropriately handling data proxies and third-party data. 00:22:54.646 --> 00:22:57.276 Model Development, Implementation, and Use 00:22:57.942 --> 00:23:02.442 The guidance emphasizes that "model risk management begins with robust model 00:23:02.442 --> 00:23:07.082 development, implementation, and use" and that the process "should include 00:23:07.082 --> 00:23:10.912 disciplined and knowledgeable development and implementation processes that are 00:23:10.912 --> 00:23:15.362 consistent with the situation and goals of the model user and with bank policy." 00:23:16.158 --> 00:23:18.188 Model Development and Implementation 00:23:18.911 --> 00:23:22.611 The handbook states that "an effective development process begins 00:23:22.611 --> 00:23:25.881 with a clear statement of purpose to ensure that model development 00:23:25.911 --> 00:23:27.671 is aligned with the intended use." 00:23:28.391 --> 00:23:32.881 It should include documentation of "the design, theory, and logic underlying 00:23:32.881 --> 00:23:36.451 the model," explanation of "the model methodologies and processing 00:23:36.451 --> 00:23:40.671 components," and comparison "with alternative theories and approaches." 00:23:41.411 --> 00:23:45.291 The development process should produce "documented evidence in support of 00:23:45.291 --> 00:23:49.261 all model choices, including the overall theoretical construction, 00:23:49.381 --> 00:23:53.441 key assumptions, data, and specific mathematical calculations." 00:23:54.194 --> 00:23:58.224 For third-party models, banks should "ensure that there are appropriate 00:23:58.224 --> 00:24:03.164 processes in place for selecting vendor models" and require vendors to "provide 00:24:03.164 --> 00:24:06.884 appropriate testing results that show their product works as expected." 00:24:07.726 --> 00:24:08.276 Testing 00:24:08.958 --> 00:24:13.108 The guidance describes testing as "an integral part of model development" 00:24:13.188 --> 00:24:17.288 that includes "checking the model's accuracy, demonstrating that the model 00:24:17.288 --> 00:24:22.378 is robust and stable, assessing potential limitations, and evaluating the model's 00:24:22.378 --> 00:24:24.958 behavior over a range of input values." 00:24:25.688 --> 00:24:30.098 Testing should be "applied to actual circumstances under a variety of market 00:24:30.098 --> 00:24:34.078 conditions, including scenarios that are outside the range of ordinary 00:24:34.078 --> 00:24:38.768 expectations," and should "encompass the variety of products or applications 00:24:38.768 --> 00:24:40.438 for which the model is intended." 00:24:41.267 --> 00:24:42.397 Ongoing Development 00:24:43.116 --> 00:24:47.206 The handbook notes that "models are regularly adjusted to take into account 00:24:47.206 --> 00:24:51.196 new data or techniques or because of deterioration in performance." 00:24:51.896 --> 00:24:56.386 It advises that "material changes in model structure or technique, and all 00:24:56.386 --> 00:25:00.596 model redevelopment, should be subject to validation activities of appropriate 00:25:00.596 --> 00:25:02.736 range and rigor before implementation." 00:25:03.548 --> 00:25:04.408 Model Use 00:25:05.013 --> 00:25:09.303 The guidance states that "model use provides additional opportunity to test 00:25:09.303 --> 00:25:13.793 whether a model is functioning effectively and to assess its performance over time as 00:25:13.793 --> 00:25:16.263 conditions and model applications change." 00:25:17.083 --> 00:25:21.193 Model users can "provide valuable business insight during the development 00:25:21.193 --> 00:25:25.673 process" and may "question the methods or assumptions underlying the models." 00:25:26.508 --> 00:25:30.948 However, the handbook cautions that "challenge from model users may be weak 00:25:30.988 --> 00:25:35.308 if the model does not materially affect their results," and that user challenges 00:25:35.568 --> 00:25:39.728 "tend not to be comprehensive because they focus on aspects of models that have the 00:25:39.728 --> 00:25:44.178 most direct impact on the user's measured business performance or compensation." 00:25:44.961 --> 00:25:46.871 Model Overlays and Adjustments 00:25:47.552 --> 00:25:52.402 The guidance defines model overlays as "judgmental or qualitative adjustments to 00:25:52.402 --> 00:25:57.592 model inputs or outputs to compensate for model, data, or other known limitations." 00:25:58.342 --> 00:26:02.122 It advises that "sound model risk management includes policies and 00:26:02.122 --> 00:26:06.272 processes regarding the review, approval, use, and back-testing of 00:26:06.272 --> 00:26:08.182 model overlays and adjustments." 00:26:08.951 --> 00:26:13.191 The handbook emphasizes that "model overlays and adjustments should not be 00:26:13.191 --> 00:26:16.761 viewed as a solution that dissuades the bank from making improvements to the 00:26:16.761 --> 00:26:21.961 model" and that banks "typically have a process to monitor and analyze overlays 00:26:21.961 --> 00:26:25.951 and adjustments over time and address underlying limitations and issues." 00:26:26.736 --> 00:26:27.376 Reporting 00:26:28.047 --> 00:26:32.077 The guidance states that "reports used for business decision making play a 00:26:32.077 --> 00:26:36.787 critical role in model risk management" and should "be clear and comprehensible." 00:26:37.587 --> 00:26:41.077 Effective reporting "enables senior management and the board to 00:26:41.077 --> 00:26:42.947 understand the bank's model risk." 00:26:43.767 --> 00:26:48.317 Reports presented to the board "typically highlight performance measures, trends, 00:26:48.387 --> 00:26:52.717 and variances, rather than presenting the information as raw data" and may 00:26:52.717 --> 00:26:57.157 include measures on "the volume of models considered high risk," "models 00:26:57.157 --> 00:27:01.957 with temporary exemptions or provisional approvals," and "underperforming models." 00:27:02.716 --> 00:27:03.826 Model Validation 00:27:04.541 --> 00:27:09.141 The handbook describes model validation as "the set of processes and activities 00:27:09.141 --> 00:27:13.421 intended to verify that models are performing as expected, in line with their 00:27:13.421 --> 00:27:15.551 design objectives and business uses." 00:27:16.301 --> 00:27:20.331 It states that "effective validation helps ensure that models are sound" 00:27:20.781 --> 00:27:24.471 and "also identifies potential limitations and assumptions, and 00:27:24.471 --> 00:27:26.311 assesses their possible impact." 00:27:27.107 --> 00:27:32.297 The guidance emphasizes that "all model components, including input, processing, 00:27:32.357 --> 00:27:36.407 and reporting, should be subject to validation" and that "the rigor and 00:27:36.407 --> 00:27:39.967 sophistication of validation should be commensurate with the bank's overall 00:27:39.967 --> 00:27:44.747 use of models, the complexity and materiality of its models, and the size 00:27:44.747 --> 00:27:46.917 and complexity of the bank's operations." 00:27:47.665 --> 00:27:51.365 An effective validation framework should include three core elements: 00:27:52.126 --> 00:27:52.456 1. 00:27:53.476 --> 00:27:57.696 "Evaluation of conceptual soundness, including developmental evidence" 00:27:58.370 --> 00:27:58.690 2. 00:27:59.770 --> 00:28:03.740 "Ongoing monitoring, including process verification and benchmarking" 00:28:04.532 --> 00:28:04.952 3. 00:28:05.922 --> 00:28:08.492 "Outcomes analysis, including back-testing" 00:28:09.300 --> 00:28:11.460 Evaluation of Conceptual Soundness 00:28:12.154 --> 00:28:16.204 The handbook describes this element as "assessing the quality of the model 00:28:16.204 --> 00:28:20.834 design and construction" which "involves review of documentation and empirical 00:28:20.864 --> 00:28:24.874 evidence supporting the methods used and variables selected for the model." 00:28:25.680 --> 00:28:29.490 Evaluation of conceptual soundness generally includes assessing whether 00:28:29.490 --> 00:28:32.940 the model achieves its intended purpose, comparing alternative 00:28:32.940 --> 00:28:37.630 theories and approaches, analyzing key assumptions and variables, evaluating 00:28:37.630 --> 00:28:41.860 data relevance, and conducting sensitivity analysis and stress testing. 00:28:42.617 --> 00:28:43.757 Ongoing Monitoring 00:28:44.476 --> 00:28:48.696 The guidance describes ongoing monitoring as confirming "that the model is 00:28:48.696 --> 00:28:52.856 appropriately implemented and is being used and is performing as intended." 00:28:53.696 --> 00:28:57.796 It is "essential to evaluate whether changes in products, exposures, 00:28:57.886 --> 00:29:02.626 activities, clients, or market conditions necessitate adjustment, redevelopment, 00:29:02.796 --> 00:29:04.126 or replacement of the model." 00:29:04.963 --> 00:29:08.593 Ongoing monitoring typically includes assessment of adherence to risk 00:29:08.643 --> 00:29:13.503 appetite and internal limits, analysis of overrides, sensitivity analysis, 00:29:13.643 --> 00:29:18.463 review of risk measurements, escalation processes, timely system updates, and 00:29:18.463 --> 00:29:20.543 process verification and benchmarking. 00:29:21.468 --> 00:29:22.718 Process Verification 00:29:23.474 --> 00:29:27.574 The handbook describes process verification as checking "that all model 00:29:27.574 --> 00:29:29.634 components are functioning as designed." 00:29:30.484 --> 00:29:34.734 This includes "verifying that internal and external data inputs continue 00:29:34.734 --> 00:29:39.094 to be accurate, complete, consistent with model purpose and design, and 00:29:39.094 --> 00:29:40.784 of the highest quality available." 00:29:41.636 --> 00:29:45.846 Process verification typically includes confirming inputs are processed as 00:29:45.846 --> 00:29:51.016 expected, reviewing reports derived from model outputs, and verifying that computer 00:29:51.016 --> 00:29:55.606 code implementing the model is subject to quality and change control procedures. 00:29:56.359 --> 00:29:56.969 Benchmarking 00:29:57.735 --> 00:30:02.345 The guidance defines benchmarking as "the comparison of a given model's inputs and 00:30:02.375 --> 00:30:06.895 outputs to estimates from alternative internal or external data or models." 00:30:07.645 --> 00:30:11.515 It notes that "discrepancies between the model output and benchmarks should 00:30:11.515 --> 00:30:15.075 trigger investigation into the sources and degree of the differences." 00:30:15.865 --> 00:30:17.095 Outcomes Analysis 00:30:17.786 --> 00:30:21.806 The handbook describes outcomes analysis as "a comparison of model 00:30:21.846 --> 00:30:25.746 outputs to corresponding actual outcomes" which "helps to evaluate 00:30:25.746 --> 00:30:29.796 model performance, by establishing expected ranges for those actual 00:30:29.826 --> 00:30:33.756 outcomes in relation to the intended objectives and assessing the reasons 00:30:33.756 --> 00:30:35.816 for observed variation between the two." 00:30:36.650 --> 00:30:40.230 Outcomes analysis approaches may vary depending on the characteristics 00:30:40.230 --> 00:30:43.640 and objectives of a model, and may include both quantitative 00:30:43.670 --> 00:30:45.110 and qualitative techniques. 00:30:45.760 --> 00:30:49.720 The guidance notes that "models are regularly adjusted to take into account 00:30:49.720 --> 00:30:54.310 new data or techniques, or because of deterioration in performance" and 00:30:54.310 --> 00:30:58.880 that "parallel outcomes analysis" is an important test of such adjustments. 00:30:59.598 --> 00:31:00.298 Back-Testing 00:31:01.044 --> 00:31:05.874 The guidance defines back-testing as "a form of outcomes analysis that involves 00:31:05.874 --> 00:31:09.964 the comparison of actual outcomes with model forecasts during a sample time 00:31:09.964 --> 00:31:14.284 period not used in model development, and at an observation frequency 00:31:14.284 --> 00:31:18.014 that matches the forecast horizon or performance window of the model." 00:31:18.785 --> 00:31:23.125 The handbook notes that "analysis of the results of even high-quality and 00:31:23.125 --> 00:31:27.525 well-designed back-testing can pose challenges" and that "models with long 00:31:27.525 --> 00:31:31.515 forecast horizons should be back-tested, but given the amount of time it would 00:31:31.515 --> 00:31:35.585 take to accumulate the necessary data, that testing should be supplemented 00:31:35.585 --> 00:31:37.885 by evaluation over shorter periods." 00:31:38.676 --> 00:31:40.096 Third-Party Risk Management 00:31:40.890 --> 00:31:44.970 The guidance states that "banks may benefit from third-party relationships 00:31:44.970 --> 00:31:49.220 by gaining operational efficiencies or improving the banks' competitive edge." 00:31:49.910 --> 00:31:54.110 However, "a bank's use of third parties does not diminish the responsibility 00:31:54.110 --> 00:31:57.260 of the bank's board and senior management to ensure that the bank 00:31:57.320 --> 00:31:59.330 operates in a safe and sound manner." 00:32:00.128 --> 00:32:04.208 The handbook advises that "third-party models should be incorporated into the 00:32:04.208 --> 00:32:09.188 bank's third-party risk management and model risk management processes" and that 00:32:09.348 --> 00:32:12.838 "management should conduct appropriate due diligence on the third-party 00:32:12.838 --> 00:32:14.998 relationship and on the model itself." 00:32:15.838 --> 00:32:17.368 Third-Party Models and Data 00:32:18.111 --> 00:32:21.771 The guidance states that "the widespread use of vendor and other third-party 00:32:21.771 --> 00:32:26.501 products—including data, parameter values, and complete models—poses 00:32:26.501 --> 00:32:30.761 unique challenges for validation and other model risk management activities 00:32:30.951 --> 00:32:34.931 because the modeling expertise is external to the user and because some 00:32:34.931 --> 00:32:37.091 components are considered proprietary." 00:32:37.915 --> 00:32:41.965 When a bank uses a third-party model, it should obtain information from the 00:32:41.965 --> 00:32:45.825 third party including developmental evidence, information about data 00:32:45.825 --> 00:32:50.105 used, testing results, documentation of limitations and assumptions, 00:32:50.255 --> 00:32:51.985 and implementation instructions. 00:32:52.666 --> 00:32:56.086 Engaging Third Parties for Model Risk Management Activities 00:32:56.763 --> 00:33:00.473 The handbook notes that "although model risk management is an internal 00:33:00.473 --> 00:33:05.213 process, a bank may decide to engage external resources to help execute 00:33:05.213 --> 00:33:08.733 certain activities related to the model risk management framework." 00:33:09.533 --> 00:33:13.813 These activities could include "model validation and review, compliance 00:33:13.813 --> 00:33:17.243 functions, or other activities in support of internal audit." 00:33:18.120 --> 00:33:22.110 Banks should "specify the activities to be conducted in a clearly written 00:33:22.110 --> 00:33:26.230 and agreed-upon scope of work" and designate an internal party to 00:33:26.510 --> 00:33:30.850 "understand and evaluate the results of validation and risk control activities 00:33:30.850 --> 00:33:32.880 conducted by external resources." 00:33:33.616 --> 00:33:34.396 IT Systems 00:33:35.062 --> 00:33:38.052 The guidance states that "banks should have appropriate risk 00:33:38.052 --> 00:33:42.062 management to maintain adequate IT systems that result in appropriate 00:33:42.092 --> 00:33:43.752 and accurate model outputs." 00:33:44.472 --> 00:33:48.572 It emphasizes that "sound model risk management depends on substantial 00:33:48.572 --> 00:33:52.642 investment in supporting systems to ensure data and reporting integrity, 00:33:52.822 --> 00:33:54.692 together with controls and testing." 00:33:55.496 --> 00:33:59.016 The handbook advises that "the IT environment supporting the bank's 00:33:59.016 --> 00:34:02.386 models has appropriate controls before model implementation." 00:34:03.216 --> 00:34:07.986 This includes controls for "the access, authentication, transmission, and storage 00:34:07.986 --> 00:34:12.546 of sensitive customer information" and appropriate "security controls related to 00:34:12.546 --> 00:34:17.626 how the bank and any third parties access, transfer, share, and store information." 00:34:18.417 --> 00:34:21.777 This concludes our comprehensive summary with direct quotes from the 00:34:21.817 --> 00:34:24.017 OCC Handbook on Model Risk Management. 00:34:24.781 --> 00:34:29.031 If your Credit union could use assistance with your exam, reach out to Mark Treichel 00:34:29.031 --> 00:34:31.681 on LinkedIn, or at mark Treichel dot com. 00:34:32.271 --> 00:34:34.891 This is Samantha Shares and we Thank you for listening.