Model risk management
How Kaimei's analyses give validators and model owners evidence for conceptual soundness and ongoing monitoring, under guidance such as SR 11-7 and SS1/23.
#What validators look for
The US supervisory guidance SR 11-7, issued by the Federal Reserve and the OCC in 2011, describes model validation in three parts: evaluating conceptual soundness, ongoing monitoring, and outcomes analysis. The Bank of England's PRA sets out similar principles for UK banks in SS1/23, in effect since May 2024.
Kaimei does not replace independent validation, which is a matter of people and process. It produces evidence that validators and model owners can examine and keep.
#Evidence Kaimei produces
| Validation activity | What Kaimei provides |
|---|---|
| Conceptual soundness Does the model behave as theory and policy expect? | Global importance; behaviour curves with their direction and consistency; learned rules in plain English; interactions between inputs. |
| Sensitivity and stability | How often decisions flip under small input noise, and which inputs are most fragile. |
| Ongoing monitoring | Drift for every input and for the output, scheduled analyses, alerts, and traffic statistics. |
| Outcomes analysis | Accuracy on labelled reference data, reported with the importance analysis. Kaimei does not track realised outcomes over time; use your own outcome data for back-testing. |
| Benchmarking | Insights for two versions or two models side by side, and explanations of the same case under each. |
| Audit trail | Every analysis is stored with its options, seed and model version, and every result can be retrieved through the API. |
#Reading the insights
Each analysis writes insights in plain English. These are the ones the credit model produced:
| Insight | What it answers | Example from the credit model |
|---|---|---|
| Global importance | Which inputs the model relies on most, across everyone. | Credit score is the most influential input, followed by debt-to-income and utilisation. Together they account for 59% of the model's output variation. |
| Learned rules | How each important input moves the prediction, in a sentence. | Higher credit score decreases the probability of decline, most sharply between 543 and 610. |
| Behaviour curves | The average prediction as one input changes, and whether it moves in one direction. | All six of the most important inputs move the prediction in one direction only. |
| Interactions | Pairs of inputs whose combined effect is more than the sum of their parts. | The strongest pair is income and loan amount, and 6% of their joint effect is non-additive. |
| Robustness | How often small noise in the inputs flips a decision. | 0.9% of decisions flip under noise of 5% of a standard deviation. |
| Drift | Whether recent traffic still looks like the reference data. Population stability under 0.10 is stable and under 0.25 moderate; above that is significant. | Moderate drift, with a stability index of 0.15; loan amount, debt-to-income and income have shifted. |
| Output behaviour | How predictions are distributed, and how many sit close to the threshold. | The model predicts decline 44% of the time, and 15% of cases sit within 0.10 of the threshold. |
#Limits to keep in mind
- Kaimei studies the model on the population you give it, so its conclusions are only as representative as that data.
- It does not test fairness, back-test against realised outcomes, or build challenger models.
- Analyses use sampling; set a seed when you need to reproduce a result exactly.