CRITICAL Tier: Credit Decisioning Support¶
The system: a model that scores credit applications to support, and in some segments automate, lending decisions. To meet data-residency and performance needs, the bank runs a self-hosted, open-weight model fine-tuned on its own historical lending data. Outputs feed an adjudication workflow; for low-value segments the decision is automated, for the rest it informs a human underwriter.
This is the example where the model is the risk. The previous examples secured what goes into and around a trusted model. Here the bank built the model, so it owns a class of risk the others could outsource: poisoning, backdoors, provenance, and fairness. It is also where runtime oversight is at its most demanding.
Why this is CRITICAL tier¶
From the classification process: automated decisions with direct financial and legal consequence, regulated fairness obligations, and sensitive personal data. Every dimension scores at the top. This is CRITICAL, the tier where automated-only operation for significant decisions is not acceptable and the control depth is maximum across the board.
The open-weight burden shifts to you¶
Choosing a self-hosted, fine-tuned open-weight model is a security decision before it is a performance one. With a managed API, the provider carries much of the model-integrity burden. Self-host and that burden shifts to you: you are now responsible for the weights, the fine-tuning pipeline, and everything that could be hiding in them.
Model poisoning and supply chain¶
These risks are decided pre-runtime, during model selection and training, which is why this example leans on this site's Model Selection and MLOps sections.
| Risk | What goes wrong | Where it is addressed |
|---|---|---|
| Backdoors and trojans in the base model | An open-weight model carries a hidden trigger that flips behaviour on a specific input | Vulnerability Scanning, SUP-01 |
| Provenance and integrity | You cannot prove the weights are the ones you vetted, unmodified | Provenance and Integrity, SUP-01 |
| Serialisation risks | Model files in unsafe formats execute code on load | Vulnerability Scanning |
| Fine-tuning data poisoning | Corrupted or biased training data bends the model's decisions, subtly and persistently | Secure ML Pipelines, SUP-04, Data Governance |
| Pipeline compromise | An insecure training pipeline means you cannot trust what was deployed | Secure ML Pipelines, CI/CD for AI |
The controls are verification and integrity at every step: verify base-model provenance and scan it before use, secure and attest the fine-tuning pipeline, govern the training data's lineage, sign the resulting model, and verify that signature at load. A poisoned model is a poisoned system, and no runtime guardrail reliably detects a well-built backdoor.
Fairness is a security property here¶
A credit model carries regulatory fairness obligations. Bias introduced through training data or fine-tuning is both a compliance failure and a poisoning symptom. Pre-deployment bias and fairness testing and adversarial evaluation are mandatory gates, not optional, alongside the explainability the decision must carry. See Adversarial Testing, Trust and Evaluation, and Regulatory Alignment.
Maximum runtime oversight¶
At CRITICAL the control matrix calls for the deepest runtime controls, and the design must honour them:
- Human oversight on every significant decision. The model recommends; a human decides for anything material. Automation is confined to a narrow, well-evidenced low-value segment with tight thresholds. See Human Oversight.
- A layered reviewing line, not one control. The reviewing layer is scanners, a semantic firewall, and a model-as-judge, and at CRITICAL you run more than one of them. Deterministic scanners catch known-bad in the hot path, a semantic firewall weighs each output against policy and context inline, and a judge evaluates the output and its reasoning. See Model-as-Judge and Judge Assurance.
- Full audit and explainability. Every decision is logged with its inputs, the model version, and the rationale, sufficient to answer a regulator and to reconstruct any decision. See Logging & Observability.
- Circuit breakers and a non-AI fallback. A tested, dependency-isolated manual adjudication path the system can fall back to. See Scaling and PACE and Circuit Breakers.
You choose the controls, and you combine them¶
The reviewing controls are not a fixed recipe; they are choices you make against risk and latency, and they compose. This matters most in the automated low-value segment, where decisions happen fast and a full inline judge may cost more latency than the interaction can spend. A workable split for that segment:
- Semantic firewall inline, for real time. It weighs each output against policy and context in the hot path at low to moderate latency, blocking a clearly non-compliant decision before it lands. This is the control that keeps up with fast-paced automated actions.
- Model-as-judge for post-action review. A larger judge audits decisions asynchronously, off the hot path, catching the subtler failures the inline firewall waves through. It does not gate the action; it reviews it after the fact and flags what needs attention.
- Humans on the alerts. When the async judge or the firewall flags something, a person is paged and acts: reverse the decision, pull the segment back to manual, or trip a circuit breaker. Post-action review is only worth running if someone is on the other end of the alert.
The same controls sit differently in other tiers. Because a semantic firewall is cheaper on latency than a full inline judge, lower tiers often lean on the firewall (or scanners alone) in the hot path and run the judge on a sample asynchronously, if at all. The control matrix and the reviewing-controls latency table give the baseline depth per tier; the point here is that you are picking and combining these controls deliberately, not switching one on in isolation.
You cannot inspect your way out of a poisoned model
Runtime monitoring catches anomalous behaviour, but a backdoor that only fires on a rare trigger may never show up in monitoring until it is exploited. That is why the integrity work happens before deployment: provenance, scanning, a secure fine-tuning pipeline, and signing. At CRITICAL, pre-runtime model security is not a nicety; it is the load-bearing control.
What this example teaches¶
When you own the model, you own its integrity. The centre of gravity moves earlier, into model selection and MLOps, and the runtime controls (real-time Judge, human decision, full audit, circuit breaker) are the maximum the framework defines. Compare this with the LOW FAQ Assistant: same framework, opposite ends of the dial. The method, Should You Use AI? → threat model → tier → controls, is identical; only the depth changes.