The Loan Letter Test: What You Should Never Hand to AI

You work at a bank. A loan application has been rejected, and you need to send the applicant a letter explaining why – using their full financial history, income details, and credit profile. AI can help. The question is with what, exactly.Sort the task into four pieces: drafting the letter’s wording, deciding whether to reject the loan, pasting the applicant’s financial details into the tool, and checking the final letter before it goes out. Two of these are safe to hand off. Two are not – and the reasons aren’t intuition, they’re documented in regulation, research, and at least one very public corporate incident.Deciding the outcome: keep human, and regulators already agreeA 2019 study published in the Journal of Financial Economics – led by researchers at University of California Berkeley and the University of Southern California – examined over seven million mortgage applications and found that algorithmic lenders charged Black and Latino borrowers 5.3 basis points more on purchase loans and 2.0 basis points more on refinance loans than equally qualified white and Asian borrowers, despite the process being “face-to-face discrimination-free.” The algorithm didn’t eliminate bias. It relocated it.Regulators have responded directly, in India as much as anywhere. The Reserve Bank of India’s FREE-AI framework – released last year by a committee chaired by IIT Bombay’s Professor Pushpak Bhattacharyya – lays out seven guiding principles for AI in Indian finance, and “People First” is explicit: final decision-making must stay with humans, AI is there to augment it, not replace it. The framework’s fairness and transparency pillar goes further, calling on banks and NBFCs to keep credit and underwriting models explainable and to give customers a grievance channel for AI-related disputes. The US Consumer Financial Protection Bureau’s 2023 guidance arrives at the same place from a different direction: a lender cannot explain a credit denial by pointing to “the algorithm” – creditors must provide the specific, accurate reasons behind a rejection, even when a complex model produced it. Two regulators in two different financial systems have independently landed on the same rule: the decision – not just the wording around it – stays accountable to a person.Handling the data: keep human, and Samsung already found out whyIn April 2023, engineers at Samsung pasted confidential source code and internal meeting notes into ChatGPT to get help debugging and summarizing – inputs that, once submitted, become data the AI provider can store and potentially use to improve its models. Samsung banned employee use of generative AI tools on internal devices within weeks of the incident becoming public. The specific leak was source code; the general lesson applies directly to any sensitive personal data, financial or otherwise: once it’s pasted into a public AI tool, an organization has lost control over where it goes.For a loan-rejection letter, that means the applicant’s income figures, credit history, and account details don’t belong in a general-purpose AI tool at all – regardless of how good the resulting letter might sound. It’s worth noting this is exactly the risk the RBI’s FREE-AI framework flags too, calling out over-collection and uncontrolled data flow as a specific hazard for regulated entities adopting AI, and pointing banks back to India’s own Digital Personal Data Protection Act as the baseline they can’t route around.Drafting the wording: safe to hand offOnce the decision is made and the data is handled correctly, drafting – structure, wording, and tone, using only what’s necessary to state the reason – is squarely in AI’s comfort zone. This is a bounded writing task with a clear, factual input the tool has been given, not a judgment call it’s making on its own.The final check: keep human, and this is where confidence quietly misleads peopleEven a well-drafted, properly-scoped letter needs a human read before it goes out – and this is the step people are most likely to skip, because the earlier steps already went well. A 2022 study published in the Journal of Public Administration Research and Theory, based on three experiments with public-sector decision-makers in the Netherlands, found something worth sitting with: people didn’t just over-rely on algorithmic advice in general — they were more likely to follow it when it happened to confirm an existing stereotype about the person being judged, even when other warning signs pointed the other way. In other words, automation bias doesn’t just make people less careful. It can make them less careful in a specific, predictable direction. Confidence in the tool, not the content, is what erodes the check – and for a loan-rejection letter, that’s precisely the moment where a biased output is most likely to slip through unquestioned.Why this generalizesSwap “loan rejection” for a performance review, a medical intake summary, or a university admissions decision, and the same four-way split holds: drafting is safe, deciding isn’t; wording is safe, raw personal data isn’t; and the final check is exactly the step a smoothly-run process makes easiest to skip. The Loan Letter Test isn’t really about lending – it’s a template for where Judgment has to hold the line, regardless of which sensitive decision is on the table.