LLM Family Trees Unraveled by Slop Forensics

🔊 Listen to the Podcast version here. 🔊

The Linguistic Fingerprint

In a bustling AI lab, a detective-like researcher peers at lines of AI-generated prose through a magnifying glass. This isn’t a murder mystery, but it’s a hunt for hidden heritage. Every AI model has a secret family history written between the lines of its output, and Samuel J. Paech is determined to unravel it. Meet slop forensics, - a cheeky new technique that reads an AI’s “linguistic DNA” to trace its lineage in the large language model (LLM) family tree. He suspects that behind many an AI’s eloquent facade lurk the speech habits of another model, - and he’s on a mission to expose these hidden kinships.

Like a forensic linguist for AI, Paech analyzes the “slop”, - the quirks and overused phrases in an AI’s responses, - to determine who its “parents” might be. It turns out, just as kids inherit mannerisms from their parents, AI models trained on another model’s outputs pick up telltale verbal tics. From favorite phrases to peculiar punctuation, these models leave breadcrumbs about their upbringing. By sifting through this textual slop, Paech can infer which models influenced a given AI, essentially building a family tree from linguistic fingerprints. For technology leaders, this kind of sleuthing is like running a background check on that shiny new AI system, - it reveals whether your prospective model grew up on a healthy diet of original data or just got by on someone else’s outputs in secret.

Paech’s approach begins with generating creative writing samples from different models and combing through them for overused words, bigrams, and trigrams, - the so-called GPT-isms. Each model gets a “slop profile,” essentially a list of phrases it uses way more often than a human writer would. Think of it as an AI’s verbal fingerprint. One model might constantly start sentences with “However,” while another can’t tell a fantasy story without naming a hero “Elara”. These recurring tells are the genetic markers in Paech’s forensic toolkit.

After profiling each model’s favorite buzzwords and phrases, Paech turns these findings into a faux DNA sequence. Imagine lining up dozens of models and marking each one with a 1 or 0 depending on whether it uses a particular phrase excessively. The result is a binary string – a slop fingerprint like 101100, - encoding each model’s quirks. By feeding these fingerprints into a bioinformatics algorithm,- the kind used for gene sequencing, a phylogenetic tree of AI models emerges. Models that share a lot of slop, let’s say that they all overuse “In a nutshell”, cluster together on the tree, hinting they might share training ancestry or influence.

DeepSeek’s Lineage Twist

This technique isn’t just theoretical. Recently, an the model DeepSeek R1 provided a perfect test case. DeepSeek R1 is an open-source model that shot to fame for punching above its weight. But after a major update, keen observers noticed something odd: it sounded different. Its style of answering questions had subtly shifted, almost like an actor adopting a new accent. Was this just a tweak in tone, or had DeepSeek gotten a new “parent” behind the scenes? Slop forensics set out to find the answer.

Paech dug into DeepSeek R1’s slop profile before and after the update. The findings were striking. The older version of DeepSeek clustered snugly with OpenAI’s models on the family tree, - its outputs were full of phrases and patterns reminiscent of GPT-3 or GPT-4. But the newer R1-0528 had jumped branches, now clustering among Google’s Gemini family. In other words, the updated DeepSeek suddenly started “speaking” with a Google-esque tongue. The slop forensics strongly suggested that DeepSeek’s makers likely switched from using OpenAI’s text to using Google’s Gemini outputs to train the new model.

Paech’s phylogenetic-style slop tree makes this lineage twist plain to see. On the chart, old DeepSeek R1 sits on a branch beside OpenAI’s models, while the revamped R1-0528 hops over to a distant limb with Gemini kin. In the visualization, it’s as if DeepSeek got a genetic transplant, - suddenly its closest “relatives” are Google’s AI models, not the OpenAI cluster.

https://github.com/sam-paech/slop-forensics

This dramatic reorganization of the family tree confirmed what many suspected. That is that DeepSeek’s creators quietly changed the model’s upbringing mid-stream. For DeepSeek’s users, this explained why the AI’s answers suddenly felt different, - the model had essentially swapped out its playbook mid-stream.

The DeepSeek case also highlighted a broader trend. Paech noticed that models from different “families” are increasingly borrowing each other’s favorite phrases. For instance, various models, - even those built independently, all started naming their fantasy story characters “Elara”, a uniquely specific name that one model made popular. Such convergence hints at an echo chamber, - models training on other models’ outputs, unintentionally sharing and amplifying quirks. It’s a bit like linguistic inbreeding in AI land, and it underscores why slop forensics is both exciting and a little unnerving.

Now, before we declare every AI’s ancestry with absolute certainty, a word of caution. This family tree is a best-guess, not a birth certificate. Paech himself emphasizes that these inferred lineages are speculative. Shared slop might mean a direct training connection, - or it might just mean two models coincidentally learned to talk alike because of similar training tactics. In other words, correlation is not confirmation. Slop forensics provides clues, not courtroom evidence, but in the world of closed-door model training, clues are a good start.

The Slop Tree Expands

DeepSeek’s tale opened the door to ask: what about other prominent LLMs? Take Meta’s LLaMA and its many offspring. LLaMA was originally a standalone model, - famously leaked into the wild, but soon it sired a family of fine-tuned variants. One prodigious child, Vicuna, was trained by feeding LLaMA a trove of ChatGPT conversations. If we apply slop forensics, we’d expect Vicuna to exhibit an intriguing mix of traits: the genetic base of LLaMA, but the adopted “accent” of OpenAI’s ChatGPT. In fact, Vicuna reportedly achieves about 90% of ChatGPT’s quality, not surprising given it literally learned to talk from ChatGPT’s transcripts.

If Vicuna’s outputs were put under the slop-scope, we’d likely see telltale OpenAI phrases popping up. Where vanilla LLaMA might produce more raw, unguarded text, Vicuna often responds with that polished ChatGPT style, - the “Sure, here’s what I found:” kind of phrasing. Those friendly formalities and cautious disclaimers didn’t come from LLaMA’s original training which had no instruction tuning They’re fingerprints left by ChatGPT’s influence. Slop forensics would place Vicuna on the family tree somewhere between its open-source parent and its ChatGPT mentor, a linguistic love-child of Meta and OpenAI lineages.

What about newcomers like Mistral or Falcon? These open-source upstarts claim independent pedigrees, - Mistral was built from scratch by a French AI lab, and Falcon by researchers in the UAE. Their creators didn’t fine-tune them on ChatGPT outputs, - at least, not openly. Slop forensics gives us a way to verify such claims. If Mistral’s slop profile came out eerily similar to LLaMA’s, for example, it might raise eyebrows about shared training data or architecture. On the other hand, if Falcon’s outputs show unique quirks not seen in any other model, that would validate it as a truly independent branch of the LLM family tree. So far, these models seem to be holding their own distinctive styles, - a relief for those who want diversity in the AI gene pool.

Then there’s Anthropic’s Claude, an AI bred for helpfulness and high-minded conversation. Claude was developed by former OpenAI researchers who struck out on their own, so you’d expect a distinct lineage. Indeed, many users note Claude’s voice feels different, - often more verbose yet conversational, where ChatGPT might be more terse or formal. If we mapped Claude on the slop tree, it would likely form its own branch separate from OpenAI’s GPT family. Any overlap in slop patterns between Claude and ChatGPT could suggest either coincidental convergent evolution, - both being trained to please human feedback, or perhaps some shared public data in their diets. But given Anthropic’s ethos, Claude’s quirks, - its “Anthropic-isms”, probably set it apart as a cousin, not a sibling, to the GPT line.

Blurred Lines, Shared Speech

Interestingly, even without direct lineage, AI models can end up speaking alike due to convergent evolution. Nearly all advanced models today undergo some form of reinforcement learning from human feedback (RLHF), which often teaches them similar manners: be polite, avoid certain words, structure answers helpfully. This means a model from Meta and one from OpenAI might both say “I’m sorry, I cannot do that” under the same conditions, even if they never met. Slop forensics has to account for these common environmental pressures. Not every shared phrase is a smoking gun of copied training data; sometimes different AI “species” just evolved the same survival traits in the wild.

Why Lineage Matters

Why do these hidden family ties matter? For one, trust and transparency. If a company claims its new model was built from scratch, but slop forensics shows it parroting another model’s pet phrases, eyebrows will rise. Executives don’t love surprises in their tech stack and neither should you. Hidden lineage can also tread into intellectual property gray zones. Imagine an open-source model unwittingly carrying licensed quirks from a proprietary system, it’s like discovering some borrowed code in an otherwise open project. Knowing a model’s ancestry helps everyone gauge how original or entangled it really is. It’s essentially an audit trail for AI development, which is increasingly crucial as models intermingle.

There’s also an AI safety angle. Hidden lineage might mean hidden risks. If Model B was trained on Model A’s outputs, any bias or flaw in A could quietly transfer to B, multiplying across the ecosystem. This feedback-loop training can create a synthetic data echo chamber where every model starts sounding the same and sharing the same blind spots. Researchers warn of model collapse, a scenario where training on AI-generated data causes generational degradation of quality and diversity. In plainer terms, - if everyone copies from each other’s homework, pretty soon all the essays look alike. And the mistakes are all the same too. For AI to be robust, we want a diversity of thought, not a monoculture of mirror images, - especially of “slop”.

Illuminating the Invisible

Slop forensics offers a peek behind the curtain of AI development. It’s a bit playful, - sniffing out AI “DNA” from the odd turns of phrase, but also profound in its implications. As LLMs proliferate, understanding their family tree isn’t just academic gossip, it could become standard due diligence. In an era when AI models are often released with vague or zero info about training data, tools like this offer a form of accountability. Perhaps in the future we’ll see formal AI lineage disclosures, or even watermarking of AI-generated text to trace origins. Until then, we have Paech and his clever slop detective work to thank for shining a light on who’s teaching whom in the model world. In the end, you might say: an AI is what it eats. Slop forensics is helping us see exactly what our models have been gorging on!

---

---

Digital Forensics in the Age of Large Language Models (Zhipeng Yin, Zichong Wang, Weifeng Xu, Jun Zhuang, Pallab Mozumder, Antoinette Smith, Wenbin Zhang - April 2025) This paper surveys how LLMs can automate and enhance digital forensic tasks, outlining their superior capabilities in evidence collection and analysis. It critically examines LLM limitations, - such as bias, hallucinations, and interpretability issues, and calls for standardized practices to ensure transparency and accountability in forensic applications.

Slop Forensics Toolkit (Samuel J. Paech, 2025) This GitHub repository introduces the Slop Forensics toolkit, designed to analyze over-represented lexical patterns, - termed “slop”, in large language model outputs. The toolkit facilitates dataset generation, slop profiling, creation of canonical slop lists, and construction of phylogenetic trees to infer model lineage. It serves as a comprehensive resource for researchers aiming to understand and trace the ancestry of AI models through linguistic analysis.

Using ‘Slop Forensics’ to Determine Model Ancestry (Drew Breunig, May 2025) In this article, Drew Breunig explores the application of Slop Forensics in uncovering the lineage of large language models. By analyzing the linguistic patterns in model outputs, the study reveals a shift in DeepSeek’s R1 series from OpenAI’s models to characteristics resembling Google’s Gemini models. The piece underscores the importance of such forensic tools in promoting transparency and understanding in AI development.

Towards a Standardized Methodology and Dataset for Evaluating LLM-Based Digital Forensic Timeline Analysis (Hudan Studiawan, Frank Breitinger, Mark Scanlon - May 2025) This work proposes a replicable framework and dataset to benchmark LLMs on forensic timeline creation, inspired by NIST standards. It details the construction of ground-truth timelines, evaluation metrics like BLEU and ROUGE (demonstrated with ChatGPT), and discusses pitfalls in current LLM-based forensic evaluations.

Escaping Collapse: The Strength of Weak Data for Large Language Model Training (Kareem Amin, Sara Babakniya, Alex Bie, Weiwei Kong, Umar Syed, Sergei Vassilvitskii - February 2025) This paper formalizes the phenomenon of model collapse when LLMs are trained predominantly on synthetic data. It introduces a theoretical framework showing minimal curation requirements to maintain continual model improvement and presents experiments validating dynamic data‐selection methods that avert collapse even when most data are low‐quality.

---

Disclaimer: The perspectives shared in this article are my own and do not represent those of my employer or any affiliated organizations. All company names, product names, logos, and brands mentioned are the property of their respective owners and are used for identification and illustrative purposes only. No endorsement, sponsorship, or affiliation is intended or implied. References to specific companies or case studies are based on publicly available information and are used solely for educational and discussion purposes.