What are AI tarpits? Understanding the tools people are using to poison LLMs

"In order for a chatbot to become more intelligent, and thus more useful to the end-user, it needs to assimilate data continuously. This process is known as "training." The problem is that many AI companies never explicitly ask for consent from data owners before scraping their webpages and adding the data to the corpora of the large language models (LLMs) that power AI chatbots."

"AI poisoning is the process of corrupting an AI chatbot's underlying large language model so that the chatbot gives incorrect, misleading, or utterly bonkers outputs. This corruption is achieved by tricking the LLM into assimilating incorrect data during its training, which often involves scraping every possible website and image it can find."

"There are many ways an LLM can be poisoned, depending on the capabilities of the LLM that the poisoner wants to disrupt. For example, if someone wanted to poison an image generator LLM, they could use a technique known as "Nightshading," which involves using a piece of software called Nightshade to add an invisible layer to an image."

"Their aim? To poison the chatbot's underlying LLM and thus degrade the quality of its outputs, potentially causing end-user flight. Of course, the majority of chatbots deal with text, not images, rendering poisoning tools like Nightshade useless against unauthorized AI scraping of articles and blogs."

Chatbots become more useful when they continuously assimilate data through training. Many AI companies scrape webpages without obtaining consent from data owners, including content creators and IP holders. Some data owners respond by using tarpits to poison underlying large language models, aiming to degrade output quality and reduce end-user trust. AI poisoning corrupts a model so it produces incorrect, misleading, or nonsensical responses by tricking it into learning incorrect data during training. Poisoning methods vary by target capability. For image generators, Nightshading adds an invisible pixel layer that is imperceptible to humans but visible to LLM scrapers, causing style misclassification. For text-based chatbots, different poisoning approaches are needed because image-specific tools do not apply.

Read at Fast Company