Before the rise of AI chatbots, websites allowed search engines to access their content in return for increased visibility, a system that rewarded them with traffic and advertising revenues.
But the rapid development of generative AI has allowed tech giants like Google and OpenAI to harvest information for their chatbots with web crawlers, without humans ever needing to visit the original sites.
Traditional content producers, such as media outlets, are being outpaced by AI crawlers, which have cut into their online operations and advertising revenues.
"Sites that gave bots access to their content used to get readers in exchange," said Kurt Muehmel, head of AI strategy at data management firm Dataiku.
But the arrival of generative AI "completely breaks" that model, he told AFP.
Wikipedia's human internet traffic fell by eight percent between 2024 and 2025 because of a rise in AI search engine summaries, the online encyclopaedia reported last month.
"The fundamental tension is that the new business of the internet that is AI-driven doesn't generate traffic," said Matthew Prince, CEO of Cloudflare, an American internet services provider.
- 'No trespassing' -
Cloudflare, which processes more than 20 percent of all internet traffic, announced this summer a new measure aimed at blocking AI crawlers from accessing content without payment or permission from website owners.
"It's basically like putting a speed limit sign or a no trespassing sign," Prince told AFP on the sidelines of the Web Summit in Lisbon.
"Badly behaving bots can get by that, but we can track that... Over time, we can tighten these controls in a way that we're confident the AI companies can't get through."
The measure, which applies to more than 10 million websites, has already "attracted the attention of artificial intelligence giants", he added.
On a smaller scale, American startup TollBit is providing online news publishers with tools to block, monitor and monetise AI crawler traffic.
"The internet is a highway," said CEO and co-founder Toshit Panigrahi, who described the company as a "tollbooth on the internet".
TollBit works with more than 5,600 sites, including USA Today, Time magazine and the Associated Press, allowing media outlets to set their own access fees for their content.
The analytics are free for publishers, but AI companies are charged a "transaction fee for every piece of content they access".
But for Muehmel, the online takeover by AI crawlers cannot be resolved with only "partial measures or by an individual company".
"This is an evolution of the entire internet economy, which will take years," he said.
If the bot swarm continues to roam freely online, "all of the incentives for content creation are going to go away," Prince said.
"That would be a loss, not just for us humans that want to consume it, but actually for the AI companies that need original content in order to train their systems."
EU bows to pressure on loosening AI, privacy rules
Brussels, Belgium (AFP) Nov 15, 2025 -
The European Union is set next week to kickstart a rollback of landmark rules on artificial intelligence and data protection that face powerful pushback on both sides of the Atlantic.
Part of a bid to slash red tape for European businesses struggling against US and Chinese rivals, the move is drawing accusations that Brussels is putting competitiveness ahead of citizens' privacy and protection.
Brussels denies that pressure from the US administration influenced its push to "simplify" the bloc's digital rules, which have drawn the wrath of President Donald Trump and American tech giants.
But the European Commission says it has heard the concerns of EU firms and wants to make it easier for them to access users' data for AI development -- a move critics attack as a threat to privacy.
One planned change could unite many Europeans in relief however: the EU wants to get rid of those pesky cookie banners seeking users' consent for tracking on websites.
According to EU officials and draft documents seen by AFP, which could change before the November 19 announcement, the European Commission will propose:
-- a one-year pause in the implementation of parts of its AI law
-- overhauling its flagship data protection rules, which privacy defenders say will make it easier for US Big Tech to "suck up Europeans' personal data".
The bloc's cornerstone General Data Protection Regulation (GDPR) enshrined users' privacy from 2018 and influenced standards around the world.
The EU says it is only proposing technical changes to streamline the rules, but rights activists and EU lawmakers paint a different picture.
- 'Biggest rollback' -
The EU executive proposes to narrow the definition of personal data, and allow companies to process such data to train AI models "for purposes of a legitimate interest", a draft document shows.
Reaction to the leaks has been swift -- and strong.
"Unless the European Commission changes course, this would be the biggest rollback of digital fundamental rights in EU history," 127 groups, including civil society organisations and trade unions, wrote in a letter on Thursday.
Online privacy activist Max Schrems warned the proposals "would be a massive downgrading of Europeans' privacy" if they stay the same.
An EU official told AFP that Brussels is also expected to propose a one-year delay on implementing many provisions on high-risk AI, for example, models that can pose dangers to safety, health or citizens' fundamental rights.
Instead of taking effect next year, they would apply from 2027.
This move comes after heavy pressure from European businesses and US Big Tech.
Dozens of Europe's biggest companies, including France's Airbus and Germany's Lufthansa and Mercedes-Benz, called for a pause in July on the AI law which they warn risks stifling innovation.
- More battles ahead -
Commission president Ursula von der Leyen faces a battle ahead as the changes will need the approval of both the EU parliament and member states.
Her conservative camp's main coalition allies have raised the alarm, with the socialists saying they oppose any delay to the AI law, and the centrists warning they would stand firm against any changes that undermine privacy.
Noyb, a campaign group founded by Schrems, published a scathing takedown of the EU's plans for the GDPR and what they entail.
The EU has pushed back against claims that Brussels will reduce privacy.
"I can confirm 100 percent that the objective... is not to lower the high privacy standards we have for our citizens," EU spokesman for digital affairs, Thomas Regnier, said.
But there are fears that more changes to digital rules are on the way.
- Simplification, not deregulation -
The proposals are part of the EU executive's so-called simplification packages to remove what they describe as administrative burdens.
Brussels rejects any influence from Trump -- despite sustained pressure since the first weeks of the new US administration, when Vice President JD Vance railed against the "excessive regulation" of AI.
This "started before the mandate of the president of the US", chief commission spokeswoman Paula Pinho said this week.
Calls for changes to AI and data rules have been growing louder in Europe.
A major report last year by Italian ex-premier Mario Draghi also warned that data rules could hamper European businesses' AI innovation.
Related Links
Space Technology News - Applications and Research
| Subscribe Free To Our Daily Newsletters |
| Subscribe Free To Our Daily Newsletters |
