Web scraping gets more expensive: how Google is driving up costs for SEO tools

You’ve built up the same routine over the years. Every day, you open the same newspaper at the kiosk for free. The owner couldn’t care less. Until now. From tomorrow onwards, everyone at that kiosk first has to perform a small circus act before getting their hands on the paper: go around the back, enter a code, wait a minute. For you, as a casual reader, that may be a minor annoyance.

But imagine someone at that same kiosk who comes to collect a hundred copies every morning to resell elsewhere. That same single minute, multiplied by a hundred, suddenly hits their wallet. Enough with the analogy, because that is roughly what happened to Google Search over the past week. And make no mistake: the consequences go far beyond a purely technical implication for developers 😉.

What exactly changes with the google.com/goto URL

Google is rolling out the google.com/goto URL as an intermediate step in its search results. Previously, a click would immediately reveal the actual web address of a result. Now, you first see a redirect through Google itself: click a result and you pass through Google’s own server, which then sends you on to the destination.

A small technical intervention with a major intention: it becomes much harder for external parties to determine, at scale, which URLs actually appear in the search results.

A Google spokesperson, who for now remains anonymous, confirmed the rollout to Search Engine Roundtable, albeit in fairly general terms: “We have a long history of deploying technical measures against evolving forms of abuse, and we regularly take steps to protect our services and users.”

The test becomes reality

The change did not come out of nowhere. Google tested the goto parameter on a small scale in July 2026, without any official communication. If you were paying close attention at the time, you could see it happening: more and more search results pages showed only goto links instead of direct URLs. Five weeks later, that test has turned into a structural rollout.

Derek Perkins, a well-known figure in technical SEO who has been following the development closely, described it as “nearly a 100% rollout across several residential ip providers.”

The real impact: not a roadblock, but a price increase

It is tempting to think that Google is stopping scraping with this move. That is not what is happening. As the SEO community aptly summed it up: this does not block scraping, it makes it more expensive. Every link now has to be decoded individually instead of being read directly. At the scale of a single search query, that is negligible. At the scale of hundreds of millions of keywords per month — the reality for rank trackers, SEO tools and AI platforms that train on or build on search results, it adds up quickly: more computing power, more requests, more proxy bandwidth, plus captchas that significantly increase the error rate.

Who this is aimed at can be guessed without Google saying it out loud. SerpApi, which is currently suing Google over the scraping of search results, is an obvious candidate. But the measure has a broader impact: rank-tracking tools, SEO platforms such as Semrush and Ahrefs, and AI assistants that use search results as fuel for their own answers.

There is a certain irony to it. Google’s business model revolves around crawling and indexing other people’s content. Now that AI Overviews synthesise that same content and display it at the top, the very click incentive that publishers have criticised Google for taking away for years is disappearing. Google is now defending itself against exactly the kind of disintermediation it has itself been accused of for years.

How tools and marketers are dealing with it

The figures immediately show why “more expensive” is not an empty claim here. Derek Perkins calculated that fully resolving all links across a ranking spanning five results pages now requires between 500 and 1,000 separate requests: the encoded goto links cannot be decoded locally, and Google no longer allows even the lighter HEAD requests that could reduce the load. Providers therefore have to fire off a full GET request for every link.

This is not the first time either. When Google removed the num=100 parameter last year, which allowed tools to retrieve one hundred results at once instead of ten at a time, Semrush said its costs increased tenfold. Ahrefs, through Tim Soulo, indicated at the time that it was considering tracking only the first two pages of results rather than the full ranking, purely to keep it financially viable.

AccuRanker and SISTRIX adjusted their collection methods in similar ways. That history is now repeating itself, only more emphatically.

In practice, the same three responses keep appearing among tool providers. Some absorb the additional cost and sooner or later pass it on to their customers, as Semrush did at the time. Others reduce the depth of what they continue to track, as Ahrefs considered doing. Still others look for technical workarounds, but these are usually blocked fairly quickly once Google notices them, so that is not a sustainable long-term strategy. Larger players such as DataForSEO and SerpApi have already adapted, but what that will mean in concrete terms for their own costs and therefore for your bill, is not yet clear everywhere.

For marketers and brands, this translates into two practical points:

• First: expect somewhat rougher and slower rank-tracking data over the coming months, with the possibility of small inconsistencies between figures from before and after the rollout, and potentially a higher bill for the tools you use today.

   

• Second, and most important: if your visibility depends entirely on third parties to tell you how you are performing, you become vulnerable as soon as that third party runs into difficulties. Your own data, such as Search Console and Analytics, therefore becomes not optional but fundamental: these are the channels Google itself keeps open precisely because they are first-party.

Want to read more?

This blog is partly based on reporting by Barry Schwartz at Search Engine Roundtable, including confirmation from Google itself.

Need advice on your online visibility?

Google is changing the rules of the game again, and it will continue to do so. We are happy to think along with you about an approach that does not depend on the latest trick in the book, but instead builds on a foundation that lasts. Feel free to get in touch.