Search Disrupted Newsletter (Issue 14)

One visitor for every 6,000 pages crawled

Per Cloudflare’s internal stats:

• Google (historically) refers one visitor for every two pages crawled.
• OpenAI/ChatGPT refers one visitor for every 250 pages crawled.
• Anthropic/Claude refers only one visitor for every 6,000 pages crawled!

This is a wild and shocking disparity in the implicit “contract” of publishing on the open web.

Much of this is due to wild inefficiencies in the crawling of AI companies, as they’re not nearly as sophisticated in forecasting and tracking page changes as Google.

But, the harder side to reckon with is that while referral traffic from AI search is up and growing, it’s still primarily an off-site game that won’t be about raw traffic.

And in that light, the question of “is it worth it” to let these AI companies crawl your site is real.

I come down on the side of “yes” for a few reasons:

1. From a purely technical standpoint, serving even 10,000 pages costs much less in real cash than getting a qualified visitor to your site via other marketing channels.
2. With more and more of the search journey happening inside AI platforms, your brand is getting value from mentions, recommendations, and awareness inside those platforms, even if it’s opaque and hard to attribute. However, we’re working on it with Knowatoa.
3. It’s far more strategic to be considering what is being crawled (are you producing the right content to teach the AIs about your brand) than how many crawls are happening.

Note: the above stats came from a discussion on the Latent Space podcast. I’ve linked directly to the timestamp of the relevant discussion and would encourage you to listen as they follow this up with a really good dissection of LLMs.txt and AI agents.

Link - Latent Space Podcast

ChatGPT’s stopped sucking up

For about a week, asking ChatGPT even the most mundane question would tell you that you were a brilliant and strategic thinker, which both my tween children and I assure you that I am not.

Internally at OpenAI, they’ve been referring to this with the technical term of “Sycophancy” and have rolled back the changes, so while your ego might take a blow, you’ll likely be better off.

I mention it here because it’s funny to think of humanity creating a superintelligence to act as a hyperactive motivation coach. It also illustrates an overlooked layer in AI search.

AI models have a data-cutoff point, and they supplement that data with things like calls to search APIs and the web, but there’s a middle layer of System Prompts, filters, and instructions that the AI platforms use to shape the output and actions of the model. OpenAI calls this their Model Spec.

This impacts search because these system prompts are updated much more frequently than the underlying data and can radically shift the rankings and output.

Consider one of the actual use cases from the Model Spec, a query of:

“Give me a list of real estate agents who specialize in residential properties in Plano, Texas.”

Depending on how the system prompt is weighting things, that might result in either a list of agents, or it might block the request as it “feels” like it might reveal personal information.

These constant system updates (at OpenAI, but all the other platforms as well) are a big reason to diligently track how your site is performing in the AI search ecosystem.

Link - OpenAI Model Spec

Reddit’s Mainstreamification of AI Search

Reddit Answers has been a little off to the side and “beta” for the past year, but they’ve announced that they’re shifting it into the main Reddit search experience.

If you haven’t tried it yet, it’s shockingly good.

• Speedy results (significantly faster than a full AI chat response)
• Wayyy less cruft and extra bits than a Google SERP
• Reads almost like a Wirecutter-style product article
• Mixes aggregated recommendations with user quotes, gives a good breakdown of price tiers, features, etc.

This is yet another example of AI as a technology enabling search to exist in many more places than just Google.

Link - Reddit Answers

Vibe Searching with Pinterest

Auto captioning of images was one of the big early breakthroughs of AI image tooling, and it’s now being used to power a new way to search on Pinterest.

They’re launching a new visual search experience powered by an unnamed generative AI and a Visual Language Model. You can long-press on a Pin and launch a visual search.

A weird twist is that this is partially driven by the need to identify AI-generated images on the platform, as Pinterest has been absolutely overrun. They’re using the same tooling to label and limit the proliferation of AI-generated images.

Link - Pinterest Visual Search

Google’s Prompt Engineering Guide

Google/Kaggle has released a new guide to help developers understand how to use prompts to get the best results from their AI tools.

While they explicitly say that you don’t need to be a developer for this, it is fairly technical. That being said, it’s worthwhile, both from a productivity and understanding standpoint, to dig into these tools.

The same techniques that they’re teaching in this are what they’re using internally for their AI search tools and processes.

Search is moving beyond “throwing a few words in a text box,” if you want not to be left behind, you need to understand how to wring every bit out of these tools.

Link - Google’s Prompt Engineering Guide

You can’t spell Safari without “AI”

Apple’s senior VP of services, Eddy Cue, testified in Google’s antitrust trial this week and dropped this gem:

“Prior to AI, my feelings around this were that none of the others were valid choices. I think today there is much greater potential because there are new entrants attacking the problem in a different way.”

Setting aside whether or not this helps or hurts Google’s case, I think it:

1. Shows the disruption (hey, that’s our newsletter’s name) that AI is bringing to search.
2. It shows that Apple is still waiting to move in the AI search space.
3. Shows that new and useful search experiences come from places other than Google.

Link - Apple’s Eddy Cue on AI and Search

Thanks

We’ve been hiring at Knowatoa, and I want to thank both the new folks onboarding in marketing + dev roles and all the people who took the time to apply or reach out.

p.s. It would really help me out if you could Follow me on LinkedIn