Half of 200 US stores shut out Black Friday bots

Black Friday is on 27 November. Between now and then, a growing share of the people shopping for your products will not be people. They will be assistants acting for them, fetching pages, comparing options and in some cases trying to complete the purchase.

On 2 November a London firm, Human Forward, tested what happens when one of those arrives. It pointed a crawler at 200 of the biggest retailers and direct-to-consumer brands in the United States. Ninety-eight of the 200 shut the door: 86 refused it outright and 12 would not load at all.

Then the part that should worry an operator more than the number does. Of the blocked sites whose robots.txt could be read, 58 do not block a single AI crawler in that file. Their published policy says come in. Something behind it says no.

Nobody at those companies decided to do this. That is exactly why it will still be true on Black Friday.

What happens when an agent tries to buy

Blocking is the loud failure. The quiet one is worse, and it is the thing the headline phrase “not ready” actually points at.

Of 48 large retailers whose sites could be read at all, 2 were set up for an AI agent to complete a checkout.

Be precise about what that counts, because the number is doing a lot of work. Agent ready here means the site carried a UCP manifest or a storefront MCP endpoint on 2 November. It is a check for whether the plumbing exists. Nobody placed a test order. So the real figure for completed agent purchases is unknown, and 2 of 48 is the ceiling rather than the measurement.

Even as a ceiling it is a stark number three weeks out. It says the infrastructure that lets an assistant finish a purchase is almost entirely absent from large US retail, and a shopper whose agent cannot check out either gets handed back to do it manually or goes to whichever competitor the agent could complete.

There is a third failure underneath both, and it needs no technology at all. Asked “What is Purple?”, seven of eight assistants described the colour rather than the mattress company. If a brand name collides with an ordinary word, the assistant never reaches the site to be blocked by it.

From the publisher

Everything we know about running Amazon, in one course.

The commercial strategy, taught by operators who run our own brands.

Check out Amazon Grand Masters

Nobody decided this

The robots.txt finding is the one to take to whoever owns your infrastructure, because it describes a failure that no one is watching for.

A retailer writes an AI policy. Someone updates robots.txt to permit the assistant crawlers, because that is the file everyone has heard of. The decision is made, minuted, and considered done.

Meanwhile the content delivery network, the web application firewall or the bot-management product sits in front of everything, scoring each request for how human it looks. An agent arrives with an unfamiliar user agent, solves no challenge and carries no session history. It scores badly and is dropped before the origin server ever sees it. Nothing in robots.txt is consulted, because robots.txt is a polite request to well-behaved crawlers, not an instruction to your own edge.

Human Forward is direct about this: the block comes from bot protection that turns away automated visitors it cannot verify.

So two layers disagree, and only one of them was in the room when the policy was written. The published position says yes. The enforced position says no. Traffic only ever meets the enforced one.

Test yours this afternoon

This is checkable without buying anything or calling a vendor, which makes it the highest-return item on any pre-peak list.

Request three of your own product pages from outside your corporate network, using an unfamiliar user agent. Then compare what comes back against what your robots.txt promises. A 403, a challenge page or a silent timeout is your enforced position, whatever the file says.

If the two disagree, the thing that needs changing is the bot-management configuration, not the file. That distinction matters because it moves the ticket from marketing to infrastructure, and those queues have very different lead times in November.

Which is the real reason to do it this week rather than next. Most retailers freeze configuration changes before peak. A bot policy discovered on 20 November is a bot policy you live with until January, through the highest-traffic fortnight of your year.

What it is worth, and what it is not

Three caveats, and they cut the claim down without removing it.

This is a single day. Everything above describes 2 November, measured against one named crawler. A site that refused it that day may behave differently on another, and a site that admitted it may still refuse a commercial assistant.

The company that ran the test sells the remedy. Human Forward calls itself the Agentic Commerce Company and sells Storefront, a product for making a site readable by shopping agents, at 199 a month. Both of its reports close by inviting you to run the same check on your own site. That does not make the measurements wrong, and the robots.txt finding in particular costs nothing to confirm for yourself. It does mean the urgency is the part that sells the product, and the method is the part worth keeping.

And the European read-through is thinner than it has been reported. Human Forward ran a separate UK study on 26 October, and its widely repeated finding, that one in five UK brands were not ready, is not what that study measured. The one-in-five figure is 15 of 75 brands that were not named by any assistant when asked for the best in their category, which is a visibility result. The UK readiness number is different again: 35 of 50 sites were not set up for agents to buy from. The two studies also used different methods, six assistants against eight, and different definitions of agent ready. A European operator should treat the US percentages as a warning to go and measure their own stack, not as a figure that describes their market.

What a seller does differently on Monday morning

• Fetch three of your own product pages from outside your network with an unfamiliar user agent, and compare the result against your robots.txt. Do it before any pre-peak configuration freeze.
• Ask whoever owns your edge whether agent traffic is in scope of the current bot policy. Most policies predate the question entirely.
• Search your own brand name in two assistants. If the answer is not your company, that is a content and entity problem, and no amount of crawler configuration fixes it.
• Treat UCP manifests and storefront MCP endpoints as a watch item rather than a November project. With 2 of 48 large US retailers carrying them, the standard is early, and arriving late to it costs nothing this year.

Where the honest uncertainty sits

No purchase was attempted in either study, so nothing here measures whether an agent checkout actually succeeds, only whether the standards are present on the page.

The sample is large US retailers and brands chosen by the researcher, not a random sample of retail, so the proportions do not transfer to smaller merchants, whose stacks are usually simpler and less aggressively defended.

And none of this measures the assistants people actually use. The crawler was Human Forward’s own, so the results describe how sites treat one unfamiliar automated visitor, which is a reasonable proxy for an agent and not the same thing as one.

The three-line version

• If you run the storefront: request your own product pages from outside your network this week, before any peak freeze. The question is not what your robots.txt says, it is what your edge actually returns.
• If you have just taken on digital: the gap to name is that AI access is declared in one file and enforced in a different system, and nobody currently owns the difference between the two.
• If you sign the budget: there is nothing to buy yet. Approve an hour of engineering time to test the two layers against each other, and refuse any agent-readiness package priced off a vendor’s own survey of how unready everyone is.

Sources

• Human Forward, US report: https://humanforward.io/us/research/black-friday-ai-shopping-2026
• Human Forward, UK report: https://humanforward.io/research/ai-shopping-2026
• ChannelX: https://channelx.world/2026/11/americas-biggest-stores-arent-ready-for-agentic-commerce-this-black-friday/