Case 001. Cloudflare Crawler Surface Policy: Controlling Bot Access to URLs

Case 001. Cloudflare Crawler Surface Policy: Controlling Bot Access to URLs

The Great and Terrible Cloudflare series. Cloudflare Operations — policy, security and fleet management.

How can a store close transactional URLs to crawlers, retain access to public content, and verify the result across a fleet of sites? This case describes a solution that developed from a particular production incident and client infrastructure.

According to the author’s account, the fleet covered 22 sites with a combined daily audience of approximately 400,000 unique users. These figures provide author-supplied context; the aggregate metrics and pilot results below have not undergone an independent audit. Client domains, credentials, and private snapshots are excluded from publication.

The management tools initially ran on the client’s shared hosting and were written in PHP: the environment already existed alongside the sites, and the operator needed no separate server. A new independent installation can use a local environment such as WSL. The architectural principle remains consistent: configuration and policy have a reviewable state, secrets and private overlays stay local, and Cloudflare changes pass through a separate controlled write path.

Standalone code and examples: IRONCREED/cloudflare, under Case 001. Every instruction below uses that repository’s structure.

A useful bot on a transactional surface

On a WooCommerce site, crawler traffic repeatedly requested routes and parameters serving the cart, checkout, and customer account:

/cart/
/checkout/
/my-account/
?add-to-cart=
?remove_item=
?wc-ajax=

These requests can invoke PHP, WooCommerce, sessions, and cart operations. Meta-WebIndexer was one conspicuous participant. Globally blocking that User-Agent would also remove its access to product pages and articles that may have search and linking value.

The decision unit becomes crawler identity × request surface × profile × zone configuration.

Three URL classes

The example paths match /cart, /checkout, /my-account, and descendants with a / after the name. A similar prefix such as /cartography is outside that list. Translated WooCommerce slugs, language prefixes, URL encoding, and query behaviour require testing on each actual site.

The demonstration rule searches for add-to-cart=, remove_item=, and wc-ajax= as query substrings. This simple example may also match another parameter’s value. Its matching precision needs review before production use. Filters and search remain outside the shared example: a useful SEO landing page on one store may resemble an unnecessary combination on another.

Identity and scope

Using all of cf.client.bot as this case’s crawler list would be too broad. Verified automation also includes monitoring, security checks, and other services. The demonstration uses narrower cf.verified_bot_category values:

Search Engine Crawler
AI Search
AI Crawler
Page Preview

It adds User-Agent fallbacks for Meta-WebIndexer, Googlebot, and Bingbot. These strings can be spoofed: a fallback classifies a request but does not authenticate its origin. As checked on 6 September 2026, Cloudflare retains these legacy categories for WAF compatibility, while its new taxonomy classifies bot behaviour differently. Verified bots

Other Cloudflare rules and the origin can change the final response. Failure to match one block rule does not create a global allow.

Why the edge

The restriction could be implemented in WordPress, Apache, nginx, or the application. Choosing Cloudflare places rejection before the origin: a crawler matching the defined transactional surface should not invoke WooCommerce. Custom Rules execute in their own ruleset-engine phase. Cloudflare Custom Rules

The initial production model combined a global baseline, site profile, and hostname exceptions. The portable tool generalises that model through explicit policy composition.

From production workspace to a standalone repository

The original shared-hosting workspace contained policies, profiles, exceptions, scripts, backups, and reports. It produced the compiler, diff, dry-run, audit, deployment, snapshots, and edge smoke. The standalone repository contains the reusable mechanism, fixtures, and demonstration configurations. Private client overlays remain outside it.

The relocation preserves the case ID, licences, and exact submodule revisions. A provenance record identifies the source tree and subsequent changes. Repository preparation and automated checks leave the final release decision with a human.

Case 001’s boundaries

The tool manages zone-level WAF Custom Rules in http_request_firewall_custom. DNS, Workers, Zero Trust, cache rules, billing, account membership, and origin configuration remain outside this case.

The files bot-example.json, country-policy.json, crawler-surface.json, named-ip-list.example.json, and scanner-user-agents.json demonstrate the mechanism’s capabilities. For example, the commerce fixture additionally includes bot-example; reproducing the pilot’s surface-only restriction requires separately reviewing and, where appropriate, removing that extra policy. The examples are not a production baseline.

Inventory, profiles, and exceptions

The inventory explicitly lists zones, phase, ownership prefix, and budget:

{
	"schema_version": 1,
	"phase": "http_request_firewall_custom",
	"managed_prefix": "[CFOPS]",
	"max_physical_rules": 5,
	"zones": ["domain-01.example", "domain-02.example", "domain-03.example"]
}

5 is an example parameter tied to the pilot’s conditions. Check it against actual zone limits; the budget must also count unmanaged rules. Custom Rules availability

A zone configuration selects a profile and modifies its composition:

{
	"zone": "domain-02.example",
	"profile": "commerce",
	"additional_policies": ["bot-example"],
	"disabled_policies": []
}

An inline rule can handle one hostname without creating another shared profile. The compiler also requires id and name:

{
	"zone": "domain-03.example",
	"profile": "restricted-geo",
	"inline_rules": [
		{
			"id": "forum-example",
			"name": "Forum hostname example",
			"action": "managed_challenge",
			"physical_group": "challenge",
			"expression": "(http.host eq \"forum.domain-03.example\" and ip.src.country ne \"UA\")"
		}
	]
}

The geographic restriction demonstrates an exception mechanism. Its suitability depends on the actual site’s audience.

Logical and physical rules

PolicyCompiler collects profile policies, adds additional_policies, removes disabled_policies, and appends inline_rules. Simple rules with the same action and physical_group are joined with or into one physical rule. Their logical files remain separate.

(crawler_surface_condition) or (bot_condition)

This is an explanatory formula, not a deployable Cloudflare expression. Grouping requires compatible actions and a review of execution order. The current compiler serves the demonstrated simple rules; it does not guarantee preservation of arbitrary action_parameters in managed policies.

The managed/unmanaged boundary

The tool owns only rules whose description begins with [CFOPS]. A new managed block replaces the first previous managed rule’s position; when none exists, it is appended. Unmanaged rules retain their relative order.

If managed and unmanaged rules were interleaved, collecting the managed portion into one block may change their positions relative to each other. Review the plan for that change. Another writer using the same prefix breaks the ownership boundary. Sequential rollout inside one process also provides no lock against simultaneous dashboard edits or another tool.

A local WSL environment

On Windows, WSL can be installed from PowerShell:

wsl --install -d Ubuntu

The command and prerequisites are described in Microsoft’s documentation. Inside Ubuntu, install Git, PHP 8.1+, and cURL:

sudo apt update
sudo apt install -y git php-cli php-curl
git clone https://github.com/IRONCREED/cloudflare.git
cd cloudflare
git submodule update --init --recursive
cd cases/001-crawler-surface-policy
php tests/run.php
cp -R examples local
export CFOPS_CONFIG_DIR="$PWD/local"
export CFOPS_RUNTIME_DIR="$PWD/runtime"

local/ is excluded from Git and holds real zones and private overlays. runtime/ holds snapshots and reports. Access to normative submodules follows their own repositories’ policies; offline engine tests do not require downloading them. Copy examples during the first installation, before editing local configuration.

Creating and storing API tokens

In Cloudflare, open your profile → API Tokens → Create Token → Create Custom Token. Read operations in this case require Zone / Zone / Read for zone-ID lookup and Zone / WAF / Read for rulesets. Give a separate write token Zone / Zone / Read and Zone / WAF / Edit; it is also used for inspection before and after writing. Restrict both tokens to the required zones and, where practical, an expiry and the operator’s IP. Check permission names and availability in the current interface. Create a token, permission reference

For a single Bash session, enter the read token without displaying it or placing its value in command history:

read -rsp 'Cloudflare read token: ' CLOUDFLARE_READ_TOKEN
printf '\n'
export CLOUDFLARE_READ_TOKEN

Enter the write token in the same way as CLOUDFLARE_WRITE_TOKEN immediately before an approved operation, then remove it from the environment:

read -rsp 'Cloudflare write token: ' CLOUDFLARE_WRITE_TOKEN
printf '\n'
export CLOUDFLARE_WRITE_TOKEN
# Run the reviewed apply command here.
unset CLOUDFLARE_WRITE_TOKEN

For reuse, use a secrets manager or a file outside the checkout, for example under ~/.config/ironcreed/cloudflare/, with directory permissions 700 and file permissions 600. Enter values through an editor or secrets manager; .gitignore cannot remove a secret already committed to Git history. Revoke and replace an exposed token. Environment variables are inherited by child processes, so load the write token only for the operation.

Fixtures, compile, and plan

php tests/run.php checks the local compilation model without API tokens. These tests do not execute expressions at the Cloudflare edge. Compilation is also local:

php bin/compile.php --zone=domain-02.example

Plan uses the read token and an actual zone’s configuration. Replace example.com in the following commands with a zone from your own inventory:

php bin/plan.php --zone=example.com

Ruleset Manager resolves the zone ID, reads the phase entry point, separates managed rules, compiles desired state, constructs the final list, checks the budget, and identifies a change. It performs no write. Expression comparison preserves whitespace inside strings; a formatting difference can produce a conservative review signal.

Apply and evidence

php bin/apply.php --zone=example.com --apply

Without --apply, the CLI refuses before creating an API client. With that flag, it uses the separate write token. Before writing, the tool reads the current state again and checks the limit. An unchanged result returns NO_CHANGE.

A change creates a unique runtime/deployments/<timestamp-random>--<zone>/ directory containing before.json, write-response.json, and after.json. An existing entry point is updated through one ruleset operation; otherwise, a zone ruleset is created for the phase. Rulesets API

Live state is read again after the write. Remaining drift ends the operation with Post-write verification failed. A successful HTTP response alone does not establish a correct deployment. Case 001 provides no automatic rollback; restoration from a snapshot is a separate reviewed operator action.

Fleet rollout, audit, and export

Audit reports drift and retains the read-only boundary. Snapshots, reports, and exports stay local. Traversing the entire inventory is also useful for demonstrating that most zones require no changes.

The original pilot’s results

The pilot ran on the earlier private EMBO tools before Case 001 was extracted. The following numbers come from the author’s pilot account; they are not results of a new production run of the standalone code.

The author’s smoke summary was 10 PASS, 0 FAIL. Meta and Googlebot name test-request profiles; a substituted User-Agent alone does not test a real verified identity. The homepage’s 200 records access in that test rather than a guarantee for every future request.

According to the pilot account, the commerce change was assigned to one zone; the other 21 retained their previous expected configuration. These aggregate results express the intended acceptance criterion. Independent reproduction requires your own configuration, a reviewed plan, and an edge smoke.

From pilot to portable pipeline

The operational discipline follows compile → diff/plan → dry-run → apply → smoke → audit. Different commands test different claims: compilation checks the model, plan identifies divergence from live state, smoke tests request behaviour, and audit checks the expected state after a change. Readback does not replace edge smoke.

Telemetry and an analytical LLM layer

Crawler telemetry can report a new traffic class across several hostnames or a zone’s departure from its usual profile. That observation becomes a proposal for a policy revision. Authority to perform a security write remains a separate operator responsibility.

The analytical layer receives no write token. This allows observation to be automated while changes to site availability remain explicit, reviewable actions.

Boundaries and result

The case addresses a bounded task: express crawler access through identity and surface, assign the policy to the intended zone, and check the rest of the inventory. Faceted navigation, filters, and search require separate decisions; a commerce profile is assigned only after reviewing the actual site.

The result is a shared policy-management system with different effective states for individual zones. The repository separates a reusable mechanism from client data: demonstration domains use domain-N.example, private overlays and tokens stay local, and writes follow a separate controlled path.

Case author and source of the original production account: Sam Starling. Translation, editorial adaptation, questions, public-documentation verification, and standalone-code preparation were assisted by AI. Platform sources were checked on 6 September 2026; preparing this article involved no repeat deployment to the client’s Cloudflare account.