The Growth Fix logoThe Growth FixGrowth diagnosis + execution

About the Marketing Priority AI crawler

MarketingPriorityAI/1.0

If you are reading this, you have most likely seen our crawler in a server log. This page tells you what it is, what it does to your server, and how to stop it.

What it is

Marketing Priority AI is a marketing analysis tool operated by The Growth Fix. When a business becomes a client and connects their website, our crawler reads that site's public pages so we can advise them on its structure — which pages exist, how they link together, whether they are indexable, and what contact routes they offer.

We only crawl sites our clients have connected themselves. We do not crawl the web at large, we do not build a search index, and we do not collect training data. If our crawler reached your site, someone with access to it asked us to look at it. If you believe that is wrong, tell us and we will stop — see below.

How to identify it

Every request we make carries this User-Agent:

MarketingPriorityAI/1.0 (+https://www.thegrowthfix.io/crawler)

We do not disguise the crawler, rotate user agents, or present ourselves as a browser. If you see a request claiming to be us that behaves differently from what is described on this page, it is not us — the user agent string is public and anyone can copy it.

What it does to your server

  • One request per second at most, per site, by default — and slower if your robots.txt asks for slower.
  • 2,000 pages at most in a single pass.
  • Only GET requests for HTML. We never submit forms, never log in, and never attempt to reach anything behind a password.
  • One crawl at a time. We do not open parallel connections to the same host.

If our crawler is causing you a problem in spite of this, we would genuinely like to know.

It obeys robots.txt

We read /robots.txt before we fetch anything, and we honour it. Any path you disallow is never requested. We match on the product token MarketingPriorityAI, which is case-insensitive, and we honour Crawl-delay as a minimum wait between requests — if you ask for ten seconds, you get ten seconds.

How to block it

To stop us from crawling anything on your site, add this to your robots.txt:

User-agent: MarketingPriorityAI
Disallow: /

To allow most of the site but keep us out of one area:

User-agent: MarketingPriorityAI
Disallow: /members/
Disallow: /checkout/

To slow us down rather than block us:

User-agent: MarketingPriorityAI
Crawl-delay: 10

These take effect on our next crawl; we do not cache your robots.txt between passes. You are of course also free to block us at your firewall or WAF, and we will not attempt to work around it.

What we store, and for how long

We keep the raw HTML of the pages we fetch for up to 30 days, so that a finding about a page can be re-checked without fetching the page again. After 30 days the HTML is deleted automatically. What remains is the structural summary we derived from it — page titles, headings, whether a page is indexable, which contact routes it offers.

Your pages may mention people — staff, customers, anyone whose details you have published. We do not use that information to contact anybody, we do not combine it with data from anywhere else, and we do not sell or share it. Our privacy policy sets out the detail, including the legal basis we rely on.

Contact

Questions, complaints, or a request that we stop crawling a site — including a request from someone named on a page we hold — go to romualdo@thegrowthfix.io and reach a person, not a queue.