{"id":695,"date":"2025-09-18T23:10:53","date_gmt":"2025-09-18T23:10:53","guid":{"rendered":"https:\/\/www.pofii.com\/blog\/?p=695"},"modified":"2025-09-18T23:11:13","modified_gmt":"2025-09-18T23:11:13","slug":"robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce","status":"publish","type":"post","link":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/","title":{"rendered":"Robots.txt &amp; Sitemaps: Clean Crawl for WordPress &amp; WooCommerce"},"content":{"rendered":"\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_84 ez-toc-wrap-center counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<span class=\"ez-toc-title-toggle\"><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#What_robotstxt_does_and_doesnt\" >What robots.txt does (and doesn\u2019t)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#A_minimal_modern_robotstxt_for_WordPress\" >A minimal, modern robots.txt for WordPress<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#What_to_noindex_not_Disallow\" >What to noindex (not Disallow)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#Sitemaps_what_to_include_and_exclude\" >Sitemaps: what to include (and exclude)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#WooCommerce_specifics_the_safe_defaults\" >WooCommerce specifics (the safe defaults)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#Validate_in_5_minutes\" >Validate in 5 minutes<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#Common_pitfalls_and_quick_fixes\" >Common pitfalls (and quick fixes)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#Example_WordPress_Woo_robots_meta_setup\" >Example: WordPress + Woo robots &amp; meta setup<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#Keep_crawl_efficient_performance_tie-ins\" >Keep crawl efficient (performance tie-ins)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#FAQ\" >FAQ<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#Final_word\" >Final word<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\" id=\"h-what-robots-txt-does-and-doesn-t\"><span class=\"ez-toc-section\" id=\"What_robotstxt_does_and_doesnt\"><\/span>What robots.txt does (and doesn\u2019t)<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Does:<\/strong> tell crawlers which paths <strong>not to crawl<\/strong>.<\/li>\n\n\n\n<li><strong>Does NOT:<\/strong> remove URLs from Google or stop indexing if other pages link to them. For that, use <strong>meta robots <code>noindex<\/code><\/strong>.<\/li>\n\n\n\n<li><strong>Rule of thumb:<\/strong> use <strong>robots.txt<\/strong> for <strong>utility paths<\/strong> (<code>\/wp-admin\/<\/code>, infinite filter URLs), and use <strong><code>noindex,follow<\/code><\/strong> for thin\/utility pages (search, cart, checkout, account).<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-a-minimal-modern-robots-txt-for-wordpress\"><span class=\"ez-toc-section\" id=\"A_minimal_modern_robotstxt_for_WordPress\"><\/span>A minimal, modern <strong>robots.txt<\/strong> for WordPress<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">Start simple. Don\u2019t copy legacy blocks like <code>\/wp-includes\/<\/code> or <code>\/wp-content\/<\/code>\u2014they often <strong>break CSS\/JS<\/strong> crawling.<\/p>\n<\/blockquote>\n\n\n\n<pre class=\"wp-block-code\"><code># robots.txt (minimal + safe)\nUser-agent: *\nDisallow: \/wp-admin\/\nAllow: \/wp-admin\/admin-ajax.php\n\n# Optional: block internal search URLs\nDisallow: \/?s=\nDisallow: \/search\/\n\n# Optional: block known noise (adjust to your site)\nDisallow: \/*add-to-cart=*\nDisallow: \/*coupon_code=*\nDisallow: \/*orderby=*\n\n# Your XML sitemap index (Yoast\/RankMath\/WP core)\nSitemap: https:\/\/example.com\/sitemap_index.xml\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>When to add parameter lines?<\/strong><br>Only if your theme\/plugins create <strong>infinite combinations<\/strong> (e.g., layered nav filters). If you rely on those for browsing, prefer <strong><code>noindex,follow<\/code><\/strong> on the template and let canonical tags point to the main category.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-what-to-noindex-not-disallow\"><span class=\"ez-toc-section\" id=\"What_to_noindex_not_Disallow\"><\/span>What to <strong>noindex<\/strong> (not Disallow)<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">These URLs should be <strong>accessible<\/strong> (so Google can see the meta tag), but <strong>not indexed<\/strong>:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Search results:<\/strong> <code>\/search\/<\/code> or <code>\/?s=\u2026<\/code> \u2192 <code>noindex,follow<\/code><\/li>\n\n\n\n<li><strong>Cart, Checkout, My Account<\/strong> \u2192 <code>noindex,follow<\/code><\/li>\n\n\n\n<li><strong>Thank you \/ order-received<\/strong> \u2192 <code>noindex,follow<\/code><\/li>\n\n\n\n<li><strong>Pagination of archives<\/strong> (optional) \u2192 many sites keep indexed; if thin, <code>noindex,follow<\/code><\/li>\n\n\n\n<li><strong>Tag archives<\/strong> (optional) \u2192 if thin\/duplicative, <code>noindex,follow<\/code><\/li>\n\n\n\n<li><strong>Author archives<\/strong> (single-author blogs) \u2192 <code>noindex,follow<\/code><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Set this in your SEO plugin (Yoast\/RankMath): <strong>Titles &amp; Metas \u2192 Archives\/Pages \u2192 noindex<\/strong> the templates above.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-sitemaps-what-to-include-and-exclude\"><span class=\"ez-toc-section\" id=\"Sitemaps_what_to_include_and_exclude\"><\/span>Sitemaps: what to include (and exclude)<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Use your SEO plugin\u2019s <strong>sitemap index<\/strong>:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Include:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Posts, Pages, <strong>Products<\/strong><\/li>\n\n\n\n<li>Product Categories (and optionally Post Categories)<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Consider excluding:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Tag archives (if thin)<\/li>\n\n\n\n<li>Author archives (single author)<\/li>\n\n\n\n<li>Test\/staging\/landing variations<\/li>\n\n\n\n<li>Anything set to <code>noindex<\/code> should <strong>not<\/strong> appear in sitemaps<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Make sure <code>lastmod<\/code> updates<\/strong> when you refresh content. After publishing or updating, request indexing in GSC.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-woocommerce-specifics-the-safe-defaults\"><span class=\"ez-toc-section\" id=\"WooCommerce_specifics_the_safe_defaults\"><\/span>WooCommerce specifics (the safe defaults)<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Noindex<\/strong>: <code>\/cart\/<\/code>, <code>\/checkout\/<\/code>, <code>\/my-account\/<\/code>, <code>\/order-received\/<\/code><\/li>\n\n\n\n<li><strong>Block crawl (optional)<\/strong>: URL parameters that explode combinations:\n<ul class=\"wp-block-list\">\n<li><code>\/*add-to-cart=*<\/code>, <code>\/*orderby=*<\/code>, <code>\/*rating=*<\/code><\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Canonical<\/strong>: category pages should canonical to themselves; filtered views should canonical back to the base category.<\/li>\n\n\n\n<li><strong>Facets\/filters<\/strong>: Prefer <strong><code>noindex,follow<\/code><\/strong> on the filtered template instead of blocking with robots.txt\u2014so link equity can still flow through the page.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-validate-in-5-minutes\"><span class=\"ez-toc-section\" id=\"Validate_in_5_minutes\"><\/span>Validate in 5 minutes<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Robots.txt Tester<\/strong> (Search Console) \u2192 Confirm rules match expectations.<\/li>\n\n\n\n<li><strong>URL Inspection<\/strong> \u2192 Check a cart\/checkout URL shows <strong>\u201cExcluded by \u2018noindex\u2019\u201d<\/strong>.<\/li>\n\n\n\n<li><strong>PageSpeed\/DevTools<\/strong> \u2192 Ensure CSS\/JS are <strong>crawlable<\/strong> (no blocked resources).<\/li>\n\n\n\n<li><strong>Sitemaps<\/strong> in GSC \u2192 Submit <code>sitemap_index.xml<\/code>, watch for <strong>Discovered<\/strong> \u2192 <strong>Indexed<\/strong> progression.<\/li>\n\n\n\n<li>Re-check your <strong>SEO audit<\/strong> items:<br><a href=\"https:\/\/www.pofii.com\/blog\/what-is-seo-and-how-to-audit-your-website-like-a-pro-for-free\/\">What is SEO &amp; How to Audit Your Website (Free)<\/a><\/li>\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-common-pitfalls-and-quick-fixes\"><span class=\"ez-toc-section\" id=\"Common_pitfalls_and_quick_fixes\"><\/span>Common pitfalls (and quick fixes)<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Blocking CSS\/JS<\/strong> in <code>\/wp-content\/<\/code> or <code>\/wp-includes\/<\/code><br>\u2192 Remove those Disallows. Google needs resources to render.<\/li>\n\n\n\n<li><strong>Disallowed but still indexed<\/strong><br>\u2192 You blocked crawl; Google can\u2019t see <code>noindex<\/code>. <strong>Allow crawl<\/strong> and set <strong><code>noindex,follow<\/code><\/strong>.<\/li>\n\n\n\n<li><strong>Sitemap lists noindexed pages<\/strong><br>\u2192 Exclude them in your SEO plugin \u2192 resubmit sitemap.<\/li>\n\n\n\n<li><strong>Infinite filter URLs indexed<\/strong><br>\u2192 Add <code>noindex,follow<\/code> on filtered templates + canonical back to the base category. Add selective Disallows if the crawl still balloons.<\/li>\n\n\n\n<li><strong>Cart\/Checkout indexed<\/strong><br>\u2192 Ensure they are <strong><code>noindex,follow<\/code><\/strong> and <strong>not<\/strong> in the sitemap. Remove any stray internal links pointing at them.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-example-wordpress-woo-robots-amp-meta-setup\"><span class=\"ez-toc-section\" id=\"Example_WordPress_Woo_robots_meta_setup\"><\/span>Example: WordPress + Woo robots &amp; meta setup<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>robots.txt<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: *\nDisallow: \/wp-admin\/\nAllow: \/wp-admin\/admin-ajax.php\nDisallow: \/?s=\nDisallow: \/search\/\nDisallow: \/*add-to-cart=*\nDisallow: \/*orderby=*\nSitemap: https:\/\/example.com\/sitemap_index.xml\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Meta robots (via SEO plugin)<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Search results: <code>noindex,follow<\/code><\/li>\n\n\n\n<li>Cart\/Checkout\/Account\/Thank-you: <code>noindex,follow<\/code><\/li>\n\n\n\n<li>Author archives (single author): <code>noindex,follow<\/code><\/li>\n\n\n\n<li>Tag archives (if thin): <code>noindex,follow<\/code><\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-keep-crawl-efficient-performance-tie-ins\"><span class=\"ez-toc-section\" id=\"Keep_crawl_efficient_performance_tie-ins\"><\/span>Keep crawl efficient (performance tie-ins)<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Speed + caching:<\/strong> fast pages get more done per crawl budget.<br><a href=\"https:\/\/www.pofii.com\/blog\/litespeed-cache-for-wordpress-safe-defaults-for-speed-woo-included\/\">LiteSpeed Cache: Safe Defaults<\/a><\/li>\n\n\n\n<li><strong>Edge\/CDN headers:<\/strong> avoid redirect chains; respect HTTPS\/www canonical.<br><a href=\"https:\/\/www.pofii.com\/blog\/zero-downtime-website-migration-the-dns-ttl-playbook\/\">Zero-Downtime DNS TTL Playbook<\/a><\/li>\n\n\n\n<li><strong>Don\u2019t waste crawl on noisy JS\/AJAX:<\/strong><br><a href=\"https:\/\/www.pofii.com\/blog\/woocommerce-cart-fragments-stop-the-ajax-drag\/\">Stop Woo Cart Fragments<\/a><\/li>\n\n\n\n<li><strong>Tidy images &amp; LCP:<\/strong> helps render and indexing quality.<br><a href=\"https:\/\/www.pofii.com\/blog\/woocommerce-image-speed-webp-avif-correct-sizes-lazy-load\/\">Woo Image Speed<\/a><\/li>\n\n\n\n<li><strong>Core Web Vitals basics:<\/strong><br><a href=\"https:\/\/www.pofii.com\/blog\/core-web-vitals-for-websites-lcp-inp-cls-in-2025\/\">Core Web Vitals, Plain English<\/a><\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-faq\"><span class=\"ez-toc-section\" id=\"FAQ\"><\/span>FAQ<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Should I disallow <code>\/wp-content\/uploads\/<\/code>?<\/strong><br>No. That breaks image discovery and hurts SEO.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Can I block <code>\/cart\/<\/code> in robots.txt?<\/strong><br>Better: keep it crawlable and set <strong><code>noindex,follow<\/code><\/strong> so link equity flows through the page.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Do sitemaps boost rankings?<\/strong><br>They don\u2019t increase rankings directly\u2014they help <strong>discovery<\/strong> and <strong>freshness<\/strong>. Rankings improve when the right pages are crawled and are <strong>fast, unique, and useful<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Do I need tag sitemaps?<\/strong><br>Only if tags have unique value. Most stores can <strong>exclude tags<\/strong> from indexing and sitemaps.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-final-word\"><span class=\"ez-toc-section\" id=\"Final_word\"><\/span>Final word<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Clean crawl \u2260 complicated rules. Keep robots.txt <strong>minimal<\/strong>, push thin\/utility pages to <strong><code>noindex,follow<\/code><\/strong>, and offer a tidy <strong>sitemap index<\/strong>. That\u2019s it. If you want a speed head start while Google crawls more efficiently, Pofii\u2019s <strong>Pofii-Tuned LiteSpeed<\/strong> stack pairs perfectly with this setup.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A clean crawl means Google spends time on the right URLs\u2014your products, posts, and key pages\u2014not search results, filters, or logins. This guide gives you a minimal robots.txt, shows what to noindex (not block), and sets up sitemaps the right way for WordPress + WooCommerce.<\/p>\n","protected":false},"author":5,"featured_media":703,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,3,4],"tags":[262,264,188],"class_list":["post-695","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ecommerce","category-hosting","category-wordpress","tag-robots-txt","tag-sitemap","tag-wordpress"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v25.9 (Yoast SEO v25.9) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Robots.txt &amp; Sitemaps: Clean Crawl for WordPress &amp; WooCommerce<\/title>\n<meta name=\"description\" content=\"Robots.txt &amp; sitemaps for WordPress\/Woo: what to Disallow vs noindex, and how to keep crawl clean. Minimal rules, examples, checklist.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Robots.txt &amp; Sitemaps: Clean Crawl for WordPress &amp; WooCommerce\" \/>\n<meta property=\"og:description\" content=\"Robots.txt &amp; sitemaps for WordPress\/Woo: what to Disallow vs noindex, and how to keep crawl clean. Minimal rules, examples, checklist.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/\" \/>\n<meta property=\"og:site_name\" content=\"Pofii Insights\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/PofiiCOM\" \/>\n<meta property=\"article:published_time\" content=\"2025-09-18T23:10:53+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-09-18T23:11:13+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2025\/09\/robots-txt-and-sitemaps.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1280\" \/>\n\t<meta property=\"og:image:height\" content=\"720\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"John Cavil\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@PofiiGlobal\" \/>\n<meta name=\"twitter:site\" content=\"@PofiiGlobal\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"John Cavil\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/\"},\"author\":{\"name\":\"John Cavil\",\"@id\":\"https:\/\/www.pofii.com\/blog\/#\/schema\/person\/82dd56bd4b7d1ae512907145b3a68873\"},\"headline\":\"Robots.txt &amp; Sitemaps: Clean Crawl for WordPress &amp; WooCommerce\",\"datePublished\":\"2025-09-18T23:10:53+00:00\",\"dateModified\":\"2025-09-18T23:11:13+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/\"},\"wordCount\":669,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.pofii.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2025\/09\/robots-txt-and-sitemaps.png\",\"keywords\":[\"Robots.txt\",\"Sitemap\",\"WordPress\"],\"articleSection\":[\"eCommerce\",\"Hosting Tips &amp; Tricks\",\"WordPress Mastery\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/\",\"url\":\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/\",\"name\":\"Robots.txt &amp; Sitemaps: Clean Crawl for WordPress &amp; WooCommerce\",\"isPartOf\":{\"@id\":\"https:\/\/www.pofii.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2025\/09\/robots-txt-and-sitemaps.png\",\"datePublished\":\"2025-09-18T23:10:53+00:00\",\"dateModified\":\"2025-09-18T23:11:13+00:00\",\"description\":\"Robots.txt & sitemaps for WordPress\/Woo: what to Disallow vs noindex, and how to keep crawl clean. Minimal rules, examples, checklist.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#primaryimage\",\"url\":\"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2025\/09\/robots-txt-and-sitemaps.png\",\"contentUrl\":\"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2025\/09\/robots-txt-and-sitemaps.png\",\"width\":1280,\"height\":720,\"caption\":\"robots txt and sitemaps\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/nl.pofii.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Robots.txt &amp; Sitemaps: Clean Crawl for WordPress &amp; WooCommerce\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.pofii.com\/blog\/#website\",\"url\":\"https:\/\/www.pofii.com\/blog\/\",\"name\":\"Pofii\",\"description\":\"Success Simplified.\",\"publisher\":{\"@id\":\"https:\/\/www.pofii.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.pofii.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.pofii.com\/blog\/#organization\",\"name\":\"Pofii\",\"url\":\"https:\/\/www.pofii.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.pofii.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-cropped-pofiiinsights-2.png\",\"contentUrl\":\"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-cropped-pofiiinsights-2.png\",\"width\":2079,\"height\":221,\"caption\":\"Pofii\"},\"image\":{\"@id\":\"https:\/\/www.pofii.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/PofiiCOM\",\"https:\/\/x.com\/PofiiGlobal\",\"https:\/\/www.instagram.com\/PofiiGlobal\",\"https:\/\/www.linkedin.com\/company\/Pofii\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.pofii.com\/blog\/#\/schema\/person\/82dd56bd4b7d1ae512907145b3a68873\",\"name\":\"John Cavil\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.pofii.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/www.pofii.com\/brand\/logo-icon-circle-friendly.png\",\"contentUrl\":\"https:\/\/www.pofii.com\/brand\/logo-icon-circle-friendly.png\",\"caption\":\"John Cavil\"},\"description\":\"WordPress Expert, Head of technical support at Pofii Ltd.\",\"sameAs\":[\"https:\/\/www.pofii.com\/blog\/author\/JohnC\"],\"url\":\"https:\/\/www.pofii.com\/blog\/author\/johnc\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Robots.txt &amp; Sitemaps: Clean Crawl for WordPress &amp; WooCommerce","description":"Robots.txt & sitemaps for WordPress\/Woo: what to Disallow vs noindex, and how to keep crawl clean. Minimal rules, examples, checklist.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/","og_locale":"en_US","og_type":"article","og_title":"Robots.txt &amp; Sitemaps: Clean Crawl for WordPress &amp; WooCommerce","og_description":"Robots.txt & sitemaps for WordPress\/Woo: what to Disallow vs noindex, and how to keep crawl clean. Minimal rules, examples, checklist.","og_url":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/","og_site_name":"Pofii Insights","article_publisher":"https:\/\/www.facebook.com\/PofiiCOM","article_published_time":"2025-09-18T23:10:53+00:00","article_modified_time":"2025-09-18T23:11:13+00:00","og_image":[{"width":1280,"height":720,"url":"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2025\/09\/robots-txt-and-sitemaps.png","type":"image\/png"}],"author":"John Cavil","twitter_card":"summary_large_image","twitter_creator":"@PofiiGlobal","twitter_site":"@PofiiGlobal","twitter_misc":{"Written by":"John Cavil","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#article","isPartOf":{"@id":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/"},"author":{"name":"John Cavil","@id":"https:\/\/www.pofii.com\/blog\/#\/schema\/person\/82dd56bd4b7d1ae512907145b3a68873"},"headline":"Robots.txt &amp; Sitemaps: Clean Crawl for WordPress &amp; WooCommerce","datePublished":"2025-09-18T23:10:53+00:00","dateModified":"2025-09-18T23:11:13+00:00","mainEntityOfPage":{"@id":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/"},"wordCount":669,"commentCount":0,"publisher":{"@id":"https:\/\/www.pofii.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#primaryimage"},"thumbnailUrl":"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2025\/09\/robots-txt-and-sitemaps.png","keywords":["Robots.txt","Sitemap","WordPress"],"articleSection":["eCommerce","Hosting Tips &amp; Tricks","WordPress Mastery"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/","url":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/","name":"Robots.txt &amp; Sitemaps: Clean Crawl for WordPress &amp; WooCommerce","isPartOf":{"@id":"https:\/\/www.pofii.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#primaryimage"},"image":{"@id":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#primaryimage"},"thumbnailUrl":"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2025\/09\/robots-txt-and-sitemaps.png","datePublished":"2025-09-18T23:10:53+00:00","dateModified":"2025-09-18T23:11:13+00:00","description":"Robots.txt & sitemaps for WordPress\/Woo: what to Disallow vs noindex, and how to keep crawl clean. Minimal rules, examples, checklist.","breadcrumb":{"@id":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#primaryimage","url":"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2025\/09\/robots-txt-and-sitemaps.png","contentUrl":"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2025\/09\/robots-txt-and-sitemaps.png","width":1280,"height":720,"caption":"robots txt and sitemaps"},{"@type":"BreadcrumbList","@id":"https:\/\/www.pofii.com\/blog\/robots-txt-sitemaps-clean-crawl-for-wordpress-woocommerce\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/nl.pofii.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Robots.txt &amp; Sitemaps: Clean Crawl for WordPress &amp; WooCommerce"}]},{"@type":"WebSite","@id":"https:\/\/www.pofii.com\/blog\/#website","url":"https:\/\/www.pofii.com\/blog\/","name":"Pofii","description":"Success Simplified.","publisher":{"@id":"https:\/\/www.pofii.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pofii.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.pofii.com\/blog\/#organization","name":"Pofii","url":"https:\/\/www.pofii.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.pofii.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-cropped-pofiiinsights-2.png","contentUrl":"https:\/\/www.pofii.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-cropped-pofiiinsights-2.png","width":2079,"height":221,"caption":"Pofii"},"image":{"@id":"https:\/\/www.pofii.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/PofiiCOM","https:\/\/x.com\/PofiiGlobal","https:\/\/www.instagram.com\/PofiiGlobal","https:\/\/www.linkedin.com\/company\/Pofii"]},{"@type":"Person","@id":"https:\/\/www.pofii.com\/blog\/#\/schema\/person\/82dd56bd4b7d1ae512907145b3a68873","name":"John Cavil","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.pofii.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/www.pofii.com\/brand\/logo-icon-circle-friendly.png","contentUrl":"https:\/\/www.pofii.com\/brand\/logo-icon-circle-friendly.png","caption":"John Cavil"},"description":"WordPress Expert, Head of technical support at Pofii Ltd.","sameAs":["https:\/\/www.pofii.com\/blog\/author\/JohnC"],"url":"https:\/\/www.pofii.com\/blog\/author\/johnc\/"}]}},"_links":{"self":[{"href":"https:\/\/www.pofii.com\/blog\/wp-json\/wp\/v2\/posts\/695","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pofii.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.pofii.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pofii.com\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pofii.com\/blog\/wp-json\/wp\/v2\/comments?post=695"}],"version-history":[{"count":1,"href":"https:\/\/www.pofii.com\/blog\/wp-json\/wp\/v2\/posts\/695\/revisions"}],"predecessor-version":[{"id":696,"href":"https:\/\/www.pofii.com\/blog\/wp-json\/wp\/v2\/posts\/695\/revisions\/696"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.pofii.com\/blog\/wp-json\/wp\/v2\/media\/703"}],"wp:attachment":[{"href":"https:\/\/www.pofii.com\/blog\/wp-json\/wp\/v2\/media?parent=695"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.pofii.com\/blog\/wp-json\/wp\/v2\/categories?post=695"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.pofii.com\/blog\/wp-json\/wp\/v2\/tags?post=695"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}