SEO Index Feature Guide

1. Introduction

SEO Index is a Webhub tool for optimizing search engine indexing. It helps you manage the settings that affect how your site is crawled, indexed, and shown in search results.

With SEO Index, you can centrally manage the following indexing-related files:

• Robots.txt: Controls which paths on your website search engine crawlers are allowed to access.
• Sitemap.xml: Submits your site structure to search engines so they can discover and index pages faster.

Properly configuring Robots and Sitemap can help improve indexing efficiency and avoid low-value pages being indexed, while also supporting better performance in search results.

2. Features

2.1 Robots

2.1.1 Introduction

Robots.txt is a standard file in your site's root directory. It tells search engine crawlers (such as Google and Bing) which paths can be crawled and which should not be crawled.

• Purpose: Controls crawler access, helps keep sensitive or irrelevant pages out of search results, and saves crawl budget for important pages.
• Common use cases: Allow crawlers to access the homepage and key sections, while blocking the admin area, temporary directories, duplicate content, and similar pages.

In SEO Index, you can edit your Robots.txt file directly or configure it with simple rules, without uploading files manually.

2.1.2 Simple Mode - Add Rules

Simple Mode lets you create Robots rules with visual options. No code is required.

• Select crawler type: Create rules for all crawlers (User-agent: *) or a specific crawler, such as Googlebot or Bingbot.
• Add allowed/disallowed paths:

○ Allow: Specify the path or directory you want crawlers to access.

○ Disallow: Specify the path or directory you do not want crawlers to crawl or index.

• Save and apply: After you save, the system creates or updates the robots.txt file in your site's root directory. Crawlers use the latest rules the next time they visit.

This mode is ideal for users who want common Robots settings without learning the Robots syntax.

2.1.3 Code Editor Mode

Code Editor Mode lets you directly edit the full robots.txt file. It is suitable for users who need precise control or advanced syntax.

• What you can edit: Edit the Robots protocol text directly in the editor. It supports multiple User-agent groups, multiple Allow/Disallow rules, and comments (starting with #).
• Syntax reference (brief guidance can be shown in the interface):

○ User-agent: * means the rule applies to all crawlers.

○ Disallow: /path/ means crawling is blocked for this path and its subpaths.

○ Allow: /path/ can be used together with Disallow to allow a small number of paths inside a blocked directory.

• Validate and publish: Run a basic format check before saving. After confirmation, publish the file so the robots.txt file in your site's root directory is updated.

When using Code Mode, please make sure the syntax is correct. Invalid rules may cause crawlers to misread the file or ignore your site.

2.2 Sitemap

2.2.1 Introduction

Sitemap is a file (usually XML) that lists important URLs on your site and some metadata. It helps search engines discover, crawl, and index your pages.

• Purpose: It helps search engines find pages they may miss through normal links. It is especially useful for new sites, large sites, or complex site structures, and can speed up indexing of important pages.
• Common formats: Sitemap.xml (or sitemap_index.xml as an index that links to multiple child sitemaps).

In SEO Index, you can manage your Sitemap file and its submission status.

2.2.2 Features

• Generate and update: Automatically generate a Sitemap from your site's pages (such as sections, articles, and products), or generate it by rule. You can also regenerate it after content updates.
• View and download: View the current URL list in your Sitemap, or open/download sitemap.xml directly to check it.
• Submit and notify: Provides guidance or an entry point for submitting your Sitemap to major search engines (such as Google Search Console and Bing Webmaster Tools), so updates can be discovered sooner.

Keeping your Sitemap updated helps improve overall indexing and the crawl priority of important pages.

3. FAQ

3.1 How long does it take for changes to Robots or Sitemap to take effect?

• Robots.txt: Crawlers read the latest robots.txt each time they crawl. So an update takes effect the next time a crawler visits. Visible index changes may still take hours or days, depending on each search engine's crawl frequency.
• Sitemap: After the file is updated, resubmitting it in tools such as Google Search Console or Bing, or triggering a new crawl, can help search engines find new content faster.

Do I have to configure both Robots and Sitemap?

No, but using them together is recommended:

• Robots.txt is used to control which pages should not be crawled;
• Sitemap is used to tell search engines which pages are recommended for crawling.

Together, they make it easier to guide crawlers toward your most important pages.

3.2 If a path is marked as Disallow in Robots, can it still be indexed?

Disallow means crawling is discouraged, and most major crawlers follow it. However, a known URL may not disappear immediately. If links to that path still exist on other pages, search engines may still find it and sometimes still crawl it. To remove it from search results, you usually also need a removal request in webmaster tools or tags such as noindex.

3.3 How many URLs should be included in a Sitemap?

A single Sitemap file should generally contain no more than 50,000 URLs, and its uncompressed size should generally stay under 50 MB. If your site is larger, use a Sitemap index file (sitemap index) to split it into multiple sub-sitemaps. SEO Index handles size limits and splitting automatically.

3.4 How can I confirm that a search engine has read my Robots.txt?

• Open https://your-domain/robots.txt in your browser and confirm that it matches the configuration in Copilot.
• Check the Robots.txt or crawl-related reports in tools such as Google Search Console and Bing Webmaster Tools to see crawler fetch and parsing status.

Note: The actual interface and steps may vary by product version. If anything differs from the descriptions above, please refer to the features currently available in your site's backend.