What Is llms.txt? How to Add llms.txt in WordPress: A Comprehensive Guide

Table of Contents#

1. What Is llms.txt?

   • 1.1 Definition and Purpose
   • 1.2 How llms.txt Differs from robots.txt
   • 1.3 The History and Evolution of llms.txt

2. Why llms.txt Matters for Website Owners

   • 2.1 Protecting Intellectual Property
   • 2.2 Controlling Content Quality and Attribution
   • 2.3 Mitigating Privacy and Compliance Risks
   • 2.4 Supporting Ethical AI Development

3. Understanding llms.txt Syntax and Directives

   • 3.1 Core Directives Explained
   • 3.2 Common Use Cases and Examples
   • 3.3 Best Practices for Writing llms.txt

4. How to Add llms.txt to Your WordPress Site

   • 4.1 Method 1: Manually Create and Upload via FTP/SFTP
   • 4.2 Method 2: Use a WordPress File Manager Plugin
   • 4.3 Method 3: Edit via cPanel File Manager
   • 4.4 Method 4: Add Custom Code to Generate llms.txt Dynamically

5. Testing Your llms.txt File

   • 5.1 Verify File Location and Accessibility
   • 5.2 Check Syntax for Errors
   • 5.3 Monitor LLM Scraping Activity

6. Common Issues and Troubleshooting

   • 6.1 llms.txt Not Loading? Check File Location
   • 6.2 Syntax Errors: How to Diagnose and Fix
   • 6.3 Permissions Issues: Ensuring Read Access
   • 6.4 LLM Providers Not Respecting llms.txt: What to Do

7. Best Practices for llms.txt in WordPress

   • 7.1 Keep It Simple and Specific
   • 7.2 Regularly Update Your llms.txt File
   • 7.3 Combine with robots.txt for Holistic Control
   • 7.4 Document Your llms.txt Policies

8. Conclusion

9. References

1. What Is llms.txt?#

1.1 Definition and Purpose#

llms.txt is a text file placed in the root directory of a website that instructs large language models (LLMs) on which parts of the site they are allowed to scrape and use for training. It is modeled after robots.txt, a long-standing standard for controlling search engine crawlers like Googlebot.

The primary goal of llms.txt is to give website owners agency over their content. As LLMs become more sophisticated, they rely on vast datasets of text scraped from the internet to learn and generate human-like responses. Without safeguards, your blog posts, articles, product descriptions, or creative work could be used to train AI models without your knowledge or permission. llms.txt aims to fix this by providing a clear, standardized way to communicate preferences to LLM developers.

1.2 How llms.txt Differs from robots.txt#

At first glance, llms.txt and robots.txt seem similar—both are text files in the root directory with directives for automated systems. However, their targets and purposes differ significantly:

In short: robots.txt helps you manage search visibility, while llms.txt helps you manage AI training data.

1.3 The History and Evolution of llms.txt#

The concept of llms.txt emerged in response to growing concerns about "AI scraping"—the practice of harvesting web content to train LLMs. In 2023, as public awareness of AI’s reliance on web data grew, content creators, journalists, and academics began advocating for tools to opt out of LLM training.

While no single organization "owns" llms.txt, it is being shaped by community-driven efforts. For example:

• In 2023, researchers at the University of California, Berkeley proposed a "Robots for LLMs" framework, which laid early groundwork for llms.txt-like standards.
• OpenAI, Anthropic, and other LLM developers have signaled openness to respecting such standards, though implementation is still in its infancy.
• As of 2024, llms.txt is not yet an official web standard (like robots.txt, which is defined by RFC 9309), but it is gaining traction as a de facto best practice.

2. Why llms.txt Matters for Website Owners#

2.1 Protecting Intellectual Property#

Your website’s content—whether blog posts, tutorials, e-books, or original research—is your intellectual property (IP). When LLMs scrape this content without permission, they may use it to generate outputs that compete with your work (e.g., a ChatGPT response that summarizes your tutorial, reducing traffic to your site). llms.txt lets you restrict access to sensitive or high-value content, ensuring your IP is used only as intended.

2.2 Controlling Content Quality and Attribution#

LLMs sometimes generate inaccurate or misleading outputs based on scraped content. If your site is a source for an LLM’s training data, you could be indirectly associated with errors or misinformation. By limiting LLM access, you reduce the risk of your brand being linked to low-quality AI-generated content. Additionally, llms.txt can help encourage LLM providers to seek explicit permission (and attribution) for using your work.

2.3 Mitigating Privacy and Compliance Risks#

If your website includes user-generated content (UGC), personal data, or sensitive information (e.g., customer reviews, forum posts), scraping by LLMs could violate privacy laws like the GDPR or CCPA. llms.txt allows you to block access to these sections, reducing legal exposure. For example, you could disallow LLM scraping of /forum/ or /customer-data/ directories.

2.4 Supporting Ethical AI Development#

By using llms.txt, you’re contributing to a more ethical AI ecosystem. LLM developers are under pressure to ensure their training data is sourced responsibly. By clearly signaling your preferences, you help push the industry toward respecting content creators’ rights—a win for both website owners and the future of AI.

3. Understanding llms.txt Syntax and Directives#

llms.txt uses a simple, human-readable syntax with directives that tell LLMs which parts of your site to crawl or avoid. While the standard is still evolving, most proposals align with the familiar robots.txt format. Below is a breakdown of core directives and how to use them.

3.1 Core Directives Explained#

llms.txt relies on a few key directives to communicate with LLMs. Here’s what each does:

User-agent#

• Purpose: Specifies which LLM(s) the following directives apply to.
• Syntax: User-agent: [LLM Name or * for all]
• Examples:
  • User-agent: ChatGPT (targets OpenAI’s ChatGPT)
  • User-agent: Claude (targets Anthropic’s Claude)
  • User-agent: * (applies to all LLMs)

Disallow#

• Purpose: Tells an LLM not to scrape a specific URL path.
• Syntax: Disallow: [URL Path]
• Notes:
  • Use / to block the entire site.
  • Use specific paths (e.g., /private/) to block directories.
  • Trailing slashes matter: /blog blocks /blog and /blog-post, while /blog/ blocks only the /blog directory.

Allow#

• Purpose: Explicitly permits scraping of a URL path, even if a broader Disallow rule would block it.
• Syntax: Allow: [URL Path]
• Use Case: Useful for allowing access to a subdirectory within a blocked parent directory.

(Proposed) Crawl-delay#

• Purpose: Specifies a delay (in seconds) between LLM scraping requests to prevent server overload.
• Syntax: Crawl-delay: [Number]
• Note: Not yet widely adopted, but may become standard as LLMs scale.

3.2 Common Use Cases and Examples#

To make this concrete, let’s walk through sample llms.txt files for different scenarios.

Example 1: Block All LLMs from Scraping Your Entire Site#

If you want to prevent all LLMs from using your content for training:

User-agent: *  
Disallow: /  

Example 2: Block Specific LLMs, Allow Others#

Suppose you want to block ChatGPT but allow Claude:

# Block ChatGPT  
User-agent: ChatGPT  
Disallow: /  
 
# Allow Claude  
User-agent: Claude  
Allow: /  

Example 3: Block Specific Directories (e.g., Private Content)#

To block LLMs from scraping your /draft-articles/ and /admin/ directories but allow access to the rest of the site:

User-agent: *  
Disallow: /draft-articles/  
Disallow: /admin/  
Allow: /  

Example 4: Allow Only a Specific Section#

If you want to allow LLMs to scrape only your /public-resources/ directory:

User-agent: *  
Disallow: /  
Allow: /public-resources/  

Example 5: Advanced: Block Pages with a Specific URL Pattern#

To block all URLs containing /category/secret/ (e.g., /category/secret/ai/ or /category/secret/research/):

User-agent: *  
Disallow: /category/secret/  

3.3 Best Practices for Writing llms.txt#

• Be Specific: Avoid overly broad rules (e.g., blocking the entire site unless necessary). Target only the content you want to protect.
• Use User-agent: * for Most Cases: Since many LLMs don’t have publicly known names yet, using * ensures your rules apply widely.
• Test Syntax: Even small errors (e.g., missing colons, typos in paths) can break your llms.txt file.
• Document Your Rules: Add comments (lines starting with #) to explain why certain paths are blocked/allowed (e.g., # Block UGC to comply with GDPR).

4. How to Add llms.txt to Your WordPress Site#

WordPress does not natively support llms.txt, but adding it is straightforward. Below are four methods, ranging from beginner-friendly (plugins) to advanced (code snippets). Choose the one that best fits your technical skill level.

Prerequisite: Create Your llms.txt File#

Before adding llms.txt to WordPress, you’ll need to create the file itself. Use a text editor like Notepad (Windows), TextEdit (Mac), or VS Code. Save it as llms.txt (lowercase, no extensions—e.g., not llms.txt.txt).

For example, use the sample from Section 3.2 to start:

# llms.txt for example.com  
User-agent: *  
Disallow: /draft-articles/  
Disallow: /admin/  
Allow: /  

4.1 Method 1: Manually Create and Upload via FTP/SFTP#

Best for: Users comfortable with FTP tools like FileZilla.

Steps:#

1. Connect to Your Server via FTP/SFTP:

   • Open your FTP client (e.g., FileZilla).
   • Enter your server credentials (host, username, password, port). You can find these in your hosting dashboard (e.g., cPanel, SiteGround, Bluehost).

2. Navigate to Your WordPress Root Directory:

   • The root directory is where your wp-config.php and index.php files live. It’s often named public_html, www, or your domain name.

3. Upload Your llms.txt File:

   • Drag and drop the llms.txt file from your computer into the root directory.

4. Verify Upload:

   • Visit https://yourdomain.com/llms.txt in a browser. You should see your file’s content.

4.2 Method 2: Use a WordPress File Manager Plugin#

Best for: Beginners who prefer a dashboard-based approach.

Plugins like File Manager (by mndpsingh287) let you upload and edit files directly from the WordPress admin.

Steps:#

1. Install the File Manager Plugin:

   • Go to Plugins → Add New in your WordPress dashboard.
   • Search for "File Manager" by mndpsingh287, install, and activate it.

2. Access the File Manager:

   • Go to WP File Manager in the left sidebar.

3. Navigate to the Root Directory:

   • The plugin opens to /wp-content/ by default. Click .. (parent directory) until you reach the root (where wp-config.php is located).

4. Upload llms.txt:

   • Click the Upload button (cloud icon) in the toolbar.
   • Select your llms.txt file from your computer and upload it.

5. Verify:

   • Visit https://yourdomain.com/llms.txt to confirm the file is live.

4.3 Method 3: Edit via cPanel File Manager#

Best for: Users with cPanel hosting (common with Bluehost, HostGator, etc.).

Steps:#

1. Log In to cPanel:

   • Go to https://yourdomain.com/cpanel and log in with your hosting credentials.

2. Open File Manager:

   • Under "Files," click File Manager.

3. Navigate to the Root Directory:

   • In the left sidebar, select the root directory (usually public_html).

4. Upload or Create llms.txt:

   • Option 1 (Upload existing file): Click Upload in the toolbar, select your llms.txt file, and confirm.
   • Option 2 (Create new file): Click New File in the toolbar, name it llms.txt, and paste your content into the editor.

5. Verify:

   • Visit https://yourdomain.com/llms.txt to check.

4.4 Method 4: Add Custom Code to Generate llms.txt Dynamically#

Best for: Developers or advanced users who want to automate llms.txt updates (e.g., via WordPress hooks).

This method uses a code snippet to generate llms.txt on the fly, which is useful if you need to dynamically update rules (e.g., block new directories based on user roles).

Steps:#

1. Add Code to Your Theme’s functions.php or a Custom Plugin:

   • Go to Appearance → Theme File Editor (or use a plugin like Code Snippets).
   • Open functions.php (child theme recommended to avoid losing changes on updates).

2. Paste the Following Snippet:

   // Generate llms.txt dynamically  
   add_action('init', 'generate_llms_txt');  
   function generate_llms_txt() {  
       // Check if the request is for llms.txt  
       if ($_SERVER['REQUEST_URI'] === '/llms.txt') {  
           header('Content-Type: text/plain');  
           // Output your llms.txt content here  
           echo "# Dynamically generated llms.txt\n";  
           echo "User-agent: *\n";  
           echo "Disallow: /draft-articles/\n";  
           echo "Disallow: /admin/\n";  
           echo "Allow: /\n";  
           exit; // Stop WordPress from loading further  
       }  
   }  

3. Save Changes:

   • Click "Update File" (if using Theme File Editor) or activate the snippet (if using Code Snippets).

4. Verify:

   • Visit https://yourdomain.com/llms.txt to see the dynamically generated content.

5. Testing Your llms.txt File#

After adding llms.txt, it’s critical to verify it works correctly. A broken or misconfigured file won’t protect your content.

5.1 Verify File Location and Accessibility#

The most common issue is uploading llms.txt to the wrong directory. To check:

• Visit https://yourdomain.com/llms.txt in a browser. If you see a "404 Not Found" error, the file is not in the root directory.
• Double-check your FTP/cPanel upload location—ensure it’s in the same folder as wp-config.php.

5.2 Check Syntax for Errors#

Even small syntax mistakes can render your llms.txt file ineffective. For example:

• Missing colons (e.g., User-agent * instead of User-agent: *).
• Typos in paths (e.g., Disallow: /drafts-articles/ instead of /draft-articles/).
• Using uppercase letters (e.g., LLMS.TXT instead of llms.txt).

Fix: Reopen your llms.txt file and compare it to the examples in Section 3.2. Tools like Robots.txt Validator (while designed for robots.txt) can also flag basic syntax errors.

5.3 Monitor LLM Scraping Activity#

To confirm LLMs are respecting your llms.txt rules, monitor your server logs for LLM-related user agents. Most LLM crawlers identify themselves in the User-Agent header.

How to Check Logs:#

• Via Hosting Dashboard: Many hosts (e.g., SiteGround, WP Engine) let you view access logs in cPanel under "Metrics → Raw Access Logs."
• Via WordPress Plugins: Use plugins like WP Security Audit Log or Query Monitor to track crawler activity.

What to Look For: Entries with user agents like ChatGPT-User, ClaudeBot, or generic LLM-related strings. If you see requests to blocked paths (e.g., /draft-articles/), your llms.txt may need adjustment.

6. Common Issues and Troubleshooting#

6.1 llms.txt Not Loading? Check File Location#

Issue: https://yourdomain.com/llms.txt returns a 404 error.
Fix: Ensure the file is in the root directory (not /wp-content/, /themes/, or a subfolder). Use FTP or cPanel to confirm the location.

6.2 Syntax Errors: How to Diagnose and Fix#

Issue: LLMs are still scraping blocked paths.
Fix:

• Use a text editor like VS Code to check for typos.
• Ensure User-agent lines are followed by directives (e.g., Disallow).
• Test with a simple rule first: User-agent: * Disallow: / (blocks the entire site). If this works, gradually add more rules.

6.3 Permissions Issues: Ensuring Read Access#

Issue: llms.txt returns a 403 Forbidden error.
Fix: The file permissions may be too restrictive. Use FTP or cPanel to set permissions to 644 (readable by everyone, writable by the owner).

6.4 LLM Providers Not Respecting llms.txt: What to Do#

Issue: Some LLMs ignore your llms.txt rules.
Fix:

• Contact the LLM Provider: Many have forms to report scraping issues (e.g., OpenAI’s Content Removal Request).
• Use Technical Measures: Block LLM IP ranges (via .htaccess or a firewall plugin like Wordfence) if scraping persists.
• Advocate for Standards: Support efforts to make llms.txt an official web standard (e.g., via the W3C).

7. Best Practices for llms.txt in WordPress#

7.1 Keep It Simple and Specific#

Avoid overcomplicating your llms.txt file. Focus on blocking only what’s necessary (e.g., sensitive content, UGC) and allow the rest. Over-blocking can limit potential partnerships with ethical LLM providers.

7.2 Regularly Update Your llms.txt File#

As your site grows, you may add new directories (e.g., /new-private-section/) or remove old ones. Review your llms.txt quarterly to ensure it reflects your current content structure.

7.3 Combine with robots.txt for Holistic Control#

llms.txt and robots.txt serve different purposes—use both! For example:

• robots.txt blocks search engines from indexing /admin/.
• llms.txt blocks LLMs from scraping /admin/.

This ensures full protection across search and AI systems.

7.4 Document Your llms.txt Policies#

Add comments to your llms.txt file explaining your rules (e.g., # Block /forum/ to protect user privacy). This helps you and your team remember why certain paths are restricted.

8. Conclusion#

llms.txt is a powerful tool for website owners to take control of their content in the age of AI. By defining clear rules for LLM scraping, you protect your intellectual property, mitigate legal risks, and support ethical AI development.

Adding llms.txt to WordPress is simple, whether you use FTP, a plugin, or custom code. By following the steps in this guide, you can ensure your site is protected today—and ready for the evolving LLM landscape tomorrow.

Remember: llms.txt is not a silver bullet (LLM compliance is still voluntary), but it’s a critical first step. As the AI industry matures, respecting llms.txt will likely become standard practice. By acting now, you position yourself as a proactive, responsible content creator.

9. References#

• Robots.txt Standard (RFC 9309)
• OpenAI’s Content Policy
• Anthropic’s Responsible Use Guide
• University of California, Berkeley: "Robots for LLMs" Proposal
• WordPress.org: Managing Files via FTP
• TechnicalSEO.com: Robots.txt Validator