Robots.txt Tester Library

A reusable JavaScript library for testing robots.txt rules against specific URLs.

Features

• Test URLs against robots.txt rules
• Highlight syntax for robots.txt content
• Validate robots.txt syntax
• Check for errors and potential issues
• Check specific user-agents & paths

Installation

npm install

Development

The project uses Webpack for JavaScript bundling and Gulp for additional build tasks:

• npm run dev - Start development mode with watch
• npm run build - Create production build with bundling and minification
• npm run styles - Build SCSS files
• npm run watch:styles - Watch SCSS files for changes

Build Output

The build process generates the following in dist:

• rtt-library.js - UMD module exporting the default component (includes bundled highlight.js core + plugins).
• main.css - Compiled and minified styles scoped to #rtt-tool.

import RttTool from 'rtt-library';
import { createRoot } from '@wordpress/element';
import 'rtt-library/dist/main.css';
// or
// import 'rtt-library/main.css';

const root = createRoot(document.getElementById('rank-math-admin-rtt'));
root.render(
	<RttTool
		ajaxUrl={window.rankMath.robotsTxtData.ajaxUrl}
		ajaxNonce={window.rankMath.robotsTxtData.nonce}
		siteUrl={window.rankMath.robotsTxtData.siteUrl}
	/>
);

Notes

• CSS import: you can use either rtt-library/dist/main.css or rtt-library/main.css.
• Peer deps/externals: jQuery and the WordPress element package are expected to be provided by the host app.
  • In the browser they should exist as globals: window.jQuery and window.wp.element.
  • In a bundler, ensure jquery and @wordpress/element are installed and resolved normally.

Configuration

You can configure the library by passing options to the RttTool component. The following options are available:

• ajaxUrl: The URL for AJAX requests
• ajaxNonce: The nonce for security
• siteUrl: The initial site URL
• robotsTxtContent: The initial content of the robots.txt file
• editorDisabled: Whether the editor is disabled (for read-only mode)
• isLocal: Local mode disables AJAX requests & prevents changing the site URL (default is false)