If you wasn’t aware, WordPress auto-generates its own robots.txt file, and that there is a filter that you can hook into to allow you to add your own rules to the file.
The filter in question is robots_txt filter. Using this information I hand crafted this handy little snippet:
/**
* Filter function used to disallow pdfs from robots.txt
*
* @param string $output The output of the robots.txt file
* @return string $public If the site is public facing
*/
function disallow_pdf_index( $output, $public ) {
if ( '1' === $public ) {
$output .= "Disallow: *.pdf\n";
}
return $output;
}
add_filter( 'robots_txt', 'disallow_pdf_index', 0, 2 );So there you go, a short, but handy little blog post detailing how you too can add entries to your robots.txt file programatically using WordPress.
