# All robots will spider the domain User-agent: * Disallow: # Disallow directory templates User-agent: * Disallow: templates # Disallow directory images User-agent: * Disallow: images # Disallow directory css User-agent: * Disallow: css # Disallow directory cgi-bin User-agent: * Disallow: cgi-bin # Disallow directory classes User-agent: * Disallow: classes