spiderbar (0.2.2)

0 users

Parse and Test Robots Exclusion Protocol Files and Rules.


The 'Robots Exclusion Protocol' documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' C++ library for processing these 'robots.txt' files.

Maintainer: Bob Rudis
Author(s): Bob Rudis (bob@rud.is) [aut, cre], SEOmoz, Inc [aut]

License: MIT + file LICENSE

Uses: Rcpp, testthat, covr, robotstxt

Released 9 months ago.

2 previous versions



  (0 votes)


  (0 votes)

Log in to vote.


No one has written a review of spiderbar yet. Want to be the first? Write one now.

Related packages: Rserve, XML, httpRequest, rjson, RCurl, OAIHarvester, RgoogleMaps, sendmailR, twitteR, scrapeR, RJSONIO, imguR, googleVis, factualR, ROAuth, Rook, osmar, whisker, FastRWeb, ggmap(20 best matches, based on common tags.)

Search for spiderbar on google, google scholar, r-help, r-devel.

Visit spiderbar on R Graphical Manual.