What Is This?

This is an HTML+RDFa representation of metadata describing this Web-addressable resource.

Why Is This Important?

The property (attributes and values pairs) links on this page unveil a different kind of link, one which enables the following on HTTP networks such as the Web:

  1. Unambiguous identity for entities (aka. strong identifiers)
  2. Implicit binding of an entity and its metadata via strong identifiers
  3. Multiple metadata representations that enable a variety of presentations
  4. High precision Search and Find queries that simply use the metadata documents (by referencing entity URIs) as the query's Data Source Name

How Do I Discover Alternative Metadata Representations?

This document exposes metadata in the following formats: (X)HTML+RDFa, Turtle, N3, RDF/JSON, or RDF/XML. In the most basic form, you can simply view the (X)HTML source markup of this page, and go directly to the <head/> section which contains a <link/> tag with relationship and type properties for each format.

In addition, you can also explicitly request a desired metadata representation for a given resource via HTTP GET requests that use the entity's strong identifier as the call target.

How Can I Expose My Web Resources In This Manner?

Simply include the following in the <head/> section of your (static or dynamically generated) (X)HTML page:

<link rel="alternate" title="My Data in RDF Linked Data form"
type="application/rdf+xml"
href="http://linkeddata.uriburner.com/about/id/<this-page-URL>/>"

How Is This Related To The Linked Data Meme?

As stated above, the links in this page expose strong identifiers for its primary topic, secondary topics, attributes, and some values. These links, via implicit association, act as conduits to their metadata-bearing documents, in a variety formats.

[OpenLink Software]

About: robots.txt - Wikipedia

An Entity of Type : Thing, from Data Source : https://en.wikipedia.org/wiki/Robots_exclusion_standard, within Data Space : dev.restore.ovi.cnr.it:8890

  • References
  • Referenced By
content

og:image
  • https://upload.wikimedia.org/wikipedia/commons/thumb/1/16/Robots_txt.svg/800px-Robots_txt.svg.png
  • https://upload.wikimedia.org/wikipedia/commons/thumb/1/16/Robots_txt.svg/640px-Robots_txt.svg.png
  • https://upload.wikimedia.org/wikipedia/commons/thumb/1/16/Robots_txt.svg/1200px-Robots_txt.svg.png
og:image:height
  • 260
  • 325
  • 488
og:image:width
  • 800
  • 1200
  • 640
og:title
  • robots.txt - Wikipedia
og:type
  • website
Title
  • robots.txt - Wikipedia
links to
  • https://en.wikipedia.org/wiki/Google
  • https://en.wikipedia.org/wiki/Larry_Page
  • https://en.wikipedia.org/wiki/Sergey_Brin
  • https://en.wikipedia.org/wiki/Google_Search_Console
  • https://en.wikipedia.org/wiki/Server_(computing)
  • https://en.wikipedia.org/wiki/The_Verge
  • https://en.wikipedia.org/wiki/Sitemaps
  • https://en.wikipedia.org/wiki/Googlebot
  • https://en.wikipedia.org/wiki/Portal:Internet
  • https://en.wikipedia.org/wiki/Wikipedia:Protection_policy#pending
  • https://en.wikipedia.org/wiki/Wikipedia:Pending_changes
  • https://en.wikipedia.org/wiki/Category:Wikipedia_pending_changes_protected_pages
  • https://en.wikipedia.org/wiki/National_Digital_Library_Program
  • https://en.wikipedia.org/wiki/Filename
  • https://en.wikipedia.org/wiki/Terminator_(character)
  • https://en.wikipedia.org/wiki/Facebook_Paper
  • https://en.wikipedia.org/wiki/Jason_Scott
  • https://en.wikipedia.org/wiki/National...frastructure_and_Preservation_Program
  • https://en.wikipedia.org/wiki/Category:Websites
  • https://wikimediafoundation.org/
  • https://donate.wikimedia.org/?wmf_sour..._campaign=en.wikipedia.org&uselang=en
  • https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Terms_of_Use
  • https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Privacy_policy
  • https://foundation.wikimedia.org/wiki/...uage/Policy:Universal_Code_of_Conduct
  • https://developer.wikimedia.org
  • https://stats.wikimedia.org/#/en.wikipedia.org
  • https://foundation.wikimedia.org/wiki/Special:MyLanguage/Policy:Cookie_statement
  • https://www.mediawiki.org/
  • https://en.wikipedia.org/wiki/ISBN_(identifier)
  • https://en.wikipedia.org/wiki/Doi_(identifier)
  • https://en.wikipedia.org/wiki/Category...escription_is_different_from_Wikidata
  • https://en.wikipedia.org/wiki/The_New_York_Times
  • https://en.wikipedia.org/wiki/BBC
  • https://en.wikipedia.org/wiki/Medium_(website)
  • https://en.wikipedia.org/wiki/All_Things_Considered
  • https://en.wikipedia.org/wiki/Request_for_Comments
  • https://en.wikipedia.org/wiki/OpenAI
  • https://en.wikipedia.org/wiki/Digital_Trends
  • https://en.wikipedia.org/wiki/Robots.txt
  • https://en.wikipedia.org/wiki/Web_crawler
  • https://en.wikipedia.org/wiki/De_facto_standard
  • https://www.digitaltrends.com/computin...net-archive-robots-txt/#ixzz4gQYOqpUi
  • https://en.wikipedia.org/wiki/Main_Page
  • https://en.wikipedia.org/wiki/Wikipedia:Contents
  • https://en.wikipedia.org/wiki/Portal:Current_events
  • https://en.wikipedia.org/wiki/Special:Random
  • https://en.wikipedia.org/wiki/Wikipedia:About
  • https://en.wikipedia.org/wiki/Wikipedia:Contact_us
  • https://en.wikipedia.org/wiki/Help:Contents
  • https://en.wikipedia.org/wiki/Help:Introduction
  • https://en.wikipedia.org/wiki/Wikipedia:Community_portal
  • https://en.wikipedia.org/wiki/Special:RecentChanges
  • https://en.wikipedia.org/wiki/Wikipedia:File_upload_wizard
  • https://en.wikipedia.org/wiki/Special:Search
  • https://en.wikipedia.org/wiki/Special:MyContributions
  • https://en.wikipedia.org/wiki/Special:MyTalk
  • https://en.wikipedia.org/wiki/Wikipedia:File_Upload_Wizard
  • https://en.wikipedia.org/wiki/Special:SpecialPages
  • https://en.wikipedia.org/wiki/Help:Category
  • https://en.wikipedia.org/wiki/Category:Articles_with_short_description
  • https://en.wikipedia.org/wiki/Wikipedi...-ShareAlike_4.0_International_License
  • https://en.wikipedia.org/wiki/Wikipedia:General_disclaimer
  • https://en.wikipedia.org/wiki/NPR
  • https://en.wikipedia.org/wiki/Security_through_obscurity
  • https://en.wikipedia.org/wiki/Perplexity.ai
  • https://en.wikipedia.org/wiki/Noindex
  • https://en.wikipedia.org/wiki/Website
  • https://en.wikipedia.org/wiki/Web_archiving
  • https://en.wikipedia.org/wiki/Charles_Stross
  • https://en.wikipedia.org/wiki/Voluntary_compliance
  • https://en.wikipedia.org/wiki/Generative_artificial_intelligence
  • https://en.wikipedia.org/wiki/Internet_Archive
  • https://en.wikipedia.org/wiki/GitHub
  • https://en.wikipedia.org/wiki/National_Institute_of_Standards_and_Technology
  • https://en.wikipedia.org/wiki/Lycos
  • https://en.wikipedia.org/wiki/Denial-of-service_attack
  • https://en.wikipedia.org/wiki/Internet_Engineering_Task_Force
  • https://en.wikipedia.org/wiki/404_Media
  • https://en.wikipedia.org/wiki/Internet_bot
  • https://www.wikimedia.org/
  • https://en.wikipedia.org/wiki/Web_developer
  • https://en.wikipedia.org/robots.txt
  • https://en.wikipedia.org/wiki/.htaccess
  • https://ru.wikipedia.org/wiki/%D0%A1%D...0%D0%BE%D0%B1%D0%BE%D1%82%D0%BE%D0%B2
  • https://blog.archive.org/2017/04/17/ro...ines-dont-work-well-for-web-archives/
  • https://en.wikipedia.org/wiki/EBay_v._Bidder%27s_Edge
  • https://en.wikipedia.org/wiki/Web_robot
  • https://en.wikipedia.org/wiki/Sitemap
  • https://en.wikipedia.org/wiki/Same_origin_policy
  • https://en.wikipedia.org/wiki/Perma.cc
  • https://en.wikipedia.org/wiki/Archive_Team
  • https://en.wikipedia.org/wiki/BotSeer
  • https://en.wikipedia.org/wiki/Anthropic
  • https://en.wikipedia.org/wiki/AltaVista
  • https://en.wikipedia.org/wiki/Nofollow
  • https://en.wikipedia.org/w/index.php?t...ecial:Log&type=review&page=Robots.txt
  • https://robotstxt.org
  • https://datatracker.ietf.org/doc/html/rfc9309
  • http://www.greenhills.co.uk/historical.html
  • https://web.archive.org/web/2017040315.../www.greenhills.co.uk/historical.html
  • http://www94.web.cern.ch/WWW94/PapersWWW94/fielding.ps
  • https://web.archive.org/web/2013092709...cern.ch/WWW94/PapersWWW94/fielding.ps
  • http://www.robotstxt.org/orig.html#status
  • https://web.archive.org/web/2014011209...://www.robotstxt.org/orig.html#status
  • https://web.archive.org/web/2013102920...//inkdroid.org/tmp/www-talk/4113.html
  • http://inkdroid.org/tmp/www-talk/4113.html
  • http://www.antipope.org/charlie/blog-s...how_i_got_here_in_the_end_part_3.html
  • https://web.archive.org/web/2013112522...how_i_got_here_in_the_end_part_3.html
  • https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders
  • http://searchengineland.com/robots-txt...-years-blocking-search-engines-195479
  • https://web.archive.org/web/2015090700...-years-blocking-search-engines-195479
  • https://webmasters.googleblog.com/2019/07/rep-id.html
  • https://web.archive.org/web/2019071006...rs.googleblog.com/2019/07/rep-id.html
  • https://doi.org/10.17487%2FRFC9309
  • https://www.youtube.com/watch?v=KBdEwpRQRD0#t=196s
  • https://web.archive.org/web/2014010622...outube.com/watch?v=KBdEwpRQRD0#t=196s
  • http://about.ask.com/docs/about/webmasters.shtml
  • https://web.archive.org/web/2013012713...t.ask.com/docs/about/webmasters.shtml
  • https://web.archive.org/web/20121213134546/http://search.aol.com/aol/about
  • http://search.aol.com/aol/about
  • http://www.baidu.com/search/spider_english.html
  • https://web.archive.org/web/2013080613....baidu.com/search/spider_english.html
  • https://blogs.bing.com/webmaster/2008/...ther-to-provide-better-documentation/
  • https://web.archive.org/web/2014081802...ther-to-provide-better-documentation/
  • https://duckduckgo.com/duckduckbot
  • https://web.archive.org/web/20170216043103/https://duckduckgo.com/duckduckbot
  • https://kagi.com/bot
  • https://web.archive.org/web/20240412192855/https://kagi.com/bot
  • https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
  • https://web.archive.org/web/2013011521...s/control-crawl-index/docs/robots_txt
  • http://help.yahoo.com/kb/index?page=co...ale=en_US&id=SLN2217&impressions=true
  • https://web.archive.org/web/2013012103...ale=en_US&id=SLN2217&impressions=true
  • http://help.yandex.com/webmaster/?id=1113851
  • https://web.archive.org/web/2013012504...help.yandex.com/webmaster/?id=1113851
  • https://wiki.archiveteam.org/index.php/ArchiveBot#Bad_behavior
  • https://web.archive.org/web/2022101003...org/index.php/ArchiveBot#Bad_behavior
  • http://www.archiveteam.org/index.php?title=Robots.txt
  • https://web.archive.org/web/2017021804...veteam.org/index.php?title=Robots.txt
  • https://web.archive.org/web/2018120413...ines-dont-work-well-for-web-archives/
  • https://web.archive.org/web/2017051613...net-archive-robots-txt/#ixzz4gQYOqpUi
  • https://www.404media.co/websites-are-b...se-ai-companies-keep-making-new-ones/
  • https://support.google.com/webmasters/answer/6062608
  • https://web.archive.org/web/2015081401....google.com/webmasters/answer/6062608
  • https://www.theregister.co.uk/2015/05/19/robotstxt/
  • https://web.archive.org/web/2015082106...eregister.co.uk/2015/05/19/robotstxt/
  • http://csrc.nist.gov/publications/nistpubs/800-123/SP800-123.pdf
  • https://doi.org/10.6028%2FNIST.SP.800-123
  • https://web.archive.org/web/2011100811...ations/nistpubs/800-123/SP800-123.pdf
  • https://books.google.com/books?id=RjVjgPQsKogC&pg=PA92
  • https://web.archive.org/web/2016040119...gle.com/books?id=RjVjgPQsKogC&pg=PA92
  • http://www.user-agents.org/
  • https://web.archive.org/web/20140107154205/http://user-agents.org/
  • https://httpd.apache.org/docs/2.2/howto/access.html
  • https://web.archive.org/web/2013122911...apache.org/docs/2.2/howto/access.html
  • https://www.iis.net/configreference/sy...teringrules/filteringrule/denystrings
  • https://web.archive.org/web/2014010111...teringrules/filteringrule/denystrings
  • https://www.google.com/humans.txt
  • https://web.archive.org/web/20170124121422/https://www.google.com/humans.txt
  • https://github.com/humans.txt
  • https://web.archive.org/web/20160530160942/https://github.com/humans.txt
  • https://slate.com/technology/2014/07/a-killer-robots-txt-google-easter-egg.html
  • https://web.archive.org/web/2018111810...ler-robots-txt-google-easter-egg.html
  • https://web.archive.org/web/2018011016...ps://www.google.com/killer-robots.txt
  • https://www.google.com/killer-robots.txt
  • https://blogs.bing.com/webmaster/2012/...t-to-crawl-that-is-bingbots-question/
  • https://web.archive.org/web/2016020314...t-to-crawl-that-is-bingbots-question/
  • https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
  • https://support.google.com/webmasters/answer/48620?hl=en
  • https://web.archive.org/web/2018111820...gle.com/webmasters/answer/48620?hl=en
  • https://web.archive.org/web/2009030506...-can-now-auto-discover-with-sitemaps/
  • http://ysearchblog.com/2007/04/11/webm...-can-now-auto-discover-with-sitemaps/
  • https://developers.google.com/search/reference/robots_txt?hl=en
  • https://web.archive.org/web/2019110219...com/search/reference/robots_txt?hl=en
  • https://developers.google.com/webmaste...trol-crawl-index/docs/robots_meta_tag
  • https://web.archive.org/web/2013080802...trol-crawl-index/docs/robots_meta_tag
  • https://web.archive.org/web/2022101710...s/crawling-indexing/robots/robots_txt
  • https://www.npr.org/2024/07/05/nx-s1-5...ligence-web-crawlers-are-running-amok
  • https://web.archive.org/web/2024070606...ligence-web-crawlers-are-running-amok
  • https://www.robotstxt.org
  • https://ar.wikipedia.org/wiki/%D9%85%D...1%D9%88%D8%A8%D9%88%D8%AA%D8%A7%D8%AA
  • https://az.wikipedia.org/wiki/Robots.txt
  • https://bar.wikipedia.org/wiki/Robots_Exclusion_Standard
  • https://ca.wikipedia.org/wiki/Protocol_d%27exclusi%C3%B3_de_robots
  • https://cs.wikipedia.org/wiki/Robots.txt
  • https://da.wikipedia.org/wiki/Robot_Exclusion_Standard
  • https://de.wikipedia.org/wiki/Robots_Exclusion_Standard
  • https://es.wikipedia.org/wiki/Est%C3%A1ndar_de_exclusi%C3%B3n_de_robots
  • https://fa.wikipedia.org/wiki/%D8%A7%D...8%A8%D8%A7%D8%AA%E2%80%8C%D9%87%D8%A7
  • https://fr.wikipedia.org/wiki/Protocole_d%27exclusion_des_robots
  • https://gl.wikipedia.org/wiki/Robots.txt
  • https://ko.wikipedia.org/wiki/%EB%A1%9...%EB%B0%B0%EC%A0%9C_%ED%91%9C%EC%A4%80
  • https://id.wikipedia.org/wiki/Robots.txt
  • https://it.wikipedia.org/wiki/Protocollo_di_esclusione_robot
  • https://he.wikipedia.org/wiki/%D7%A4%D...8%D7%95%D7%91%D7%95%D7%98%D7%99%D7%9D
  • https://nl.wikipedia.org/wiki/Robots_Exclusion_Protocol
  • https://ja.wikipedia.org/wiki/Robots_Exclusion_Standard
  • https://pl.wikipedia.org/wiki/Robots_Exclusion_Protocol
  • https://pt.wikipedia.org/wiki/Protocolo_de_exclus%C3%A3o_de_rob%C3%B4s
  • https://simple.wikipedia.org/wiki/Robots_exclusion_standard
  • https://sr.wikipedia.org/wiki/Standard_isklju%C4%8Denja_robota
  • https://fi.wikipedia.org/wiki/Robots.txt
  • https://sv.wikipedia.org/wiki/Robots_Exclusion_Standard
  • https://tr.wikipedia.org/wiki/Robot_engelleme_standard%C4%B1
  • https://uk.wikipedia.org/wiki/Robots.txt
  • https://zh.wikipedia.org/wiki/Robots.txt
  • https://en.wikipedia.org/wiki/WebCrawler
  • https://en.wikipedia.org/wiki/Focused_crawler
  • https://en.wikipedia.org/wiki/Spider_trap
  • https://en.wikipedia.org/wiki/Distributed_web_crawling
  • https://en.wikipedia.org/wiki/Cloaking
  • https://en.wikipedia.org/wiki/Category:Search_engine_optimization
  • https://en.wikipedia.org/wiki/Category:Web_scraping
  • https://en.wikipedia.org/wiki/Martijn_Koster
  • https://en.wikipedia.org/wiki/Nexor
  • https://en.wikipedia.org/wiki/User-agent
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#bodyContent
  • https://en.wikipedia.org/w/index.php?t...ial:CreateAccount&returnto=Robots.txt
  • https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Robots.txt
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#History
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Standard
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Compliance
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Search_engines
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Archival_sites
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Artificial_intelligence
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Security
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Alternatives
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Examples
  • https://en.wikipedia.org/wiki/Robots_e...rd#The_use_of_the_wildcard_*_in_rules
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Nonstandard_extensions
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Sitemap
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Universal_"*"_match
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Meta_tags_and_headers
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#A_"noindex"_meta_tag
  • https://en.wikipedia.org/wiki/Robots_e...dard#A_"noindex"_HTTP_response_header
  • https://en.wikipedia.org/wiki/Robots_e...ard#Maximum_size_of_a_robots.txt_file
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#See_also
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#References
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#Further_reading
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#External_links
  • https://www.wikidata.org/wiki/Special:EntityPage/Q80776#sitelinks-wikipedia
  • https://en.wikipedia.org/wiki/Talk:Robots.txt
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&action=edit
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&action=history
  • https://en.wikipedia.org/wiki/Special:WhatLinksHere/Robots.txt
  • https://en.wikipedia.org/wiki/Special:RecentChangesLinked/Robots.txt
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&oldid=1286725209
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&action=info
  • https://en.wikipedia.org/w/index.php?t...1286725209&wpFormIdentifier=titleform
  • https://en.wikipedia.org/w/index.php?t...Fen.wikipedia.org%2Fwiki%2FRobots.txt
  • https://en.wikipedia.org/w/index.php?t...Fen.wikipedia.org%2Fwiki%2FRobots.txt
  • https://en.wikipedia.org/w/index.php?t...obots.txt&action=show-download-screen
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&printable=yes
  • https://commons.wikimedia.org/wiki/Category:Robots.txt
  • https://www.wikidata.org/wiki/Special:EntityPage/Q80776
  • https://en.wikipedia.org/w/index.php?title=Robots_exclusion_standard&redirect=no
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&action=edit§ion=1
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-1
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-2
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-:0-3
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-4
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-5
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-Verge-6
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-sear_Robo-7
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-8
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-rfc9309-9
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&action=edit§ion=2
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-10
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&action=edit§ion=3
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&action=edit§ion=4
  • https://en.wikipedia.org/wiki/Robots_e..._standard#cite_note-ask-webmasters-11
  • https://en.wikipedia.org/wiki/Robots_e...tandard#cite_note-about-aol-search-12
  • https://en.wikipedia.org/wiki/Robots_e...on_standard#cite_note-baidu-spider-13
  • https://en.wikipedia.org/wiki/Robots_e...tandard#cite_note-bing-blog-robots-14
  • https://en.wikipedia.org/wiki/Robots_e..._standard#cite_note-duckduckgo-bot-15
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-KagiBot-16
  • https://en.wikipedia.org/wiki/Robots_e...d#cite_note-google-webmasters-spec-17
  • https://en.wikipedia.org/wiki/Robots_e...ard#cite_note-yahoo-search-is-bing-18
  • https://en.wikipedia.org/wiki/Robots_e...n_standard#cite_note-yandex-robots-19
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&action=edit§ion=5
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-20
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-21
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-22
  • https://en.wikipedia.org/wiki/Robots_e...tandard#cite_note-Internet_Archive-23
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&action=edit§ion=6
  • https://en.wikipedia.org/wiki/Blocklist
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-24
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&action=edit§ion=7
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-25
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-26
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-27
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-28
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&action=edit§ion=8
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-29
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-30
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-31
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard#cite_note-32
opl:providedBy
  • http://www.stackoverflow.com/#this
xhv:alternate
  • http://dev.restore.ovi.cnr.it:8890/abo...title=Special:RecentChanges&feed=atom
  • https://en.wikipedia.org/w/index.php?title=Special:RecentChanges&feed=atom
  • https://en.m.wikipedia.org/wiki/Robots.txt
  • http://dev.restore.ovi.cnr.it:8890/abo...ndex.php?title=Robots.txt&action=edit
  • http://dev.restore.ovi.cnr.it:8890/abo...ps/en.m.wikipedia.org/wiki/Robots.txt
  • https://en.wikipedia.org/w/index.php?title=Robots.txt&action=edit
xhv:icon
xhv:license
  • Deed - Attribution-ShareAlike 4.0 International - Creative Commons
xhv:stylesheet
  • http://dev.restore.ovi.cnr.it:8890/abo...e.styles&only=styles&skin=vector-2022
  • http://dev.restore.ovi.cnr.it:8890/abo...ent.init&only=styles&skin=vector-2022
mw:PageProp/toc
primary topic
  • https://en.wikipedia.org/wiki/Robots_exclusion_standard
Alternative Linked Data Views: Facets | iSPARQL | ODE     Raw Linked Data formats: CXML | CSV | RDF ( N-Triples N3/Turtle JSON XML ) | OData ( Atom JSON ) | Microdata ( JSON HTML) | JSON-LD
This material is Open Knowledge   W3C Semantic Web Technology     This material is Open Knowledge Creative Commons License Valid XHTML + RDFa
This work is licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License.
OpenLink Virtuoso version 07.20.3231, on Linux (x86_64-generic_glibc25-linux-gnu), Single Edition