# Robots Txt
**Source:** https://glossary.keenfunnel.com/terms/robots-txt
**Language:** German

---

## Technische Erklärung

Rules identify user agents and paths that are allowed or disallowed for crawling and may declare sitemap locations. The file is publicly accessible, path matching is sensitive to syntax, and support for directives varies among crawlers.

## Geschäftliche Relevanz

A correct robots.txt helps manage crawler access and protect crawl capacity from low-value paths while pointing crawlers to sitemaps.

## Implementierungsbeispiel

A site allows public pages, disallows an internal API path, and declares the canonical XML sitemap while keeping CSS and JavaScript required for rendering accessible.

## Einschränkungen und häufige Missverständnisse

Robots.txt is not an access-control or confidentiality mechanism and does not itself prevent indexing. Blocked URLs may still appear without content, and sensitive resources require authentication or removal.

## Themen

Suchoptimierung Systemarchitektur

## Quellen

Google Search Central — robots.txt — RFC 9309 — Robots Exclusion Protocol — https://www.rfc-editor.org/rfc/rfc9309

## Besprechen Sie Ihre Systeme

Benötigen Sie Hilfe bei der Implementierung oder Bewertung dieses Konzepts? Keenfunnel entwirft vernetzte KI-, Automatisierungs- und Datensysteme.

Discovery-Session buchen