ByteDance
Bytespider
A crawler identified as Bytespider in request user-agents. Public documentation of its policy is thin.
- Who runs it
- ByteDance
- robots.txt token
- Bytespider
- User-agent
- See the operator's documentation for the current full string.
Requests observed on the web identify this crawler with the token Bytespider, sometimes with a link to ByteDance's webmaster host. We are not linking a policy page we cannot treat as a stable official spec. If you name it in robots.txt, the token is Bytespider. We do not claim the crawler honors that file.
Bytespider is the name ByteDance's crawler uses in its user-agent. English documentation from the operator is sparse, so this page does not invent a purpose, a training claim, or a promise that robots.txt is respected.
If you still want a robots.txt group, name the token Bytespider. That is a request in your file. It is not proof the crawler will follow it. A firewall rule is the hard stop, and it needs the user-agent you actually see in your logs.
The example rules lower on this page are examples of the robots.txt format. They are not snippets published by ByteDance.
How robots.txt groups work (Google's intro, for the file format only)
Example robots.txt rules
Examples for your file, using the documented token. They are not a copy of the operator's own robots.txt.
Allow
User-agent: Bytespider Allow: /
Block
User-agent: Bytespider Disallow: /
Related guides
Other crawlers
Questions
- Why is the official link a Google page?
- That link explains the robots.txt format. It is not a ByteDance policy. We would rather link the format than invent a docs URL.