For website owners
If you've seen OpenCourseSpaceBot in your logs: it collects course titles, short descriptions and links so learners can find your free courses. Here's what it does and how to control it.
How the crawler behaves
- It identifies itself as
OpenCourseSpaceBot/0.1 (+https://opencourse.space/about/bot/). - It obeys
robots.txt, includingCrawl-delay. - It makes at most one request per second to a site by default, one at a time.
- It uses conditional requests (
ETag,If-Modified-Since) so unchanged pages cost you almost nothing. - It visits each source daily, weekly or monthly, never continuously.
- It never tries to get past logins, paywalls, CAPTCHAs or bot-protection challenges.
- It stores metadata and links only: never your videos, documents or other course content.
Limiting or stopping it
To block it entirely, add this to your robots.txt:
User-agent: OpenCourseSpaceBot Disallow: /
You can also slow it with Crawl-delay, or email us at hello@opencourse.spaceand we'll change or remove your source by hand. We'd much rather use an official feed or API if you have one: just tell us where it is.
Why list your courses?
Every listing links straight to your site and credits your institution. We don't host your content, show ads, or put anything between learners and you.