Skip to main content
POST
/
{guru_slug}
/
crawl
/
start
/
Start Website Crawl
curl --request POST \
  --url https://api.gurubase.io/api/v1/{guru_slug}/crawl/start/ \
  --header 'Content-Type: application/json' \
  --header 'x-api-key: <api-key>' \
  --data '{
  "url": "<string>"
}'
{
    "id": 211,
    "url": "https://getanteon.com/",
    "status": "RUNNING",
    "guru_type": "anteon",
    "discovered_urls": [],
    "start_time": "2025-02-21T10:25:22.710211Z",
    "end_time": null,
    "link_limit": 1500
}
Starts an asynchronous website crawl operation to find all the sub URLs of the provided root URL. You can check the status of created crawl operation using the Get Crawl Status endpoint or stop it using the Stop Crawl endpoint. Crawling does not mean that the discovered URLs are indexed immediately. You need to manually add the discovered URLs as a data source by passing them to Create Data Source endpoint.
Crawls are rate limited to 1 concurrent operation per Guru type. Subsequent requests will fail if a crawl is already running.

​
Path Parameters

​
guru_slug
string
required
The slug of the Guru type to associate the crawled content with

​
Headers

​
x-api-key
string
required
Your API key for authentication. You can obtain your API key from the Gurubase.io dashboard.

​
Body Parameters

​
url
string
required
The root URL to start crawling from. Must include http:// or https:// protocol.

​
Response

​
id
string
Unique identifier for the crawl operation
​
url
string
The root URL to be crawled. The crawler will start from this URL and follow all links that begin with it. For example, if the URL is https://example.com/a/b/c, the crawler will extract all links that start with https://example.com/a/b/c.
​
status
string
Current status of the crawl operation
​
guru_type
string
The Guru type that the crawl was initiated for
​
discovered_urls
list
List of URLs discovered during crawling
​
start_time
string
Timestamp when crawl started (ISO 8601 format)
​
end_time
string
Timestamp when crawl ended (ISO 8601 format)
{
    "id": 211,
    "url": "https://getanteon.com/",
    "status": "RUNNING",
    "guru_type": "anteon",
    "discovered_urls": [],
    "start_time": "2025-02-21T10:25:22.710211Z",
    "end_time": null,
    "link_limit": 1500
}