{ "msg": "Request was throttled. Expected available in 56 seconds."}
Starts an asynchronous website crawl operation to find all the sub URLs of the provided root URL.You can check the status of created crawl operation using the Get Crawl Status endpoint or stop it using the Stop Crawl endpoint.Crawling does not mean that the discovered URLs are indexed immediately. You need to manually add the discovered URLs as a data source by passing them to Create Data Source endpoint.
Crawls are rate limited to 1 concurrent operation per Guru type. Subsequent requests will fail if a crawl is already running.
Skipped Paths: The crawler automatically skips non-content paths like /feed/, /rss/, /static/, /assets/, /media/, /wp-admin/, /wp-json/, /_static/, /_sources/, and common file extensions (images, PDFs, CSS, JS, etc.).
The root URL to start crawling from. Must include http:// or https:// protocol.Important: The crawler only discovers URLs that start with this path. For example:
https://example.com/ → crawls entire site
https://example.com/docs/ → only crawls URLs under /docs/
When true, query parameters are stripped from discovered URLs (e.g., ?utm_source=..., ?id=123).
Set to false if you need to crawl paginated content like ?page=1, ?page=2.
The root URL to be crawled. The crawler will start from this URL and follow all links that begin with it. For example, if the URL is https://example.com/a/b/c, the crawler will extract all links that start with https://example.com/a/b/c.
{ "msg": "Request was throttled. Expected available in 56 seconds."}
Was this page helpful?
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"Gurubase","colors":{"primary":"#006BFF","light":"#3389FF","dark":"#006BFF"},"logo":{"light":"https://mintcdn.com/gurubase/0kE9Hd9k_SJDKu8Q/logo/light.svg?fit=max\u0026auto=format\u0026n=0kE9Hd9k_SJDKu8Q\u0026q=85\u0026s=e80be9ea5250513afdd5cac9e9cf40eb","dark":"https://mintcdn.com/gurubase/0kE9Hd9k_SJDKu8Q/logo/dark.svg?fit=max\u0026auto=format\u0026n=0kE9Hd9k_SJDKu8Q\u0026q=85\u0026s=95fad17b670ff55e15b56ec9a86a5e6a"},"favicon":"https://mintcdn.com/gurubase/7wgpr3jQ02zKlT61/favicon.svg?fit=max\u0026auto=format\u0026n=7wgpr3jQ02zKlT61\u0026q=85\u0026s=3c3872b3bfdd7703dfb3c23febfa08dd","api":{"mdx":{"auth":{"method":"key","name":"x-api-key"},"server":"https://api.gurubase.io/api/v1/"}},"navbar":{"links":[],"primary":{"type":"button","label":"Gurubase.io","href":"https://gurubase.io/"}},"footer":{"socials":{"twitter":"https://x.com/gurubaseio","linkedin":"https://www.linkedin.com/company/gurubase"},"links":[{"header":"Legal","items":[{"label":"Privacy Policy","href":"https://gurubase.io/privacy-policy"},{"label":"Cookie Policy","href":"https://gurubase.io/cookie-policy"}]},{"header":"More","items":[{"label":"Siper (Turkish PII Masking)","href":"https://siper.gurubase.ai/"}]}]},"seo":{"metatags":{"og:site_name":"Gurubase Documentation","og:type":"website","og:locale":"en_US","twitter:card":"summary_large_image","twitter:site":"@gurubaseio","robots":"index, follow","author":"Gurubase","keywords":"AI agents, hallucination-resistant AI, RAG, trust score, citations, agentic actions, MCP, MCP server, MCP client, on-premise AI, enterprise AI, knowledge base, documentation"},"indexing":"navigable"},"icons":{"library":"lucide"},"contextual":{"options":["copy","view","chatgpt","claude"],"display":"header"},"styling":{"latex":false},"navigation":{"global":{"anchors":[{"anchor":"Try Gurubase","icon":"sparkles","href":"https://gurubase.io/"}]},"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"gurubase","hasAudioNarration":false,"actualSubdomain":"gurubase","customDomains":["docs.gurubase.ai"],"gitSource":{"type":"github","owner":"gurubase","repo":"gurubase-docs","deployBranch":"master","contentDirectory":"","isPrivate":true,"mountPath":"","label":""},"chroma":{"liveDeploymentHistoryId":"6addf60abcfaf11b9abaf72b","collectionId":"a39040e4-e46a-488c-98eb-b9a1cba646c5","schemaVersion":"v3","latestBuiltDeploymentHistoryId":"6addf60abcfaf11b9abaf72b"},"feedback":{"thumbs":true},"entitlements":{"AI_CHAT":{"status":"DISABLED"}},"buildId":"6addf60abcfaf11b9abaf72b","clientVersion":"0.0.3966","searchFilterCounts":[{"language":"default","tag":"Documentation","count":47},{"language":"default","tag":"API Reference","count":26},{"language":"default","tag":"Guides","count":25},{"language":"default","tag":"Endpoints","count":24},{"language":"default","tag":"Integrations","count":15},{"language":"default","tag":"Bots","count":7},{"language":"default","tag":"Ingestion","count":6},{"language":"default","tag":"Get Started","count":4},{"language":"default","tag":"Advanced","count":3},{"language":"default","tag":"API Documentation","count":2}]},"assistantConfig":{},"hasChatPermissions":false,"hasWebMcpPermissions":false,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-v7-file-tree","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"pause-universal-search-reconcile","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"per-site-search-delta","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-private-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-nav-generations","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"byok-sep-25","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"vite-origin","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-agent-model-picker","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-updated-9-15","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"isAtlas":false,"googleFontsInlined":false},"customCss":[],"page":{"pagePath":"api-reference/endpoints/crawler-start","slug":"/api-reference/endpoints/crawler-start","pageMetadata":{"title":"Start Website Crawl","icon":"play","api":"POST /{guru_slug}/crawl/start/","description":"Initiate a crawl to explore the URLs of a website","href":"/api-reference/endpoints/crawler-start"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Initiate a crawl to explore the URLs of a website\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={a:`a`,code:`code`,li:`li`,p:`p`,strong:`strong`,ul:`ul`,..._provideComponents(),...props.components},{Heading,Note,ParamField,ResponseField}=_components;if(!Heading)_missingMdxReference(`Heading`,true);if(!Note)_missingMdxReference(`Note`,true);if(!ParamField)_missingMdxReference(`ParamField`,true);if(!ResponseField)_missingMdxReference(`ResponseField`,true);return _jsxs(_Fragment,{children:[_jsxs(_components.p,{children:[`Starts an `,_jsx(_components.strong,{children:`asynchronous`}),` website crawl operation to find all the sub URLs of the provided root URL.`]}),`\n`,_jsxs(_components.p,{children:[`You can check the status of created crawl operation using the `,_jsx(_components.a,{href:`/api-reference/endpoints/crawler-status`,children:`Get Crawl Status`}),` endpoint or stop it using the `,_jsx(_components.a,{href:`/api-reference/endpoints/crawler-stop`,children:`Stop Crawl`}),` endpoint.`]}),`\n`,_jsxs(_components.p,{children:[`Crawling does not mean that the discovered URLs are indexed immediately. You need to manually add the discovered URLs as a data source by passing them to `,_jsx(_components.a,{href:`/api-reference/endpoints/create-datasources`,children:`Create Data Source`}),` endpoint.`]}),`\n`,_jsx(Note,{children:_jsx(_components.p,{children:`Crawls are rate limited to 1 concurrent operation per Guru type. Subsequent requests will fail if a crawl is already running.`})}),`\n`,_jsx(Note,{children:_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Skipped Paths`}),`: The crawler automatically skips non-content paths like `,_jsx(_components.code,{children:`/feed/`}),`, `,_jsx(_components.code,{children:`/rss/`}),`, `,_jsx(_components.code,{children:`/static/`}),`, `,_jsx(_components.code,{children:`/assets/`}),`, `,_jsx(_components.code,{children:`/media/`}),`, `,_jsx(_components.code,{children:`/wp-admin/`}),`, `,_jsx(_components.code,{children:`/wp-json/`}),`, `,_jsx(_components.code,{children:`/_static/`}),`, `,_jsx(_components.code,{children:`/_sources/`}),`, and common file extensions (images, PDFs, CSS, JS, etc.).`]})}),`\n`,_jsx(Heading,{level:`2`,id:`path-parameters`,children:`Path Parameters`}),`\n`,_jsx(ParamField,{path:`guru_slug`,type:`string`,required:true,id:`param-guru-slug`,children:_jsx(_components.p,{children:`The slug of the Guru type to associate the crawled content with`})}),`\n`,_jsx(Heading,{level:`2`,id:`headers`,children:`Headers`}),`\n`,_jsx(ParamField,{header:`x-api-key`,type:`string`,required:true,id:`param-x-api-key`,children:_jsxs(_components.p,{children:[`Your API key for authentication. You can obtain your API key from the `,_jsx(_components.a,{href:`https://app.gurubase.io/api-keys`,children:`Gurubase dashboard`}),`.`]})}),`\n`,_jsx(Heading,{level:`3`,id:`body-parameters`,children:`Body Parameters`}),`\n`,_jsxs(ParamField,{body:`url`,type:`string`,required:true,id:`param-url`,children:[_jsx(_components.p,{children:`The root URL to start crawling from. Must include http:// or https:// protocol.`}),_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Important`}),`: The crawler only discovers URLs that start with this path. For example:`]}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`https://example.com/`}),` → crawls entire site`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`https://example.com/docs/`}),` → only crawls URLs under `,_jsx(_components.code,{children:`/docs/`})]}),`\n`]})]}),`\n`,_jsx(ParamField,{body:`ignore_query_params`,type:`boolean`,default:`true`,id:`param-ignore-query-params`,children:_jsxs(_components.p,{children:[`When `,_jsx(_components.code,{children:`true`}),`, query parameters are stripped from discovered URLs (e.g., `,_jsx(_components.code,{children:`?utm_source=...`}),`, `,_jsx(_components.code,{children:`?id=123`}),`).\nSet to `,_jsx(_components.code,{children:`false`}),` if you need to crawl paginated content like `,_jsx(_components.code,{children:`?page=1`}),`, `,_jsx(_components.code,{children:`?page=2`}),`.`]})}),`\n`,_jsx(Heading,{level:`3`,id:`response`,children:`Response`}),`\n`,_jsx(ResponseField,{name:`id`,type:`string`,id:`param-id`,children:_jsx(_components.p,{children:`Unique identifier for the crawl operation`})}),`\n`,_jsx(ResponseField,{name:`url`,type:`string`,id:`param-url-1`,children:_jsxs(_components.p,{children:[`The root URL to be crawled. The crawler will start from this URL and follow all links that begin with it. For example, if the URL is `,_jsx(_components.a,{href:`https://example.com/a/b/c`,children:`https://example.com/a/b/c`}),`, the crawler will extract all links that start with `,_jsx(_components.a,{href:`https://example.com/a/b/c`,children:`https://example.com/a/b/c`}),`.`]})}),`\n`,_jsx(ResponseField,{name:`status`,type:`string`,enum:[`RUNNING`,`COMPLETED`,`FAILED`,`STOPPED`],id:`param-status`,children:_jsx(_components.p,{children:`Current status of the crawl operation`})}),`\n`,_jsx(ResponseField,{name:`guru_type`,type:`string`,id:`param-guru-type`,children:_jsx(_components.p,{children:`The Guru type that the crawl was initiated for`})}),`\n`,_jsx(ResponseField,{name:`discovered_urls`,type:`list`,id:`param-discovered-urls`,children:_jsx(_components.p,{children:`List of URLs discovered during crawling`})}),`\n`,_jsx(ResponseField,{name:`start_time`,type:`string`,id:`param-start-time`,children:_jsx(_components.p,{children:`Timestamp when crawl started (ISO 8601 format)`})}),`\n`,_jsx(ResponseField,{name:`end_time`,type:`string`,id:`param-end-time`,children:_jsx(_components.p,{children:`Timestamp when crawl ended (ISO 8601 format)`})}),`\n`,_jsx(ResponseField,{name:`ignore_query_params`,type:`boolean`,id:`param-ignore-query-params-1`,children:_jsx(_components.p,{children:`Whether query parameters are being stripped from discovered URLs`})})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Start Website Crawl","icon":"play","api":"POST /{guru_slug}/crawl/start/","description":"Initiate a crawl to explore the URLs of a website","href":"/api-reference/endpoints/crawler-start"}}},"mdxSourceWithNoJs":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={a:`a`,code:`code`,li:`li`,p:`p`,pre:`pre`,span:`span`,strong:`strong`,ul:`ul`,..._provideComponents(),...props.components},{CodeBlock,Heading,Note,ParamField,ResponseExample,ResponseField}=_components;if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!Note)_missingMdxReference(`Note`,true);if(!ParamField)_missingMdxReference(`ParamField`,true);if(!ResponseExample)_missingMdxReference(`ResponseExample`,true);if(!ResponseField)_missingMdxReference(`ResponseField`,true);return _jsxs(_Fragment,{children:[_jsxs(_components.p,{children:[`Starts an `,_jsx(_components.strong,{children:`asynchronous`}),` website crawl operation to find all the sub URLs of the provided root URL.`]}),`\n`,_jsxs(_components.p,{children:[`You can check the status of created crawl operation using the `,_jsx(_components.a,{href:`/api-reference/endpoints/crawler-status`,children:`Get Crawl Status`}),` endpoint or stop it using the `,_jsx(_components.a,{href:`/api-reference/endpoints/crawler-stop`,children:`Stop Crawl`}),` endpoint.`]}),`\n`,_jsxs(_components.p,{children:[`Crawling does not mean that the discovered URLs are indexed immediately. You need to manually add the discovered URLs as a data source by passing them to `,_jsx(_components.a,{href:`/api-reference/endpoints/create-datasources`,children:`Create Data Source`}),` endpoint.`]}),`\n`,_jsx(Note,{children:_jsx(_components.p,{children:`Crawls are rate limited to 1 concurrent operation per Guru type. Subsequent requests will fail if a crawl is already running.`})}),`\n`,_jsx(Note,{children:_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Skipped Paths`}),`: The crawler automatically skips non-content paths like `,_jsx(_components.code,{children:`/feed/`}),`, `,_jsx(_components.code,{children:`/rss/`}),`, `,_jsx(_components.code,{children:`/static/`}),`, `,_jsx(_components.code,{children:`/assets/`}),`, `,_jsx(_components.code,{children:`/media/`}),`, `,_jsx(_components.code,{children:`/wp-admin/`}),`, `,_jsx(_components.code,{children:`/wp-json/`}),`, `,_jsx(_components.code,{children:`/_static/`}),`, `,_jsx(_components.code,{children:`/_sources/`}),`, and common file extensions (images, PDFs, CSS, JS, etc.).`]})}),`\n`,_jsx(Heading,{level:`2`,id:`path-parameters`,children:`Path Parameters`}),`\n`,_jsx(ParamField,{path:`guru_slug`,type:`string`,required:true,id:`param-guru-slug`,children:_jsx(_components.p,{children:`The slug of the Guru type to associate the crawled content with`})}),`\n`,_jsx(Heading,{level:`2`,id:`headers`,children:`Headers`}),`\n`,_jsx(ParamField,{header:`x-api-key`,type:`string`,required:true,id:`param-x-api-key`,children:_jsxs(_components.p,{children:[`Your API key for authentication. You can obtain your API key from the `,_jsx(_components.a,{href:`https://app.gurubase.io/api-keys`,children:`Gurubase dashboard`}),`.`]})}),`\n`,_jsx(Heading,{level:`3`,id:`body-parameters`,children:`Body Parameters`}),`\n`,_jsxs(ParamField,{body:`url`,type:`string`,required:true,id:`param-url`,children:[_jsx(_components.p,{children:`The root URL to start crawling from. Must include http:// or https:// protocol.`}),_jsxs(_components.p,{children:[_jsx(_components.strong,{children:`Important`}),`: The crawler only discovers URLs that start with this path. For example:`]}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`https://example.com/`}),` → crawls entire site`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`https://example.com/docs/`}),` → only crawls URLs under `,_jsx(_components.code,{children:`/docs/`})]}),`\n`]})]}),`\n`,_jsx(ParamField,{body:`ignore_query_params`,type:`boolean`,default:`true`,id:`param-ignore-query-params`,children:_jsxs(_components.p,{children:[`When `,_jsx(_components.code,{children:`true`}),`, query parameters are stripped from discovered URLs (e.g., `,_jsx(_components.code,{children:`?utm_source=...`}),`, `,_jsx(_components.code,{children:`?id=123`}),`).\nSet to `,_jsx(_components.code,{children:`false`}),` if you need to crawl paginated content like `,_jsx(_components.code,{children:`?page=1`}),`, `,_jsx(_components.code,{children:`?page=2`}),`.`]})}),`\n`,_jsx(Heading,{level:`3`,id:`response`,children:`Response`}),`\n`,_jsx(ResponseField,{name:`id`,type:`string`,id:`param-id`,children:_jsx(_components.p,{children:`Unique identifier for the crawl operation`})}),`\n`,_jsx(ResponseField,{name:`url`,type:`string`,id:`param-url-1`,children:_jsxs(_components.p,{children:[`The root URL to be crawled. The crawler will start from this URL and follow all links that begin with it. For example, if the URL is `,_jsx(_components.a,{href:`https://example.com/a/b/c`,children:`https://example.com/a/b/c`}),`, the crawler will extract all links that start with `,_jsx(_components.a,{href:`https://example.com/a/b/c`,children:`https://example.com/a/b/c`}),`.`]})}),`\n`,_jsx(ResponseField,{name:`status`,type:`string`,enum:[`RUNNING`,`COMPLETED`,`FAILED`,`STOPPED`],id:`param-status`,children:_jsx(_components.p,{children:`Current status of the crawl operation`})}),`\n`,_jsx(ResponseField,{name:`guru_type`,type:`string`,id:`param-guru-type`,children:_jsx(_components.p,{children:`The Guru type that the crawl was initiated for`})}),`\n`,_jsx(ResponseField,{name:`discovered_urls`,type:`list`,id:`param-discovered-urls`,children:_jsx(_components.p,{children:`List of URLs discovered during crawling`})}),`\n`,_jsx(ResponseField,{name:`start_time`,type:`string`,id:`param-start-time`,children:_jsx(_components.p,{children:`Timestamp when crawl started (ISO 8601 format)`})}),`\n`,_jsx(ResponseField,{name:`end_time`,type:`string`,id:`param-end-time`,children:_jsx(_components.p,{children:`Timestamp when crawl ended (ISO 8601 format)`})}),`\n`,_jsx(ResponseField,{name:`ignore_query_params`,type:`boolean`,id:`param-ignore-query-params-1`,children:_jsx(_components.p,{children:`Whether query parameters are being stripped from discovered URLs`})}),`\n`,_jsxs(ResponseExample,{children:[_jsx(CodeBlock,{filename:`200`,numberOfLines:`11`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`11`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"id\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`211`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"url\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"https://getanteon.com/\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"status\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"RUNNING\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"guru_type\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"anteon\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"discovered_urls\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: [],`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"start_time\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"2025-02-21T10:25:22.710211Z\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"end_time\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`null`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"link_limit\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1500`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"ignore_query_params\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`true`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})}),`\n`]})})}),_jsx(CodeBlock,{filename:`400`,numberOfLines:`3`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`3`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"msg\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Invalid URL format\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})}),`\n`]})})}),_jsx(CodeBlock,{filename:`429`,numberOfLines:`3`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`3`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"msg\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Request was throttled. Expected available in 56 seconds.\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})}),`\n`]})})})]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Start Website Crawl","icon":"play","api":"POST /{guru_slug}/crawl/start/","description":"Initiate a crawl to explore the URLs of a website","href":"/api-reference/endpoints/crawler-start"}}}},"mdxExtracts":{"tableOfContents":[{"title":"Path Parameters","slug":"path-parameters","depth":2,"children":[]},{"title":"Headers","slug":"headers","depth":2,"children":[{"title":"Body Parameters","slug":"body-parameters","depth":3,"children":[]},{"title":"Response","slug":"response","depth":3,"children":[]}]}],"codeExamples":{"response":{"type":"ResponseExample","children":[{"filename":"200","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"json\">\u003ccode language=\"json\" numberOfLines=\"11\">\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"id\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#B5CEA8\">211\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"url\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"https://getanteon.com/\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"status\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"RUNNING\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"guru_type\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"anteon\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"discovered_urls\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: [],\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"start_time\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"2025-02-21T10:25:22.710211Z\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"end_time\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\">null\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"link_limit\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#B5CEA8\">1500\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"ignore_query_params\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\">true\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"},{"filename":"400","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"json\">\u003ccode language=\"json\" numberOfLines=\"3\">\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"msg\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Invalid URL format\"\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"},{"filename":"429","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"json\">\u003ccode language=\"json\" numberOfLines=\"3\">\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"msg\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Request was throttled. Expected available in 56 seconds.\"\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"}],"attributes":[]}},"endpoint":{"title":"Start Website Crawl","path":"/{guru_slug}/crawl/start/","servers":[{"url":"https://api.gurubase.io/api/v1/"}],"method":"post","request":{"security":[{"title":"API Key Auth","parameters":{"header":{"x-api-key":{"type":"apiKey"}},"query":{},"cookie":{}}}],"parameters":{"query":{},"header":{"x-api-key":{"schema":[{"type":"string","description":"Your API key for authentication. You can obtain your API key from the Gurubase dashboard .","required":true}]}},"cookie":{},"path":{"guru_slug":{"schema":[{"type":"string","description":"The slug of the Guru type to associate the crawled content with","required":true}]}}},"body":{"application/json":{"schemaArray":[{"type":"object","properties":{"url":[{"type":"string","description":"The root URL to start crawling from. Must include http:// or https:// protocol. Important : The crawler only discovers URLs that start with this path. For example: `https://example.com/` → crawls entire site `https://example.com/docs/` → only crawls URLs under `/docs/`","required":true}],"ignore_query_params":[{"type":"boolean","description":"When `true` , query parameters are stripped from discovered URLs (e.g., `?utm_source=...` , `?id=123` ). Set to `false` if you need to crawl paginated content like `?page=1` , `?page=2` ."}]}}],"examples":{"example":{"value":{"url":"\u003cstring>","ignore_query_params":true}}}}}},"response":{"200":{"application/json":{"schemaArray":[{"type":"object","properties":{"id":[{"type":"string","description":"Unique identifier for the crawl operation"}],"url":[{"type":"string","description":"The root URL to be crawled. The crawler will start from this URL and follow all links that begin with it. For example, if the URL is https://example.com/a/b/c , the crawler will extract all links that start with https://example.com/a/b/c ."}],"status":[{"type":"string","description":"Current status of the crawl operation"}],"guru_type":[{"type":"string","description":"The Guru type that the crawl was initiated for"}],"discovered_urls":[{"type":"array","items":[{"type":"object","properties":{}}],"description":"List of URLs discovered during crawling"}],"start_time":[{"type":"string","description":"Timestamp when crawl started (ISO 8601 format)"}],"end_time":[{"type":"string","description":"Timestamp when crawl ended (ISO 8601 format)"}],"ignore_query_params":[{"type":"boolean","description":"Whether query parameters are being stripped from discovered URLs"}]}}],"examples":{"example":{"value":{"id":"\u003cstring>","url":"\u003cstring>","status":"\u003cstring>","guru_type":"\u003cstring>","discovered_urls":[{}],"start_time":"\u003cstring>","end_time":"\u003cstring>","ignore_query_params":true}}}}}},"deprecated":false}},"jsLinks":["/_mintlify/custom/gurubase/12071eb9505de623/gurubase.js"],"subdomain":"gurubase","siteName":"Gurubase","title":"Start Website Crawl - Gurubase","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-01-27T11:15:55.210Z","originalFileLocation":"api-reference/endpoints/crawler-start.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Initiate a crawl to explore the URLs of a website\">\u003cmeta data-mint-head name=\"keywords\" content=\"AI agents, hallucination-resistant AI, RAG, trust score, citations, agentic actions, MCP, MCP server, MCP client, on-premise AI, enterprise AI, knowledge base, documentation\">\u003cmeta data-mint-head name=\"application-name\" content=\"Gurubase\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003cmeta data-mint-head name=\"robots\" content=\"index, follow\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/gurubase/VZQ0YRO71cVM_Oe2/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/gurubase/VZQ0YRO71cVM_Oe2/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/gurubase/VZQ0YRO71cVM_Oe2/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/gurubase/VZQ0YRO71cVM_Oe2/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/gurubase/VZQ0YRO71cVM_Oe2/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/gurubase/VZQ0YRO71cVM_Oe2/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/gurubase/VZQ0YRO71cVM_Oe2/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/gurubase/VZQ0YRO71cVM_Oe2/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/gurubase/VZQ0YRO71cVM_Oe2/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Start Website Crawl - Gurubase\">\u003cmeta data-mint-head property=\"og:description\" content=\"Initiate a crawl to explore the URLs of a website\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Gurubase Documentation\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://docs.gurubase.ai/api-reference/endpoints/crawler-start\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://gurubase.mintlify.app/_mintlify/api/og/v2?division=Endpoints\u0026amp;title=Start+Website+Crawl\u0026amp;description=Initiate+a+crawl+to+explore+the+URLs+of+a+website\u0026amp;theme=58798cb8c98022377cbe4604\u0026amp;signature=f636ab21969d6341fe5c2babdb898b8999046afd4a120225a6cf34dded8a72b9\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Start Website Crawl - Gurubase\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Initiate a crawl to explore the URLs of a website\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://gurubase.mintlify.app/_mintlify/api/og/v2?division=Endpoints\u0026amp;title=Start+Website+Crawl\u0026amp;description=Initiate+a+crawl+to+explore+the+URLs+of+a+website\u0026amp;theme=58798cb8c98022377cbe4604\u0026amp;signature=f636ab21969d6341fe5c2babdb898b8999046afd4a120225a6cf34dded8a72b9\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/gurubase/VZQ0YRO71cVM_Oe2/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Gurubase\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#006BFF\">\u003cmeta data-mint-head name=\"og:locale\" content=\"en_US\">\u003cmeta data-mint-head name=\"author\" content=\"Gurubase\">\u003clink data-mint-head rel=\"canonical\" href=\"https://docs.gurubase.ai/api-reference/endpoints/crawler-start\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/api-reference/endpoints/crawler-start.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://docs.gurubase.ai/#organization\",\"name\":\"Gurubase\",\"url\":\"https://docs.gurubase.ai\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/gurubase/0kE9Hd9k_SJDKu8Q/logo/light.svg?fit=max\u0026auto=format\u0026n=0kE9Hd9k_SJDKu8Q\u0026q=85\u0026s=e80be9ea5250513afdd5cac9e9cf40eb\"}},{\"@type\":\"WebSite\",\"@id\":\"https://docs.gurubase.ai#website\",\"name\":\"Gurubase\",\"url\":\"https://docs.gurubase.ai\",\"publisher\":{\"@id\":\"https://docs.gurubase.ai/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://docs.gurubase.ai/api-reference/endpoints/crawler-start#webpage\",\"url\":\"https://docs.gurubase.ai/api-reference/endpoints/crawler-start\",\"name\":\"Start Website Crawl\",\"description\":\"Initiate a crawl to explore the URLs of a website\",\"dateModified\":\"2026-01-27T11:15:55.210Z\",\"isPartOf\":{\"@id\":\"https://docs.gurubase.ai#website\"},\"breadcrumb\":{\"@id\":\"https://docs.gurubase.ai/api-reference/endpoints/crawler-start#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://docs.gurubase.ai/api-reference/endpoints/crawler-start#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Endpoints\",\"item\":\"https://docs.gurubase.ai/api-reference/endpoints/ask-question\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Start Website Crawl\",\"item\":\"https://docs.gurubase.ai/api-reference/endpoints/crawler-start\"}]},{\"@type\":[\"TechArticle\",\"APIReference\"],\"@id\":\"https://docs.gurubase.ai/api-reference/endpoints/crawler-start#apireference\",\"headline\":\"Start Website Crawl\",\"name\":\"Start Website Crawl\",\"description\":\"Initiate a crawl to explore the URLs of a website\",\"url\":\"https://docs.gurubase.ai/api-reference/endpoints/crawler-start\",\"mainEntityOfPage\":{\"@id\":\"https://docs.gurubase.ai/api-reference/endpoints/crawler-start#webpage\"},\"dateModified\":\"2026-01-27T11:15:55.210Z\",\"publisher\":{\"@id\":\"https://docs.gurubase.ai/#organization\"},\"isPartOf\":{\"@id\":\"https://docs.gurubase.ai#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"Gurubase\",\"creator\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","siteName":"Gurubase","nav":{"global":{"anchors":[{"anchor":"Try Gurubase","icon":"sparkles","href":"https://gurubase.io/"}]},"tabs":[{"tab":"Documentation","pages":[{"title":"Documentation","href":"/introduction"}]},{"tab":"API Reference","groups":[{"group":"API Documentation","icon":"book-text","pages":[{"title":"API Reference","icon":"book-open","href":"/api-reference/introduction"},{"title":"Authentication","icon":"key","href":"/api-reference/authentication"}]},{"group":"Endpoints","icon":"route","pages":[{"title":"Ask Question","icon":"message-circle-question-mark","api":"POST /{guru_slug}/answer/","href":"/api-reference/endpoints/ask-question"},{"title":"Answer Progress","icon":"activity","api":"GET /{guru_slug}/answer-progress/","href":"/api-reference/endpoints/answer-progress"},{"title":"Create Data Sources","icon":"folder-plus","api":"POST /{guru_slug}/data-sources/","href":"/api-reference/endpoints/create-datasources"},{"title":"Get Data Sources","icon":"folder","api":"GET /{guru_slug}/data-sources/","href":"/api-reference/endpoints/get-datasources"},{"title":"Delete Data Sources","icon":"trash-2","api":"DELETE /{guru_slug}/data-sources/","href":"/api-reference/endpoints/delete-datasources"},{"title":"Reindex Data Sources","icon":"refresh-cw","api":"POST /{guru_slug}/data-sources/reindex/","description":"Reindex one or more data sources","href":"/api-reference/endpoints/reindex-datasources"},{"title":"Start Website Crawl","icon":"play","api":"POST /{guru_slug}/crawl/start/","href":"/api-reference/endpoints/crawler-start"},{"title":"Stop Website Crawl","icon":"circle-stop","api":"POST /{guru_slug}/crawl/{crawl_id}/stop/","description":"Stop an ongoing website crawl operation","href":"/api-reference/endpoints/crawler-stop"},{"title":"Get Crawl Status","icon":"activity","api":"GET /{guru_slug}/crawl/{crawl_id}/status/","href":"/api-reference/endpoints/crawler-status"},{"title":"Import YouTube Channel","icon":"https://mintlify.s3.us-west-1.amazonaws.com/gurubase/logo/brands/youtube.svg","api":"POST /youtube/channel/","href":"/api-reference/endpoints/import-youtube-channel"},{"title":"Import YouTube Playlist","icon":"https://mintlify.s3.us-west-1.amazonaws.com/gurubase/logo/brands/youtube.svg","api":"POST /youtube/playlist/","href":"/api-reference/endpoints/import-youtube-playlist"},{"title":"Export Analytics Data","icon":"download","api":"POST /{guru_slug}/analytics/export/","href":"/api-reference/endpoints/analytics-export"},{"title":"Record Vote","icon":"thumbs-up","api":"POST /{guru_type}/record_vote/","href":"/api-reference/endpoints/record-vote"},{"title":"Semantic Search","icon":"search","api":"POST /{guru_slug}/semantic-search/","href":"/api-reference/endpoints/semantic-search"},{"title":"Transcribe Audio","icon":"audio-lines","api":"POST /{guru_type}/transcribe/","href":"/api-reference/endpoints/transcribe-audio"},{"title":"Text-to-Speech Stream","icon":"volume-2","api":"POST /{guru_type}/text-to-speech/stream/","href":"/api-reference/endpoints/text-to-speech"},{"title":"Text-to-Speech Stream V2","icon":"volume-2","api":"POST /{guru_type}/text-to-speech-v2/stream/","href":"/api-reference/endpoints/text-to-speech-v2"},{"title":"List Sessions","icon":"list","api":"POST /{guru_slug}/sessions/","href":"/api-reference/endpoints/list-sessions"},{"title":"Get Session","icon":"eye","api":"POST /sessions/{session_id}/conversation/","href":"/api-reference/endpoints/get-session"},{"title":"Delete Session","icon":"trash-2","api":"DELETE /sessions/{session_id}/delete/","href":"/api-reference/endpoints/delete-session"},{"title":"Get End-User Data","icon":"shield-user","api":"GET /{guru_slug}/end-users/{external_user_id}/data/","href":"/api-reference/endpoints/end-user-data"},{"title":"Erase End-User Data","icon":"shield-x","api":"DELETE /{guru_slug}/end-users/{external_user_id}/data/","href":"/api-reference/endpoints/erase-end-user-data"},{"title":"Follow-up Examples","icon":"message-square-plus","api":"POST /{guru_type}/follow-up/examples/","href":"/api-reference/endpoints/follow-up-examples"},{"title":"Upsert User Profiles","icon":"user-pen","api":"POST /{guru_slug}/profiles/","href":"/api-reference/endpoints/upsert-profiles"}]}]}]},"title":"Start Website Crawl - Gurubase","buildId":"dpl_cadrRc5H9XQiJQbDFN5mcYRs2PAj","deploymentId":"dpl_cadrRc5H9XQiJQbDFN5mcYRs2PAj","dynamics":["c2aa737661","2b790eda92","b0f84f53ff"],"chunks":["OpenApiReferenceProvider-DKJwRUt2.js","utils-CWD_VS3u.js","cloneDeep-izs3Hzxb.js","_Uint8Array-BVIqJ90l.js","isObjectLike-C8vmzPEN.js","isArray-Dcu1dQ5t.js","utils-oFjciODD.js","toString-BX-iyh8G.js","isSymbol-CSH9U2Gm.js","addUniqueKeysToSchema-C76WsxC3.js","snippetPresets-DvPMxpX_.js","style-to-object-BY3JEAdk.js","v4-DIgz8zy6.js","ApiExamples-CuNQfndH.js","isEqual-BfaiBbcl.js","usePlaygroundInputsStore-BVFpVUoF.js","shallow-t59Jg5Kr.js","getApiPlaygroundDisplay-C_JZ4amX.js","CodeSnippet-CLRDnx_h.js","CodeSnippets-C2myU4kc.js","Page-7TwLLapg.js","getPastedJson-B-JaVXw7.js","icon-BA-iDhH3.js","MethodPill-D4qZABFp.js","DuplicateIcon-THIDKO1O.js","createIcon-CYYJwgFd.js","DynamicMarkdown-UGQuxDKp.js","SectionHeading-jYELb23h.js","axios-vUo_bRtp.js"]}