Crawl a website for indexed search

Recursively crawl a website to make it available for indexed search.

GET /integrations/web_crawler/index

JavaScript

JavaScript

import Hyperspell from '@hyperspell/hyperspell';

const client = new Hyperspell({
  apiKey: process.env['HYPERSPELL_API_KEY'], // This is the default and can be omitted
});

const response = await client.integrations.webCrawler.index({ url: 'url' });

console.log(response.resource_id);

200

{
  "resource_id": "<string>"
}

Authorizations

APIKeyAsUser

Authorization · string · header · required

API Key or JWT User Token. If using an API Key, set the X-As-User header to act as a specific user. A JWT User Token is always scoped to a specific user.

Query Parameters

url · string · required

The base URL of the website to crawl

max_depth · integer · default: 2

Maximum depth of links to follow during crawling

Required range: 0 <= x <= 5

limit · integer · default: 20

Maximum number of pages to crawl in total

Required range: 1 <= x <= 100

Response

200

application/json

Successful Response

source · enum<string> · required

Available options: reddit, notion, slack, google_calendar, google_mail, box, dropbox, github, google_drive, vault, web_crawler, trace, microsoft_teams, gmail_actions, granola, fathom, linear, hubspot, salesforce, coda

resource_id · string · required

status · enum<string> · required

Available options: pending, processing, completed, failed, pending_review, skipped