Back to top

DeepCrawl API Documentation

Introduction

The DeepCrawl API provides programmatic access to the DeepCrawl web site crawler. Using the API, you can write applications that utilize DeepCrawl’s capabilities, deployed in your own environment, using the programming language and platform you prefer.

The API uses REST: the HTTP request methods GET, POST, PATCH and DELETE are used to fetch, create, modify, and delete records. JSON is returned in all responses, including errors. The response codes correspond to the success or failure of the request.

POST and PATCH requests use JSON payloads to specify the request data.

Authentication

To authenticate API calls, you must first generate a session to get a token. Then you must send the session token as the value of the X-Auth-Token header to authenticate API calls.

To create a session you first need to manually generate an API Key. This is a key/value pair that can be generated in the ‘API Access’ page. When you click ‘Generate New API Key’, a popup will be displayed with the API Key Value and in the table of Active Keys you can find the API Key ID. This is then sent via Basic Authentication in a POST call to the sessions route.

curl -X POST -u '123:abcdef' https://prod-1-dc-api-oopeix3r.deepcrawl.com/sessions

{
  "token":"abcdef123",
  "_user_href":"/users/example-user",
  ...
}

The token returned from the call to sessions is then passed to all API calls as the X-Auth-Token header:

curl -X GET -H 'X-Auth-Token:abcdef123' https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/1/projects

The session token remains valid for 30 days, then must be regenerated.

Errors

Validation errors return a response code of 422 and a JSON payload with the following form:

{
  "errors": [
    {
      "code": "missing_field",
      "field": "name",
      "message": "Name is missing"
    },
  ...
}

Pagination

Lists of records (for example, lists of projects) are paginated. You can specify from 1 to 200 records per page. If not specified, the default is 50 records per page.

The number of records per page is specified using the per_page parameter:

?per_page=100

To specifiy which page to fetch, use the page parameter:

?page=15

If no page number is specified, the first page of records is fetched (page number 1).

Pagination Information

Pagination information is included in the Link header.

An example Link header is:

Link: </accounts?page=1&per_page=5>; rel='first', 
  </accounts?page=1&per_page=5>; rel='prev',
  </accounts?page=3&per_page=5>; rel='next', 
  </accounts?page=7&per_page=5>; rel='last'

Possible values of the rel attribute are:

Value Description
next URL of the next page
last URL of the last page
first URL of the first page
prev URL of the previous page

Filtering a List

You can filter a list of records (for example, a list of projects) by using a simple filtering syntax in a GET query string. A filter contains one or more conditions, which are combined with a logical AND.

A condition has the form:

q[<field_name>_<operator>]=<value>

where <field_name> is the field you wish to filter by, <operator> is the boolean operator you wish to use, and <value> is the value that the field must have to match.

For example, to filter a list of projects to only those with a name equal to “MyProject”, use the following query string:

?q[name_eql]=MyProject

The default boolean operator is eql. Therefore, the following query string will also filter a list of projects to those with a name equal to “MyProject”:

?q[name]=MyProject

To filter a list of projects to only those with a name containing the string “city”, and a crawl rate less than or equal to 10, use the following query string:

?q[name_cont]=city&q[crawl_rate_lte]=10

The available operators are:

Operator Description Examples
eql equals (text or number) ?q[name_eql]=MyProject
cont contains the specified text ?q[name_cont]=Tourist
lt less than ?q[crawl_rate_lt]=10
lte less than or equal to ?q[crawl_rate_lte]=10
gt greater than ?q[crawl_rate_gt]=10
gte greater than or equal to ?q[crawl_rate_gte]=10
rgx regex ?q[name_rgx]=My.*Project
nrgx negative regex ?q[name_nrgx]=My.*Project

Sorting a List

You can sort a list of records (for example, a list of projects). Use the sort parameter to specify one or more fields to sort by, and a sorting direction for each field.

By default, the sorting direction is ascending. To sort by a field descending, place a - before the field:

?sort=<field_name>      // Ascending
?sort=-<field_name>     // Descending

For example, to sort a list of projects by their finished at date, descending, use the following query string:

?sort=-finished_at

To sort a list by multiple fields, separate the fields with commas. For example, to sort a list of users by their last name descending, and then by their email ascending, use the following query string:

?sort=-name_last,email

Extending Resources

You can extend a resource with an associated resource. For example, you can extend a project resource with its associated account.

Extension is available for single associated resources (not lists of associated resources). In this documentation, the associations available to extend are noted for each resource.

To extend resources, use an extend parameter to specify the name of the associated resource.

For each association, you must also specify the fields you want in the associated resource. You can do this by listing the fields you want in parentheses, separated by commas, after the association name. This allows you to reduce the amount of data retrieved by limiting the associated resource to only the fields you need.

For example, if you want to extend a project resource with its associated account resource, and only want the city and state address fields, use the following query string:

?extend=account(address_city,address_state)

To include all fields in the associated resource, use a * instead of the list of fields:

?extend=account(*)

You can extend a resource with multiple associated resources by providing a list of associations, separated by commas. For example, to extend a project by its associated account and user_agent, use the following query string:

?extend=account(*),user_agent(*)

Selecting Specific Fields

You can limit the resources in a response to just certain fields in order to minimize the size of the payload. Specify the fields you want in a response resource using the select parameter, listing the fields separated by commas.

For example, if you want a list of projects, but want only their name and site_primary fields, use the following query string:

?select=name,site_primary

API Callbacks

You can specify an api_callback to be called when a crawl is finished or a report download has been generated. The api_callback can be specified in the Account object or Project object. If it is not specified in a project, then the account’s callback value is used.

If a callback URL has been specified, it will be called with a POST request, content type application/x-www-form-urlencoded, with the following form fields:

Crawl Finished:

  "message": crawl_status
  "resource": crawl
  "crawl_id": 12345
  "project_id": 1234
  "account_id": 123
  "user_id": 321
  "partner_user_id":
  "partner_account_id":
  "status": finished
  "href": url_of_crawl_api_route

Report Download Available:

  "file_url": url_of_zipped_csv_file
  "partner_user_id": 
  "project_id": 1234
  "status": generated
  "report_id": 123456
  "href": url_of_report_download_api_route
  "partner_account_id": 
  "crawl_id": 12345
  "user_id": 321
  "resource": report_download
  "report_download_id": 1234567
  "message": report_download_available
  "account_id": 123

Accounts

An Account represents your account with DeepCrawl. It contains details such as your email address and street address, time zone, and custom branding settings. It is the parent of a User object, which contains login details.

Accounts

List Accounts
GET/accounts

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts
Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
Link: </accounts?per_page=2&page=1>; rel='first', </accounts?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
Body
[
  {
    "_href": "/accounts/123789",
    "_projects_href": "/accounts/123789/projects",
    "_account_packages_href": "/accounts/123789/account_packages",
    "_crawls_href": "/accounts/123789/crawls",
    "address_city": "The End of the Universe",
    "address_state": "",
    "address_street": "Restaurant",
    "address_zip": "TEOTU 42",
    "api_callback": "http://example.com?someParam=1",
    "country": "Magrathea",
    "custom_color_header": null,
    "custom_color_menu": null,
    "custom_domain": null,
    "custom_email_footer": null,
    "custom_logo_file": null,
    "custom_support_email": null,
    "custom_support_phone": null,
    "finance_vat": "42",
    "static_location": null,
    "name": "Milliways",
    "phone": "42 42 42",
    "pref_email_support": false,
    "timezone": "UTC"
  }
]

Account

Read Account
GET/accounts/{account_id}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_href": "/accounts/123789",
  "_projects_href": "/accounts/123789/projects",
  "_account_packages_href": "/accounts/123789/account_packages",
  "_crawls_href": "/accounts/123789/crawls",
  "address_city": "The End of the Universe",
  "address_state": "",
  "address_street": "Restaurant",
  "address_zip": "TEOTU 42",
  "api_callback": "http://example.com?someParam=1",
  "country": "Magrathea",
  "custom_color_header": null,
  "custom_color_menu": null,
  "custom_domain": null,
  "custom_email_footer": null,
  "custom_logo_file": null,
  "custom_support_email": null,
  "custom_support_phone": null,
  "finance_vat": "42",
  "static_location": null,
  "name": "Milliways",
  "phone": "42 42 42",
  "pref_email_support": false,
  "timezone": "UTC"
}
Response  404

Update Account
PATCH/accounts/{account_id}

Example URI

PATCH https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

Request
HideShow
Headers
Content-Type: application/json
Body
{
  "address_city": "The Beginning of the Universe"
}
Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_href": "/accounts/123789",
  "_projects_href": "/accounts/123789/projects",
  "_account_packages_href": "/accounts/123789/account_packages",
  "_crawls_href": "/accounts/123789/crawls",
  "_primary_account_package_href": null,
  "address_city": "The Beginning of the Universe",
  "address_state": "",
  "address_street": "Restaurant",
  "address_zip": "TEOTU 42",
  "api_callback": "http://example.com?someParam=1",
  "country": "Magrathea",
  "custom_color_header": null,
  "custom_color_menu": null,
  "custom_domain": null,
  "custom_email_footer": null,
  "custom_logo_file": null,
  "custom_support_email": null,
  "custom_support_phone": null,
  "finance_vat": "42",
  "static_location": null,
  "name": "Milliways",
  "phone": "42 42 42",
  "pref_email_support": false,
  "timezone": "UTC"
}
Response  422
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "errors": [
    {
      "code": "missing_field",
      "field": "address_city",
      "message": "Address city is missing"
    },
    {
      "code": "missing_field",
      "field": "address_street",
      "message": "Address street is missing"
    },
    {
      "code": "missing_field",
      "field": "address_zip",
      "message": "Address zip is missing"
    },
    {
      "code": "missing_field",
      "field": "country",
      "message": "Country is missing"
    },
    {
      "code": "missing_field",
      "field": "name",
      "message": "Name is missing"
    },
    {
      "code": "missing_field",
      "field": "phone",
      "message": "Phone number is missing"
    }
  ]
}
Response  404

Delete Account
DELETE/accounts/{account_id}

Example URI

DELETE https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

Response  204
HideShow
Headers
X-Content-Type-Options: nosniff
Response  409
Response  404

Account Errors

Projects

Project Object

Argument Description
name string (required)
The name of the project.
Eg, Web crawl of example.com
site_primary string (required)
The main domain of the project, including the protocol.
Eg, https://www.example.com
crawl_subdomains boolean Default: false
Whether to crawl all subdomains of the site_primary domain.
Eg, true
v2_crawl_type_codes array[string] (required)
Choices: web sitemap list google_analytics backlinks
The crawl types to include in this project.
Eg, ["web", "list"]
crawl_rate integer (required)
Min: 1 ; Max: Account maximum, eg 50
The maximum number of pages per second to crawl.
Eg, 3
limit_levels_max integer (required)
Min: 1 ; Max: Account maximum, eg 100
The maximum number of levels from the starting URL to crawl.
Eg, 10
limit_pages_max integer (required)
Min: 1 ; Max: Account maximum, eg 3,000,000
The maximum number of pages to crawl.
Eg, 5000
auto_finalize boolean Default: false
Whether to generate reports and finalize the crawl once the level or page limit is reached, or pause and allow resuming instead.
Eg, true
site_secondaries array[string]
Additional domains that the crawler should consider internal to the site being crawled. Protocols should be included. A * can be used as a wildcard.
Eg, ["https://foobar.com", "http://*.foobar.co.uk"]
start_urls array[string]
A list of URLs that a web crawl should start at, instead of the site_primary domain. If left empty, a web crawl will start at the site_primary domain.
Eg, ["https://foo.com/food", "https://foo.com/drink"]
urls_included array[string]
A list of regular expressions. The crawler will only crawl URLs that match any of the regular expressions. Patterns match against the entire URL, including protocol.
Eg, ["/food/", "https?://foo.com/drink/1/.*?/4"]
urls_excluded array[string]
A list of regular expressions. The crawler will not crawl any URLs that match any of the regular expressions. Patterns match against the entire URL, including protocol.
Eg, ["/snacks/", "https?://foo.com/food/3.*?/4"]
page_groupings array[object]
A list of page grouping objects. Page groupings allow you to crawl only a percentage of URLs that match a pattern – for example, 20% of URLs that match the pattern /products/. A page grouping object contains the following fields:
  • name: The name of the group
  • url_match: A regular expression that URLs must match
  • crawl_sample: The fraction of matching URLs to crawl, between 0 and 1
Eg, [{"name":"Products", "url_match":"/products/", "crawl_sample":"0.2"}]
crawl_css_js boolean Default: false
Whether to crawl linked CSS and JS files.
Eg, true
crawl_disallowed_pages boolean Default: false
Whether to crawl the first level of URLs marked disallowed in robots.txt.
Eg, true
crawl_external_urls boolean Default: false
Whether to crawl the first level of URLs considered external (not matching the site_primary or site_secondaries).
Eg, true
crawl_nofollow_links boolean Default: false
Whether to follow links that are nofollowed.
Eg, true
crawl_noindex_pages boolean Default: false
Whether to crawl URLs marked noindex in robots.txt.
Eg, true
crawl_non_html_file_types boolean Default: false
Whether to fetch the headers for linked non-HTML URLs.
Eg, true
crawl_not_included_urls boolean Default: false
Whether to crawl the first level of URLs that are outside the urls_included restriction.
Eg, true
location string Default: dip

Choices:
  • dip Dynamic IP
  • gb United Kingdom
  • us United States
  • de Germany
  • fr France
  • cn China
  • sg Singapore
  • sip Static IP (use the account’s static_location)
  • it Italy
A code indicating the IP address the crawler should crawl from.
Eg, cn
is_stealth_mode boolean Default: false
Whether to crawl in Stealth mode. If set to true, the crawler will ignore other IP location, user agent, and crawl rate settings, and crawl with a randomized location and user agent, and a very slow crawl rate of less than 1 page per second.
Eg, true
user_agent string Default: googlebot
Choices: applebot bingbot bingbot-mobile chrome deepcrawl facebook firefox generic google-web-preview googlebot googlebot-image googlebot-mobile-feature-phone googlebot-news googlebot-smartphone googlebot-video internet-explorer internet-explorer-6 iphone
A code indicating the user agent to send in request headers.
Eg, iphone
custom_header_user_agent string
A user agent string to send in request headers.
Eg, My custom user agent
custom_extractions array[object]
An array of custom extraction objects. A custom extraction object contains the following fields:
  • label: The column header for this extraction to appear in reports
  • regex: A regular expression to extract text
  • match_number_from: An integer; take only instances of matches starting from the nth match
  • match_number_to: An integer; take only instances of matches up to and including the nth match
  • clean_html_tags: A boolean; if true, strip HTML tags from extracted text
  • filter: A regular expression; matching text will be filtered out of extraction results
Eg, [{"label":"Telephone","regex":"tel:([^\"]*)","clean_html_tags":true, "match_number_from":2, "match_number_to":4, "filter":"^020"}]
robots_overwrite string
Text representing a robots.txt file to use instead of the site’s robots.txt file.
Eg,
User-agent: *
Disallow: /foo
custom_dns array[object]
An array of custom DNS objects, to specify IP addresses for domains that do not have public DNS records. A custom DNS object contains the following fields:
  • hostname: The domain to use custom DNS for
  • ip_address: The IP address the hostname should resolve to

Eg, [{"hostname":"staging.example.com", "ip_address":"12.34.56.78"}]
site_test string
The domain of a staging version of the project’s primary site. Include protocol.
Eg, http://staging.example.com
crawl_test_site boolean Default: false
Whether to crawl the site_test domain rather than the project’s primary domain.
Eg, true
site_test_user string
The Basic Authentication username to send with requests when crawling the site_test domain.
Eg, myusername
site_test_pass string
The Basic Authentication password to send with requests when crawling the site_test domain.
Eg, mypassword
url_rewrite_query_parameters array[string]
An array of strings representing parameters to remove from URL query strings.
Eg, ["page", "sessionId"]
url_rewrite_regex_parameters array[object]
An array of rewrite objects describing rewrites to apply to all discovered URLs. A rewrite object contains the following fields:
  • match_from: A regex describing the match.
  • match_to: Text describing the replacement for matched URLs, with variables $1, $2, etc representing matched groups.
  • case_option: change the case of all matched text: u = uppercase, l = lowercase, null = no change

Eg, [{"match_from":"^https?://example\.org(.*)", "match_to":"http://mirror.org$1"}]
use_rewrite_rules boolean Default: false
Whether to apply the rewrite rules defined in url_rewrite_regex_parameters during the crawl.
Eg, true
url_rewrite_strip_fragment boolean Default: false
Whether to strip fragments (’#’ and all following text) from all discovered URLs.
Eg, true
api_callback string
A URL that the crawler should call when a crawl finishes, or a report download has been generated.
Eg, http://example.com/webhooks/dc_crawl_finished
alert_emails array[string]
An array of email addresses to send alerts to.
Eg, ["ab@example.org", "cd@example.org"]
alert_setting string Default: always
Choices: always never
When to send alerts.
Eg, never

Projects

List Projects
GET/accounts/{account_id}/projects

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects
URI Parameters
HideShow
account_id
integer (required) 

Account ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
Link: </accounts/123789/projects?per_page=2&page=1>; rel='first', </accounts/123789/projects?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
Body
[
  {
    "_account_href": "/accounts/123789",
    "_alert_setting_href": "/alert_settings/always",
    "_crawls_finished_last_href": "/accounts/123789/projects/152511/crawls/166183",
    "_crawls_href": "/accounts/123789/projects/152511/crawls",
    "_crawls_last_href": "/accounts/123789/projects/152511/crawls/166183",
    "_google_analytics_project_view_href": "/accounts/123789/projects/152511/google_analytics_project_view",
    "_href": "/accounts/123789/projects/152511",
    "_issues_href": "/accounts/123789/projects/152511/issues",
    "_location_href": "/locations/dip",
    "_schedules_href": "/accounts/123789/projects/152511/schedules",
    "_uploads_href": "/accounts/123789/projects/152511/uploads",
    "_user_agent_href": null,
    "_crawl_settings_last_href": "/accounts/123789/projects/152511/crawl_settings/152444",
    "alert_emails": [
      "sample@email.com"
    ],
    "_sitemaps_href": "/accounts/123789/projects/152511/sitemaps",
    "_crawls_finished_last_finished_at": "1952-03-11T00:00:00+00:00",
    "_crawls_finished_last_progress_crawled": 0,
    "id": 123,
    "api_callback": null,
    "attach_pdf": true,
    "auto_finalize": false,
    "clean_custom_extraction": false,
    "compare_to": "last_crawl",
    "crawl_css_js": true,
    "crawl_disallowed_pages": false,
    "crawl_external_urls": false,
    "crawl_nofollow_links": true,
    "crawl_noindex_pages": true,
    "crawl_non_html_file_types": false,
    "crawl_not_included_urls": false,
    "crawl_rate": 3,
    "crawl_subdomains": false,
    "crawl_test_site": false,
    "crawl_types": [
      "web"
    ],
    "crawls_count": 1,
    "custom_dns": null,
    "custom_header_user_agent": null,
    "duplicate_precision": 2,
    "is_stealth_mode": false,
    "limit_levels_max": 4,
    "limit_pages_max": 100000,
    "max_content_size": 50,
    "max_description_length": 160,
    "max_external_links": 10,
    "max_html_size": 200,
    "max_links": 100,
    "max_load_time": 2,
    "max_redirections": 4,
    "max_title_width": 502,
    "max_url_length": 1024,
    "min_content_ratio": 0.1,
    "min_content_size": 3,
    "min_description_length": 50,
    "min_title_length": 10,
    "name": "Table",
    "page_groupings": [],
    "robots_overwrite": null,
    "safe_title_width": 480,
    "site_primary": "http://example.com/",
    "site_secondaries": [],
    "site_test": null,
    "site_test_pass": null,
    "site_test_user": null,
    "start_urls": [
      "http://example.com/"
    ],
    "twitter_description_length": 200,
    "url_rewrite_query_parameters": [],
    "url_rewrite_regex_parameters": [],
    "url_rewrite_strip_fragment": true,
    "urls_excluded": [],
    "urls_included": [],
    "use_rewrite_rules": false,
    "use_robots_overwrite": false,
    "next_run_time": "2015-08-01T08:00:00.000+00:00"
  }
]
Response  404

Create Project
POST/accounts/{account_id}/projects

Example URI

POST https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects
URI Parameters
HideShow
account_id
integer (required) 

Account ID

Request
HideShow
Headers
Content-Type: application/json
Body
{
  "crawl_types": [
    "web"
  ],
  "name": "Table",
  "site_primary": "http://example.com/"
}
Response  201
HideShow
Headers
Content-Type: application/json
Location: /accounts/123789/projects/152511
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_alert_setting_href": "/alert_settings/always",
  "_crawls_href": "/accounts/123789/projects/152511/crawls",
  "_href": "/accounts/123789/projects/152511",
  "_issues_href": "/accounts/123789/projects/152511/issues",
  "_location_href": "/locations/dip",
  "_schedules_href": "/accounts/123789/projects/152511/schedules",
  "_uploads_href": "/accounts/123789/projects/152511/uploads",
  "_user_agent_href": "/user_agents/googlebot",
  "_crawl_settings_last_href": "/accounts/123789/projects/152511/crawl_settings/152444",
  "_google_analytics_project_view_href": "/accounts/123789/projects/152511/google_analytics_project_view",
  "alert_emails": [
    "sample@email.com"
  ],
  "_sitemaps_href": "/accounts/123789/projects/152511/sitemaps",
  "id": 123,
  "alert_setting": "always",
  "api_callback": null,
  "attach_pdf": true,
  "auto_finalize": false,
  "clean_custom_extraction": false,
  "compare_to": "last_crawl",
  "crawl_css_js": true,
  "crawl_disallowed_pages": false,
  "crawl_external_urls": false,
  "crawl_nofollow_links": true,
  "crawl_noindex_pages": true,
  "crawl_non_html_file_types": false,
  "crawl_not_included_urls": false,
  "crawl_rate": 3,
  "crawl_subdomains": false,
  "crawl_test_site": false,
  "crawl_types": [
    "web"
  ],
  "crawls_count": 0,
  "custom_dns": null,
  "custom_header_user_agent": null,
  "duplicate_precision": 2,
  "is_stealth_mode": false,
  "limit_levels_max": 4,
  "limit_pages_max": 100000,
  "location": "dip",
  "max_content_size": 51200,
  "max_description_length": 160,
  "max_external_links": 10,
  "max_html_size": 204800,
  "max_links": 100,
  "max_load_time": 2,
  "max_redirections": 4,
  "max_title_width": 502,
  "max_url_length": 1024,
  "min_content_ratio": 0.1,
  "min_content_size": 3072,
  "min_description_length": 50,
  "min_title_length": 10,
  "name": "Table",
  "page_groupings": [],
  "robots_overwrite": null,
  "safe_title_width": 480,
  "site_primary": "http://example.com/",
  "site_secondaries": [],
  "site_test": null,
  "site_test_pass": null,
  "site_test_user": null,
  "start_urls": [],
  "twitter_description_length": 200,
  "url_rewrite_query_parameters": [],
  "url_rewrite_regex_parameters": [],
  "url_rewrite_strip_fragment": true,
  "urls_excluded": [],
  "urls_included": [],
  "use_rewrite_rules": false,
  "use_robots_overwrite": false,
  "user_agent": "googlebot",
  "next_run_time": null,
  "use_robots_for_sitemaps": true,
  "custom_extractions": []
}
Response  422
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "errors": [
    {
      "code": "missing_field",
      "field": "name",
      "message": "Name is missing"
    },
    {
      "code": "missing_field",
      "field": "site_primary",
      "message": "Site primary is missing"
    }
  ]
}
Response  404

Project

Read Project
GET/accounts/{account_id}/projects/{project_id}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_alert_setting_href": "/alert_settings/always",
  "_crawls_finished_last_href": "/accounts/123789/projects/152511/crawls/166183",
  "_crawls_href": "/accounts/123789/projects/152511/crawls",
  "_crawls_last_href": "/accounts/123789/projects/152511/crawls/166183",
  "_google_analytics_project_view_href": "/accounts/123789/projects/152511/google_analytics_project_view",
  "_href": "/accounts/123789/projects/152511",
  "_issues_href": "/accounts/123789/projects/152511/issues",
  "_location_href": "/locations/dip",
  "_schedules_href": "/accounts/123789/projects/152511/schedules",
  "_uploads_href": "/accounts/123789/projects/152511/uploads",
  "_user_agent_href": null,
  "_crawl_settings_last_href": "/accounts/123789/projects/152511/crawl_settings/152444",
  "alert_emails": [
    "sample@email.com"
  ],
  "_sitemaps_href": "/accounts/123789/projects/152511/sitemaps",
  "_crawls_finished_last_finished_at": "1952-03-11T00:00:00+00:00",
  "_crawls_finished_last_progress_crawled": 0,
  "id": 123,
  "api_callback": null,
  "attach_pdf": true,
  "auto_finalize": false,
  "clean_custom_extraction": false,
  "compare_to": "last_crawl",
  "crawl_css_js": true,
  "crawl_disallowed_pages": false,
  "crawl_external_urls": false,
  "crawl_nofollow_links": true,
  "crawl_noindex_pages": true,
  "crawl_non_html_file_types": false,
  "crawl_not_included_urls": false,
  "crawl_rate": 3,
  "crawl_subdomains": false,
  "crawl_test_site": false,
  "crawl_types": [
    "web"
  ],
  "crawls_count": 1,
  "custom_dns": null,
  "custom_header_user_agent": null,
  "duplicate_precision": 2,
  "limit_levels_max": 4,
  "limit_pages_max": 100000,
  "max_content_size": 51200,
  "max_description_length": 160,
  "max_external_links": 10,
  "max_html_size": 204800,
  "max_links": 100,
  "max_load_time": 2,
  "max_redirections": 4,
  "max_title_width": 502,
  "max_url_length": 1024,
  "min_content_ratio": 0.1,
  "min_content_size": 3072,
  "min_description_length": 50,
  "min_title_length": 10,
  "name": "Table",
  "page_groupings": [],
  "robots_overwrite": null,
  "safe_title_width": 480,
  "site_primary": "http://example.com/",
  "site_secondaries": [],
  "site_test": null,
  "site_test_pass": [],
  "site_test_user": [],
  "start_urls": [
    "http://example.com/"
  ],
  "twitter_description_length": 200,
  "url_rewrite_query_parameters": null,
  "url_rewrite_regex_parameters": null,
  "url_rewrite_strip_fragment": true,
  "urls_excluded": null,
  "urls_included": null,
  "use_rewrite_rules": false,
  "use_robots_overwrite": false,
  "next_run_time": "2015-08-01T08:00:00.000+00:00"
}
Response  404

Update Project
PATCH/accounts/{account_id}/projects/{project_id}

Example URI

PATCH https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

Request
HideShow
Headers
Content-Type: application/json
Body
{
  "name": "Matrix"
}
Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_alert_setting_href": "/alert_settings/always",
  "_crawls_finished_last_href": "/accounts/123789/projects/152511/crawls/166183",
  "_crawls_href": "/accounts/123789/projects/152511/crawls",
  "_crawls_last_href": "/accounts/123789/projects/152511/crawls/166183",
  "_google_analytics_project_view_href": "/accounts/123789/projects/152511/google_analytics_project_view",
  "_href": "/accounts/123789/projects/152511",
  "_issues_href": "/accounts/123789/projects/152511/issues",
  "_location_href": "/locations/dip",
  "_schedules_href": "/accounts/123789/projects/152511/schedules",
  "_uploads_href": "/accounts/123789/projects/152511/uploads",
  "_user_agent_href": null,
  "_crawl_settings_last_href": "/accounts/123789/projects/152511/crawl_settings/152444",
  "alert_emails": [
    "sample@email.com"
  ],
  "_sitemaps_href": "/accounts/123789/projects/152511/sitemaps",
  "_crawls_finished_last_finished_at": "1952-03-11T00:00:00+00:00",
  "_crawls_finished_last_progress_crawled": 0,
  "id": 123,
  "api_callback": null,
  "attach_pdf": true,
  "auto_finalize": false,
  "clean_custom_extraction": false,
  "compare_to": "last_crawl",
  "crawl_css_js": true,
  "crawl_disallowed_pages": false,
  "crawl_external_urls": false,
  "crawl_nofollow_links": true,
  "crawl_noindex_pages": true,
  "crawl_non_html_file_types": false,
  "crawl_not_included_urls": false,
  "crawl_rate": 3,
  "crawl_subdomains": false,
  "crawl_test_site": false,
  "crawl_types": [
    "web"
  ],
  "crawls_count": 1,
  "custom_dns": null,
  "custom_header_user_agent": null,
  "duplicate_precision": 2,
  "limit_levels_max": 4,
  "limit_pages_max": 100000,
  "max_content_size": 51200,
  "max_description_length": 160,
  "max_external_links": 10,
  "max_html_size": 204800,
  "max_links": 100,
  "max_load_time": 2,
  "max_redirections": 4,
  "max_title_width": 502,
  "max_url_length": 1024,
  "min_content_ratio": 0.1,
  "min_content_size": 3072,
  "min_description_length": 50,
  "min_title_length": 10,
  "name": "Matrix",
  "page_groupings": [],
  "robots_overwrite": null,
  "safe_title_width": 480,
  "site_primary": "http://example.com/",
  "site_secondaries": [],
  "site_test": null,
  "site_test_pass": null,
  "site_test_user": null,
  "start_urls": [
    "http://example.com/"
  ],
  "twitter_description_length": 200,
  "url_rewrite_query_parameters": [],
  "url_rewrite_regex_parameters": [],
  "url_rewrite_strip_fragment": true,
  "urls_excluded": [],
  "urls_included": [],
  "use_rewrite_rules": false,
  "use_robots_overwrite": false,
  "next_run_time": "2015-08-01T08:00:00.000+00:00"
}
Response  422
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "errors": [
    {
      "code": "missing_field",
      "field": "name",
      "message": "Name is missing"
    },
    {
      "code": "missing_field",
      "field": "site_primary",
      "message": "Site primary is missing"
    }
  ]
}
Response  404

Delete Project
DELETE/accounts/{account_id}/projects/{project_id}

Example URI

DELETE https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

Response  204
HideShow
Headers
X-Content-Type-Options: nosniff
Response  409
Response  404

Project Errors

Project Uploads

Project Uploads

List Project Uploads
GET/accounts/{account_id}/projects/{project_id}/uploads

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/uploads
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
Link: </accounts/123789/projects/152511/uploads?per_page=2&page=1>; rel='first', </accounts/123789/projects/152511/uploads?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
Body
[
  {
    "_account_href": "/accounts/123789",
    "_href": "/accounts/123789/projects/152511/uploads/81651",
    "_project_href": "/accounts/123789/projects/152511",
    "_project_upload_type_href": "analytics_csv",
    "enabled": true,
    "file_name": "analytics_csv_small.csv",
    "status": "processing",
    "total_rows": 0
  }
]
Response  404

Create Project Upload --NOTEST--
POST/accounts/{account_id}/projects/{project_id}/uploads

Example URI

POST https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/uploads
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

Request
HideShow
Headers
Content-Type: application/json
Body
{
  "file": "<MULTIPART>",
  "project_upload_type": "analytics_csv",
  "crawl_type": "google_analytics"
}
Response  201
HideShow
Headers
Content-Type: application/json
Location: /accounts/123789/projects/152511/uploads/81651
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_href": "/accounts/123789/projects/152511/uploads/81651",
  "_project_href": "/accounts/123789/projects/152511",
  "_project_upload_type_href": "analytics_csv",
  "enabled": true,
  "file_name": "analytics_csv_small.csv",
  "status": "processing",
  "total_rows": 0
}
Response  422
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{}
Response  404

Project Upload

Read Project Upload
GET/accounts/{account_id}/projects/{project_id}/uploads/{upload_id}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/uploads/upload_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

upload_id
integer (required) 

Upload ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_href": "/accounts/123789/projects/152511/uploads/81651",
  "_project_href": "/accounts/123789/projects/152511",
  "_project_upload_type_href": "analytics_csv",
  "enabled": true,
  "file_name": "analytics_csv_small.csv",
  "status": "processing",
  "total_rows": 0
}
Response  404

Update Project Upload
PATCH/accounts/{account_id}/projects/{project_id}/uploads/{upload_id}

Example URI

PATCH https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/uploads/upload_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

upload_id
integer (required) 

Upload ID

Request
HideShow
Headers
Content-Type: application/json
Body
{
  "enabled": "false"
}
Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_href": "/accounts/123789/projects/152511/uploads/81651",
  "_project_href": "/accounts/123789/projects/152511",
  "_project_upload_type_href": "analytics_csv",
  "enabled": false,
  "file_name": "analytics_csv_small.csv",
  "status": "processing",
  "total_rows": 0
}
Response  422
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{}
Response  404

Delete Project Upload
DELETE/accounts/{account_id}/projects/{project_id}/uploads/{upload_id}

Example URI

DELETE https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/uploads/upload_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

upload_id
integer (required) 

Upload ID

Response  204
HideShow
Headers
X-Content-Type-Options: nosniff
Response  404

Project Upload Errors

Crawls

Starting a Crawl

To start a crawl, POST to the crawls href, with the status field set to crawling:

curl -X POST -H 'X-Auth-Token:abcdef123' 'https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/1/projects/2/crawls' -d 'status=crawling'

Crawls

List Crawls
GET/accounts/{account_id}/projects/{project_id}/crawls

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
Link: </accounts/123789/projects/152511/crawls?per_page=2&page=1>; rel='first', </accounts/123789/projects/152511/crawls?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
Body
[
  {
    "_account_href": "/accounts/123789",
    "_href": "/accounts/123789/projects/152511/crawls/166183",
    "_project_href": "/accounts/123789/projects/152511",
    "_reports_href": "/accounts/123789/projects/152511/crawls/166183/reports",
    "_sitemaps_href": "/accounts/123789/projects/152511/crawls/166183/sitemaps",
    "_crawl_settings_last_href": "/accounts/123789/projects/152511/crawl_settings/152444",
    "_statistics_href": "/accounts/123789/projects/152511/crawls/166183/statistics",
    "crawling_at": "1952-03-11T00:00:00+00:00",
    "finished_at": "1952-03-11T00:00:00+00:00",
    "progress_crawled": 0,
    "progress_discovered": 0,
    "progress_enqueued": 0,
    "stats_crawled": {
      "crawled_/_s": {}
    },
    "status": "finished",
    "total_steps": 0,
    "total_step_links": 0
  }
]
Response  404

Create Crawl
POST/accounts/{account_id}/projects/{project_id}/crawls

Example URI

POST https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

Request
HideShow
Headers
Content-Type: application/json
Body
{
  "status": "draft"
}
Request  --422--
HideShow
Headers
Content-Type: application/json
Body
{
  "status": "finished"
}
Response  201
HideShow
Headers
Content-Type: application/json
Location: /accounts/123789/projects/152511/crawls/166183
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_href": "/accounts/123789/projects/152511/crawls/166183",
  "_project_href": "/accounts/123789/projects/152511",
  "_reports_href": "/accounts/123789/projects/152511/crawls/166183/reports",
  "_sitemaps_href": "/accounts/123789/projects/152511/crawls/166183/sitemaps",
  "_crawl_settings_last_href": "/accounts/123789/projects/152511/crawl_settings/152444",
  "_statistics_href": "/accounts/123789/projects/152511/crawls/166183/statistics",
  "crawling_at": null,
  "finished_at": null,
  "progress_crawled": 0,
  "progress_discovered": 0,
  "progress_enqueued": 0,
  "stats_crawled": null,
  "status": "draft",
  "total_steps": 0,
  "total_step_links": 0,
  "v1_migration_status": null
}
Response  422
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "errors": [
    {
      "field": "status",
      "code": "invalid",
      "message": "Status is not valid"
    }
  ]
}
Response  409
Response  404

Crawl

Read Crawl
GET/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_href": "/accounts/123789/projects/152511/crawls/166183",
  "_project_href": "/accounts/123789/projects/152511",
  "_reports_href": "/accounts/123789/projects/152511/crawls/166183/reports",
  "_sitemaps_href": "/accounts/123789/projects/152511/crawls/166183/sitemaps",
  "_crawl_settings_last_href": "/accounts/123789/projects/152511/crawl_settings/152444",
  "_statistics_href": "/accounts/123789/projects/152511/crawls/166183/statistics",
  "crawling_at": "1952-03-11T00:00:00+00:00",
  "finished_at": "1952-03-11T00:00:00+00:00",
  "progress_crawled": 0,
  "progress_discovered": 0,
  "progress_enqueued": 0,
  "stats_crawled": {
    "crawled_/_s": {}
  },
  "status": "finished",
  "total_steps": 0,
  "total_step_links": 0
}
Response  404

Update Crawl
PATCH/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}

Example URI

PATCH https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

Request
HideShow
Headers
Content-Type: application/json
Body
{
  "limit_levels_max": 42
}
Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_href": "/accounts/123789/projects/152511/crawls/166183",
  "_project_href": "/accounts/123789/projects/152511",
  "_reports_href": "/accounts/123789/projects/152511/crawls/166183/reports",
  "_sitemaps_href": "/accounts/123789/projects/152511/crawls/166183/sitemaps",
  "_crawl_settings_last_href": "/accounts/123789/projects/152511/crawl_settings/152444",
  "_statistics_href": "/accounts/123789/projects/152511/crawls/166183/statistics",
  "crawling_at": "1952-03-11T00:00:00+00:00",
  "finished_at": "1952-03-11T00:00:00+00:00",
  "progress_crawled": 0,
  "progress_discovered": 0,
  "progress_enqueued": 0,
  "stats_crawled": {
    "crawled_/_s": {}
  },
  "status": "finished",
  "total_steps": 0,
  "total_step_links": 0
}
Response  422
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "errors": [
    {
      "field": "status",
      "code": "invalid",
      "message": "Status is not valid"
    }
  ]
}
Response  404

Delete Crawl
DELETE/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}

Example URI

DELETE https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

Response  204
HideShow
Headers
X-Content-Type-Options: nosniff
Response  404

Crawl Errors

Reports

Reports

List Reports
GET/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}/reports

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id/reports
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
Link: </accounts/123789/projects/152511/crawls/166183/reports?per_page=2&page=1>; rel='first', </accounts/123789/projects/152511/crawls/166183/reports?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
Body
[
  {
    "_account_href": "/accounts/123789",
    "_crawl_href": "/accounts/123789/projects/152511/crawls/166183",
    "_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656",
    "_project_href": "/accounts/123789/projects/152511",
    "_report_downloads_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/downloads",
    "_report_rows_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/report_rows",
    "_report_template_href": "/datasources/crawl_urls/report_templates/62711",
    "_report_type_href": "/report_types/basic",
    "_statistics_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/statistics",
    "_recent_report_trend_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/statistics/recent_report_trend",
    "_issues_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/issues",
    "id": 123,
    "report_type": "basic",
    "total_rows": 1
  }
]
Response  404

Report

Read Report
GET/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}/reports/{report_id}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id/reports/report_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

report_id
integer (required) 

Report ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_crawl_href": "/accounts/123789/projects/152511/crawls/166183",
  "_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656",
  "_project_href": "/accounts/123789/projects/152511",
  "_report_downloads_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/downloads",
  "_report_rows_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/report_rows",
  "_report_template_href": "/datasources/crawl_urls/report_templates/62711",
  "_report_type_href": "/report_types/basic",
  "_statistics_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/statistics",
  "_recent_report_trend_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/statistics/recent_report_trend",
  "_issues_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/issues",
  "id": 123,
  "report_type": "basic",
  "total_rows": 1
}
Response  404

Report Rows

Report Rows

List Report Rows
GET/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}/reports/{report_id}/report_rows

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id/reports/report_id/report_rows
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

report_id
integer (required) 

Report ID

Request  --422--
HideShow
Headers
Content-Type: application/json
Body
{
  "filter": "something invalid"
}
Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
Link: </accounts/123789/projects/152511/crawls/166183/reports/5656/report_rows?per_page=2&page=1>; rel='first', </accounts/123789/projects/152511/crawls/166183/reports/5656/report_rows?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
Body
[
  {
    "_account_href": "/accounts/123789",
    "_crawl_href": "/accounts/123789/projects/152511/crawls/166183",
    "_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/report_rows/1",
    "_project_href": "/accounts/123789/projects/152511",
    "_report_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656",
    "_data_pages_href": {
      "url": "/accounts/123789/projects/152511/crawls/166183/pages/e77DS1P3wj4slksSpe-1tloS01m9iznHhRzXK9kTZ2mykqGv67w3fkhUTON39kvVuQk68Wa9No4yzS0B763irQ=="
    },
    "data": {
      "ga_avg_page_load_time": null,
      "ga_avg_time_on_page": null,
      "ga_pageviews_per_visits": null,
      "ga_visit_bounce_rate": null,
      "ga_visits": null,
      "canonical_header_url": null,
      "canonical_html_url": null,
      "canonical_url": null,
      "canonical_url_digest": null,
      "canonical_url_is_consistent": null,
      "content_html_ratio": null,
      "content_size": null,
      "crawl_id": 166183,
      "custom_extraction_1": null,
      "custom_extraction_2": null,
      "custom_extraction_3": null,
      "custom_extraction_4": null,
      "custom_extraction_5": null,
      "custom_extraction_6": null,
      "custom_extraction_7": null,
      "custom_extraction_8": null,
      "custom_extraction_9": null,
      "custom_extraction_10": null,
      "custom_extraction_11": null,
      "custom_extraction_12": null,
      "custom_extraction_13": null,
      "custom_extraction_14": null,
      "custom_extraction_15": null,
      "custom_extraction_16": null,
      "custom_extraction_17": null,
      "custom_extraction_18": null,
      "custom_extraction_19": null,
      "custom_extraction_20": null,
      "custom_extraction_21": null,
      "custom_extraction_22": null,
      "custom_extraction_23": null,
      "custom_extraction_24": null,
      "custom_extraction_25": null,
      "custom_extraction_26": null,
      "custom_extraction_27": null,
      "custom_extraction_28": null,
      "custom_extraction_29": null,
      "custom_extraction_30": null,
      "deeprank": null,
      "description": null,
      "description_length": null,
      "description_length_px": null,
      "disallowed_page": null,
      "duplicate_body": null,
      "duplicate_body_non_indexable": null,
      "duplicate_body_primary_url": null,
      "duplicate_body_primary_url_digest": null,
      "duplicate_description": null,
      "duplicate_description_non_indexable": null,
      "duplicate_description_primary_url": null,
      "duplicate_description_primary_url_digest": null,
      "duplicate_page": null,
      "duplicate_page_non_indexable": null,
      "duplicate_title": null,
      "duplicate_title_non_indexable": null,
      "duplicate_title_primary_url": null,
      "duplicate_title_primary_url_digest": null,
      "encoding_header": null,
      "external_links": null,
      "fb_admins": null,
      "fb_app_id": null,
      "found_at": null,
      "found_at_sitemap": null,
      "h1_count": null,
      "h1_length": null,
      "h1_tag": null,
      "h2_tag": null,
      "h3_tag": null,
      "header_content_type": null,
      "header_noarchive": null,
      "header_nofollow": null,
      "header_noindex": null,
      "header_noodp": null,
      "header_nosnippet": null,
      "header_noydir": null,
      "hreflang_combination": null,
      "hreflang_on_page": null,
      "hreflang_on_header": null,
      "hreflang_on_sitemap": null,
      "hreflang_url_count": null,
      "hsts": null,
      "html_size": null,
      "indexable": null,
      "internal": null,
      "internal_links": null,
      "is_redirect": null,
      "is_redirect_loop": null,
      "language": null,
      "level": null,
      "link_nofollow": null,
      "links_in": null,
      "links_out": null,
      "meta_charset": null,
      "meta_content_type": null,
      "meta_disabled_sitelinks": null,
      "meta_noarchive": null,
      "meta_nofollow": null,
      "meta_noindex": null,
      "meta_noodp": null,
      "meta_nosnippet": null,
      "meta_noydir": null,
      "meta_redirect": null,
      "no_redirects": null,
      "noarchive": null,
      "nofollow": null,
      "noindex": null,
      "noodp": null,
      "nosnippet": null,
      "noydir": null,
      "og_description": null,
      "og_image": null,
      "og_locale": null,
      "og_site_name": null,
      "og_title": null,
      "og_type": null,
      "og_url": null,
      "page_1": null,
      "page_title": null,
      "page_title_length": null,
      "page_title_length_px": null,
      "paginated_page": null,
      "primary_url": null,
      "primary_url_digest": null,
      "primary_url_non_indexable": null,
      "redirect_chain": null,
      "redirected_to": null,
      "redirected_to_url_digest": null,
      "redirected_to_request_status": null,
      "rel_alternate_header_url": null,
      "rel_alternate_html_url": null,
      "rel_alternate_url": null,
      "rel_alternate_url_is_consistent": null,
      "rel_next_header_url": null,
      "rel_next_html_url": null,
      "rel_next_url": null,
      "rel_next_url_is_consistent": null,
      "rel_prev_header_url": null,
      "rel_prev_html_url": null,
      "rel_prev_url": null,
      "rel_prev_url_is_consistent": null,
      "request_duration": null,
      "request_status": null,
      "robots_noindex": null,
      "twitter_card": null,
      "twitter_description": null,
      "twitter_description_length": null,
      "twitter_image": null,
      "twitter_site": null,
      "twitter_title": null,
      "url": null,
      "url_digest": "e77DS1P3wj4slksSpe+1tloS01m9iznHhRzXK9kTZ2mykqGv67w3fkhUTON39kvVuQk68Wa9No4yzS0B763irQ==",
      "url_alias": null,
      "url_length": null,
      "valid_open_graph": null,
      "valid_twitter_card": null,
      "vary_user_agent": null,
      "viewport": null,
      "word_count": null
    }
  }
]
Response  422
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "errors": [
    {
      "field": "filter",
      "code": "invalid",
      "message": "Filter is not valid"
    }
  ]
}
Response  503
Response  404

Report Row

Read Report Row
GET/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}/reports/{report_id}/report_rows/{report_row_id}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id/reports/report_id/report_rows/report_row_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

report_id
integer (required) 

Report ID

report_row_id
integer (required) 

Report Row ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_crawl_href": "/accounts/123789/projects/152511/crawls/166183",
  "_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/report_rows/1",
  "_project_href": "/accounts/123789/projects/152511",
  "_report_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656",
  "_data_pages_href": {
    "url": "/accounts/123789/projects/152511/crawls/166183/pages/e77DS1P3wj4slksSpe-1tloS01m9iznHhRzXK9kTZ2mykqGv67w3fkhUTON39kvVuQk68Wa9No4yzS0B763irQ=="
  },
  "data": {
    "ga_avg_page_load_time": null,
    "ga_avg_time_on_page": null,
    "ga_pageviews_per_visits": null,
    "ga_visit_bounce_rate": null,
    "ga_visits": null,
    "canonical_header_url": null,
    "canonical_html_url": null,
    "canonical_url": null,
    "canonical_url_digest": null,
    "canonical_url_is_consistent": null,
    "content_html_ratio": null,
    "content_size": null,
    "crawl_id": 166183,
    "custom_extraction_1": null,
    "custom_extraction_2": null,
    "custom_extraction_3": null,
    "custom_extraction_4": null,
    "custom_extraction_5": null,
    "custom_extraction_6": null,
    "custom_extraction_7": null,
    "custom_extraction_8": null,
    "custom_extraction_9": null,
    "custom_extraction_10": null,
    "custom_extraction_11": null,
    "custom_extraction_12": null,
    "custom_extraction_13": null,
    "custom_extraction_14": null,
    "custom_extraction_15": null,
    "custom_extraction_16": null,
    "custom_extraction_17": null,
    "custom_extraction_18": null,
    "custom_extraction_19": null,
    "custom_extraction_20": null,
    "custom_extraction_21": null,
    "custom_extraction_22": null,
    "custom_extraction_23": null,
    "custom_extraction_24": null,
    "custom_extraction_25": null,
    "custom_extraction_26": null,
    "custom_extraction_27": null,
    "custom_extraction_28": null,
    "custom_extraction_29": null,
    "custom_extraction_30": null,
    "deeprank": null,
    "description": null,
    "description_length": null,
    "description_length_px": null,
    "disallowed_page": null,
    "duplicate_body": null,
    "duplicate_body_non_indexable": null,
    "duplicate_body_primary_url": null,
    "duplicate_body_primary_url_digest": null,
    "duplicate_description": null,
    "duplicate_description_non_indexable": null,
    "duplicate_description_primary_url": null,
    "duplicate_description_primary_url_digest": null,
    "duplicate_page": null,
    "duplicate_page_non_indexable": null,
    "duplicate_title": null,
    "duplicate_title_non_indexable": null,
    "duplicate_title_primary_url": null,
    "duplicate_title_primary_url_digest": null,
    "encoding_header": null,
    "external_links": null,
    "fb_admins": null,
    "fb_app_id": null,
    "found_at": null,
    "found_at_sitemap": null,
    "h1_count": null,
    "h1_length": null,
    "h1_tag": null,
    "h2_tag": null,
    "h3_tag": null,
    "header_content_type": null,
    "header_noarchive": null,
    "header_nofollow": null,
    "header_noindex": null,
    "header_noodp": null,
    "header_nosnippet": null,
    "header_noydir": null,
    "hreflang_combination": null,
    "hreflang_on_page": null,
    "hreflang_on_header": null,
    "hreflang_on_sitemap": null,
    "hreflang_url_count": null,
    "hsts": null,
    "html_size": null,
    "indexable": null,
    "internal": null,
    "internal_links": null,
    "is_redirect": null,
    "is_redirect_loop": null,
    "language": null,
    "level": null,
    "link_nofollow": null,
    "links_in": null,
    "links_out": null,
    "meta_charset": null,
    "meta_content_type": null,
    "meta_disabled_sitelinks": null,
    "meta_noarchive": null,
    "meta_nofollow": null,
    "meta_noindex": null,
    "meta_noodp": null,
    "meta_nosnippet": null,
    "meta_noydir": null,
    "meta_redirect": null,
    "no_redirects": null,
    "noarchive": null,
    "nofollow": null,
    "noindex": null,
    "noodp": null,
    "nosnippet": null,
    "noydir": null,
    "og_description": null,
    "og_image": null,
    "og_locale": null,
    "og_site_name": null,
    "og_title": null,
    "og_type": null,
    "og_url": null,
    "page_1": null,
    "page_title": null,
    "page_title_length": null,
    "page_title_length_px": null,
    "paginated_page": null,
    "primary_url": null,
    "primary_url_digest": null,
    "primary_url_non_indexable": null,
    "redirect_chain": null,
    "redirected_to": null,
    "redirected_to_url_digest": null,
    "redirected_to_request_status": null,
    "rel_alternate_header_url": null,
    "rel_alternate_html_url": null,
    "rel_alternate_url": null,
    "rel_alternate_url_is_consistent": null,
    "rel_next_header_url": null,
    "rel_next_html_url": null,
    "rel_next_url": null,
    "rel_next_url_is_consistent": null,
    "rel_prev_header_url": null,
    "rel_prev_html_url": null,
    "rel_prev_url": null,
    "rel_prev_url_is_consistent": null,
    "request_duration": null,
    "request_status": null,
    "robots_noindex": null,
    "twitter_card": null,
    "twitter_description": null,
    "twitter_description_length": null,
    "twitter_image": null,
    "twitter_site": null,
    "twitter_title": null,
    "url": null,
    "url_digest": "e77DS1P3wj4slksSpe+1tloS01m9iznHhRzXK9kTZ2mykqGv67w3fkhUTON39kvVuQk68Wa9No4yzS0B763irQ==",
    "url_alias": null,
    "url_length": null,
    "valid_open_graph": null,
    "valid_twitter_card": null,
    "vary_user_agent": null,
    "viewport": null,
    "word_count": null
  }
}
Response  503
Response  404

Report Row Errors

Report Downloads

Report Downloads

List Report Downloads
GET/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}/reports/{report_id}/downloads

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id/reports/report_id/downloads
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

report_id
string (required) 

Report ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
Link: </accounts/123789/projects/152511/crawls/166183/reports/5656/downloads?per_page=2&page=1>; rel='first', </accounts/123789/projects/152511/crawls/166183/reports/5656/downloads?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
Body
[
  {
    "_account_href": "/accounts/123789",
    "_crawl_href": "/accounts/123789/projects/152511/crawls/166183",
    "_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/downloads/7465",
    "_project_href": "/accounts/123789/projects/152511",
    "_report_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656",
    "_report_template_href": "/datasources/crawl_urls/report_templates/62711",
    "_report_type_href": "/report_types/basic",
    "id": 123,
    "report_file": "http://example.com/download",
    "report_type": "basic",
    "status": "generated",
    "filter": null
  }
]
Response  404

Create Report Download --NOTEST--
POST/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}/reports/{report_id}/downloads

Example URI

POST https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id/reports/report_id/downloads
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

report_id
string (required) 

Report ID

Request
HideShow
Headers
Content-Type: application/json
Response  201
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_crawl_href": "/accounts/123789/projects/152511/crawls/166183",
  "_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/downloads/7465",
  "_project_href": "/accounts/123789/projects/152511",
  "_report_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656",
  "_report_template_href": "/datasources/crawl_urls/report_templates/62711",
  "_report_type_href": "/report_types/basic",
  "id": 123,
  "report_file": null,
  "report_type": "basic",
  "status": "generating",
  "filter": null
}
Response  409
Response  422
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "errors": [
    {
      "code": "invalid",
      "field": "filter",
      "message": "Filter is not valid"
    }
  ]
}
Response  404

Report Download

Read Report Download
GET/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}/reports/{report_id}/downloads/{report_download_id}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id/reports/report_id/downloads/report_download_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

report_id
string (required) 

Report ID

report_download_id
integer (required) 

Report Download ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_crawl_href": "/accounts/123789/projects/152511/crawls/166183",
  "_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/downloads/7465",
  "_project_href": "/accounts/123789/projects/152511",
  "_report_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656",
  "_report_template_href": "/datasources/crawl_urls/report_templates/62711",
  "_report_type_href": "/report_types/basic",
  "id": 123,
  "report_file": "http://example.com/download",
  "report_type": "basic",
  "status": "generated",
  "filter": null
}
Response  404

Delete Report Download
DELETE/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}/reports/{report_id}/downloads/{report_download_id}

Example URI

DELETE https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id/reports/report_id/downloads/report_download_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

report_id
string (required) 

Report ID

report_download_id
integer (required) 

Report Download ID

Response  204
HideShow
Headers
X-Content-Type-Options: nosniff
Response  404

Report Download Errors

Pages

Pages

Read Pages
GET/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}/pages/{page_id}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id/pages/page_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

page_id
string (required) 

Page ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_crawl_href": "/accounts/123789/projects/152511/crawls/166183",
  "_href": "/accounts/123789/projects/152511/crawls/166183/pages/dVhCJaL06EyqHYMP9hlc2vD2Z9awv5Kn_PGGi9Ci10boLYctQm4N8H4dXsYOZTTGtpOpYy_1SolH3r3U3wLn5g==",
  "_project_href": "/accounts/123789/projects/152511",
  "url": null,
  "reports": [],
  "request_status": null,
  "indexable": null,
  "duplicate_page": null,
  "level": null,
  "deeprank": null,
  "title": null
}
Response  404

Locations

Locations

List Locations
GET/locations

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/locations
Response  200
HideShow
Headers
Content-Type: application/json
Link: </locations?per_page=2&page=1>; rel='first', </locations?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
X-Records: 2
Body
[
  {
    "_href": "/locations/cod",
    "code": "cod",
    "type": "dynamic_ip",
    "name": "Cod",
    "enabled": false,
    "ip_addresses": []
  },
  {
    "_href": "/locations/dip",
    "code": "dip",
    "type": "Dynamic",
    "enabled": true,
    "ip_addresses": []
  }
]

Location

Read Location
GET/locations/{location_code}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/locations/location_code
URI Parameters
HideShow
location_code
string (required) 

Location Code

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_href": "/locations/cod",
  "code": "cod",
  "type": "dynamic_ip",
  "name": "Cod",
  "enabled": false,
  "ip_addresses": []
}
Response  404

Static Locations

Static Locations

List Static Locations
GET/static_locations

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/static_locations
Response  200
HideShow
Headers
Content-Type: application/json
Link: </static_locations?per_page=2&page=1>; rel='first', </static_locations?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
X-Records: 1
Body
[
  {
    "_href": "/static_locations/sip-pl-222.222.222.222",
    "code": "sip-pl-222.222.222.222",
    "enabled": true,
    "territory": "PL",
    "address": "222.222.222.222"
  }
]

Static Location

Read Static Location
GET/static_locations/{static_location_code}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/static_locations/static_location_code
URI Parameters
HideShow
static_location_code
string (required) 

Static Location Code

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_href": "/static_locations/sip-pl-222.222.222.222",
  "code": "sip-pl-222.222.222.222",
  "enabled": true,
  "territory": "PL",
  "address": "222.222.222.222"
}
Response  404

Domain Responses

Read Domain Responses --USER--
GET/domain_responses

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/domain_responses
Request
HideShow
Headers
Content-Type: application/json
Body
{
  "url": "somefakedomainblabla.com"
}
Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
[
  {
    "domain": "https://www.somefakedomainblabla.com/",
    "response": null
  },
  {
    "domain": "http://www.somefakedomainblabla.com/",
    "response": null
  },
  {
    "domain": "https://somefakedomainblabla.com/",
    "response": null
  },
  {
    "domain": "http://somefakedomainblabla.com/",
    "response": null
  }
]

Support Email

Send Support Email --USER--
POST/send_support_email

Example URI

POST https://prod-1-dc-api-oopeix3r.deepcrawl.com/send_support_email
Request
HideShow
Headers
Content-Type: application/json
Body
{
  "username": "example_username",
  "email": "user_email@example.com",
  "url": "/some_deepcrawl_url",
  "message": "Some message"
}
Response  204
HideShow
Headers
X-Content-Type-Options: nosniff

Issues

Issues

List Issues
GET/accounts/{account_id}/projects/{project_id}/issues

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/issues
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
Link: </accounts/123789/projects/152511/issues?per_page=2&page=1>; rel='first', </accounts/123789/projects/152511/issues?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
Body
[
  {
    "_account_href": "/accounts/123789",
    "_href": "/accounts/123789/projects/152511/issues/26872",
    "_project_href": "/accounts/123789/projects/152511",
    "_issue_notes_href": "/accounts/123789/projects/152511/issues/26872/issue_notes",
    "actions": null,
    "assigned_to": null,
    "created_at": "1952-03-11T00:00:00Z",
    "deadline_at": null,
    "description": "There is a bug in my soup!!",
    "discovered_at": null,
    "dismissed": false,
    "filters": null,
    "fixed_at": null,
    "identified": null,
    "priority": "medium",
    "remaining": null,
    "report_template": null,
    "report_type": null,
    "title": "Excuse me sir, I have an issue",
    "trend": null,
    "site_primary": "http://example.com/",
    "_crawls_finished_last_href": "/accounts/123789/projects/152511/crawls/166183",
    "_reports_last_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656"
  }
]
Response  404

Create Issue
POST/accounts/{account_id}/projects/{project_id}/issues

Example URI

POST https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/issues
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

Request
HideShow
Headers
Content-Type: application/json
Body
{
  "title": "Excuse me sir, I have an issue",
  "description": "There is a bug in my soup!!",
  "priority": "medium"
}
Response  201
HideShow
Headers
Content-Type: application/json
Location: /accounts/123789/projects/152511/issues/26872
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_href": "/accounts/123789/projects/152511/issues/26872",
  "_project_href": "/accounts/123789/projects/152511",
  "_issue_notes_href": "/accounts/123789/projects/152511/issues/26872/issue_notes",
  "actions": null,
  "assigned_to": null,
  "created_at": "1952-03-11T00:00:00Z",
  "deadline_at": null,
  "description": "There is a bug in my soup!!",
  "discovered_at": null,
  "dismissed": false,
  "filters": null,
  "fixed_at": null,
  "identified": null,
  "priority": "medium",
  "remaining": null,
  "report_template": null,
  "report_type": null,
  "title": "Excuse me sir, I have an issue",
  "trend": [
    [
      "1952-03-11T00:00:00+00:00",
      null
    ]
  ],
  "site_primary": "http://example.com/",
  "_crawls_finished_last_href": "/accounts/123789/projects/152511/crawls/166183"
}
Response  422
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "errors": [
    {
      "field": "description",
      "code": "missing_field",
      "message": "Description is missing"
    },
    {
      "field": "priority",
      "code": "missing_field",
      "message": "Priority is missing"
    },
    {
      "field": "title",
      "code": "missing_field",
      "message": "Title is missing"
    }
  ]
}
Response  409
Response  404

Issue

Read Issue
GET/accounts/{account_id}/projects/{project_id}/issues/{issue_id}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/issues/issue_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

issue_id
integer (required) 

Issue ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_href": "/accounts/123789/projects/152511/issues/26872",
  "_project_href": "/accounts/123789/projects/152511",
  "_issue_notes_href": "/accounts/123789/projects/152511/issues/26872/issue_notes",
  "actions": null,
  "assigned_to": null,
  "created_at": "1952-03-11T00:00:00Z",
  "deadline_at": null,
  "description": "There is a bug in my soup!!",
  "discovered_at": null,
  "dismissed": false,
  "filters": null,
  "fixed_at": null,
  "identified": null,
  "priority": "medium",
  "remaining": null,
  "report_template": null,
  "report_type": null,
  "title": "Excuse me sir, I have an issue",
  "trend": null,
  "site_primary": "http://example.com/",
  "_crawls_finished_last_href": "/accounts/123789/projects/152511/crawls/166183",
  "_reports_last_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656"
}
Response  404

Update Issue
PATCH/accounts/{account_id}/projects/{project_id}/issues/{issue_id}

Example URI

PATCH https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/issues/issue_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

issue_id
integer (required) 

Issue ID

Request
HideShow
Headers
Content-Type: application/json
Body
{
  "description": "It may even be a beetle :-o"
}
Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_href": "/accounts/123789/projects/152511/issues/26872",
  "_project_href": "/accounts/123789/projects/152511",
  "_issue_notes_href": "/accounts/123789/projects/152511/issues/26872/issue_notes",
  "actions": null,
  "assigned_to": null,
  "created_at": "1952-03-11T00:00:00Z",
  "deadline_at": null,
  "description": "It may even be a beetle :-o",
  "discovered_at": null,
  "dismissed": false,
  "filters": null,
  "fixed_at": null,
  "identified": null,
  "priority": "medium",
  "remaining": null,
  "report_template": null,
  "report_type": null,
  "title": "Excuse me sir, I have an issue",
  "trend": null,
  "site_primary": "http://example.com/",
  "_crawls_finished_last_href": "/accounts/123789/projects/152511/crawls/166183",
  "_reports_last_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656"
}
Response  422
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "errors": [
    {
      "field": "description",
      "code": "missing_field",
      "message": "Description is missing"
    },
    {
      "field": "priority",
      "code": "missing_field",
      "message": "Priority is missing"
    },
    {
      "field": "title",
      "code": "missing_field",
      "message": "Title is missing"
    }
  ]
}
Response  404

Delete Issue
DELETE/accounts/{account_id}/projects/{project_id}/issues/{issue_id}

Example URI

DELETE https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/issues/issue_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

issue_id
integer (required) 

Issue ID

Response  204
HideShow
Headers
X-Content-Type-Options: nosniff
Response  422
Response  404

Issue Errors

Account Issues

Account Issues

List Account Issues
GET/accounts/{account_id}/issues

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/issues
URI Parameters
HideShow
account_id
integer (required) 

Account ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
Link: </accounts/123789/issues?per_page=2&page=1>; rel='first', </accounts/123789/issues?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
Body
[
  {
    "_account_href": "/accounts/123789",
    "_href": "/accounts/123789/projects/152511/issues/26872",
    "_project_href": "/accounts/123789/projects/152511",
    "_issue_notes_href": "/accounts/123789/projects/152511/issues/26872/issue_notes",
    "actions": null,
    "assigned_to": null,
    "created_at": "1952-03-11T00:00:00Z",
    "deadline_at": null,
    "description": "There is a bug in my soup!!",
    "discovered_at": null,
    "dismissed": false,
    "filters": null,
    "fixed_at": null,
    "identified": null,
    "priority": "medium",
    "remaining": null,
    "report_template": null,
    "report_type": null,
    "title": "Excuse me sir, I have an issue",
    "trend": null,
    "site_primary": "http://example.com/",
    "_crawls_finished_last_href": "/accounts/123789/projects/152511/crawls/166183",
    "_reports_last_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656"
  }
]
Response  404

Crawl Downloads

Crawl Downloads

List Crawl Downloads
GET/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}/downloads

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id/downloads
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
Link: </accounts/123789/projects/152511/crawls/166183/downloads?per_page=2&page=1>; rel='first', </accounts/123789/projects/152511/crawls/166183/downloads?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
Body
[
  {
    "_account_href": "/accounts/123789",
    "_crawl_href": "/accounts/123789/projects/152511/crawls/166183",
    "_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656/downloads/7465",
    "_project_href": "/accounts/123789/projects/152511",
    "_report_href": "/accounts/123789/projects/152511/crawls/166183/reports/5656",
    "_report_template_href": "/datasources/crawl_urls/report_templates/62711",
    "_report_type_href": "/report_types/basic",
    "report_file": "http://example.com/download",
    "report_type": "basic",
    "status": "generated",
    "filter": null
  }
]
Response  404

User Agents

User Agents

List User Agents
GET/user_agents

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/user_agents
Response  200
HideShow
Headers
Content-Type: application/json
Link: </user_agents?per_page=2&page=1>; rel='first', </user_agents?per_page=2&page=2>; rel='next', </user_agents?per_page=2&page=9>; rel='last'
X-Content-Type-Options: nosniff
X-Records: 17
Body
[
  {
    "_href": "/user_agents/bingbot",
    "code": "bingbot",
    "name": "Bingbot",
    "user_agent_strings": [
      "Mozilla/5.0 (compatible; bingbot/2.0; +https://deepcrawl.com/bot)"
    ]
  },
  {
    "_href": "/user_agents/bingbot-mobile",
    "code": "bingbot-mobile",
    "name": "Bingbot Mobile",
    "user_agent_strings": [
      "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b https://deepcrawl.com/bot"
    ]
  }
]

User Agent

Read User Agent
GET/user_agents/{user_agent_code}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/user_agents/user_agent_code
URI Parameters
HideShow
user_agent_code
string (required) 

User Agent Code

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_href": "/user_agents/custom",
  "code": "custom",
  "name": "Custom",
  "user_agent_strings": [
    "Mozilla/5.0 (compatible; custom/2.0; +https://deepcrawl.com/bot)"
  ]
}
Response  404

Crawl Sitemaps

Crawl Sitemaps

List Crawl Sitemaps
GET/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}/sitemaps

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id/sitemaps
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
Link: </accounts/123789/projects/152511/crawls/166183/sitemaps?per_page=2&page=1>; rel='first', </accounts/123789/projects/152511/crawls/166183/sitemaps?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
Body
[
  {
    "_account_href": "/accounts/123789",
    "_crawl_href": "/accounts/123789/projects/152511/crawls/166183",
    "_href": "/accounts/123789/projects/152511/crawls/166183/sitemaps/dVhCJaL06EyqHYMP9hlc2vD2Z9awv5Kn_PGGi9Ci10boLYctQm4N8H4dXsYOZTTGtpOpYy_1SolH3r3U3wLn5g==",
    "_project_href": "/accounts/123789/projects/152511",
    "links_out": 10,
    "level": null,
    "parent": null,
    "parent_digest": null,
    "child": "http://example.com",
    "child_digest": "dVhCJaL06EyqHYMP9hlc2vD2Z9awv5Kn_PGGi9Ci10boLYctQm4N8H4dXsYOZTTGtpOpYy_1SolH3r3U3wLn5g=="
  }
]
Response  202
Response  503
Response  404

Crawl Sitemap

Read Crawl Sitemap
GET/accounts/{account_id}/projects/{project_id}/crawls/{crawl_id}/sitemaps/{sitemap_code}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/crawls/crawl_id/sitemaps/sitemap_code
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

crawl_id
integer (required) 

Crawl ID

sitemap_code
string (required) 

Sitemap Code

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_crawl_href": "/accounts/123789/projects/152511/crawls/166183",
  "_href": "/accounts/123789/projects/152511/crawls/166183/sitemaps/dVhCJaL06EyqHYMP9hlc2vD2Z9awv5Kn_PGGi9Ci10boLYctQm4N8H4dXsYOZTTGtpOpYy_1SolH3r3U3wLn5g==",
  "_project_href": "/accounts/123789/projects/152511",
  "links_out": 10,
  "level": null,
  "parent": null,
  "parent_digest": null,
  "child": "http://example.com",
  "child_digest": "dVhCJaL06EyqHYMP9hlc2vD2Z9awv5Kn_PGGi9Ci10boLYctQm4N8H4dXsYOZTTGtpOpYy_1SolH3r3U3wLn5g=="
}
Response  503
Response  404

Project Sitemaps

Project Sitemaps

List Project Sitemaps
GET/accounts/{account_id}/projects/{project_id}/sitemaps

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/sitemaps
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
X-Content-Type-Options: nosniff
Body
[
  {
    "_href": "/accounts/123789/projects/152511/sitemaps/dVhCJaL06EyqHYMP9hlc2vD2Z9awv5Kn_PGGi9Ci10boLYctQm4N8H4dXsYOZTTGtpOpYy_1SolH3r3U3wLn5g==",
    "links_out": 10,
    "level": null,
    "url": "http://example.com",
    "enabled": true,
    "children": [],
    "type": "discovered"
  }
]
Response  202
Response  503
Response  404

Create Project Sitemap
POST/accounts/{account_id}/projects/{project_id}/sitemaps

Example URI

POST https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/sitemaps
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

Request
HideShow
Headers
Content-Type: application/json
Body
{
  "url": "http://example.com/sitemap.xml"
}
Response  201
HideShow
Headers
Content-Type: application/json
Location: /accounts/123789/projects/152511/sitemaps/frIDRy4URyIpZ-cH6Byt_JlyAfeBDsooTBZsk5rrAJrIKQOe6zEKEZKydcYNpsjXxLyu0pSEz5-noAZnuT_lyA==
X-Content-Type-Options: nosniff
Body
{
  "_href": "/accounts/123789/projects/152511/sitemaps/frIDRy4URyIpZ-cH6Byt_JlyAfeBDsooTBZsk5rrAJrIKQOe6zEKEZKydcYNpsjXxLyu0pSEz5-noAZnuT_lyA==",
  "links_out": null,
  "level": null,
  "parent": null,
  "url": "http://example.com/sitemap.xml",
  "enabled": true,
  "type": "custom"
}
Response  404

Project Sitemap

Update Project Sitemap
PATCH/accounts/{account_id}/projects/{project_id}/{sitemap_code}

Example URI

PATCH https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/sitemap_code
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

sitemap_code
string (required) 

Sitemap Code

Request
HideShow
Headers
Content-Type: application/json
Body
{
  "enabled": "false"
}
Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_href": "/accounts/123789/projects/152511/sitemaps/dVhCJaL06EyqHYMP9hlc2vD2Z9awv5Kn_PGGi9Ci10boLYctQm4N8H4dXsYOZTTGtpOpYy_1SolH3r3U3wLn5g==",
  "links_out": 10,
  "level": null,
  "url": "http://example.com",
  "enabled": false,
  "parent": null,
  "type": "discovered"
}
Response  503
Response  404

Delete Project Sitemap
DELETE/accounts/{account_id}/projects/{project_id}/{sitemap_code}

Example URI

DELETE https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/sitemap_code
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

sitemap_code
string (required) 

Sitemap Code

Response  204
HideShow
Headers
X-Content-Type-Options: nosniff
Response  404

Schedules

Schedules

List Schedules
GET/accounts/{account_id}/projects/{project_id}/schedules

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/schedules
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
Link: </accounts/123789/projects/152511/schedules?per_page=2&page=1>; rel='first', </accounts/123789/projects/152511/schedules?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
Body
[
  {
    "_account_href": "/accounts/123789",
    "_href": "/accounts/123789/projects/152511/schedules/12673",
    "_project_href": "/accounts/123789/projects/152511",
    "_schedule_frequency_href": "/schedule_frequencies/one_time",
    "next_run_time": "2015-08-01T08:00:00+00:00",
    "next_run_time_local": "2015-08-01T08:00:00+00:00",
    "schedule_frequency": "one_time"
  }
]
Response  404

Create Schedule
POST/accounts/{account_id}/projects/{project_id}/schedules

Example URI

POST https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/schedules
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

Request
HideShow
Headers
Content-Type: application/json
Body
{
  "next_run_time": "2015-08-01T08:00:00+0000",
  "schedule_frequency": "one_time"
}
Response  201
HideShow
Headers
Content-Type: application/json
Location: /accounts/123789/projects/152511/schedules/12673
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_href": "/accounts/123789/projects/152511/schedules/12673",
  "_project_href": "/accounts/123789/projects/152511",
  "_schedule_frequency_href": "/schedule_frequencies/one_time",
  "next_run_time": "2015-08-01T08:00:00+00:00",
  "next_run_time_local": "2015-08-01T08:00:00+00:00",
  "schedule_frequency": "one_time"
}
Response  404

Schedule

Read Schedule
GET/accounts/{account_id}/projects/{project_id}/schedules/{schedule_id}

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/schedules/schedule_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

schedule_id
integer (required) 

Schedule ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_href": "/accounts/123789/projects/152511/schedules/12673",
  "_project_href": "/accounts/123789/projects/152511",
  "_schedule_frequency_href": "/schedule_frequencies/one_time",
  "next_run_time": "2015-08-01T08:00:00+00:00",
  "next_run_time_local": "2015-08-01T08:00:00+00:00",
  "schedule_frequency": "one_time"
}
Response  404

Update Schedule
PATCH/accounts/{account_id}/projects/{project_id}/schedules/{schedule_id}

Example URI

PATCH https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/schedules/schedule_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

schedule_id
integer (required) 

Schedule ID

Request
HideShow
Headers
Content-Type: application/json
Body
{
  "next_run_time": "2015-08-02T08:00+0000"
}
Response  200
HideShow
Headers
Content-Type: application/json
X-Content-Type-Options: nosniff
Body
{
  "_account_href": "/accounts/123789",
  "_href": "/accounts/123789/projects/152511/schedules/12673",
  "_project_href": "/accounts/123789/projects/152511",
  "_schedule_frequency_href": "/schedule_frequencies/one_time",
  "next_run_time": "2015-08-02T08:00:00+00:00",
  "next_run_time_local": "2015-08-02T08:00:00+00:00",
  "schedule_frequency": "one_time"
}
Response  404

Delete Schedule
DELETE/accounts/{account_id}/projects/{project_id}/schedules/{schedule_id}

Example URI

DELETE https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/projects/project_id/schedules/schedule_id
URI Parameters
HideShow
account_id
integer (required) 

Account ID

project_id
integer (required) 

Project ID

schedule_id
integer (required) 

Schedule ID

Response  204
HideShow
Headers
X-Content-Type-Options: nosniff
Response  409
Response  404

Account Schedules

Schedules for Account

List Schedules for Account
GET/accounts/{account_id}/schedules

Example URI

GET https://prod-1-dc-api-oopeix3r.deepcrawl.com/accounts/account_id/schedules
URI Parameters
HideShow
account_id
integer (required) 

Account ID

Response  200
HideShow
Headers
Content-Type: application/json
X-Records: 1
Link: </accounts/123789/schedules?per_page=2&page=1>; rel='first', </accounts/123789/schedules?per_page=2&page=1>; rel='last'
X-Content-Type-Options: nosniff
Body
[
  {
    "_account_href": "/accounts/123789",
    "_href": "/accounts/123789/projects/152511/schedules/12673",
    "_project_href": "/accounts/123789/projects/152511",
    "_schedule_frequency_href": "/schedule_frequencies/one_time",
    "next_run_time": "2015-08-01T08:00:00+00:00",
    "next_run_time_local": "2015-08-01T08:00:00+00:00",
    "schedule_frequency": "one_time"
  }
]
Response  404

Generated by aglio on 15 Sep 2016