# Request<!-- -->

Represents a request in the Crawlee framework, containing the necessary information for crawling operations.

The `Request` class is one of the core components in Crawlee, utilized by various components such as request providers, HTTP clients, crawlers, and more. It encapsulates the essential data for executing web requests, including the URL, HTTP method, headers, payload, and user data. The user data allows custom information to be stored and persisted throughout the request lifecycle, including its retries.

Key functionalities include managing the request's identifier (`id`), unique key (`unique_key`) that is used for request deduplication, controlling retries, handling state management, and enabling configuration for session rotation and proxy handling.

The recommended way to create a new instance is by using the `Request.from_url` constructor, which automatically generates a unique key and identifier based on the URL and request parameters.

### Usage

```
from crawlee import Request



request = Request.from_url('https://crawlee.dev')
```

### Hierarchy

* *Request*
  * [RequestWithLock](https://crawlee.dev/python/api/class/RequestWithLock)

## Index[](#Index)

### Methods

* [crawl_depth](https://docs.apify.com/sdk/python/reference/class/Request.md#crawl_depth)
* [enqueue_strategy](https://docs.apify.com/sdk/python/reference/class/Request.md#enqueue_strategy)
* [forefront](https://docs.apify.com/sdk/python/reference/class/Request.md#forefront)
* [from_url](https://docs.apify.com/sdk/python/reference/class/Request.md#from_url)
* [get_query_param_from_url](https://docs.apify.com/sdk/python/reference/class/Request.md#get_query_param_from_url)
* [last_proxy_tier](https://docs.apify.com/sdk/python/reference/class/Request.md#last_proxy_tier)
* [session_rotation_count](https://docs.apify.com/sdk/python/reference/class/Request.md#session_rotation_count)
* [state](https://docs.apify.com/sdk/python/reference/class/Request.md#state)

### Properties

* [crawl_depth](https://docs.apify.com/sdk/python/reference/class/Request.md#crawl_depth)
* [crawlee_data](https://docs.apify.com/sdk/python/reference/class/Request.md#crawlee_data)
* [enqueue_strategy](https://docs.apify.com/sdk/python/reference/class/Request.md#enqueue_strategy)
* [forefront](https://docs.apify.com/sdk/python/reference/class/Request.md#forefront)
* [handled_at](https://docs.apify.com/sdk/python/reference/class/Request.md#handled_at)
* [label](https://docs.apify.com/sdk/python/reference/class/Request.md#label)
* [last_proxy_tier](https://docs.apify.com/sdk/python/reference/class/Request.md#last_proxy_tier)
* [loaded_url](https://docs.apify.com/sdk/python/reference/class/Request.md#loaded_url)
* [max_retries](https://docs.apify.com/sdk/python/reference/class/Request.md#max_retries)
* [method](https://docs.apify.com/sdk/python/reference/class/Request.md#method)
* [model_config](https://docs.apify.com/sdk/python/reference/class/Request.md#model_config)
* [no_retry](https://docs.apify.com/sdk/python/reference/class/Request.md#no_retry)
* [payload](https://docs.apify.com/sdk/python/reference/class/Request.md#payload)
* [retry_count](https://docs.apify.com/sdk/python/reference/class/Request.md#retry_count)
* [session_id](https://docs.apify.com/sdk/python/reference/class/Request.md#session_id)
* [session_rotation_count](https://docs.apify.com/sdk/python/reference/class/Request.md#session_rotation_count)
* [state](https://docs.apify.com/sdk/python/reference/class/Request.md#state)
* [unique_key](https://docs.apify.com/sdk/python/reference/class/Request.md#unique_key)
* [url](https://docs.apify.com/sdk/python/reference/class/Request.md#url)
* [was_already_handled](https://docs.apify.com/sdk/python/reference/class/Request.md#was_already_handled)

## Methods<!-- -->[](#Methods)

### [](#crawl_depth)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L373)crawl_depth

* **crawl_depth**(new_value): None

- #### Parameters

  * ##### new_value: int

  #### Returns None

### [](#enqueue_strategy)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L405)enqueue_strategy

* **enqueue_strategy**(new_enqueue_strategy): None

- #### Parameters

  * ##### new_enqueue_strategy: [EnqueueStrategy](https://crawlee.dev/python/api#EnqueueStrategy)

  #### Returns None

### [](#forefront)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L423)forefront

* **forefront**(new_value): None

- #### Parameters

  * ##### new_value: bool

  #### Returns None

### [](#from_url)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L244)from_url

* **from_url**(url, \*, method, headers, payload, label, session_id, unique_key, keep_url_fragment, use_extended_unique_key, always_enqueue, enqueue_strategy, max_retries, kwargs): Self

- Create a new `Request` instance from a URL.

  This is recommended constructor for creating new `Request` instances. It generates a `Request` object from a given URL with additional options to customize HTTP method, payload, unique key, and other request properties. If no `unique_key` or `id` is provided, they are computed automatically based on the URL, method and payload. It depends on the `keep_url_fragment` and `use_extended_unique_key` flags.

  ***

  #### Parameters

  * ##### url: str

    The URL of the request.

  * ##### optionalkeyword-onlymethod: [HttpMethod](https://crawlee.dev/python/api#HttpMethod) = <!-- -->'GET'

    The HTTP method of the request.

  * ##### optionalkeyword-onlyheaders: ([HttpHeaders](https://crawlee.dev/python/api/class/HttpHeaders) | dict\[str, str]) | None = <!-- -->None

    The HTTP headers of the request.

  * ##### optionalkeyword-onlypayload: ([HttpPayload](https://crawlee.dev/python/api#HttpPayload) | str) | None = <!-- -->None

    The data to be sent as the request body. Typically used with 'POST' or 'PUT' requests.

  * ##### optionalkeyword-onlylabel: str | None = <!-- -->None

    A custom label to differentiate between request types. This is stored in `user_data`, and it is used for request routing (different requests go to different handlers).

  * ##### optionalkeyword-onlysession_id: str | None = <!-- -->None

    ID of a specific `Session` to which the request will be strictly bound. If the session becomes unavailable when the request is processed, a `RequestCollisionError` will be raised.

  * ##### optionalkeyword-onlyunique_key: str | None = <!-- -->None

    A unique key identifying the request. If not provided, it is automatically computed based on the URL and other parameters. Requests with the same `unique_key` are treated as identical.

  * ##### optionalkeyword-onlykeep_url_fragment: bool = <!-- -->False

    Determines whether the URL fragment (e.g., `#section`) should be included in the `unique_key` computation. This is only relevant when `unique_key` is not provided.

  * ##### optionalkeyword-onlyuse_extended_unique_key: bool = <!-- -->False

    Determines whether to include the HTTP method, ID Session and payload in the `unique_key` computation. This is only relevant when `unique_key` is not provided.

  * ##### optionalkeyword-onlyalways_enqueue: bool = <!-- -->False

    If set to `True`, the request will be enqueued even if it is already present in the queue. Using this is not allowed when a custom `unique_key` is also provided and will result in a `ValueError`.

  * ##### optionalkeyword-onlyenqueue_strategy: [EnqueueStrategy](https://crawlee.dev/python/api#EnqueueStrategy) | None = <!-- -->None

    The strategy that will be used for enqueuing the request.

  * ##### optionalkeyword-onlymax_retries: int | None = <!-- -->None

    Maximum number of retries for this request. Allows to override the global `max_request_retries` option of `BasicCrawler`.

  * ##### kwargs: Any

  #### Returns Self

### [](#get_query_param_from_url)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L343)get_query_param_from_url

* **get_query_param_from_url**(param, \*, default): str | None

- Get the value of a specific query parameter from the URL.

  ***

  #### Parameters

  * ##### param: str
  * ##### optionalkeyword-onlydefault: str | None = <!-- -->None

  #### Returns str | None

### [](#last_proxy_tier)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L414)last_proxy_tier

* **last_proxy_tier**(new_value): None

- #### Parameters

  * ##### new_value: int

  #### Returns None

### [](#session_rotation_count)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L396)session_rotation_count

* **session_rotation_count**(new_session_rotation_count): None

- #### Parameters

  * ##### new_session_rotation_count: int

  #### Returns None

### [](#state)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L382)state

* **state**(new_state): None

- #### Parameters

  * ##### new_state: [RequestState](https://crawlee.dev/python/api/class/RequestState)

  #### Returns None

## Properties<!-- -->[](#Properties)

### [](#crawl_depth)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L368)crawl_depth

crawl_depth: int

The depth of the request in the crawl tree.

### [](#crawlee_data)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L359)crawlee_data

crawlee_data: [CrawleeRequestData](https://crawlee.dev/python/api/class/CrawleeRequestData)

Crawlee-specific configuration stored in the `user_data`.

### [](#enqueue_strategy)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L400)enqueue_strategy

enqueue_strategy: [EnqueueStrategy](https://crawlee.dev/python/api#EnqueueStrategy)

The strategy that was used for enqueuing the request.

### [](#forefront)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L418)forefront

forefront: bool

Indicate whether the request should be enqueued at the front of the queue.

### [](#handled_at)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L240)handled_at

handled_at: datetime | None

Timestamp when the request was handled.

### [](#label)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L349)label

label: str | None

A string used to differentiate between arbitrary request types.

### [](#last_proxy_tier)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L409)last_proxy_tier

last_proxy_tier: int | None

The last proxy tier used to process the request.

### [](#loaded_url)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L237)loaded_url

loaded_url: str | None

URL of the web page that was loaded. This can differ from the original URL in case of redirects.

### [](#max_retries)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L386)max_retries

max_retries: int | None

Crawlee-specific limit on the number of retries of the request.

### [](#method)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L187)method

method: [HttpMethod](https://crawlee.dev/python/api#HttpMethod)

HTTP request method.

### [](#model_config)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L169)model_config

model_config: Undefined

### [](#no_retry)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L234)no_retry

no_retry: bool

If set to `True`, the request will not be retried in case of failure.

### [](#payload)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L190)payload

payload: [HttpPayload](https://crawlee.dev/python/api#HttpPayload) | None

HTTP request payload.

### [](#retry_count)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L231)retry_count

retry_count: int

Number of times the request has been retried.

### [](#session_id)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L354)session_id

session_id: str | None

The ID of the bound session, if there is any.

### [](#session_rotation_count)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L391)session_rotation_count

session_rotation_count: int | None

Crawlee-specific number of finished session rotations for the request.

### [](#state)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L377)state

state: [RequestState](https://crawlee.dev/python/api/class/RequestState)

Crawlee-specific request handling state.

### [](#unique_key)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L171)unique_key

unique_key: str

A unique key identifying the request. Two requests with the same `unique_key` are considered as pointing to the same URL.

If `unique_key` is not provided, then it is automatically generated by normalizing the URL. For example, the URL of `HTTP://www.EXAMPLE.com/something/` will produce the `unique_key` of `http://www.example.com/something`.

Pass an arbitrary non-empty text value to the `unique_key` property to override the default behavior and specify which URLs shall be considered equal.

### [](#url)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L183)url

url: str

The URL of the web page to crawl. Must be a valid HTTP or HTTPS URL, and may include query parameters and fragments.

### [](#was_already_handled)[](https://github.com/apify/crawlee-python/blob/dbfd4197a2100c638ec6a6280971f161c8db01bc//src/crawlee/_request.py#L427)was_already_handled

was_already_handled: bool

Indicates whether the request was handled.
