◐ Shell
clean mode source ↗

RequestQueue | API | Crawlee for Python · Fast, reliable Python web crawlers.

Index

Methods

  • __init__(client, id, name): None
  • async add_requests(requests, *, forefront, batch_size, wait_time_between_batches, wait_for_all_requests_to_be_added, wait_for_all_requests_to_be_added_timeout): None

  • Parameters

    • requests: Sequence[str | Request]
    • optionalkeyword-onlyforefront: bool = False
    • optionalkeyword-onlybatch_size: int = 1000
    • optionalkeyword-onlywait_time_between_batches: timedelta = timedelta(seconds=1)
    • optionalkeyword-onlywait_for_all_requests_to_be_added: bool = False
    • optionalkeyword-onlywait_for_all_requests_to_be_added_timeout: timedelta | None = None

    Returns None

  • async drop(): None

  • Returns None

  • async fetch_next_request(): Request | None
  • async get_handled_count(): int

  • Returns int

  • async get_request(unique_key): Request | None

  • Parameters

    • unique_key: str

    Returns Request | None

  • async get_total_count(): int

  • Returns int

  • async is_empty(): bool

  • Returns bool

  • async is_finished(): bool

  • Returns bool

  • async open(*, id, name, alias, configuration, storage_client): Storage

  • Parameters

    • optionalkeyword-onlyid: str | None = None
    • optionalkeyword-onlyname: str | None = None
    • optionalkeyword-onlyalias: str | None = None
    • optionalkeyword-onlyconfiguration: Configuration | None = None
    • optionalkeyword-onlystorage_client: StorageClient | None = None

    Returns Storage

  • async purge(): None

  • Returns None

Properties