head	1.19;
access;
symbols
	pkgsrc-2026Q2:1.18.0.2
	pkgsrc-2026Q2-base:1.18
	pkgsrc-2026Q1:1.16.0.2
	pkgsrc-2026Q1-base:1.16
	pkgsrc-2025Q4:1.15.0.6
	pkgsrc-2025Q4-base:1.15
	pkgsrc-2025Q3:1.15.0.4
	pkgsrc-2025Q3-base:1.15
	pkgsrc-2025Q2:1.15.0.2
	pkgsrc-2025Q2-base:1.15
	pkgsrc-2025Q1:1.13.0.4
	pkgsrc-2025Q1-base:1.13
	pkgsrc-2024Q4:1.13.0.2
	pkgsrc-2024Q4-base:1.13
	pkgsrc-2024Q3:1.12.0.4
	pkgsrc-2024Q3-base:1.12
	pkgsrc-2024Q2:1.12.0.2
	pkgsrc-2024Q2-base:1.12
	pkgsrc-2024Q1:1.11.0.2
	pkgsrc-2024Q1-base:1.11
	pkgsrc-2023Q4:1.10.0.2
	pkgsrc-2023Q4-base:1.10
	pkgsrc-2023Q3:1.9.0.4
	pkgsrc-2023Q3-base:1.9
	pkgsrc-2023Q2:1.9.0.2
	pkgsrc-2023Q2-base:1.9
	pkgsrc-2023Q1:1.8.0.18
	pkgsrc-2023Q1-base:1.8
	pkgsrc-2022Q4:1.8.0.16
	pkgsrc-2022Q4-base:1.8
	pkgsrc-2022Q3:1.8.0.14
	pkgsrc-2022Q3-base:1.8
	pkgsrc-2022Q2:1.8.0.12
	pkgsrc-2022Q2-base:1.8
	pkgsrc-2022Q1:1.8.0.10
	pkgsrc-2022Q1-base:1.8
	pkgsrc-2021Q4:1.8.0.8
	pkgsrc-2021Q4-base:1.8
	pkgsrc-2021Q3:1.8.0.6
	pkgsrc-2021Q3-base:1.8
	pkgsrc-2021Q2:1.8.0.4
	pkgsrc-2021Q2-base:1.8
	pkgsrc-2021Q1:1.8.0.2
	pkgsrc-2021Q1-base:1.8
	pkgsrc-2020Q4:1.7.0.8
	pkgsrc-2020Q4-base:1.7
	pkgsrc-2020Q3:1.7.0.6
	pkgsrc-2020Q3-base:1.7
	pkgsrc-2020Q2:1.7.0.4
	pkgsrc-2020Q2-base:1.7
	pkgsrc-2020Q1:1.7.0.2
	pkgsrc-2020Q1-base:1.7
	pkgsrc-2019Q4:1.6.0.6
	pkgsrc-2019Q4-base:1.6
	pkgsrc-2019Q3:1.6.0.2
	pkgsrc-2019Q3-base:1.6
	pkgsrc-2019Q2:1.5.0.4
	pkgsrc-2019Q2-base:1.5
	pkgsrc-2019Q1:1.5.0.2
	pkgsrc-2019Q1-base:1.5
	pkgsrc-2018Q4:1.3.0.8
	pkgsrc-2018Q4-base:1.3
	pkgsrc-2018Q3:1.3.0.6
	pkgsrc-2018Q3-base:1.3
	pkgsrc-2018Q2:1.3.0.4
	pkgsrc-2018Q2-base:1.3
	pkgsrc-2018Q1:1.3.0.2
	pkgsrc-2018Q1-base:1.3
	pkgsrc-2017Q4:1.2.0.8
	pkgsrc-2017Q4-base:1.2
	pkgsrc-2017Q3:1.2.0.6
	pkgsrc-2017Q3-base:1.2
	pkgsrc-2017Q2:1.2.0.2
	pkgsrc-2017Q2-base:1.2
	pkgsrc-2017Q1:1.1.0.2
	pkgsrc-2017Q1-base:1.1;
locks; strict;
comment	@# @;


1.19
date	2026.09.03.08.35.37;	author adam;	state Exp;
branches;
next	1.18;
commitid	eKUG4JmJzNVKv9UG;

1.18
date	2026.05.20.12.47.09;	author adam;	state Exp;
branches;
next	1.17;
commitid	NowtzCK3S1fOiyGG;

1.17
date	2026.04.22.11.13.56;	author adam;	state Exp;
branches;
next	1.16;
commitid	JI7mRIWsAR79HWCG;

1.16
date	2026.01.08.16.07.35;	author adam;	state Exp;
branches;
next	1.15;
commitid	oreeHmbxAFj7FBpG;

1.15
date	2025.05.15.09.21.59;	author adam;	state Exp;
branches;
next	1.14;
commitid	HdAhaaOhettd4ZUF;

1.14
date	2025.04.14.20.28.02;	author adam;	state Exp;
branches;
next	1.13;
commitid	UEGjPtvhiyhEK3RF;

1.13
date	2024.11.30.06.56.49;	author adam;	state Exp;
branches;
next	1.12;
commitid	wokNRJGH9n1dBDzF;

1.12
date	2024.05.14.19.15.59;	author adam;	state Exp;
branches;
next	1.11;
commitid	4S5NYv2usH3n70aF;

1.11
date	2024.02.16.19.02.45;	author adam;	state Exp;
branches;
next	1.10;
commitid	i6P7L6ng8lxhSGYE;

1.10
date	2023.10.10.17.18.23;	author triaxx;	state Exp;
branches;
next	1.9;
commitid	tLzh8iEqeU6qr6IE;

1.9
date	2023.04.27.09.33.44;	author adam;	state Exp;
branches;
next	1.8;
commitid	tlgNed5lNpikcJmE;

1.8
date	2021.03.22.08.56.56;	author triaxx;	state Exp;
branches;
next	1.7;
commitid	Do2yerH4BnWuChMC;

1.7
date	2020.01.29.22.06.30;	author adam;	state Exp;
branches;
next	1.6;
commitid	6gSvVBpWVN1PqDUB;

1.6
date	2019.08.22.08.21.11;	author adam;	state Exp;
branches;
next	1.5;
commitid	4ykVR3aKtVPZZZzB;

1.5
date	2019.01.31.09.07.46;	author adam;	state Exp;
branches;
next	1.4;
commitid	YqCcULhRsPC1NU9B;

1.4
date	2019.01.24.14.11.48;	author adam;	state Exp;
branches;
next	1.3;
commitid	D549NF1qLDkjH29B;

1.3
date	2018.01.04.21.31.41;	author adam;	state Exp;
branches;
next	1.2;
commitid	AYVt976oH416vBlA;

1.2
date	2017.05.20.06.25.36;	author adam;	state Exp;
branches;
next	1.1;
commitid	SHXh0bHW2C98R5Sz;

1.1
date	2017.02.13.21.25.33;	author adam;	state Exp;
branches;
next	;
commitid	RFPMXVz9YZD0VPFz;


desc
@@


1.19
log
@py-scrapy: updated to 2.18.0

Scrapy 2.18.0 (2026-08-20)

Highlights:

-   ``HttpxDownloadHandler`` now uses `httpx2 <https://httpx2.pydantic.dev/>`__

-   The Twisted-based HTTP/2 download handler is no longer experimental

-   ``brotli`` and Zstandard support are now always available, and :ref:`optional
    extras <extras>` cover the rest of the optional features

-   Late :class:`~scrapy.crawler.Crawler` attributes, such as
    :attr:`~scrapy.crawler.Crawler.stats`, now raise :exc:`RuntimeError`
    instead of being ``None`` before the crawl starts

-   Item exporters now export fields in declaration order

-   New :class:`~scrapy.spidermiddlewares.metacopy.MetaCopyDetectionMiddleware`

-   New :ref:`optimization <optimize>` page and :ref:`built-in stats reference
    <topics-stats-reference>`
@
text
@@@comment $NetBSD: PLIST,v 1.18 2026/05/20 12:47:09 adam Exp $
bin/scrapy-${PYVERSSUFFIX}
${PYSITELIB}/${WHEEL_INFODIR}/METADATA
${PYSITELIB}/${WHEEL_INFODIR}/RECORD
${PYSITELIB}/${WHEEL_INFODIR}/WHEEL
${PYSITELIB}/${WHEEL_INFODIR}/entry_points.txt
${PYSITELIB}/${WHEEL_INFODIR}/licenses/AUTHORS
${PYSITELIB}/${WHEEL_INFODIR}/licenses/LICENSE
${PYSITELIB}/scrapy/VERSION
${PYSITELIB}/scrapy/__init__.py
${PYSITELIB}/scrapy/__init__.pyc
${PYSITELIB}/scrapy/__init__.pyo
${PYSITELIB}/scrapy/__main__.py
${PYSITELIB}/scrapy/__main__.pyc
${PYSITELIB}/scrapy/__main__.pyo
${PYSITELIB}/scrapy/addons.py
${PYSITELIB}/scrapy/addons.pyc
${PYSITELIB}/scrapy/addons.pyo
${PYSITELIB}/scrapy/cmdline.py
${PYSITELIB}/scrapy/cmdline.pyc
${PYSITELIB}/scrapy/cmdline.pyo
${PYSITELIB}/scrapy/commands/__init__.py
${PYSITELIB}/scrapy/commands/__init__.pyc
${PYSITELIB}/scrapy/commands/__init__.pyo
${PYSITELIB}/scrapy/commands/bench.py
${PYSITELIB}/scrapy/commands/bench.pyc
${PYSITELIB}/scrapy/commands/bench.pyo
${PYSITELIB}/scrapy/commands/check.py
${PYSITELIB}/scrapy/commands/check.pyc
${PYSITELIB}/scrapy/commands/check.pyo
${PYSITELIB}/scrapy/commands/crawl.py
${PYSITELIB}/scrapy/commands/crawl.pyc
${PYSITELIB}/scrapy/commands/crawl.pyo
${PYSITELIB}/scrapy/commands/edit.py
${PYSITELIB}/scrapy/commands/edit.pyc
${PYSITELIB}/scrapy/commands/edit.pyo
${PYSITELIB}/scrapy/commands/fetch.py
${PYSITELIB}/scrapy/commands/fetch.pyc
${PYSITELIB}/scrapy/commands/fetch.pyo
${PYSITELIB}/scrapy/commands/genspider.py
${PYSITELIB}/scrapy/commands/genspider.pyc
${PYSITELIB}/scrapy/commands/genspider.pyo
${PYSITELIB}/scrapy/commands/list.py
${PYSITELIB}/scrapy/commands/list.pyc
${PYSITELIB}/scrapy/commands/list.pyo
${PYSITELIB}/scrapy/commands/parse.py
${PYSITELIB}/scrapy/commands/parse.pyc
${PYSITELIB}/scrapy/commands/parse.pyo
${PYSITELIB}/scrapy/commands/runspider.py
${PYSITELIB}/scrapy/commands/runspider.pyc
${PYSITELIB}/scrapy/commands/runspider.pyo
${PYSITELIB}/scrapy/commands/settings.py
${PYSITELIB}/scrapy/commands/settings.pyc
${PYSITELIB}/scrapy/commands/settings.pyo
${PYSITELIB}/scrapy/commands/shell.py
${PYSITELIB}/scrapy/commands/shell.pyc
${PYSITELIB}/scrapy/commands/shell.pyo
${PYSITELIB}/scrapy/commands/startproject.py
${PYSITELIB}/scrapy/commands/startproject.pyc
${PYSITELIB}/scrapy/commands/startproject.pyo
${PYSITELIB}/scrapy/commands/version.py
${PYSITELIB}/scrapy/commands/version.pyc
${PYSITELIB}/scrapy/commands/version.pyo
${PYSITELIB}/scrapy/commands/view.py
${PYSITELIB}/scrapy/commands/view.pyc
${PYSITELIB}/scrapy/commands/view.pyo
${PYSITELIB}/scrapy/contracts/__init__.py
${PYSITELIB}/scrapy/contracts/__init__.pyc
${PYSITELIB}/scrapy/contracts/__init__.pyo
${PYSITELIB}/scrapy/contracts/default.py
${PYSITELIB}/scrapy/contracts/default.pyc
${PYSITELIB}/scrapy/contracts/default.pyo
${PYSITELIB}/scrapy/core/__init__.py
${PYSITELIB}/scrapy/core/__init__.pyc
${PYSITELIB}/scrapy/core/__init__.pyo
${PYSITELIB}/scrapy/core/_http2/__init__.py
${PYSITELIB}/scrapy/core/_http2/__init__.pyc
${PYSITELIB}/scrapy/core/_http2/__init__.pyo
${PYSITELIB}/scrapy/core/_http2/agent.py
${PYSITELIB}/scrapy/core/_http2/agent.pyc
${PYSITELIB}/scrapy/core/_http2/agent.pyo
${PYSITELIB}/scrapy/core/_http2/protocol.py
${PYSITELIB}/scrapy/core/_http2/protocol.pyc
${PYSITELIB}/scrapy/core/_http2/protocol.pyo
${PYSITELIB}/scrapy/core/_http2/stream.py
${PYSITELIB}/scrapy/core/_http2/stream.pyc
${PYSITELIB}/scrapy/core/_http2/stream.pyo
${PYSITELIB}/scrapy/core/downloader/__init__.py
${PYSITELIB}/scrapy/core/downloader/__init__.pyc
${PYSITELIB}/scrapy/core/downloader/__init__.pyo
${PYSITELIB}/scrapy/core/downloader/_idna_patch.py
${PYSITELIB}/scrapy/core/downloader/_idna_patch.pyc
${PYSITELIB}/scrapy/core/downloader/_idna_patch.pyo
${PYSITELIB}/scrapy/core/downloader/contextfactory.py
${PYSITELIB}/scrapy/core/downloader/contextfactory.pyc
${PYSITELIB}/scrapy/core/downloader/contextfactory.pyo
${PYSITELIB}/scrapy/core/downloader/handlers/__init__.py
${PYSITELIB}/scrapy/core/downloader/handlers/__init__.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/__init__.pyo
${PYSITELIB}/scrapy/core/downloader/handlers/_base_http.py
${PYSITELIB}/scrapy/core/downloader/handlers/_base_http.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/_base_http.pyo
${PYSITELIB}/scrapy/core/downloader/handlers/_base_streaming.py
${PYSITELIB}/scrapy/core/downloader/handlers/_base_streaming.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/_base_streaming.pyo
${PYSITELIB}/scrapy/core/downloader/handlers/_httpx.py
${PYSITELIB}/scrapy/core/downloader/handlers/_httpx.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/_httpx.pyo
${PYSITELIB}/scrapy/core/downloader/handlers/base.py
${PYSITELIB}/scrapy/core/downloader/handlers/base.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/base.pyo
${PYSITELIB}/scrapy/core/downloader/handlers/datauri.py
${PYSITELIB}/scrapy/core/downloader/handlers/datauri.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/datauri.pyo
${PYSITELIB}/scrapy/core/downloader/handlers/file.py
${PYSITELIB}/scrapy/core/downloader/handlers/file.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/file.pyo
${PYSITELIB}/scrapy/core/downloader/handlers/ftp.py
${PYSITELIB}/scrapy/core/downloader/handlers/ftp.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/ftp.pyo
${PYSITELIB}/scrapy/core/downloader/handlers/http.py
${PYSITELIB}/scrapy/core/downloader/handlers/http.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/http.pyo
${PYSITELIB}/scrapy/core/downloader/handlers/http11.py
${PYSITELIB}/scrapy/core/downloader/handlers/http11.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/http11.pyo
${PYSITELIB}/scrapy/core/downloader/handlers/http2.py
${PYSITELIB}/scrapy/core/downloader/handlers/http2.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/http2.pyo
${PYSITELIB}/scrapy/core/downloader/handlers/s3.py
${PYSITELIB}/scrapy/core/downloader/handlers/s3.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/s3.pyo
${PYSITELIB}/scrapy/core/downloader/middleware.py
${PYSITELIB}/scrapy/core/downloader/middleware.pyc
${PYSITELIB}/scrapy/core/downloader/middleware.pyo
${PYSITELIB}/scrapy/core/downloader/tls.py
${PYSITELIB}/scrapy/core/downloader/tls.pyc
${PYSITELIB}/scrapy/core/downloader/tls.pyo
${PYSITELIB}/scrapy/core/engine.py
${PYSITELIB}/scrapy/core/engine.pyc
${PYSITELIB}/scrapy/core/engine.pyo
${PYSITELIB}/scrapy/core/scheduler.py
${PYSITELIB}/scrapy/core/scheduler.pyc
${PYSITELIB}/scrapy/core/scheduler.pyo
${PYSITELIB}/scrapy/core/scraper.py
${PYSITELIB}/scrapy/core/scraper.pyc
${PYSITELIB}/scrapy/core/scraper.pyo
${PYSITELIB}/scrapy/core/spidermw.py
${PYSITELIB}/scrapy/core/spidermw.pyc
${PYSITELIB}/scrapy/core/spidermw.pyo
${PYSITELIB}/scrapy/crawler.py
${PYSITELIB}/scrapy/crawler.pyc
${PYSITELIB}/scrapy/crawler.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/__init__.py
${PYSITELIB}/scrapy/downloadermiddlewares/__init__.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/__init__.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/cookies.py
${PYSITELIB}/scrapy/downloadermiddlewares/cookies.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/cookies.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/defaultheaders.py
${PYSITELIB}/scrapy/downloadermiddlewares/defaultheaders.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/defaultheaders.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/downloadtimeout.py
${PYSITELIB}/scrapy/downloadermiddlewares/downloadtimeout.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/downloadtimeout.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/httpauth.py
${PYSITELIB}/scrapy/downloadermiddlewares/httpauth.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/httpauth.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/httpcache.py
${PYSITELIB}/scrapy/downloadermiddlewares/httpcache.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/httpcache.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/httpcompression.py
${PYSITELIB}/scrapy/downloadermiddlewares/httpcompression.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/httpcompression.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/httpproxy.py
${PYSITELIB}/scrapy/downloadermiddlewares/httpproxy.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/httpproxy.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/offsite.py
${PYSITELIB}/scrapy/downloadermiddlewares/offsite.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/offsite.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/redirect.py
${PYSITELIB}/scrapy/downloadermiddlewares/redirect.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/redirect.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/retry.py
${PYSITELIB}/scrapy/downloadermiddlewares/retry.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/retry.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/robotstxt.py
${PYSITELIB}/scrapy/downloadermiddlewares/robotstxt.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/robotstxt.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/stats.py
${PYSITELIB}/scrapy/downloadermiddlewares/stats.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/stats.pyo
${PYSITELIB}/scrapy/downloadermiddlewares/useragent.py
${PYSITELIB}/scrapy/downloadermiddlewares/useragent.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/useragent.pyo
${PYSITELIB}/scrapy/dupefilters.py
${PYSITELIB}/scrapy/dupefilters.pyc
${PYSITELIB}/scrapy/dupefilters.pyo
${PYSITELIB}/scrapy/exceptions.py
${PYSITELIB}/scrapy/exceptions.pyc
${PYSITELIB}/scrapy/exceptions.pyo
${PYSITELIB}/scrapy/exporters.py
${PYSITELIB}/scrapy/exporters.pyc
${PYSITELIB}/scrapy/exporters.pyo
${PYSITELIB}/scrapy/extension.py
${PYSITELIB}/scrapy/extension.pyc
${PYSITELIB}/scrapy/extension.pyo
${PYSITELIB}/scrapy/extensions/__init__.py
${PYSITELIB}/scrapy/extensions/__init__.pyc
${PYSITELIB}/scrapy/extensions/__init__.pyo
${PYSITELIB}/scrapy/extensions/closespider.py
${PYSITELIB}/scrapy/extensions/closespider.pyc
${PYSITELIB}/scrapy/extensions/closespider.pyo
${PYSITELIB}/scrapy/extensions/corestats.py
${PYSITELIB}/scrapy/extensions/corestats.pyc
${PYSITELIB}/scrapy/extensions/corestats.pyo
${PYSITELIB}/scrapy/extensions/debug.py
${PYSITELIB}/scrapy/extensions/debug.pyc
${PYSITELIB}/scrapy/extensions/debug.pyo
${PYSITELIB}/scrapy/extensions/feedexport.py
${PYSITELIB}/scrapy/extensions/feedexport.pyc
${PYSITELIB}/scrapy/extensions/feedexport.pyo
${PYSITELIB}/scrapy/extensions/httpcache.py
${PYSITELIB}/scrapy/extensions/httpcache.pyc
${PYSITELIB}/scrapy/extensions/httpcache.pyo
${PYSITELIB}/scrapy/extensions/logcount.py
${PYSITELIB}/scrapy/extensions/logcount.pyc
${PYSITELIB}/scrapy/extensions/logcount.pyo
${PYSITELIB}/scrapy/extensions/logstats.py
${PYSITELIB}/scrapy/extensions/logstats.pyc
${PYSITELIB}/scrapy/extensions/logstats.pyo
${PYSITELIB}/scrapy/extensions/memdebug.py
${PYSITELIB}/scrapy/extensions/memdebug.pyc
${PYSITELIB}/scrapy/extensions/memdebug.pyo
${PYSITELIB}/scrapy/extensions/memusage.py
${PYSITELIB}/scrapy/extensions/memusage.pyc
${PYSITELIB}/scrapy/extensions/memusage.pyo
${PYSITELIB}/scrapy/extensions/periodic_log.py
${PYSITELIB}/scrapy/extensions/periodic_log.pyc
${PYSITELIB}/scrapy/extensions/periodic_log.pyo
${PYSITELIB}/scrapy/extensions/postprocessing.py
${PYSITELIB}/scrapy/extensions/postprocessing.pyc
${PYSITELIB}/scrapy/extensions/postprocessing.pyo
${PYSITELIB}/scrapy/extensions/spiderstate.py
${PYSITELIB}/scrapy/extensions/spiderstate.pyc
${PYSITELIB}/scrapy/extensions/spiderstate.pyo
${PYSITELIB}/scrapy/extensions/statsmailer.py
${PYSITELIB}/scrapy/extensions/statsmailer.pyc
${PYSITELIB}/scrapy/extensions/statsmailer.pyo
${PYSITELIB}/scrapy/extensions/telnet.py
${PYSITELIB}/scrapy/extensions/telnet.pyc
${PYSITELIB}/scrapy/extensions/telnet.pyo
${PYSITELIB}/scrapy/extensions/throttle.py
${PYSITELIB}/scrapy/extensions/throttle.pyc
${PYSITELIB}/scrapy/extensions/throttle.pyo
${PYSITELIB}/scrapy/http/__init__.py
${PYSITELIB}/scrapy/http/__init__.pyc
${PYSITELIB}/scrapy/http/__init__.pyo
${PYSITELIB}/scrapy/http/cookies.py
${PYSITELIB}/scrapy/http/cookies.pyc
${PYSITELIB}/scrapy/http/cookies.pyo
${PYSITELIB}/scrapy/http/headers.py
${PYSITELIB}/scrapy/http/headers.pyc
${PYSITELIB}/scrapy/http/headers.pyo
${PYSITELIB}/scrapy/http/request/__init__.py
${PYSITELIB}/scrapy/http/request/__init__.pyc
${PYSITELIB}/scrapy/http/request/__init__.pyo
${PYSITELIB}/scrapy/http/request/form.py
${PYSITELIB}/scrapy/http/request/form.pyc
${PYSITELIB}/scrapy/http/request/form.pyo
${PYSITELIB}/scrapy/http/request/json_request.py
${PYSITELIB}/scrapy/http/request/json_request.pyc
${PYSITELIB}/scrapy/http/request/json_request.pyo
${PYSITELIB}/scrapy/http/request/rpc.py
${PYSITELIB}/scrapy/http/request/rpc.pyc
${PYSITELIB}/scrapy/http/request/rpc.pyo
${PYSITELIB}/scrapy/http/response/__init__.py
${PYSITELIB}/scrapy/http/response/__init__.pyc
${PYSITELIB}/scrapy/http/response/__init__.pyo
${PYSITELIB}/scrapy/http/response/html.py
${PYSITELIB}/scrapy/http/response/html.pyc
${PYSITELIB}/scrapy/http/response/html.pyo
${PYSITELIB}/scrapy/http/response/json.py
${PYSITELIB}/scrapy/http/response/json.pyc
${PYSITELIB}/scrapy/http/response/json.pyo
${PYSITELIB}/scrapy/http/response/text.py
${PYSITELIB}/scrapy/http/response/text.pyc
${PYSITELIB}/scrapy/http/response/text.pyo
${PYSITELIB}/scrapy/http/response/xml.py
${PYSITELIB}/scrapy/http/response/xml.pyc
${PYSITELIB}/scrapy/http/response/xml.pyo
${PYSITELIB}/scrapy/interfaces.py
${PYSITELIB}/scrapy/interfaces.pyc
${PYSITELIB}/scrapy/interfaces.pyo
${PYSITELIB}/scrapy/item.py
${PYSITELIB}/scrapy/item.pyc
${PYSITELIB}/scrapy/item.pyo
${PYSITELIB}/scrapy/link.py
${PYSITELIB}/scrapy/link.pyc
${PYSITELIB}/scrapy/link.pyo
${PYSITELIB}/scrapy/linkextractors/__init__.py
${PYSITELIB}/scrapy/linkextractors/__init__.pyc
${PYSITELIB}/scrapy/linkextractors/__init__.pyo
${PYSITELIB}/scrapy/linkextractors/lxmlhtml.py
${PYSITELIB}/scrapy/linkextractors/lxmlhtml.pyc
${PYSITELIB}/scrapy/linkextractors/lxmlhtml.pyo
${PYSITELIB}/scrapy/loader/__init__.py
${PYSITELIB}/scrapy/loader/__init__.pyc
${PYSITELIB}/scrapy/loader/__init__.pyo
${PYSITELIB}/scrapy/logformatter.py
${PYSITELIB}/scrapy/logformatter.pyc
${PYSITELIB}/scrapy/logformatter.pyo
${PYSITELIB}/scrapy/mail.py
${PYSITELIB}/scrapy/mail.pyc
${PYSITELIB}/scrapy/mail.pyo
${PYSITELIB}/scrapy/middleware.py
${PYSITELIB}/scrapy/middleware.pyc
${PYSITELIB}/scrapy/middleware.pyo
${PYSITELIB}/scrapy/mime.types
${PYSITELIB}/scrapy/pipelines/__init__.py
${PYSITELIB}/scrapy/pipelines/__init__.pyc
${PYSITELIB}/scrapy/pipelines/__init__.pyo
${PYSITELIB}/scrapy/pipelines/files.py
${PYSITELIB}/scrapy/pipelines/files.pyc
${PYSITELIB}/scrapy/pipelines/files.pyo
${PYSITELIB}/scrapy/pipelines/images.py
${PYSITELIB}/scrapy/pipelines/images.pyc
${PYSITELIB}/scrapy/pipelines/images.pyo
${PYSITELIB}/scrapy/pipelines/media.py
${PYSITELIB}/scrapy/pipelines/media.pyc
${PYSITELIB}/scrapy/pipelines/media.pyo
${PYSITELIB}/scrapy/pqueues.py
${PYSITELIB}/scrapy/pqueues.pyc
${PYSITELIB}/scrapy/pqueues.pyo
${PYSITELIB}/scrapy/py.typed
${PYSITELIB}/scrapy/resolver.py
${PYSITELIB}/scrapy/resolver.pyc
${PYSITELIB}/scrapy/resolver.pyo
${PYSITELIB}/scrapy/responsetypes.py
${PYSITELIB}/scrapy/responsetypes.pyc
${PYSITELIB}/scrapy/responsetypes.pyo
${PYSITELIB}/scrapy/robotstxt.py
${PYSITELIB}/scrapy/robotstxt.pyc
${PYSITELIB}/scrapy/robotstxt.pyo
${PYSITELIB}/scrapy/selector/__init__.py
${PYSITELIB}/scrapy/selector/__init__.pyc
${PYSITELIB}/scrapy/selector/__init__.pyo
${PYSITELIB}/scrapy/selector/unified.py
${PYSITELIB}/scrapy/selector/unified.pyc
${PYSITELIB}/scrapy/selector/unified.pyo
${PYSITELIB}/scrapy/settings/__init__.py
${PYSITELIB}/scrapy/settings/__init__.pyc
${PYSITELIB}/scrapy/settings/__init__.pyo
${PYSITELIB}/scrapy/settings/default_settings.py
${PYSITELIB}/scrapy/settings/default_settings.pyc
${PYSITELIB}/scrapy/settings/default_settings.pyo
${PYSITELIB}/scrapy/shell.py
${PYSITELIB}/scrapy/shell.pyc
${PYSITELIB}/scrapy/shell.pyo
${PYSITELIB}/scrapy/signalmanager.py
${PYSITELIB}/scrapy/signalmanager.pyc
${PYSITELIB}/scrapy/signalmanager.pyo
${PYSITELIB}/scrapy/signals.py
${PYSITELIB}/scrapy/signals.pyc
${PYSITELIB}/scrapy/signals.pyo
${PYSITELIB}/scrapy/spiderloader.py
${PYSITELIB}/scrapy/spiderloader.pyc
${PYSITELIB}/scrapy/spiderloader.pyo
${PYSITELIB}/scrapy/spidermiddlewares/__init__.py
${PYSITELIB}/scrapy/spidermiddlewares/__init__.pyc
${PYSITELIB}/scrapy/spidermiddlewares/__init__.pyo
${PYSITELIB}/scrapy/spidermiddlewares/base.py
${PYSITELIB}/scrapy/spidermiddlewares/base.pyc
${PYSITELIB}/scrapy/spidermiddlewares/base.pyo
${PYSITELIB}/scrapy/spidermiddlewares/depth.py
${PYSITELIB}/scrapy/spidermiddlewares/depth.pyc
${PYSITELIB}/scrapy/spidermiddlewares/depth.pyo
${PYSITELIB}/scrapy/spidermiddlewares/httperror.py
${PYSITELIB}/scrapy/spidermiddlewares/httperror.pyc
${PYSITELIB}/scrapy/spidermiddlewares/httperror.pyo
${PYSITELIB}/scrapy/spidermiddlewares/metacopy.py
${PYSITELIB}/scrapy/spidermiddlewares/metacopy.pyc
${PYSITELIB}/scrapy/spidermiddlewares/metacopy.pyo
${PYSITELIB}/scrapy/spidermiddlewares/referer.py
${PYSITELIB}/scrapy/spidermiddlewares/referer.pyc
${PYSITELIB}/scrapy/spidermiddlewares/referer.pyo
${PYSITELIB}/scrapy/spidermiddlewares/start.py
${PYSITELIB}/scrapy/spidermiddlewares/start.pyc
${PYSITELIB}/scrapy/spidermiddlewares/start.pyo
${PYSITELIB}/scrapy/spidermiddlewares/urllength.py
${PYSITELIB}/scrapy/spidermiddlewares/urllength.pyc
${PYSITELIB}/scrapy/spidermiddlewares/urllength.pyo
${PYSITELIB}/scrapy/spiders/__init__.py
${PYSITELIB}/scrapy/spiders/__init__.pyc
${PYSITELIB}/scrapy/spiders/__init__.pyo
${PYSITELIB}/scrapy/spiders/crawl.py
${PYSITELIB}/scrapy/spiders/crawl.pyc
${PYSITELIB}/scrapy/spiders/crawl.pyo
${PYSITELIB}/scrapy/spiders/feed.py
${PYSITELIB}/scrapy/spiders/feed.pyc
${PYSITELIB}/scrapy/spiders/feed.pyo
${PYSITELIB}/scrapy/spiders/sitemap.py
${PYSITELIB}/scrapy/spiders/sitemap.pyc
${PYSITELIB}/scrapy/spiders/sitemap.pyo
${PYSITELIB}/scrapy/squeues.py
${PYSITELIB}/scrapy/squeues.pyc
${PYSITELIB}/scrapy/squeues.pyo
${PYSITELIB}/scrapy/statscollectors.py
${PYSITELIB}/scrapy/statscollectors.pyc
${PYSITELIB}/scrapy/statscollectors.pyo
${PYSITELIB}/scrapy/templates/project/module/__init__.py
${PYSITELIB}/scrapy/templates/project/module/__init__.pyc
${PYSITELIB}/scrapy/templates/project/module/__init__.pyo
${PYSITELIB}/scrapy/templates/project/module/items.py.tmpl
${PYSITELIB}/scrapy/templates/project/module/middlewares.py.tmpl
${PYSITELIB}/scrapy/templates/project/module/pipelines.py.tmpl
${PYSITELIB}/scrapy/templates/project/module/settings.py.tmpl
${PYSITELIB}/scrapy/templates/project/module/spiders/__init__.py
${PYSITELIB}/scrapy/templates/project/module/spiders/__init__.pyc
${PYSITELIB}/scrapy/templates/project/module/spiders/__init__.pyo
${PYSITELIB}/scrapy/templates/project/scrapy.cfg
${PYSITELIB}/scrapy/templates/spiders/basic.tmpl
${PYSITELIB}/scrapy/templates/spiders/crawl.tmpl
${PYSITELIB}/scrapy/templates/spiders/csvfeed.tmpl
${PYSITELIB}/scrapy/templates/spiders/xmlfeed.tmpl
${PYSITELIB}/scrapy/utils/__init__.py
${PYSITELIB}/scrapy/utils/__init__.pyc
${PYSITELIB}/scrapy/utils/__init__.pyo
${PYSITELIB}/scrapy/utils/_compression.py
${PYSITELIB}/scrapy/utils/_compression.pyc
${PYSITELIB}/scrapy/utils/_compression.pyo
${PYSITELIB}/scrapy/utils/_deps_compat.py
${PYSITELIB}/scrapy/utils/_deps_compat.pyc
${PYSITELIB}/scrapy/utils/_deps_compat.pyo
${PYSITELIB}/scrapy/utils/_download_handlers.py
${PYSITELIB}/scrapy/utils/_download_handlers.pyc
${PYSITELIB}/scrapy/utils/_download_handlers.pyo
${PYSITELIB}/scrapy/utils/asyncgen.py
${PYSITELIB}/scrapy/utils/asyncgen.pyc
${PYSITELIB}/scrapy/utils/asyncgen.pyo
${PYSITELIB}/scrapy/utils/asyncio.py
${PYSITELIB}/scrapy/utils/asyncio.pyc
${PYSITELIB}/scrapy/utils/asyncio.pyo
${PYSITELIB}/scrapy/utils/benchserver.py
${PYSITELIB}/scrapy/utils/benchserver.pyc
${PYSITELIB}/scrapy/utils/benchserver.pyo
${PYSITELIB}/scrapy/utils/boto.py
${PYSITELIB}/scrapy/utils/boto.pyc
${PYSITELIB}/scrapy/utils/boto.pyo
${PYSITELIB}/scrapy/utils/conf.py
${PYSITELIB}/scrapy/utils/conf.pyc
${PYSITELIB}/scrapy/utils/conf.pyo
${PYSITELIB}/scrapy/utils/console.py
${PYSITELIB}/scrapy/utils/console.pyc
${PYSITELIB}/scrapy/utils/console.pyo
${PYSITELIB}/scrapy/utils/curl.py
${PYSITELIB}/scrapy/utils/curl.pyc
${PYSITELIB}/scrapy/utils/curl.pyo
${PYSITELIB}/scrapy/utils/datatypes.py
${PYSITELIB}/scrapy/utils/datatypes.pyc
${PYSITELIB}/scrapy/utils/datatypes.pyo
${PYSITELIB}/scrapy/utils/decorators.py
${PYSITELIB}/scrapy/utils/decorators.pyc
${PYSITELIB}/scrapy/utils/decorators.pyo
${PYSITELIB}/scrapy/utils/defer.py
${PYSITELIB}/scrapy/utils/defer.pyc
${PYSITELIB}/scrapy/utils/defer.pyo
${PYSITELIB}/scrapy/utils/deprecate.py
${PYSITELIB}/scrapy/utils/deprecate.pyc
${PYSITELIB}/scrapy/utils/deprecate.pyo
${PYSITELIB}/scrapy/utils/display.py
${PYSITELIB}/scrapy/utils/display.pyc
${PYSITELIB}/scrapy/utils/display.pyo
${PYSITELIB}/scrapy/utils/engine.py
${PYSITELIB}/scrapy/utils/engine.pyc
${PYSITELIB}/scrapy/utils/engine.pyo
${PYSITELIB}/scrapy/utils/ftp.py
${PYSITELIB}/scrapy/utils/ftp.pyc
${PYSITELIB}/scrapy/utils/ftp.pyo
${PYSITELIB}/scrapy/utils/gz.py
${PYSITELIB}/scrapy/utils/gz.pyc
${PYSITELIB}/scrapy/utils/gz.pyo
${PYSITELIB}/scrapy/utils/httpobj.py
${PYSITELIB}/scrapy/utils/httpobj.pyc
${PYSITELIB}/scrapy/utils/httpobj.pyo
${PYSITELIB}/scrapy/utils/iterators.py
${PYSITELIB}/scrapy/utils/iterators.pyc
${PYSITELIB}/scrapy/utils/iterators.pyo
${PYSITELIB}/scrapy/utils/job.py
${PYSITELIB}/scrapy/utils/job.pyc
${PYSITELIB}/scrapy/utils/job.pyo
${PYSITELIB}/scrapy/utils/log.py
${PYSITELIB}/scrapy/utils/log.pyc
${PYSITELIB}/scrapy/utils/log.pyo
${PYSITELIB}/scrapy/utils/misc.py
${PYSITELIB}/scrapy/utils/misc.pyc
${PYSITELIB}/scrapy/utils/misc.pyo
${PYSITELIB}/scrapy/utils/ossignal.py
${PYSITELIB}/scrapy/utils/ossignal.pyc
${PYSITELIB}/scrapy/utils/ossignal.pyo
${PYSITELIB}/scrapy/utils/project.py
${PYSITELIB}/scrapy/utils/project.pyc
${PYSITELIB}/scrapy/utils/project.pyo
${PYSITELIB}/scrapy/utils/python.py
${PYSITELIB}/scrapy/utils/python.pyc
${PYSITELIB}/scrapy/utils/python.pyo
${PYSITELIB}/scrapy/utils/reactor.py
${PYSITELIB}/scrapy/utils/reactor.pyc
${PYSITELIB}/scrapy/utils/reactor.pyo
${PYSITELIB}/scrapy/utils/reactorless.py
${PYSITELIB}/scrapy/utils/reactorless.pyc
${PYSITELIB}/scrapy/utils/reactorless.pyo
${PYSITELIB}/scrapy/utils/request.py
${PYSITELIB}/scrapy/utils/request.pyc
${PYSITELIB}/scrapy/utils/request.pyo
${PYSITELIB}/scrapy/utils/response.py
${PYSITELIB}/scrapy/utils/response.pyc
${PYSITELIB}/scrapy/utils/response.pyo
${PYSITELIB}/scrapy/utils/serialize.py
${PYSITELIB}/scrapy/utils/serialize.pyc
${PYSITELIB}/scrapy/utils/serialize.pyo
${PYSITELIB}/scrapy/utils/signal.py
${PYSITELIB}/scrapy/utils/signal.pyc
${PYSITELIB}/scrapy/utils/signal.pyo
${PYSITELIB}/scrapy/utils/sitemap.py
${PYSITELIB}/scrapy/utils/sitemap.pyc
${PYSITELIB}/scrapy/utils/sitemap.pyo
${PYSITELIB}/scrapy/utils/spider.py
${PYSITELIB}/scrapy/utils/spider.pyc
${PYSITELIB}/scrapy/utils/spider.pyo
${PYSITELIB}/scrapy/utils/ssl.py
${PYSITELIB}/scrapy/utils/ssl.pyc
${PYSITELIB}/scrapy/utils/ssl.pyo
${PYSITELIB}/scrapy/utils/template.py
${PYSITELIB}/scrapy/utils/template.pyc
${PYSITELIB}/scrapy/utils/template.pyo
${PYSITELIB}/scrapy/utils/test.py
${PYSITELIB}/scrapy/utils/test.pyc
${PYSITELIB}/scrapy/utils/test.pyo
${PYSITELIB}/scrapy/utils/trackref.py
${PYSITELIB}/scrapy/utils/trackref.pyc
${PYSITELIB}/scrapy/utils/trackref.pyo
${PYSITELIB}/scrapy/utils/url.py
${PYSITELIB}/scrapy/utils/url.pyc
${PYSITELIB}/scrapy/utils/url.pyo
${PYSITELIB}/scrapy/utils/versions.py
${PYSITELIB}/scrapy/utils/versions.pyc
${PYSITELIB}/scrapy/utils/versions.pyo
@


1.18
log
@py-scrapy: updated to 2.16.0

Scrapy 2.16.0 (2026-05-19)

Highlights:

-   Official support for Python 3.14
-   Support for Twisted 26.4.0+

Modified requirements

-   Increased the minimum versions of the following dependencies:

    - service_identity_: 18.1.0 → 23.1.0

-   Added support for Twisted 26.4.0+.
-   Added support for Python 3.14.

Backward-incompatible changes

-   The following classes and functions, intended for internal use by
    :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`
    and :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler`, have
    been made private:

    - ``scrapy.core.downloader.handlers.http11.ScrapyAgent``
    - ``scrapy.core.downloader.handlers.http11.ScrapyProxyAgent``
    - ``scrapy.core.downloader.handlers.http11.TunnelingAgent``
    - ``scrapy.core.downloader.handlers.http11.TunnelingTCP4ClientEndpoint``
    - ``scrapy.core.downloader.handlers.http11.tunnel_request_data()``
    - ``scrapy.core.downloader.handlers.http2.ScrapyH2Agent``

Deprecations

-   ``scrapy.FormRequest`` is deprecated. You can use the :doc:`form2request
    <form2request:index>` library instead, see :ref:`form`.
-   ``scrapy.utils.python.MutableChain`` is deprecated.

Deprecation removals

-   The ``start_requests()`` method of :class:`~scrapy.Spider`, deprecated in
    2.13.0, is removed and no longer called. Use :meth:`~scrapy.Spider.start`
    instead, or both to maintain support for lower Scrapy versions.
-   Support for ``process_start_requests()`` methods of :ref:`spider middlewares
    <topics-spider-middleware>`, deprecated in 2.13.0, is removed. Use
    :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` instead,
    or both to maintain support for lower Scrapy versions.
-   Support for synchronous ``process_spider_output()`` methods of spider
    middlewares, deprecated in Scrapy 2.13.0, is removed. You should upgrade
    the affected middlewares to have asynchronous ``process_spider_output()``
    methods.
-   The ``spider`` arguments of the following methods of
    :class:`~scrapy.core.scraper.Scraper`, deprecated in Scrapy 2.13.0, are
    removed:

    - ``close_spider()``
    - ``enqueue_scrape()``
    - ``handle_spider_error()``
    - ``handle_spider_output()``

-   HTTP/1.0 support code, deprecated in Scrapy 2.13.0, is removed. This
    includes:

    - ``scrapy.core.downloader.handlers.http10.HTTP10DownloadHandler``
    - The ``scrapy.core.downloader.webclient`` module.
    - The ``DOWNLOADER_HTTPCLIENTFACTORY`` setting.

-   The following functions, deprecated in Scrapy 2.13.0, are removed, you
    should import them from :mod:`w3lib.url` directly instead:

    - ``scrapy.utils.url.add_or_replace_parameter()``
    - ``scrapy.utils.url.add_or_replace_parameters()``
    - ``scrapy.utils.url.any_to_uri()``
    - ``scrapy.utils.url.canonicalize_url()``
    - ``scrapy.utils.url.file_uri_to_path()``
    - ``scrapy.utils.url.is_url()``
    - ``scrapy.utils.url.parse_data_uri()``
    - ``scrapy.utils.url.parse_url()``
    - ``scrapy.utils.url.path_to_file_uri()``
    - ``scrapy.utils.url.safe_download_url()``
    - ``scrapy.utils.url.safe_url_string()``
    - ``scrapy.utils.url.url_query_cleaner()``
    - ``scrapy.utils.url.url_query_parameter()``

-   The following test-related code, deprecated in Scrapy 2.13.0, is removed:

    - the ``scrapy.utils.testproc`` module
    - the ``scrapy.utils.testsite`` module
    - ``scrapy.utils.test.assert_gcs_environ()``
    - ``scrapy.utils.test.get_ftp_content_and_delete()``
    - ``scrapy.utils.test.get_gcs_content_and_delete()``
    - ``scrapy.utils.test.mock_google_cloud_storage()``
    - ``scrapy.utils.test.skip_if_no_boto()``
    - ``scrapy.utils.test.TestSpider``

-   ``scrapy.utils.versions.scrapy_components_versions()``, deprecated in
    Scrapy 2.13.0, is removed, you can use
    :func:`scrapy.utils.versions.get_versions` instead.
-   ``scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware`` and
    ``scrapy.utils.url.escape_ajax()``, deprecated in Scrapy 2.13.0, are
    removed.
-   The ``__init__()`` method of priority queue classes (see
    :setting:`SCHEDULER_PRIORITY_QUEUE`) now needs to support a keyword-only
    ``start_queue_cls`` parameter, not supporting it was deprecated in Scrapy
    2.13.0.

-   ``scrapy.spiders.init.InitSpider``, deprecated in Scrapy 2.13.0, is
    removed.

New features

-   New features and improvements for
    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`:

    - Support for proxies.
    - Support for the :reqmeta:`download_latency` meta key.
    - Support for :attr:`Response.certificate
      <scrapy.http.Response.certificate>`.
    - Default headers set by the ``httpx`` library are no longer added to
      requests.

-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now
    skips HTTPS proxy certificate verification when the
    :setting:`DOWNLOAD_VERIFY_CERTIFICATES` setting is set to ``False``.

Improvements

-   :func:`time.monotonic` is used instead of :func:`time.time` to calculate
    elapsed time in various places.
-   Improved extraction of the file extension from the URL in
    :class:`~scrapy.pipelines.files.FilesPipeline`.
-   Other code refactoring and improvements.

Bug fixes

-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now
    raises an exception when a request has an ``https://`` destination and an
    ``https://`` proxy, which is not supported by this handler. Previously it
    tried to connect to the proxy via HTTP in this case.
-   :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now
    raises an exception for requests with ``http://`` URLs instead of trying to
    connect, which is not supported by this handler.
-   :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` no longer
    adds the ``:status`` pseudo-header to :attr:`Response.headers
    <scrapy.http.Response.headers>`.
-   Fixed :func:`scrapy.utils.response.open_in_browser` removing the ``<head>``
    tag when adding the ``<base>`` tag.

Documentation

-   Documented that
    :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`
    doesn't support HTTPS proxies for HTTPS destinations and that
    :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` doesn't
    support proxies at all.
-   Added an example of using
    :class:`logging.handlers.TimedRotatingFileHandler` to rotate Scrapy logs.
-   Added a ``CITATION.cff`` file.
-   Mentioned :setting:`DOWNLOADER_CLIENT_TLS_METHOD` in :ref:`bans`.
-   Other documentation improvements and fixes.
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.17 2026/04/22 11:13:56 adam Exp $
d76 12
d91 3
a141 12
${PYSITELIB}/scrapy/core/http2/__init__.py
${PYSITELIB}/scrapy/core/http2/__init__.pyc
${PYSITELIB}/scrapy/core/http2/__init__.pyo
${PYSITELIB}/scrapy/core/http2/agent.py
${PYSITELIB}/scrapy/core/http2/agent.pyc
${PYSITELIB}/scrapy/core/http2/agent.pyo
${PYSITELIB}/scrapy/core/http2/protocol.py
${PYSITELIB}/scrapy/core/http2/protocol.pyc
${PYSITELIB}/scrapy/core/http2/protocol.pyo
${PYSITELIB}/scrapy/core/http2/stream.py
${PYSITELIB}/scrapy/core/http2/stream.pyc
${PYSITELIB}/scrapy/core/http2/stream.pyo
d381 3
@


1.17
log
@py-scrapy: updated to 2.15.0

Scrapy 2.15.0 (2026-04-09)

Highlights:

- Experimental support for running without a Twisted reactor
- Experimental ``httpx``-based download handler
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.16 2026/01/08 16:07:35 adam Exp $
d85 6
a108 3
${PYSITELIB}/scrapy/core/downloader/handlers/http10.py
${PYSITELIB}/scrapy/core/downloader/handlers/http10.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/http10.pyo
a123 3
${PYSITELIB}/scrapy/core/downloader/webclient.py
${PYSITELIB}/scrapy/core/downloader/webclient.pyc
${PYSITELIB}/scrapy/core/downloader/webclient.pyo
a153 3
${PYSITELIB}/scrapy/downloadermiddlewares/ajaxcrawl.py
${PYSITELIB}/scrapy/downloadermiddlewares/ajaxcrawl.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/ajaxcrawl.pyo
a395 3
${PYSITELIB}/scrapy/spiders/init.py
${PYSITELIB}/scrapy/spiders/init.pyc
${PYSITELIB}/scrapy/spiders/init.pyo
a533 6
${PYSITELIB}/scrapy/utils/testproc.py
${PYSITELIB}/scrapy/utils/testproc.pyc
${PYSITELIB}/scrapy/utils/testproc.pyo
${PYSITELIB}/scrapy/utils/testsite.py
${PYSITELIB}/scrapy/utils/testsite.pyc
${PYSITELIB}/scrapy/utils/testsite.pyo
@


1.16
log
@py-scrapy: updated to 2.14.0

Scrapy 2.14.0 (2026-01-05)

Highlights:

- More coroutine-based replacements for Deferred-based APIs
- The default priority queue is now ``DownloaderAwarePriorityQueue``
- Dropped support for Python 3.9 and PyPy 3.10
- Improved and documented the API for custom download handlers
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.15 2025/05/15 09:21:59 adam Exp $
d85 3
d432 6
d510 3
@


1.15
log
@py-scrapy: updated to 2.13.0

Scrapy 2.13.0 (2025-05-08)

Highlights:

- The asyncio reactor is now enabled by default

- Replaced ``start_requests()`` (sync) with :meth:`~scrapy.Spider.start`
  (async) and changed how it is iterated.

- Added the :reqmeta:`allow_offsite` request meta key

- :ref:`Spider middlewares that don't support asynchronous spider output
  <sync-async-spider-middleware>` are deprecated

- Added a base class for :ref:`universal spider middlewares
  <universal-spider-middleware>`
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.14 2025/04/14 20:28:02 adam Exp $
d85 3
d223 3
a377 3
${PYSITELIB}/scrapy/spidermiddlewares/offsite.py
${PYSITELIB}/scrapy/spidermiddlewares/offsite.pyc
${PYSITELIB}/scrapy/spidermiddlewares/offsite.pyo
d432 3
@


1.14
log
@Fix PLIST after py-setuptools update; bump depends and revision
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.13 2024/11/30 06:56:49 adam Exp $
a8 1
${PYSITELIB}/${WHEEL_INFODIR}/top_level.txt
d363 3
d378 3
@


1.13
log
@py-scrapy: updated to 2.12.0

Scrapy 2.12.0 (2024-11-18)

Highlights:

- Dropped support for Python 3.8, added support for Python 3.13
- :meth:`~scrapy.Spider.start_requests` can now yield items
- Added :class:`~scrapy.http.JsonResponse`
- Added :setting:`CLOSESPIDER_PAGECOUNT_NO_ITEM`
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.12 2024/05/14 19:15:59 adam Exp $
a2 2
${PYSITELIB}/${WHEEL_INFODIR}/AUTHORS
${PYSITELIB}/${WHEEL_INFODIR}/LICENSE
d7 2
@


1.12
log
@py-scrapy: updated to 2.11.2

Scrapy 2.11.2 (2024-05-14)
--------------------------

Security bug fixes
~~~~~~~~~~~~~~~~~~

-   Redirects to non-HTTP protocols are no longer followed. Please, see the
    `23j4-mw76-5v7h security advisory`_ for more information. (:issue:`457`)

    .. _23j4-mw76-5v7h security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-23j4-mw76-5v7h

-   The ``Authorization`` header is now dropped on redirects to a different
    scheme (``http://`` or ``https://``) or port, even if the domain is the
    same. Please, see the `4qqq-9vqf-3h3f security advisory`_ for more
    information.

    .. _4qqq-9vqf-3h3f security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-4qqq-9vqf-3h3f

-   When using system proxy settings that are different for ``http://`` and
    ``https://``, redirects to a different URL scheme will now also trigger the
    corresponding change in proxy settings for the redirected request. Please,
    see the `jm3v-qxmh-hxwv security advisory`_ for more information.
    (:issue:`767`)

    .. _jm3v-qxmh-hxwv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-jm3v-qxmh-hxwv

-   :attr:`Spider.allowed_domains <scrapy.Spider.allowed_domains>` is now
    enforced for all requests, and not only requests from spider callbacks.
    (:issue:`1042`, :issue:`2241`, :issue:`6358`)

-   :func:`~scrapy.utils.iterators.xmliter_lxml` no longer resolves XML
    entities. (:issue:`6265`)

-   defusedxml_ is now used to make
    :class:`scrapy.http.request.rpc.XmlRpcRequest` more secure.
    (:issue:`6250`, :issue:`6251`)

    .. _defusedxml: https://github.com/tiran/defusedxml

Bug fixes
~~~~~~~~~

-   Restored support for brotlipy_, which had been dropped in Scrapy 2.11.1 in
    favor of brotli_. (:issue:`6261`)

    .. _brotli: https://github.com/google/brotli

    .. note:: brotlipy is deprecated, both in Scrapy and upstream. Use brotli
        instead if you can.

-   Make :setting:`METAREFRESH_IGNORE_TAGS` ``["noscript"]`` by default. This
    prevents
    :class:`~scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware` from
    following redirects that would not be followed by web browsers with
    JavaScript enabled. (:issue:`6342`, :issue:`6347`)

-   During :ref:`feed export <topics-feed-exports>`, do not close the
    underlying file from :ref:`built-in post-processing plugins
    <builtin-plugins>`.
    (:issue:`5932`, :issue:`6178`, :issue:`6239`)

-   :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
    now properly applies the ``unique`` and ``canonicalize`` parameters.
    (:issue:`3273`, :issue:`6221`)

-   Do not initialize the scheduler disk queue if :setting:`JOBDIR` is an empty
    string. (:issue:`6121`, :issue:`6124`)

-   Fix :attr:`Spider.logger <scrapy.Spider.logger>` not logging custom extra
    information. (:issue:`6323`, :issue:`6324`)

-   ``robots.txt`` files with a non-UTF-8 encoding no longer prevent parsing
    the UTF-8-compatible (e.g. ASCII) parts of the document.
    (:issue:`6292`, :issue:`6298`)

-   :meth:`scrapy.http.cookies.WrappedRequest.get_header` no longer raises an
    exception if ``default`` is ``None``.
    (:issue:`6308`, :issue:`6310`)

-   :class:`~scrapy.selector.Selector` now uses
    :func:`scrapy.utils.response.get_base_url` to determine the base URL of a
    given :class:`~scrapy.http.Response`. (:issue:`6265`)

-   The :meth:`media_to_download` method of :ref:`media pipelines
    <topics-media-pipeline>` now logs exceptions before stripping them.
    (:issue:`5067`, :issue:`5068`)

-   When passing a callback to the :command:`parse` command, build the callback
    callable with the right signature.
    (:issue:`6182`)

Documentation
~~~~~~~~~~~~~

-   Add a FAQ entry about :ref:`creating blank requests <faq-blank-request>`.
    (:issue:`6203`, :issue:`6208`)

-   Document that :attr:`scrapy.selector.Selector.type` can be ``"json"``.
    (:issue:`6328`, :issue:`6334`)

Quality assurance
~~~~~~~~~~~~~~~~~

-   Make builds reproducible. (:issue:`5019`, :issue:`6322`)

-   Packaging and test fixes.
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.11 2024/02/16 19:02:45 adam Exp $
a250 3
${PYSITELIB}/scrapy/http/common.py
${PYSITELIB}/scrapy/http/common.pyc
${PYSITELIB}/scrapy/http/common.pyo
d275 3
a301 6
${PYSITELIB}/scrapy/loader/common.py
${PYSITELIB}/scrapy/loader/common.pyc
${PYSITELIB}/scrapy/loader/common.pyo
${PYSITELIB}/scrapy/loader/processors.py
${PYSITELIB}/scrapy/loader/processors.pyc
${PYSITELIB}/scrapy/loader/processors.pyo
d327 1
@


1.11
log
@py-scrapy: updated to 2.11.1

Scrapy 2.11.1 (2024-02-14)
--------------------------

Highlights:

-   Security bug fixes.

-   Support for Twisted >= 23.8.0.

-   Documentation improvements.

Security bug fixes
~~~~~~~~~~~~~~~~~~

-   Addressed `ReDoS vulnerabilities`_:

    -   ``scrapy.utils.iterators.xmliter`` is now deprecated in favor of
        :func:`~scrapy.utils.iterators.xmliter_lxml`, which
        :class:`~scrapy.spiders.XMLFeedSpider` now uses.

        To minimize the impact of this change on existing code,
        :func:`~scrapy.utils.iterators.xmliter_lxml` now supports indicating
        the node namespace with a prefix in the node name, and big files with
        highly nested trees when using libxml2 2.7+.

    -   Fixed regular expressions in the implementation of the
        :func:`~scrapy.utils.response.open_in_browser` function.

    Please, see the `cc65-xxvf-f7r9 security advisory`_ for more information.

    .. _ReDoS vulnerabilities: https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS
    .. _cc65-xxvf-f7r9 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cc65-xxvf-f7r9

-   :setting:`DOWNLOAD_MAXSIZE` and :setting:`DOWNLOAD_WARNSIZE` now also apply
    to the decompressed response body. Please, see the `7j7m-v7m3-jqm7 security
    advisory`_ for more information.

    .. _7j7m-v7m3-jqm7 security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-7j7m-v7m3-jqm7

-   Also in relation with the `7j7m-v7m3-jqm7 security advisory`_, the
    deprecated ``scrapy.downloadermiddlewares.decompression`` module has been
    removed.

-   The ``Authorization`` header is now dropped on redirects to a different
    domain. Please, see the `cw9j-q3vf-hrrv security advisory`_ for more
    information.

    .. _cw9j-q3vf-hrrv security advisory: https://github.com/scrapy/scrapy/security/advisories/GHSA-cw9j-q3vf-hrrv

Modified requirements
~~~~~~~~~~~~~~~~~~~~~

-   The Twisted dependency is no longer restricted to < 23.8.0. (:issue:`6024`,
    :issue:`6064`, :issue:`6142`)

Bug fixes
~~~~~~~~~

-   The OS signal handling code was refactored to no longer use private Twisted
    functions. (:issue:`6024`, :issue:`6064`, :issue:`6112`)

Documentation
~~~~~~~~~~~~~

-   Improved documentation for :class:`~scrapy.crawler.Crawler` initialization
    changes made in the 2.11.0 release. (:issue:`6057`, :issue:`6147`)

-   Extended documentation for :attr:`Request.meta <scrapy.http.Request.meta>`.
    (:issue:`5565`)

-   Fixed the :reqmeta:`dont_merge_cookies` documentation. (:issue:`5936`,
    :issue:`6077`)

-   Added a link to Zyte's export guides to the :ref:`feed exports
    <topics-feed-exports>` documentation. (:issue:`6183`)

-   Added a missing note about backward-incompatible changes in
    :class:`~scrapy.exporters.PythonItemExporter` to the 2.11.0 release notes.
    (:issue:`6060`, :issue:`6081`)

-   Added a missing note about removing the deprecated
    ``scrapy.utils.boto.is_botocore()`` function to the 2.8.0 release notes.
    (:issue:`6056`, :issue:`6061`)

-   Other documentation improvements. (:issue:`6128`, :issue:`6144`,
    :issue:`6163`, :issue:`6190`, :issue:`6192`)

Quality assurance
~~~~~~~~~~~~~~~~~

-   Added Python 3.12 to the CI configuration, re-enabled tests that were
    disabled when the pre-release support was added. (:issue:`5985`,
    :issue:`6083`, :issue:`6098`)

-   Fixed a test issue on PyPy 7.3.14. (:issue:`6204`, :issue:`6205`)
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.10 2023/10/10 17:18:23 triaxx Exp $
d173 3
@


1.10
log
@py-scrapy: Update to 2.11.0

upstream changes:
-----------------
  * 2.11.0: https://docs.scrapy.org/en/latest/news.html#scrapy-2-11-0-2023-09-18
  * 2.10.0: https://docs.scrapy.org/en/2.10/news.html#scrapy-2-10-0-2023-08-04
@
text
@d1 1
a1 1
@@comment $NetBSD$
d3 7
a9 7
${PYSITELIB}/${EGG_INFODIR}/PKG-INFO
${PYSITELIB}/${EGG_INFODIR}/SOURCES.txt
${PYSITELIB}/${EGG_INFODIR}/dependency_links.txt
${PYSITELIB}/${EGG_INFODIR}/entry_points.txt
${PYSITELIB}/${EGG_INFODIR}/not-zip-safe
${PYSITELIB}/${EGG_INFODIR}/requires.txt
${PYSITELIB}/${EGG_INFODIR}/top_level.txt
a154 3
${PYSITELIB}/scrapy/downloadermiddlewares/decompression.py
${PYSITELIB}/scrapy/downloadermiddlewares/decompression.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/decompression.pyo
d420 3
@


1.9
log
@py-scrapy: updated to 2.8.0

Scrapy 2.8.0 (2023-02-02)
-------------------------

This is a maintenance release, with minor features, bug fixes, and cleanups.

Deprecation removals
~~~~~~~~~~~~~~~~~~~~
-   The ``scrapy.utils.gz.read1`` function, deprecated in Scrapy 2.0, has now
    been removed. Use the :meth:`~io.BufferedIOBase.read1` method of
    :class:`~gzip.GzipFile` instead.
-   The ``scrapy.utils.python.to_native_str`` function, deprecated in Scrapy
    2.0, has now been removed. Use :func:`scrapy.utils.python.to_unicode`
    instead.
-   The ``scrapy.utils.python.MutableChain.next`` method, deprecated in Scrapy
    2.0, has now been removed. Use
    :meth:`~scrapy.utils.python.MutableChain.__next__` instead.
-   The ``scrapy.linkextractors.FilteringLinkExtractor`` class, deprecated
    in Scrapy 2.0, has now been removed. Use
    :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
    instead.
-   Support for using environment variables prefixed with ``SCRAPY_`` to
    override settings, deprecated in Scrapy 2.0, has now been removed.
-   Support for the ``noconnect`` query string argument in proxy URLs,
    deprecated in Scrapy 2.0, has now been removed. We expect proxies that used
    to need it to work fine without it.
-   The ``scrapy.utils.python.retry_on_eintr`` function, deprecated in Scrapy
    2.3, has now been removed.
-   The ``scrapy.utils.python.WeakKeyCache`` class, deprecated in Scrapy 2.4,
    has now been removed.

Deprecations
~~~~~~~~~~~~
-   :exc:`scrapy.pipelines.images.NoimagesDrop` is now deprecated.
-   :meth:`ImagesPipeline.convert_image
    <scrapy.pipelines.images.ImagesPipeline.convert_image>` must now accept a
    ``response_body`` parameter.

New features
~~~~~~~~~~~~
-   Applied black_ coding style to files generated with the
    :command:`genspider` and :command:`startproject` commands.
    .. _black: https://black.readthedocs.io/en/stable/

-   :setting:`FEED_EXPORT_ENCODING` is now set to ``"utf-8"`` in the
    ``settings.py`` file that the :command:`startproject` command generates.
    With this value, JSON exports won’t force the use of escape sequences for
    non-ASCII characters.
-   The :class:`~scrapy.extensions.memusage.MemoryUsage` extension now logs the
    peak memory usage during checks, and the binary unit MiB is now used to
    avoid confusion.
-   The ``callback`` parameter of :class:`~scrapy.http.Request` can now be set
    to :func:`scrapy.http.request.NO_CALLBACK`, to distinguish it from
    ``None``, as the latter indicates that the default spider callback
    (:meth:`~scrapy.Spider.parse`) is to be used.

Bug fixes
~~~~~~~~~
-   Enabled unsafe legacy SSL renegotiation to fix access to some outdated
    websites.
-   Fixed STARTTLS-based email delivery not working with Twisted 21.2.0 and
    better.
-   Fixed the :meth:`finish_exporting` method of :ref:`item exporters
    <topics-exporters>` not being called for empty files.
-   Fixed HTTP/2 responses getting only the last value for a header when
    multiple headers with the same name are received.
-   Fixed an exception raised by the :command:`shell` command on some cases
    when :ref:`using asyncio <using-asyncio>`.
-   When using :class:`~scrapy.spiders.CrawlSpider`, callback keyword arguments
    (``cb_kwargs``) added to a request in the ``process_request`` callback of a
    :class:`~scrapy.spiders.Rule` will no longer be ignored.
-   The :ref:`images pipeline <images-pipeline>` no longer re-encodes JPEG
    files.
-   Fixed the handling of transparent WebP images by the :ref:`images pipeline
    <images-pipeline>`.
-   :func:`scrapy.shell.inspect_response` no longer inhibits ``SIGINT``
    (Ctrl+C).
-   :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
    with ``unique=False`` no longer filters out links that have identical URL
    *and* text.
-   :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` now
    ignores URL protocols that do not support ``robots.txt`` (``data://``,
    ``file://``).
-   Silenced the ``filelock`` debug log messages introduced in Scrapy 2.6.
-   Fixed the output of ``scrapy -h`` showing an unintended ``**commands**``
    line.
-   Made the active project indication in the output of :ref:`commands
    <topics-commands>` more clear.

Documentation
~~~~~~~~~~~~~
-   Documented how to :ref:`debug spiders from Visual Studio Code
    <debug-vscode>`.
-   Documented how :setting:`DOWNLOAD_DELAY` affects per-domain concurrency.
-   Improved consistency.
-   Fixed typos.

Quality assurance
~~~~~~~~~~~~~~~~~
-   Applied :ref:`black coding style <coding-style>`, sorted import statements,
    and introduced :ref:`pre-commit <scrapy-pre-commit>`.
-   Switched from :mod:`os.path` to :mod:`pathlib`.
-   Addressed many issues reported by Pylint.
-   Improved code readability.
-   Improved package metadata.
-   Removed direct invocations of ``setup.py``.
-   Removed unnecessary :class:`~collections.OrderedDict` usages.
-   Removed unnecessary ``__str__`` definitions.
-   Removed obsolete code and comments.
-   Fixed test and CI issues.
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.8 2021/03/22 08:56:56 triaxx Exp $
d17 3
d230 3
a491 3
${PYSITELIB}/scrapy/utils/reqser.py
${PYSITELIB}/scrapy/utils/reqser.pyc
${PYSITELIB}/scrapy/utils/reqser.pyo
@


1.8
log
@py-scrapy: Update to 2.4.1

upstream cheanges:
------------------
A lot of changes listed at https://github.com/scrapy/scrapy/blob/master/docs/news.rst
@
text
@d1 1
a1 1
@@comment $NetBSD$
d101 3
d119 12
d227 3
d417 3
a458 3
${PYSITELIB}/scrapy/utils/http.py
${PYSITELIB}/scrapy/utils/http.pyc
${PYSITELIB}/scrapy/utils/http.pyo
a470 3
${PYSITELIB}/scrapy/utils/markup.py
${PYSITELIB}/scrapy/utils/markup.pyc
${PYSITELIB}/scrapy/utils/markup.pyo
a473 3
${PYSITELIB}/scrapy/utils/multipart.py
${PYSITELIB}/scrapy/utils/multipart.pyc
${PYSITELIB}/scrapy/utils/multipart.pyo
a479 3
${PYSITELIB}/scrapy/utils/py36.py
${PYSITELIB}/scrapy/utils/py36.pyc
${PYSITELIB}/scrapy/utils/py36.pyo
@


1.7
log
@py-scrapy: updated to 1.8.0

Scrapy 1.8.0:

Highlights:
* Dropped Python 3.4 support and updated minimum requirements; made Python 3.8
  support official
* New :meth:`Request.from_curl <scrapy.http.Request.from_curl>` class method
* New :setting:`ROBOTSTXT_PARSER` and :setting:`ROBOTSTXT_USER_AGENT` settings
* New :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` and
  :setting:`DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING` settings
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.6 2019/08/22 08:21:11 adam Exp $
a16 3
${PYSITELIB}/scrapy/_monkeypatches.py
${PYSITELIB}/scrapy/_monkeypatches.pyc
${PYSITELIB}/scrapy/_monkeypatches.pyo
a133 3
${PYSITELIB}/scrapy/downloadermiddlewares/chunked.py
${PYSITELIB}/scrapy/downloadermiddlewares/chunked.pyc
${PYSITELIB}/scrapy/downloadermiddlewares/chunked.pyo
a271 3
${PYSITELIB}/scrapy/linkextractors/htmlparser.py
${PYSITELIB}/scrapy/linkextractors/htmlparser.pyc
${PYSITELIB}/scrapy/linkextractors/htmlparser.pyo
a274 6
${PYSITELIB}/scrapy/linkextractors/regex.py
${PYSITELIB}/scrapy/linkextractors/regex.pyc
${PYSITELIB}/scrapy/linkextractors/regex.pyo
${PYSITELIB}/scrapy/linkextractors/sgml.py
${PYSITELIB}/scrapy/linkextractors/sgml.pyc
${PYSITELIB}/scrapy/linkextractors/sgml.pyo
a329 3
${PYSITELIB}/scrapy/settings/deprecated.py
${PYSITELIB}/scrapy/settings/deprecated.pyc
${PYSITELIB}/scrapy/settings/deprecated.pyo
d468 3
@


1.6
log
@py-scrapy: updated to 1.7.3

Scrapy 1.7.3:
Enforce lxml 4.3.5 or lower for Python 3.4 (issue 3912, issue 3918).

Scrapy 1.7.2:
Fix Python 2 support (issue 3889, issue 3893, issue 3896).

Scrapy 1.7.1:
Re-packaging of Scrapy 1.7.0, which was missing some changes in PyPI.

Scrapy 1.7.0:
Highlights:
Improvements for crawls targeting multiple domains
A cleaner way to pass arguments to callbacks
A new class for JSON requests
Improvements for rule-based spiders
New features for feed exports


Backward-incompatible changes

429 is now part of the RETRY_HTTP_CODES setting by default
This change is backward incompatible. If you don’t want to retry 429, you must override RETRY_HTTP_CODES accordingly.

Crawler, CrawlerRunner.crawl and CrawlerRunner.create_crawler no longer accept a Spider subclass instance, they only accept a Spider subclass now.
Spider subclass instances were never meant to work, and they were not working as one would expect: instead of using the passed Spider subclass instance, their from_crawler method was called to generate a new instance.

Non-default values for the SCHEDULER_PRIORITY_QUEUE setting may stop working. Scheduler priority queue classes now need to handle Request objects instead of arbitrary Python data structures.


New features

A new scheduler priority queue, scrapy.pqueues.DownloaderAwarePriorityQueue, may be enabled for a significant scheduling improvement on crawls targetting multiple web domains, at the cost of no CONCURRENT_REQUESTS_PER_IP support (issue 3520)
A new Request.cb_kwargs attribute provides a cleaner way to pass keyword arguments to callback methods (issue 1138, issue 3563)
A new JSONRequest class offers a more convenient way to build JSON requests (issue 3504, issue 3505)
A process_request callback passed to the Rule constructor now receives the Response object that originated the request as its second argument (issue 3682)
A new restrict_text parameter for the LinkExtractor constructor allows filtering links by linking text (issue 3622, issue 3635)
A new FEED_STORAGE_S3_ACL setting allows defining a custom ACL for feeds exported to Amazon S3 (issue 3607)
A new FEED_STORAGE_FTP_ACTIVE setting allows using FTP’s active connection mode for feeds exported to FTP servers (issue 3829)
A new METAREFRESH_IGNORE_TAGS setting allows overriding which HTML tags are ignored when searching a response for HTML meta tags that trigger a redirect (issue 1422, issue 3768)
A new redirect_reasons request meta key exposes the reason (status code, meta refresh) behind every followed redirect (issue 3581, issue 3687)
The SCRAPY_CHECK variable is now set to the true string during runs of the check command, which allows detecting contract check runs from code (issue 3704, issue 3739)
A new Item.deepcopy() method makes it easier to deep-copy items (issue 1493, issue 3671)
CoreStats also logs elapsed_time_seconds now (issue 3638)
Exceptions from ItemLoader input and output processors are now more verbose (issue 3836, issue 3840)
Crawler, CrawlerRunner.crawl and CrawlerRunner.create_crawler now fail gracefully if they receive a Spider subclass instance instead of the subclass itself (issue 2283, issue 3610, issue 3872)


Bug fixes

process_spider_exception() is now also invoked for generators (issue 220, issue 2061)
System exceptions like KeyboardInterrupt are no longer caught (issue 3726)
ItemLoader.load_item() no longer makes later calls to ItemLoader.get_output_value() or ItemLoader.load_item() return empty data (issue 3804, issue 3819)
The images pipeline (ImagesPipeline) no longer ignores these Amazon S3 settings: AWS_ENDPOINT_URL, AWS_REGION_NAME, AWS_USE_SSL, AWS_VERIFY (issue 3625)
Fixed a memory leak in MediaPipeline affecting, for example, non-200 responses and exceptions from custom middlewares (issue 3813)
Requests with private callbacks are now correctly unserialized from disk (issue 3790)
FormRequest.from_response() now handles invalid methods like major web browsers
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.5 2019/01/31 09:07:46 adam Exp $
d330 3
d429 3
d513 3
a536 9
${PYSITELIB}/scrapy/xlib/__init__.py
${PYSITELIB}/scrapy/xlib/__init__.pyc
${PYSITELIB}/scrapy/xlib/__init__.pyo
${PYSITELIB}/scrapy/xlib/pydispatch.py
${PYSITELIB}/scrapy/xlib/pydispatch.pyc
${PYSITELIB}/scrapy/xlib/pydispatch.pyo
${PYSITELIB}/scrapy/xlib/tx.py
${PYSITELIB}/scrapy/xlib/tx.pyc
${PYSITELIB}/scrapy/xlib/tx.pyo
@


1.5
log
@py-scrapy: updated to 1.6.0

Scrapy 1.6.0:

Highlights:
* better Windows support;
* Python 3.7 compatibility;
* big documentation improvements, including a switch
  from .extract_first() + .extract() API to .get() + .getall()
  API;
* feed exports, FilePipeline and MediaPipeline improvements;
* better extensibility: :signal:item_error and
  :signal:request_reached_downloader signals; from_crawler support
  for feed exporters, feed storages and dupefilters.
* scrapy.contracts fixes and new features;
* telnet console security improvements, first released as a
  backport in :ref:release-1.5.2;
* clean-up of the deprecated code;
* various bug fixes, small new features and usability improvements across
  the codebase.
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.4 2019/01/24 14:11:48 adam Exp $
a67 3
${PYSITELIB}/scrapy/conf.py
${PYSITELIB}/scrapy/conf.pyc
${PYSITELIB}/scrapy/conf.pyo
d248 3
a298 3
${PYSITELIB}/scrapy/log.py
${PYSITELIB}/scrapy/log.pyc
${PYSITELIB}/scrapy/log.pyo
d321 3
a332 6
${PYSITELIB}/scrapy/selector/csstranslator.py
${PYSITELIB}/scrapy/selector/csstranslator.pyc
${PYSITELIB}/scrapy/selector/csstranslator.pyo
${PYSITELIB}/scrapy/selector/lxmlsel.py
${PYSITELIB}/scrapy/selector/lxmlsel.pyc
${PYSITELIB}/scrapy/selector/lxmlsel.pyo
a395 3
${PYSITELIB}/scrapy/telnet.py
${PYSITELIB}/scrapy/telnet.pyc
${PYSITELIB}/scrapy/telnet.pyo
@


1.4
log
@py-scrapy: updated to 1.5.2

Scrapy 1.5.2:

* *Security bugfix*: Telnet console extension can be easily exploited by rogue
  websites POSTing content to http://localhost:6023, we haven't found a way to
  exploit it from Scrapy, but it is very easy to trick a browser to do so and
  elevates the risk for local development environment.

  *The fix is backwards incompatible*, it enables telnet user-password
  authentication by default with a random generated password. If you can't
  upgrade right away, please consider setting :setting:TELNET_CONSOLE_PORT
  out of its default value.

  See :ref:telnet console <topics-telnetconsole> documentation for more info

* Backport CI build failure under GCE environemnt due to boto import error.
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.3 2018/01/04 21:31:41 adam Exp $
a22 3
${PYSITELIB}/scrapy/command.py
${PYSITELIB}/scrapy/command.pyc
${PYSITELIB}/scrapy/command.pyo
a76 168
${PYSITELIB}/scrapy/contrib/__init__.py
${PYSITELIB}/scrapy/contrib/__init__.pyc
${PYSITELIB}/scrapy/contrib/__init__.pyo
${PYSITELIB}/scrapy/contrib/closespider.py
${PYSITELIB}/scrapy/contrib/closespider.pyc
${PYSITELIB}/scrapy/contrib/closespider.pyo
${PYSITELIB}/scrapy/contrib/corestats.py
${PYSITELIB}/scrapy/contrib/corestats.pyc
${PYSITELIB}/scrapy/contrib/corestats.pyo
${PYSITELIB}/scrapy/contrib/debug.py
${PYSITELIB}/scrapy/contrib/debug.pyc
${PYSITELIB}/scrapy/contrib/debug.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/__init__.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/__init__.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/__init__.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/ajaxcrawl.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/ajaxcrawl.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/ajaxcrawl.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/chunked.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/chunked.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/chunked.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/cookies.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/cookies.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/cookies.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/decompression.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/decompression.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/decompression.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/defaultheaders.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/defaultheaders.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/defaultheaders.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/downloadtimeout.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/downloadtimeout.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/downloadtimeout.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpauth.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpauth.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpauth.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpcache.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpcache.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpcache.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpcompression.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpcompression.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpcompression.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpproxy.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpproxy.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/httpproxy.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/redirect.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/redirect.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/redirect.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/retry.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/retry.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/retry.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/robotstxt.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/robotstxt.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/robotstxt.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/stats.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/stats.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/stats.pyo
${PYSITELIB}/scrapy/contrib/downloadermiddleware/useragent.py
${PYSITELIB}/scrapy/contrib/downloadermiddleware/useragent.pyc
${PYSITELIB}/scrapy/contrib/downloadermiddleware/useragent.pyo
${PYSITELIB}/scrapy/contrib/exporter/__init__.py
${PYSITELIB}/scrapy/contrib/exporter/__init__.pyc
${PYSITELIB}/scrapy/contrib/exporter/__init__.pyo
${PYSITELIB}/scrapy/contrib/feedexport.py
${PYSITELIB}/scrapy/contrib/feedexport.pyc
${PYSITELIB}/scrapy/contrib/feedexport.pyo
${PYSITELIB}/scrapy/contrib/httpcache.py
${PYSITELIB}/scrapy/contrib/httpcache.pyc
${PYSITELIB}/scrapy/contrib/httpcache.pyo
${PYSITELIB}/scrapy/contrib/linkextractors/__init__.py
${PYSITELIB}/scrapy/contrib/linkextractors/__init__.pyc
${PYSITELIB}/scrapy/contrib/linkextractors/__init__.pyo
${PYSITELIB}/scrapy/contrib/linkextractors/htmlparser.py
${PYSITELIB}/scrapy/contrib/linkextractors/htmlparser.pyc
${PYSITELIB}/scrapy/contrib/linkextractors/htmlparser.pyo
${PYSITELIB}/scrapy/contrib/linkextractors/lxmlhtml.py
${PYSITELIB}/scrapy/contrib/linkextractors/lxmlhtml.pyc
${PYSITELIB}/scrapy/contrib/linkextractors/lxmlhtml.pyo
${PYSITELIB}/scrapy/contrib/linkextractors/regex.py
${PYSITELIB}/scrapy/contrib/linkextractors/regex.pyc
${PYSITELIB}/scrapy/contrib/linkextractors/regex.pyo
${PYSITELIB}/scrapy/contrib/linkextractors/sgml.py
${PYSITELIB}/scrapy/contrib/linkextractors/sgml.pyc
${PYSITELIB}/scrapy/contrib/linkextractors/sgml.pyo
${PYSITELIB}/scrapy/contrib/loader/__init__.py
${PYSITELIB}/scrapy/contrib/loader/__init__.pyc
${PYSITELIB}/scrapy/contrib/loader/__init__.pyo
${PYSITELIB}/scrapy/contrib/loader/common.py
${PYSITELIB}/scrapy/contrib/loader/common.pyc
${PYSITELIB}/scrapy/contrib/loader/common.pyo
${PYSITELIB}/scrapy/contrib/loader/processor.py
${PYSITELIB}/scrapy/contrib/loader/processor.pyc
${PYSITELIB}/scrapy/contrib/loader/processor.pyo
${PYSITELIB}/scrapy/contrib/logstats.py
${PYSITELIB}/scrapy/contrib/logstats.pyc
${PYSITELIB}/scrapy/contrib/logstats.pyo
${PYSITELIB}/scrapy/contrib/memdebug.py
${PYSITELIB}/scrapy/contrib/memdebug.pyc
${PYSITELIB}/scrapy/contrib/memdebug.pyo
${PYSITELIB}/scrapy/contrib/memusage.py
${PYSITELIB}/scrapy/contrib/memusage.pyc
${PYSITELIB}/scrapy/contrib/memusage.pyo
${PYSITELIB}/scrapy/contrib/pipeline/__init__.py
${PYSITELIB}/scrapy/contrib/pipeline/__init__.pyc
${PYSITELIB}/scrapy/contrib/pipeline/__init__.pyo
${PYSITELIB}/scrapy/contrib/pipeline/files.py
${PYSITELIB}/scrapy/contrib/pipeline/files.pyc
${PYSITELIB}/scrapy/contrib/pipeline/files.pyo
${PYSITELIB}/scrapy/contrib/pipeline/images.py
${PYSITELIB}/scrapy/contrib/pipeline/images.pyc
${PYSITELIB}/scrapy/contrib/pipeline/images.pyo
${PYSITELIB}/scrapy/contrib/pipeline/media.py
${PYSITELIB}/scrapy/contrib/pipeline/media.pyc
${PYSITELIB}/scrapy/contrib/pipeline/media.pyo
${PYSITELIB}/scrapy/contrib/spidermiddleware/__init__.py
${PYSITELIB}/scrapy/contrib/spidermiddleware/__init__.pyc
${PYSITELIB}/scrapy/contrib/spidermiddleware/__init__.pyo
${PYSITELIB}/scrapy/contrib/spidermiddleware/depth.py
${PYSITELIB}/scrapy/contrib/spidermiddleware/depth.pyc
${PYSITELIB}/scrapy/contrib/spidermiddleware/depth.pyo
${PYSITELIB}/scrapy/contrib/spidermiddleware/httperror.py
${PYSITELIB}/scrapy/contrib/spidermiddleware/httperror.pyc
${PYSITELIB}/scrapy/contrib/spidermiddleware/httperror.pyo
${PYSITELIB}/scrapy/contrib/spidermiddleware/offsite.py
${PYSITELIB}/scrapy/contrib/spidermiddleware/offsite.pyc
${PYSITELIB}/scrapy/contrib/spidermiddleware/offsite.pyo
${PYSITELIB}/scrapy/contrib/spidermiddleware/referer.py
${PYSITELIB}/scrapy/contrib/spidermiddleware/referer.pyc
${PYSITELIB}/scrapy/contrib/spidermiddleware/referer.pyo
${PYSITELIB}/scrapy/contrib/spidermiddleware/urllength.py
${PYSITELIB}/scrapy/contrib/spidermiddleware/urllength.pyc
${PYSITELIB}/scrapy/contrib/spidermiddleware/urllength.pyo
${PYSITELIB}/scrapy/contrib/spiders/__init__.py
${PYSITELIB}/scrapy/contrib/spiders/__init__.pyc
${PYSITELIB}/scrapy/contrib/spiders/__init__.pyo
${PYSITELIB}/scrapy/contrib/spiders/crawl.py
${PYSITELIB}/scrapy/contrib/spiders/crawl.pyc
${PYSITELIB}/scrapy/contrib/spiders/crawl.pyo
${PYSITELIB}/scrapy/contrib/spiders/feed.py
${PYSITELIB}/scrapy/contrib/spiders/feed.pyc
${PYSITELIB}/scrapy/contrib/spiders/feed.pyo
${PYSITELIB}/scrapy/contrib/spiders/init.py
${PYSITELIB}/scrapy/contrib/spiders/init.pyc
${PYSITELIB}/scrapy/contrib/spiders/init.pyo
${PYSITELIB}/scrapy/contrib/spiders/sitemap.py
${PYSITELIB}/scrapy/contrib/spiders/sitemap.pyc
${PYSITELIB}/scrapy/contrib/spiders/sitemap.pyo
${PYSITELIB}/scrapy/contrib/spiderstate.py
${PYSITELIB}/scrapy/contrib/spiderstate.pyc
${PYSITELIB}/scrapy/contrib/spiderstate.pyo
${PYSITELIB}/scrapy/contrib/statsmailer.py
${PYSITELIB}/scrapy/contrib/statsmailer.pyc
${PYSITELIB}/scrapy/contrib/statsmailer.pyo
${PYSITELIB}/scrapy/contrib/throttle.py
${PYSITELIB}/scrapy/contrib/throttle.pyc
${PYSITELIB}/scrapy/contrib/throttle.pyo
${PYSITELIB}/scrapy/contrib_exp/__init__.py
${PYSITELIB}/scrapy/contrib_exp/__init__.pyc
${PYSITELIB}/scrapy/contrib_exp/__init__.pyo
${PYSITELIB}/scrapy/contrib_exp/downloadermiddleware/__init__.py
${PYSITELIB}/scrapy/contrib_exp/downloadermiddleware/__init__.pyc
${PYSITELIB}/scrapy/contrib_exp/downloadermiddleware/__init__.pyo
${PYSITELIB}/scrapy/contrib_exp/downloadermiddleware/decompression.py
${PYSITELIB}/scrapy/contrib_exp/downloadermiddleware/decompression.pyc
${PYSITELIB}/scrapy/contrib_exp/downloadermiddleware/decompression.pyo
${PYSITELIB}/scrapy/contrib_exp/iterators.py
${PYSITELIB}/scrapy/contrib_exp/iterators.pyc
${PYSITELIB}/scrapy/contrib_exp/iterators.pyo
a181 3
${PYSITELIB}/scrapy/dupefilter.py
${PYSITELIB}/scrapy/dupefilter.pyc
${PYSITELIB}/scrapy/dupefilter.pyo
a274 3
${PYSITELIB}/scrapy/linkextractor.py
${PYSITELIB}/scrapy/linkextractor.pyc
${PYSITELIB}/scrapy/linkextractor.pyo
a323 3
${PYSITELIB}/scrapy/project.py
${PYSITELIB}/scrapy/project.pyc
${PYSITELIB}/scrapy/project.pyo
a359 3
${PYSITELIB}/scrapy/spider.py
${PYSITELIB}/scrapy/spider.pyc
${PYSITELIB}/scrapy/spider.pyo
a362 3
${PYSITELIB}/scrapy/spidermanager.py
${PYSITELIB}/scrapy/spidermanager.pyc
${PYSITELIB}/scrapy/spidermanager.pyo
a395 3
${PYSITELIB}/scrapy/squeue.py
${PYSITELIB}/scrapy/squeue.pyc
${PYSITELIB}/scrapy/squeue.pyo
a398 6
${PYSITELIB}/scrapy/stats.py
${PYSITELIB}/scrapy/stats.pyc
${PYSITELIB}/scrapy/stats.pyo
${PYSITELIB}/scrapy/statscol.py
${PYSITELIB}/scrapy/statscol.pyc
${PYSITELIB}/scrapy/statscol.pyo
a437 3
${PYSITELIB}/scrapy/utils/decorator.py
${PYSITELIB}/scrapy/utils/decorator.pyc
${PYSITELIB}/scrapy/utils/decorator.pyo
@


1.3
log
@py-scrapy: updated to 1.5.0

Scrapy 1.5.0:
This release brings small new features and improvements across the codebase.
Some highlights:

* Google Cloud Storage is supported in FilesPipeline and ImagesPipeline.
* Crawling with proxy servers becomes more efficient, as connections
  to proxies can be reused now.
* Warnings, exception and logging messages are improved to make debugging
  easier.
* scrapy parse command now allows to set custom request meta via
  --meta argument.
* Compatibility with Python 3.6, PyPy and PyPy3 is improved;
  PyPy and PyPy3 are now supported officially, by running tests on CI.
* Better default handling of HTTP 308, 522 and 524 status codes.
* Documentation is improved, as usual.

Backwards Incompatible Changes
* Scrapy 1.5 drops support for Python 3.3.
* Default Scrapy User-Agent now uses https link to scrapy.org.
  **This is technically backwards-incompatible**; override
  :setting:USER_AGENT if you relied on old value.
* Logging of settings overridden by custom_settings is fixed;
  **this is technically backwards-incompatible** because the logger
  changes from [scrapy.utils.log] to [scrapy.crawler]. If you're
  parsing Scrapy logs, please update your log parsers.
* LinkExtractor now ignores m4v extension by default, this is change
  in behavior.
* 522 and 524 status codes are added to RETRY_HTTP_CODES

New features
- Support <link> tags in Response.follow
- Support for ptpython REPL
- Google Cloud Storage support for FilesPipeline and ImagesPipeline
- New --meta option of the "scrapy parse" command allows to pass additional
  request.meta
- Populate spider variable when using shell.inspect_response
- Handle HTTP 308 Permanent Redirect
- Add 522 and 524 to RETRY_HTTP_CODES
- Log versions information at startup
- scrapy.mail.MailSender now works in Python 3 (it requires Twisted 17.9.0)
- Connections to proxy servers are reused
- Add template for a downloader middleware
- Explicit message for NotImplementedError when parse callback not defined
- CrawlerProcess got an option to disable installation of root log handler
- LinkExtractor now ignores m4v extension by default
- Better log messages for responses over :setting:DOWNLOAD_WARNSIZE and
  :setting:DOWNLOAD_MAXSIZE limits
- Show warning when a URL is put to Spider.allowed_domains instead of
  a domain.

Bug fixes
- Fix logging of settings overridden by custom_settings;
  **this is technically backwards-incompatible** because the logger
  changes from [scrapy.utils.log] to [scrapy.crawler], so please
  update your log parsers if needed
- Default Scrapy User-Agent now uses https link to scrapy.org.
  **This is technically backwards-incompatible**; override
  :setting:USER_AGENT if you relied on old value.
- Fix PyPy and PyPy3 test failures, support them officially
- Fix DNS resolver when DNSCACHE_ENABLED=False
- Add cryptography for Debian Jessie tox test env
- Add verification to check if Request callback is callable
- Port extras/qpsclient.py to Python 3
- Use getfullargspec under the scenes for Python 3 to stop DeprecationWarning
- Update deprecated test aliases
- Fix SitemapSpider support for alternate links
@
text
@d1 2
a2 2
@@comment $NetBSD: PLIST,v 1.2 2017/05/20 06:25:36 adam Exp $
bin/scrapy${PYVERSSUFFIX}
@


1.2
log
@Scrapy 1.4 does not bring that many breathtaking new features
but quite a few handy improvements nonetheless.

Scrapy now supports anonymous FTP sessions with customizable user and
password via the new :setting:`FTP_USER` and :setting:`FTP_PASSWORD` settings.
And if you're using Twisted version 17.1.0 or above, FTP is now available
with Python 3.

There's a new :meth:`response.follow <scrapy.http.TextResponse.follow>` method
for creating requests; **it is now a recommended way to create Requests
in Scrapy spiders**. This method makes it easier to write correct
spiders; ``response.follow`` has several advantages over creating
``scrapy.Request`` objects directly:

* it handles relative URLs;
* it works properly with non-ascii URLs on non-UTF8 pages;
* in addition to absolute and relative URLs it supports Selectors;
  for ``<a>`` elements it can also extract their href values.
@
text
@d1 1
a1 1
@@comment $NetBSD: PLIST,v 1.1 2017/02/13 21:25:33 adam Exp $
d732 3
@


1.1
log
@Added www/py-scrapy version 1.3.2

Scrapy is a fast high-level web crawling and web scraping framework, used to
crawl websites and extract structured data from their pages. It can be used for
a wide range of purposes, from data mining to monitoring and automated testing.
@
text
@d1 2
a2 2
@@comment $NetBSD$
bin/scrapy
d14 3
d260 3
@