- Update to v2.6.2 
  Security bug fix:
  * When HttpProxyMiddleware processes a request with proxy metadata, and that proxy metadata includes proxy credentials,
    HttpProxyMiddleware sets the Proxy-Authentication header, but only if that header is not already set.
  * There are third-party proxy-rotation downloader middlewares that set different proxy metadata every time they process a request.
  * Because of request retries and redirects, the same request can be processed by downloader middlewares more than once,
    including both HttpProxyMiddleware and any third-party proxy-rotation downloader middleware.
  * These third-party proxy-rotation downloader middlewares could change the proxy metadata of a request to a new value,
    but fail to remove the Proxy-Authentication header from the previous value of the proxy metadata, causing the credentials of one
    proxy to be sent to a different proxy.
  * To prevent the unintended leaking of proxy credentials, the behavior of HttpProxyMiddleware is now as follows when processing a request:
    + If the request being processed defines proxy metadata that includes credentials, the Proxy-Authorization header is always updated 
    to feature those credentials.
    + If the request being processed defines proxy metadata without credentials, the Proxy-Authorization header is removed unless
    it was originally defined for the same proxy URL.
    + To remove proxy credentials while keeping the same proxy URL, remove the Proxy-Authorization header.
    + If the request has no proxy metadata, or that metadata is a falsy value (e.g. None), the Proxy-Authorization header is removed.
    + It is no longer possible to set a proxy URL through the proxy metadata but set the credentials through the Proxy-Authorization header.
    Set proxy credentials through the proxy metadata instead.
  * Also fixes the following regressions introduced in 2.6.0:
    + CrawlerProcess supports again crawling multiple spiders (issue 5435, issue 5436)
    + Installing a Twisted reactor before Scrapy does (e.g. importing twisted.internet.reactor somewhere at the module level)
    no longer prevents Scrapy from starting, as long as a different reactor is not specified in TWISTED_REACTOR (issue 5525, issue 5528)
    + Fixed an exception that was being logged after the spider finished under certain conditions (issue 5437, issue 5440)
    + The --output/-o command-line parameter supports again a value starting with a hyphen (issue 5444, issue 5445)
    + The scrapy parse -h command no longer throws an error (issue 5481, issue 5482)

• Files Changed
• Browse Source