Scrapy爬虫的模板比较单一,每次新建爬虫程序要么重新手敲一遍,要么复制粘贴
- 从头手敲:效率较低,容易出错,浪费时间
- 复制粘贴:老代码需要改动的地方较多,容易漏掉,导致出错
所以,pycharm中配置一个模板文件就很重要了
# -*- encoding: utf-8 -*-
"""
@Date : ${YEAR}-${MONTH}-${DAY}
@Author : Peng Shiyu
"""
from scrapy import Spider, Request, cmdline
class XXXSpider(Spider):
name = "${NAME}"
start_urls = []
def start_request(self):
for url in self.start_urls:
yield Request(url)
def parse(self, response):
pass
if __name__ == "__main__":
cmdline.execute("scrapy crawl ${NAME}".split())