Ошибка развертывания проекта scrapyd

Question

Ошибка развертывания проекта scrapyd

При попытке выполнить эту команду:

scrapyd-deploy test -p project=myProject

Я получаю следующую ошибку:

Traceback (most recent call last):
      File "/usr/bin/scrapyd-deploy", line 269, in <module>
        main()
      File "/usr/bin/scrapyd-deploy", line 95, in main
        egg, tmpdir = _build_egg()
      File "/usr/bin/scrapyd-deploy", line 236, in _build_egg
        retry_on_eintr(check_call, [sys.executable, 'setup.py', 'clean', '-a', 'bdist_egg', '-d', d], stdout=o, stderr=e)
      File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/python.py", line 331, in retry_on_eintr
        return function(*args, **kw)
      File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['/usr/bin/python', 'setup.py', 'clean', '-a', 'bdist_egg', '-d', '/tmp/scrapydeploy-wV3h4k']' returned non-zero exit status 1

Я установил scrapyd-deploy и scrapyd-client, а также setuptools и scrapyd, конечно.

Эта команда: python setup.py clean -a bdist_egg

производит вывод:

running clean
removing 'build/lib.linux-x86_64-2.7' (and everything under it)
removing 'build/bdist.linux-x86_64' (and everything under it)
'build/scripts-2.7' does not exist -- can't clean it
removing 'build'
running bdist_egg
running egg_info
writing NOAA.egg-info/PKG-INFO
writing top-level names to NOAA.egg-info/top_level.txt
writing dependency_links to NOAA.egg-info/dependency_links.txt
writing entry points to NOAA.egg-info/entry_points.txt
reading manifest file 'NOAA.egg-info/SOURCES.txt'
writing manifest file 'NOAA.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/NOAA
copying NOAA/pipelines.py -> build/lib.linux-x86_64-2.7/NOAA
copying NOAA/__init__.py -> build/lib.linux-x86_64-2.7/NOAA
copying NOAA/noaa-template.py -> build/lib.linux-x86_64-2.7/NOAA
copying NOAA/noaa-original.py -> build/lib.linux-x86_64-2.7/NOAA
copying NOAA/settings-template.py -> build/lib.linux-x86_64-2.7/NOAA
copying NOAA/tika-python.py -> build/lib.linux-x86_64-2.7/NOAA
copying NOAA/settings.py -> build/lib.linux-x86_64-2.7/NOAA
copying NOAA/mysqldb.py -> build/lib.linux-x86_64-2.7/NOAA
copying NOAA/noaa.py -> build/lib.linux-x86_64-2.7/NOAA
copying NOAA/items.py -> build/lib.linux-x86_64-2.7/NOAA
creating build/lib.linux-x86_64-2.7/NOAA/spiders
copying NOAA/spiders/spider1.py -> build/lib.linux-x86_64-2.7/NOAA/spiders
copying NOAA/spiders/spider2.py -> build/lib.linux-x86_64-2.7/NOAA/spiders
(output omitted... There are a lot of spiders)

creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/NOAA
copying build/lib.linux-x86_64-2.7/NOAA/pipelines.py -> build/bdist.linux-x86_64/egg/NOAA
creating build/bdist.linux-x86_64/egg/NOAA/spiders
copying build/lib.linux-x86_64-2.7/NOAA/spiders/spider1.py -> build/bdist.linux-x86_64/egg/NOAA/spiders
copying build/lib.linux-x86_64-2.7/NOAA/spiders/spider2.py -> build/bdist.linux-x86_64/egg/NOAA/spiders
(output omitted... There are a lot of spiders)


byte-compiling build/bdist.linux-x86_64/egg/NOAA/pipelines.py to pipelines.pyc
byte-compiling build/bdist.linux-x86_64/egg/NOAA/spiders/spider1.py to spider1.pyc
byte-compiling build/bdist.linux-x86_64/egg/NOAA/spiders/spider2.py to spider2.pyc
(output omitted... There are a lot of spiders)


byte-compiling build/bdist.linux-x86_64/egg/NOAA/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/NOAA/noaa-template.py to noaa-template.pyc
byte-compiling build/bdist.linux-x86_64/egg/NOAA/noaa-original.py to noaa-original.pyc
byte-compiling build/bdist.linux-x86_64/egg/NOAA/settings-template.py to settings-template.pyc
byte-compiling build/bdist.linux-x86_64/egg/NOAA/tika-python.py to tika-python.pyc
byte-compiling build/bdist.linux-x86_64/egg/NOAA/settings.py to settings.pyc
byte-compiling build/bdist.linux-x86_64/egg/NOAA/mysqldb.py to mysqldb.pyc
byte-compiling build/bdist.linux-x86_64/egg/NOAA/noaa.py to noaa.pyc
byte-compiling build/bdist.linux-x86_64/egg/NOAA/items.py to items.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying NOAA.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying NOAA.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying NOAA.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying NOAA.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying NOAA.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/NOAA-1.0-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)

При попытке запланировать паука с помощью этой команды:

curl http://localhost:6800/schedule.json -d project=myproject -d spider=spider1.py

Я получаю эту ошибку:

Traceback (most recent call last):
          File "/usr/local/lib/python2.7/dist-packages/twisted/web
            req.requestReceived(command, path, version)
          File "/usr/local/lib/python2.7/dist-packages/twisted/web
            self.process()
          File "/usr/local/lib/python2.7/dist-packages/twisted/web
            self.render(resrc)
          File "/usr/local/lib/python2.7/dist-packages/twisted/web
            body = resrc.render(self)
        --- <exception caught here> ---
          File "/usr/local/lib/python2.7/dist-packages/scrapyd/web
            return JsonResource.render(self, txrequest)
          File "/usr/local/lib/python2.7/dist-packages/scrapyd/uti
            r = resource.Resource.render(self, txrequest)
          File "/usr/local/lib/python2.7/dist-packages/twisted/web
            return m(request)
          File "/usr/local/lib/python2.7/dist-packages/scrapyd/web
            spiders = get_spider_list(project)
          File "/usr/local/lib/python2.7/dist-packages/scrapyd/uti
            runner = Config().get('runner')
          File "/usr/local/lib/python2.7/dist-packages/scrapyd/con
            self.cp.read(sources)
          File "/usr/lib/python2.7/ConfigParser.py", line 305, in
            self._read(fp, filename)
          File "/usr/lib/python2.7/ConfigParser.py", line 512, in
            raise MissingSectionHeaderError(fpname, lineno, line)
        ConfigParser.MissingSectionHeaderError: File contains no s
        file: /etc/scrapyd/conf.d/twistd.pid, line: 1
        '24262'

Забавно, что если я добавляю в этот файл twistd.pid то, что я считаю заголовком раздела ([header заголовка раздела]), я получаю сообщение об ошибке, говорящее о том, что он содержит нечто иное, чем числовое значение pid twistd.

Эти вопросы взаимосвязаны?

2

scrapy scrapyd

Источник

user7032282 21 ноя '16 в 23:35

1 ответ

Другие вопросы по тегам scrapy scrapyd

user5957834 16 май '18 в 07:26 2018-05-16 07:26 · Answer 1 · 2018-05-16 07:26

Я получаю ту же ошибку... что мне помогло, так это использование команды с sudo.

Ошибка может быть из-за того, что команда не получает надлежащие разрешения.

sudo scrapyd-deploy test -p project=myProject

0

Источник

user5957834 16 май '18 в 07:26