javascript - Unable to parse selector? -



javascript - Unable to parse selector? -

i using scrapy + splash + python in centos. have write next code extracting content here.

unable extract info java script popup windows illustration "href="javascript:void(0);" 1 guide me crawl.

spider code:

def parse(self,response): selector = selector(response) links = [] link in selector.css('div.job_list div.cont > span.style.display:block;vertical-align: bottom;::attr(href)').extract(): yield request(urlparse.urljoin(response.url, link), callback=self.parse_listing_page, #meta={"use_splash": false} )

following error:

[testtim@dpitstsvr015 samsungcom]$ scrapy crawl samsung 2014-11-11 18:05:33+0530 [scrapy] info: scrapy 0.24.4 started (bot: samsungcom) 2014-11-11 18:05:33+0530 [scrapy] info: optional features available: ssl, http11 2014-11-11 18:05:33+0530 [scrapy] info: overridden settings: {'newspider_module': 'samsungcom.spiders', 'concurrent_requests_per_domain': 1, 'concurrent_requests': 1, 'retry_enabled': false, 'spider_modules': ['samsungcom.spiders'], 'bot_name': 'samsungcom'} 2014-11-11 18:05:33+0530 [scrapy] info: enabled extensions: logstats, telnetconsole, closespider, webservice, corestats, spiderstate 2014-11-11 18:05:33+0530 [scrapy] info: enabled downloader middlewares: httpauthmiddleware, downloadtimeoutmiddleware, useragentmiddleware, defaultheadersmiddleware, metarefreshmiddleware, httpcompressionmiddleware, redirectmiddleware, cookiesmiddleware, splashmiddleware, chunkedtransfermiddleware, downloaderstats 2014-11-11 18:05:33+0530 [scrapy] info: enabled spider middlewares: httperrormiddleware, offsitemiddleware, referermiddleware, urllengthmiddleware, depthmiddleware 2014-11-11 18:05:33+0530 [scrapy] info: enabled item pipelines: 2014-11-11 18:05:33+0530 [samsung] info: spider opened 2014-11-11 18:05:33+0530 [samsung] info: crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2014-11-11 18:05:33+0530 [scrapy] debug: telnet console listening on 127.0.0.1:6023 2014-11-11 18:05:33+0530 [scrapy] debug: web service listening on 127.0.0.1:6080 https://careers.us.samsung.com/careers/svc/app/viewsearchjob http://localhost:9999/render.html?url=https%3a%2f%2fcareers.us.samsung.com%2fcareers%2fsvc%2fapp%2fviewsearchjob 2014-11-11 18:05:58+0530 [samsung] debug: crawled (200) <get http://localhost:9999/render.html?url=https%3a%2f%2fcareers.us.samsung.com%2fcareers%2fsvc%2fapp%2fviewsearchjob&wait=2&images=0> (referer: none) 2014-11-11 18:05:58+0530 [samsung] error: spider error processing <get http://localhost:9999/render.html?url=https%3a%2f%2fcareers.us.samsung.com%2fcareers%2fsvc%2fapp%2fviewsearchjob&wait=2&images=0> traceback (most recent phone call last): file "/usr/lib64/python2.7/site-packages/twisted/internet/base.py", line 824, in rununtilcurrent call.func(*call.args, **call.kw) file "/usr/lib64/python2.7/site-packages/twisted/internet/task.py", line 638, in _tick taskobj._oneworkunit() file "/usr/lib64/python2.7/site-packages/twisted/internet/task.py", line 484, in _oneworkunit result = next(self._iterator) file "/usr/lib64/python2.7/site-packages/scrapy/utils/defer.py", line 57, in <genexpr> work = (callable(elem, *args, **named) elem in iterable) --- <exception caught here> --- file "/usr/lib64/python2.7/site-packages/scrapy/utils/defer.py", line 96, in iter_errback yield next(it) file "/usr/lib64/python2.7/site-packages/scrapy/contrib/spidermiddleware/offsite.py", line 26, in process_spider_output x in result: file "/usr/lib64/python2.7/site-packages/scrapy/contrib/spidermiddleware/referer.py", line 22, in <genexpr> homecoming (_set_referer(r) r in result or ()) file "/usr/lib64/python2.7/site-packages/scrapy/contrib/spidermiddleware/urllength.py", line 33, in <genexpr> homecoming (r r in result or () if _filter(r)) file "/usr/lib64/python2.7/site-packages/scrapy/contrib/spidermiddleware/depth.py", line 50, in <genexpr> homecoming (r r in result or () if _filter(r)) file "/home/sureshp/downloads/samsungcom/samsungcom/spiders/samsung.py", line 24, in parse link in selector.css('div.job_list div.cont > span.style.display:block;vertical-align: bottom;::attr(href)').extract(): file "/usr/lib64/python2.7/site-packages/scrapy/selector/unified.py", line 110, in css homecoming self.xpath(self._css2xpath(query)) file "/usr/lib64/python2.7/site-packages/scrapy/selector/unified.py", line 113, in _css2xpath homecoming self._csstranslator.css_to_xpath(query) file "/usr/lib/python2.7/site-packages/cssselect/xpath.py", line 192, in css_to_xpath selector in parse(css)) file "/usr/lib/python2.7/site-packages/cssselect/parser.py", line 355, in parse homecoming list(parse_selector_group(stream)) file "/usr/lib/python2.7/site-packages/cssselect/parser.py", line 370, in parse_selector_group yield selector(*parse_selector(stream)) file "/usr/lib/python2.7/site-packages/cssselect/parser.py", line 396, in parse_selector next_selector, pseudo_element = parse_simple_selector(stream) file "/usr/lib/python2.7/site-packages/cssselect/parser.py", line 477, in parse_simple_selector **"expected selector, got %s" % (peek,)) cssselect.parser.selectorsyntaxerror: expected selector, got <delim ';' @ 48>** 2014-11-11 18:05:58+0530 [samsung] info: closing spider (finished) 2014-11-11 18:05:58+0530 [samsung] info: dumping scrapy stats: {'downloader/request_bytes': 314, 'downloader/request_count': 1, 'downloader/request_method_count/get': 1, 'downloader/response_bytes': 111927, 'downloader/response_count': 1, 'downloader/response_status_count/200': 1, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2014, 11, 11, 12, 35, 58, 732318), 'log_count/debug': 3, 'log_count/error': 1, 'log_count/info': 7, 'response_received_count': 1, 'scheduler/dequeued': 2, 'scheduler/dequeued/memory': 2, 'scheduler/enqueued': 2, 'scheduler/enqueued/memory': 2, 'spider_exceptions/selectorsyntaxerror': 1, 'splash/unwrapped': 1, 'splash/wrapped': 1, 'start_time': datetime.datetime(2014, 11, 11, 12, 35, 33, 902315)} 2014-11-11 18:05:58+0530 [samsung] info: spider closed (finished)

div.job_list div.cont > span.style.display:block;vertical-align: bottom;::attr(href) not valid css selector.

try:

'div.job_list div.cont > span[style="display:block;vertical-align: bottom;"]::attr(href)'

note ::attr(href) scrapy extension css selectors, select attribute values using pseudo-element syntax.

javascript python scrapy splash scrapinghub

-

Comments

Post a Comment

Popular posts from this blog

formatting - SAS SQL Datepart function returning odd values -

formatting - SAS SQL Datepart function returning odd values - i'm having massive problem project can't seem right. i'm trying modify info variable in datetime format. first info 30mar12:00:00:00. i've been advised utilize code below returns value 30mar60:5:30:00. want date component , have no thought how info got exported in format wasn't in microsoft access. i've tried several mods code 1 time again comes blank or sends error message. proc sql; update dataset set date = datepart(date); quit; any advice appreciated. i'm guessing variable date numeric , formatted datetime. 1 time you've carried out conversion, alter format date9. sas datetime value number of seconds since 01 jan 1960, whereas dates number of days since 01 jan 1960. both stored numbers, using right format display value key. illustration below. data _null_; a='30mar12:00:00:00'dt; b=datepart(a); c=b; format b datetime. c date9.; set b c; run; ...
Read more

c++ - Apple Mach-O Linker Error(Duplicate Symbols For Architecture armv7) -

c++ - Apple Mach-O Linker Error(Duplicate Symbols For Architecture armv7) - here error: duplicate symbol _objc_metaclass_$_bfapplinkreturntorefererview in: /users/user/documents/facebooksdk/facebooksdk.framework/facebooksdk(bfapplinkreturntorefererview.o) /users/user/desktop/project/project/project/assets/facebook/editor/ios/facebooksdk/facebooksdk.a(bfapplinkreturntorefererview.o) ld: 889 duplicate symbols architecture armv7 clang: error: linker command failed exit code 1 (use -v see invocation) i have searched find solution of error couldn't. there tons of similar error this, 1 hasn't been asked yet. problem shows when add together "-objc" "other linker flags". need add together utilize admob service. problem? how can solve this? i faced same issue while integrating facebook-ios-sdk. because using 3rd party library had bolts.framework linked against it. solution found incorporate facebook-ios-sdk source code hosted on github. ...
Read more

php - Yii 2: Unable to find a class into the extension 'yii2-admin' -

php - Yii 2: Unable to find a class into the extension 'yii2-admin' - extension's github project: https://github.com/mdmsoft/yii2-admin i'm using advanced template of yii2, i've backend , frontend , common folders advanced folder. the advanced folder on same level of vendor folder. i've others extensions , working then installed yii2-admin composer php composer.phar require mdmsoft/yii2-admin "*" installation worked well. into vendor folder i've mdmsoft folder yii2-admin subfolder it. this actual content of v endor/yiisoft/extensions.php (modified composer installation, haven't manually touched it) <?php $vendordir = dirname(__dir__); homecoming array ( 'yiisoft/yii2-jui' => array ( 'name' => 'yiisoft/yii2-jui', 'version' => '2.0.0.0', 'alias' => array ( '@yii/jui' => $vendordir . '/yiisoft/yii2-jui', ...
Read more