Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Contribute to GitLab
Sign in / Register
Toggle navigation
E
esi-table-data
Project
Project
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
esi-data-scrapping
esi-table-data
Commits
73ef5c55
Commit
73ef5c55
authored
Oct 17, 2017
by
Vasyl Bodnaruk
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Add callback for parse tag
parent
47b2e13a
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
8 additions
and
1 deletion
+8
-1
nana.py
exa/exa/spiders/nana.py
+8
-1
No files found.
exa/exa/spiders/nana.py
View file @
73ef5c55
...
...
@@ -31,6 +31,13 @@ class NanaSpider(BaseSpider):
if
self
.
pipeline
.
check_url
(
item
[
'url'
]):
is_duplicate
=
True
break
else
:
yield
scrapy
.
Request
(
item
[
'url'
],
callback
=
self
.
parse_tag
,
meta
=
{
'item'
:
item
})
next_url
=
response
.
xpath
(
'.//div[@class="pagination"]/ul//a[text()="Next"]/@href'
)
.
extract_first
()
if
self
.
can_follow
(
next_url
,
is_duplicate
):
yield
scrapy
.
Request
(
next_url
,
callback
=
self
.
parse
)
\ No newline at end of file
yield
scrapy
.
Request
(
next_url
,
callback
=
self
.
parse
,
meta
=
response
.
meta
)
def
parse_tag
(
self
,
response
):
item
=
response
.
meta
[
'item'
]
item
[
'tags'
]
=
response
.
xpath
(
'.//ul[@class="tag-list"]/li/a/text()'
)
.
extract
()
yield
item
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment