Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Contribute to GitLab
Sign in / Register
Toggle navigation
E
esi-table-data
Project
Project
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
esi-data-scrapping
esi-table-data
Commits
e75c2c21
Commit
e75c2c21
authored
Aug 11, 2017
by
Vasyl Bodnaruk
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Get crawlera key
parent
4874adac
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
8 additions
and
2 deletions
+8
-2
settings.py
exa/exa/settings.py
+5
-0
cb.py
exa/exa/spiders/cb.py
+3
-2
No files found.
exa/exa/settings.py
View file @
e75c2c21
...
@@ -109,6 +109,11 @@ CRAWLERA_MAXBANS = 3
...
@@ -109,6 +109,11 @@ CRAWLERA_MAXBANS = 3
try
:
try
:
from
.local_settings
import
*
from
.local_settings
import
*
except
ImportError
:
pass
# Import .env like DB pass, CRAWLERA_APIKEY
try
:
from
.env
import
*
from
.env
import
*
except
ImportError
:
except
ImportError
:
pass
pass
exa/exa/spiders/cb.py
View file @
e75c2c21
# -*- coding: utf-8 -*-
# -*- coding: utf-8 -*-
import
dateparser
import
dateparser
import
scrapy
import
scrapy
from
scrapy.utils.project
import
get_project_settings
from
.base
import
BaseSpider
from
.base
import
BaseSpider
from
..items
import
ExaItem
from
..items
import
ExaItem
from
..settings
import
get_env
class
CbSpider
(
BaseSpider
):
class
CbSpider
(
BaseSpider
):
name
=
"cb"
name
=
"cb"
allowed_domains
=
[
"www.crunchbase.com"
]
allowed_domains
=
[
"www.crunchbase.com"
]
crawlera_enabled
=
True
crawlera_enabled
=
True
crawlera_apikey
=
get_
env
(
'CRAWLERA_APIKEY'
)
crawlera_apikey
=
get_
project_settings
()
.
get
(
'CRAWLERA_APIKEY'
)
# start_urls = ['http://www.crunchbase.com/organization/sense-ly/press/']
# start_urls = ['http://www.crunchbase.com/organization/sense-ly/press/']
co
=
0
co
=
0
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment