处理无限滚动页面

{
"has_next": true,
"page": 2,
"quotes": [{
"author": {
"goodreads_link": "/author/show/82952.Marilyn_Monroe",
"name": "Marilyn Monroe",
"slug": "Marilyn-Monroe"
},
"tags": ["friends", "heartbreak", "inspirational", "life", "love",
"sisters"],
"text": "\u201cThis life is what you make it...."
}, {
"author": {
"goodreads_link": "/author/show/1077326.J_K_Rowling",
"name": "J.K. Rowling",
"slug": "J-K-Rowling"
},
"tags": ["courage", "friends"],
"text": "\u201cIt takes a great deal of bravery to stand up to our enemies,
but just as much to stand up to our friends.\u201d"
},

<200 http://spidyquotes.herokuapp.com/api/quotes?page=2>
2017-10-29 16:17:37 [scrapy.core.scraper] DEBUG: Scraped from <200
http://spidyquotes.herokuapp.com/api/quotes?page=2>
{'text': "“This life is what you make it. No matter what, you're going to
mess up sometimes, it's a universal truth. But the good part is you get to
decide how you're going to mess it up. Girls will be your friends - they'll
act like it anyway. But just remember, some come, some go. The ones that
stay with you through everything - they're your true best friends. Don't
let go of them. Also remember, sisters make the best friends in the world.
As for lovers, well, they'll come and go too. And baby, I hate to say it,
most of them - actually pretty much all of them are going to break your
heart, but you can't give up because if you give up, you'll never find your
soulmate. You'll never find that half who makes you whole and that goes for
everything. Just because you fail once, doesn't mean you're gonna fail at
everything. Keep trying, hold on, and always, always, always believe in
yourself, because if you don't, then who will, sweetie? So keep your head
high, keep your chin up, and most importantly, keep smiling, because life's
a beautiful thing and there's so much to smile about.”", 'author': 'Marilyn
Monroe', 'tags': ['friends', 'heartbreak', 'inspirational', 'life', 'love',
'sisters']}
2017-10-29 16:17:37 [scrapy.core.scraper] DEBUG: Scraped from <200
http://spidyquotes.herokuapp.com/api/quotes?page=2>
{'text': '“It takes a great deal of bravery to stand up to our enemies, but
just as much to stand up to our friends.”', 'author': 'J.K. Rowling',
'tags': ['courage', 'friends']}
2017-10-29 16:17:37 [scrapy.core.scraper] DEBUG: Scraped from <200
http://spidyquotes.herokuapp.com/api/quotes?page=2>
{'text': "“If you can't explain it to a six year old, you don't understand
it yourself.”", 'author': 'Albert Einstein', 'tags': ['simplicity',
'understand']}

class Spider(scrapy.Spider):
    name = 'spidyquotes'
    quotes_base_url = 'http://spidyquotes.herokuapp.com/api/quotes'
    start_urls = [quotes_base_url]
    download_delay = 1.5

def parse(self, response):
    print(response)
    data = json.loads(response.body)

for item in data.get('quotes', []):
    yield {
        'text': item.get('text'),
        'author': item.get('author', {}).get('name'),
        'tags': item.get('tags'),
    }

if data['has_next']:
    next_page = data['page'] + 1
    yield scrapy.Request(self.quotes_base_url + "?page=%s" % next_page)

from selenium import webdriver
import time

driver = webdriver.PhantomJS()

print("Starting")
driver.get("https://twitter.com")
scroll_pause_time = 1.5

# Get scroll height
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
    print(last_height)
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait to load page
    time.sleep(scroll_pause_time)

    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    print(new_height, last_height)

    if new_height == last_height:
        break
    last_height = new_height

Starting
4882
8139 4882
8139
11630 8139
11630
15055 11630
15055
15055 15055
Process finished with exit code 0