{"page":"<link rel=\"stylesheet\" href=\"https://lessonplanet.com/assets/packs/css/resources-572d6a42.css\" />\n<link rel=\"stylesheet\" href=\"https://lessonplanet.com/assets/packs/css/lp_boclips_stylesheets-f4d0de30.css\" media=\"all\" />\n<div data-title='Web Scraping Tutorial with Scrapy and Python for Beginners - Setting Up Custom Callback Function' data-url='/boclips/videos/63eb8c2ec063434929ca08f8' data-video-url='/boclips/videos/63eb8c2ec063434929ca08f8' id='bo_player_modal'>\n<div class='boclips-resource-page modal-dialog panel-container'>\n<div class='react-notifications-root'></div>\n<div class='rp-header'>\n<div class='rp-type'>\n<i aria-hidden='true' class='fai fa-regular fa-circle-play'></i>\nVideo\n</div>\n<h1 class='rp-title' id='video-title'>\nWeb Scraping Tutorial with Scrapy and Python for Beginners - Setting Up Custom Callback Function\n</h1>\n<div class='rp-actions'>\n<div class='mr-1'>\n<a class=\"btn btn-success\" data-posthog-event=\"Signup: LP Signup Activity\" data-posthog-location=\"body_link_boclips\" data-remote=\"true\" href=\"/subscription/new\"><span><span>Get Free Access</span><span class=\"\"> for 10 Days</span><span>!</span></span></a>\n</div>\n</div>\n</div>\n<div class='rp-body'>\n<div class='rp-info'>\n<div aria-label='Hide resource details' class='rp-hide-info' role='button' tabindex='0'>&times;</div>\n<i aria-label='Expand resource details' class='rp-expand-info fai fa-solid fa-up-right-and-down-left-from-center' role='button' tabindex='0'></i>\n<i aria-label='Compress resource details' class='rp-compress-info fai fa-solid fa-down-left-and-up-right-to-center' role='button' tabindex='0'></i>\n<div class='rp-rating'>\n<span class='resource-pool'>\n<span class='pool-label'>Publisher:</span>\n<span class='pool-name'>\n<span class='text'><a data-publisher-id=\"30356011\" href=\"/search?publisher_ids%5B%5D=30356011\">Curated Video</a></span>\n</span>\n</span>\n</div>\n<div class='rp-description'>\n<span class='short-description'>In this video, we will learn how to apply the concepts we learned in Scrapy shell to our spider, allowing us to extract specific data from websites more efficiently. The video walks us through the steps of selecting items, using loops to...</span>\n<span class='full-description hide'>In this video, we will learn how to apply the concepts we learned in Scrapy shell to our spider, allowing us to extract specific data from websites more efficiently. The video walks us through the steps of selecting items, using loops to extract data, and creating URLs through concatenation or string formatting. We will also learn about using callbacks to handle product details and define our own custom method to handle website responses.</span>\n</div>\n<div class='action-container flex justify-between'>\n<button aria-expanded='false' aria-label='Read more description' class='rp-full-description' type='button'>\n<i class='fai fa-solid fa-align-left'></i>\n<span id='read_more'>Read More</span>\n</button>\n<div class='rp-report'>\n</div>\n</div>\n<div aria-labelledby='resource-details-heading' class='rp-info-section'>\n<h2 class='title' id='resource-details-heading'>Resource Details</h2>\n<div class='rp-resource-details clearfix'>\n<div class='detail'>\n<dl>\n<dt>Curator Rating</dt>\n<dd><span class=\"star-rating\" aria-label=\"4.0 out of 5 stars\" role=\"img\"><i class=\"fa-solid fa-star text-action\" aria-hidden=\"true\"></i><i class=\"fa-solid fa-star text-action\" aria-hidden=\"true\"></i><i class=\"fa-solid fa-star text-action\" aria-hidden=\"true\"></i><i class=\"fa-solid fa-star text-action\" aria-hidden=\"true\"></i><i class=\"fa-regular fa-star text-action\" aria-hidden=\"true\"></i></span></dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt class=\"educator-rating-title\">Educator Rating</dt><dd><div class=\"educator-rating-details\" data-path=\"/educator_ratings/rrp_data?resourceable_id=40956&amp;resourceable_type=Boclips%3A%3AVideoMetadata\"><span class=\"not-yet-rated\">Not yet Rated</span></div></dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Media Length</dt>\n<dd>3:17</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Grade</dt><dd title=\"Grade\">Higher Ed</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Subjects</dt><dd><span><a href=\"/search?grade_ids%5B%5D=259&amp;search_tab_id=1&amp;subject_ids%5B%5D=378077\">STEM</a></span></dd><dd class=\"text-muted\"><i class=\"fa-solid fa-lock mr5\"></i>3 more...</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Media Type</dt><dd><span><a href=\"/search?grade_ids%5B%5D=259&amp;search_tab_id=2&amp;type_ids%5B%5D=4543647\">Instructional Videos</a></span></dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Source:</dt>\n<div class='preview-source' data-animation='true' data-boundary='.rp-info' data-container='.rp-resource-details' data-html='false' data-title='Instructional videos for technology and computing.' data-trigger='hover focus'>\n<span>Packt -  Programming</span>\n<i aria-hidden='true' class='fa-solid fa-circle-info channel-tooltip-icon' id='channel-tooltip'></i>\n</div>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Date</dt>\n<dd>2023</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<i aria-hidden='true' class='fai fa-solid fa-language'></i>\n<dt>Language</dt><dd>English</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Audiences</dt><dd><span><a href=\"/search?audience_ids%5B%5D=371079&amp;grade_ids%5B%5D=259&amp;search_tab_id=1\">For Teacher Use</a></span></dd><dd class=\"text-muted\"><i class=\"fa-solid fa-lock mr5\"></i>2 more...</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Usage Permissions</dt><dd>Fine Print: Educational Use</dd>\n</dl>\n</div>\n</div>\n</div>\n<div aria-labelledby='additional-materials-heading' class='rp-info-section'>\n<h2 class='title' id='additional-materials-heading'>Additional Materials</h2>\n<div class='additional-material'>\n<i aria-hidden='true' class='fai fa-solid fa-lock'></i>\n<a class=\"text-muted\" title=\"Video Transcript\" data-html=\"true\" data-placement=\"bottom\" data-trigger=\"click\" data-content=\"<div class=&quot;text-center py-2&quot;><a class=&quot;bold&quot; href=&quot;/auth/users/sign_in&quot;>Sign in</a> or <a class=&quot;bold text-danger&quot; data-posthog-event=&quot;Signup: LP Signup Activity&quot; data-posthog-location=&quot;body_link_boclips&quot; data-remote=&quot;true&quot; href=&quot;/subscription/new&quot;>Join Now</a></div>\" data-title=\"Get Full Access\" data-container=\"body\" rel=\"popover\" tabindex=\"0\" href=\"/subscription/new\">Video Transcript</a>\n</div>\n<div class='additional-material'>\n<i aria-hidden='true' class='fai fa-solid fa-lock'></i>\n<a class=\"text-muted\" title=\"Video Preview\" data-html=\"true\" data-placement=\"bottom\" data-trigger=\"click\" data-content=\"<div class=&quot;text-center py-2&quot;><a class=&quot;bold&quot; href=&quot;/auth/users/sign_in&quot;>Sign in</a> or <a class=&quot;bold text-danger&quot; data-posthog-event=&quot;Signup: LP Signup Activity&quot; data-posthog-location=&quot;body_link_boclips&quot; data-remote=&quot;true&quot; href=&quot;/subscription/new&quot;>Join Now</a></div>\" data-title=\"Get Full Access\" data-container=\"body\" rel=\"popover\" tabindex=\"0\" href=\"/subscription/new\">Video Preview</a>\n</div>\n</div>\n<div aria-labelledby='concepts-heading' class='rp-info-section'>\n<h2 class='title' id='concepts-heading'>Concepts</h2>\n<div class='clearfix'>\n<div class='details-list concepts' data-identifier='Boclips::VideoDecorator63eb8c2ec063434929ca08f8' data-type='concepts'>details</div>\n<div class='concepts-toggle-buttons' data-identifier='Boclips::VideoDecorator63eb8c2ec063434929ca08f8'>\n<button aria-expanded='false' class='more btn-link' type='button'>\n<span>Show More</span>\n<i aria-hidden='true' class='fa-solid fa-caret-down ml5'></i>\n</button>\n<button aria-expanded='true' class='less btn-link' style='display: none;' type='button'>\n<span>Show Less</span>\n<i aria-hidden='true' class='fa-solid fa-caret-up ml5'></i>\n</button>\n</div>\n</div>\n</div>\n<div aria-labelledby='additional-tags-heading' class='rp-info-section'>\n<h2 class='title' id='additional-tags-heading'>Additional Tags</h2>\n<div class='clearfix'>\n<div class='details-list keyterms' data-identifier='Boclips::VideoDecorator63eb8c2ec063434929ca08f8' data-type='keyterms'>web scraping, scraping, scrapy, python, website, dynamic website scraping, pycharm 2021.1.3, visual studio code 2022, python 3.10 and 3.8, scrapy 2.6.1, windows 10, ubuntu 20.0.2, mongodb 5.0.2, scrapy-playwright 0.0.12, json, csv, xlsx (excel), xml, keyword argument, start urls, arse method, custom method, product details, parse method, parse, back, send, callback, response, request, ahead, ebook, select, create, item, call, called, default, yield, case, define, receive</div>\n<div class='keyterms-toggle-buttons' data-identifier='Boclips::VideoDecorator63eb8c2ec063434929ca08f8'>\n<button aria-expanded='false' class='more btn-link' type='button'>\n<span>Show More</span>\n<i aria-hidden='true' class='fa-solid fa-caret-down ml5'></i>\n</button>\n<button aria-expanded='true' class='less btn-link' style='display: none;' type='button'>\n<span>Show Less</span>\n<i aria-hidden='true' class='fa-solid fa-caret-up ml5'></i>\n</button>\n</div>\n</div>\n</div>\n<div aria-labelledby='classroom-considerations-heading' class='rp-info-section'>\n<h2 class='title' id='classroom-considerations-heading'>Classroom Considerations</h2>\n<div class='classroom-considerations'><div class='fai fa-solid fa-bell'></div>Best For: Explaining a topic</div><div class='classroom-considerations'><div class='fai fa-solid fa-bell'></div>Video is ad-free</div> \n</div>\n<div aria-labelledby='educator-ratings-heading' class='rp-info-section'>\n<h2 class='title sr-only' id='educator-ratings-heading'>Educator Ratings</h2>\n<div id=\"educator-ratings-root\"></div><div id=\"all-educator-ratings-root\"></div><div id=\"educator-rating-form-root\"></div>\n</div>\n</div>\n<div class='rp-resource'>\n<div aria-label='Show resource details' class='rp-show-info' role='button' tabindex='0'>\n<i class='fai fa-solid fa-align-left'></i>\nShow resource details\n</div>\n<div aria-label='Video player' class='player' id='player-wrapper' role='region'>\n<div class='relative container mx-auto' id='lp-boclips-visitor-thumbnail'>\n<a class=\"block\" data-html=\"true\" data-placement=\"bottom\" data-trigger=\"click\" data-content=\"<div class=&quot;text-center py-2&quot;><a class=&quot;bold&quot; href=&quot;/auth/users/sign_in&quot;>Sign in</a> or <a class=&quot;bold text-danger&quot; data-posthog-event=&quot;Signup: LP Signup Activity&quot; data-posthog-location=&quot;body_link_boclips&quot; data-remote=&quot;true&quot; href=&quot;/subscription/new&quot;>Join Now</a></div>\" data-title=\"Get Full Access\" data-container=\"body\" rel=\"popover\" tabindex=\"0\" aria-label=\"Play video: Web Scraping Tutorial with Scrapy and Python for Beginners - Setting Up Custom Callback Function\" href=\"/subscription/new\"><img class=\"resource-img img-thumbnail img-responsive z-10 lp-boclips-thumbnail w-full h-full lozad\" alt=\"Web Scraping Tutorial with Scrapy and Python for Beginners - Setting Up Custom Callback Function\" title=\"Web Scraping Tutorial with Scrapy and Python for Beginners - Setting Up Custom Callback Function\" onError=\"handleImageNotLoadedError(this)\" data-default-image=\"https://statictemp.lp.lexp.cloud/images/attachment_defaults/resource/large/missing.png\" data-src=\"https://cdnapisec.kaltura.com/p/1776261/thumbnail/entry_id/1_1h5gpgx5/width/250/vid_slices/3/vid_slice/1\" width=\"315\" height=\"220\" src=\"data:image/png;base64,R0lGODlhAQABAAD/ACwAAAAAAQABAAACADs\" />\n<span aria-hidden='true' class='flex justify-center items-center bg-white rounded-full w-16 h-16 absolute top-1/2 left-1/2 -mt-8 -ml-8 cursor-pointer z-0 border-2 border-primary drop-shadow-md lp-boclips-thumbnail-playBtn'>\n<i class='fa-solid fa-play text-primary text-3xl ml-1 drop-shadow-xl'></i>\n</span>\n</a></div>\n</div>\n</div>\n</div>\n</div>\n</div>\n"}