{"page":"<link rel=\"stylesheet\" href=\"https://lessonplanet.com/assets/packs/css/resources-572d6a42.css\" />\n<link rel=\"stylesheet\" href=\"https://lessonplanet.com/assets/packs/css/lp_boclips_stylesheets-f4d0de30.css\" media=\"all\" />\n<div data-title='Web Scraping Tutorial with Scrapy and Python for Beginners - Wait for Selector/Elements Using Page Co-routines' data-url='/boclips/videos/63eb8cc008d4467b93270636' data-video-url='/boclips/videos/63eb8cc008d4467b93270636' id='bo_player_modal'>\n<div class='boclips-resource-page modal-dialog panel-container'>\n<div class='react-notifications-root'></div>\n<div class='rp-header'>\n<div class='rp-type'>\n<i aria-hidden='true' class='fai fa-regular fa-circle-play'></i>\nVideo\n</div>\n<h1 class='rp-title' id='video-title'>\nWeb Scraping Tutorial with Scrapy and Python for Beginners - Wait for Selector/Elements Using Page Co-routines\n</h1>\n<div class='rp-actions'>\n<div class='mr-1'>\n<a class=\"btn btn-success\" data-posthog-event=\"Signup: LP Signup Activity\" data-posthog-location=\"body_link_boclips\" data-remote=\"true\" href=\"/subscription/new\"><span><span>Get Free Access</span><span class=\"\"> for 10 Days</span><span>!</span></span></a>\n</div>\n</div>\n</div>\n<div class='rp-body'>\n<div class='rp-info'>\n<div aria-label='Hide resource details' class='rp-hide-info' role='button' tabindex='0'>&times;</div>\n<i aria-label='Expand resource details' class='rp-expand-info fai fa-solid fa-up-right-and-down-left-from-center' role='button' tabindex='0'></i>\n<i aria-label='Compress resource details' class='rp-compress-info fai fa-solid fa-down-left-and-up-right-to-center' role='button' tabindex='0'></i>\n<div class='rp-rating'>\n<span class='resource-pool'>\n<span class='pool-label'>Publisher:</span>\n<span class='pool-name'>\n<span class='text'><a data-publisher-id=\"30356011\" href=\"/search?publisher_ids%5B%5D=30356011\">Curated Video</a></span>\n</span>\n</span>\n</div>\n<div class='rp-description'>\n<span class='short-description'>In this video, we'll learn about using code routines with Playwright to perform certain tasks before calling the parse method. You'll see how a page code routine is defined inside the metadata using playwright_page go routine and how to...</span>\n<span class='full-description hide'>In this video, we'll learn about using code routines with Playwright to perform certain tasks before calling the parse method. You'll see how a page code routine is defined inside the metadata using playwright_page go routine and how to wait for a specific selector using the page wait for selector method. Additionally, we'll explore how to extract data from all rows of a table and create a for loop to iterate through each. The video also demonstrates how to yield data and store it in a file.</span>\n</div>\n<div class='action-container flex justify-between'>\n<button aria-expanded='false' aria-label='Read more description' class='rp-full-description' type='button'>\n<i class='fai fa-solid fa-align-left'></i>\n<span id='read_more'>Read More</span>\n</button>\n<div class='rp-report'>\n</div>\n</div>\n<div aria-labelledby='resource-details-heading' class='rp-info-section'>\n<h2 class='title' id='resource-details-heading'>Resource Details</h2>\n<div class='rp-resource-details clearfix'>\n<div class='detail'>\n<dl>\n<dt>Curator Rating</dt>\n<dd><span class=\"star-rating\" aria-label=\"4.0 out of 5 stars\" role=\"img\"><i class=\"fa-solid fa-star text-action\" aria-hidden=\"true\"></i><i class=\"fa-solid fa-star text-action\" aria-hidden=\"true\"></i><i class=\"fa-solid fa-star text-action\" aria-hidden=\"true\"></i><i class=\"fa-solid fa-star text-action\" aria-hidden=\"true\"></i><i class=\"fa-regular fa-star text-action\" aria-hidden=\"true\"></i></span></dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt class=\"educator-rating-title\">Educator Rating</dt><dd><div class=\"educator-rating-details\" data-path=\"/educator_ratings/rrp_data?resourceable_id=40946&amp;resourceable_type=Boclips%3A%3AVideoMetadata\"><span class=\"not-yet-rated\">Not yet Rated</span></div></dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Media Length</dt>\n<dd>9:14</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Grade</dt><dd title=\"Grade\">Higher Ed</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Subjects</dt><dd><span><a href=\"/search?grade_ids%5B%5D=259&amp;search_tab_id=1&amp;subject_ids%5B%5D=378077\">STEM</a></span></dd><dd class=\"text-muted\"><i class=\"fa-solid fa-lock mr5\"></i>3 more...</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Media Type</dt><dd><span><a href=\"/search?grade_ids%5B%5D=259&amp;search_tab_id=2&amp;type_ids%5B%5D=4543647\">Instructional Videos</a></span></dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Source:</dt>\n<div class='preview-source' data-animation='true' data-boundary='.rp-info' data-container='.rp-resource-details' data-html='false' data-title='Instructional videos for technology and computing.' data-trigger='hover focus'>\n<span>Packt -  Programming</span>\n<i aria-hidden='true' class='fa-solid fa-circle-info channel-tooltip-icon' id='channel-tooltip'></i>\n</div>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Date</dt>\n<dd>2023</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<i aria-hidden='true' class='fai fa-solid fa-language'></i>\n<dt>Language</dt><dd>English</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Audiences</dt><dd><span><a href=\"/search?audience_ids%5B%5D=371079&amp;grade_ids%5B%5D=259&amp;search_tab_id=1\">For Teacher Use</a></span></dd><dd class=\"text-muted\"><i class=\"fa-solid fa-lock mr5\"></i>2 more...</dd>\n</dl>\n</div>\n<div class='detail'>\n<dl>\n<dt>Usage Permissions</dt><dd>Fine Print: Educational Use</dd>\n</dl>\n</div>\n</div>\n</div>\n<div aria-labelledby='additional-materials-heading' class='rp-info-section'>\n<h2 class='title' id='additional-materials-heading'>Additional Materials</h2>\n<div class='additional-material'>\n<i aria-hidden='true' class='fai fa-solid fa-lock'></i>\n<a class=\"text-muted\" title=\"Video Transcript\" data-html=\"true\" data-placement=\"bottom\" data-trigger=\"click\" data-content=\"<div class=&quot;text-center py-2&quot;><a class=&quot;bold&quot; href=&quot;/auth/users/sign_in&quot;>Sign in</a> or <a class=&quot;bold text-danger&quot; data-posthog-event=&quot;Signup: LP Signup Activity&quot; data-posthog-location=&quot;body_link_boclips&quot; data-remote=&quot;true&quot; href=&quot;/subscription/new&quot;>Join Now</a></div>\" data-title=\"Get Full Access\" data-container=\"body\" rel=\"popover\" tabindex=\"0\" href=\"/subscription/new\">Video Transcript</a>\n</div>\n<div class='additional-material'>\n<i aria-hidden='true' class='fai fa-solid fa-lock'></i>\n<a class=\"text-muted\" title=\"Video Preview\" data-html=\"true\" data-placement=\"bottom\" data-trigger=\"click\" data-content=\"<div class=&quot;text-center py-2&quot;><a class=&quot;bold&quot; href=&quot;/auth/users/sign_in&quot;>Sign in</a> or <a class=&quot;bold text-danger&quot; data-posthog-event=&quot;Signup: LP Signup Activity&quot; data-posthog-location=&quot;body_link_boclips&quot; data-remote=&quot;true&quot; href=&quot;/subscription/new&quot;>Join Now</a></div>\" data-title=\"Get Full Access\" data-container=\"body\" rel=\"popover\" tabindex=\"0\" href=\"/subscription/new\">Video Preview</a>\n</div>\n</div>\n<div aria-labelledby='concepts-heading' class='rp-info-section'>\n<h2 class='title' id='concepts-heading'>Concepts</h2>\n<div class='clearfix'>\n<div class='details-list concepts' data-identifier='Boclips::VideoDecorator63eb8cc008d4467b93270636' data-type='concepts'>film, data</div>\n<div class='concepts-toggle-buttons' data-identifier='Boclips::VideoDecorator63eb8cc008d4467b93270636'>\n<button aria-expanded='false' class='more btn-link' type='button'>\n<span>Show More</span>\n<i aria-hidden='true' class='fa-solid fa-caret-down ml5'></i>\n</button>\n<button aria-expanded='true' class='less btn-link' style='display: none;' type='button'>\n<span>Show Less</span>\n<i aria-hidden='true' class='fa-solid fa-caret-up ml5'></i>\n</button>\n</div>\n</div>\n</div>\n<div aria-labelledby='additional-tags-heading' class='rp-info-section'>\n<h2 class='title' id='additional-tags-heading'>Additional Tags</h2>\n<div class='clearfix'>\n<div class='details-list keyterms' data-identifier='Boclips::VideoDecorator63eb8cc008d4467b93270636' data-type='keyterms'>web scraping, scraping, scrapy, python, website, dynamic website scraping, pycharm 2021.1.3, visual studio code 2022, python 3.10 and 3.8, scrapy 2.6.1, windows 10, ubuntu 20.0.2, mongodb 5.0.2, scrapy-playwright 0.0.12, json, csv, xlsx (excel), xml, row css, parse method, table row, table body, film title, film awards, table data, page wait, method, table, title, page, inside, awards, routine, playwright, spider, text, element, selector, wait, perform, metadata, define, make, list, ahead, case</div>\n<div class='keyterms-toggle-buttons' data-identifier='Boclips::VideoDecorator63eb8cc008d4467b93270636'>\n<button aria-expanded='false' class='more btn-link' type='button'>\n<span>Show More</span>\n<i aria-hidden='true' class='fa-solid fa-caret-down ml5'></i>\n</button>\n<button aria-expanded='true' class='less btn-link' style='display: none;' type='button'>\n<span>Show Less</span>\n<i aria-hidden='true' class='fa-solid fa-caret-up ml5'></i>\n</button>\n</div>\n</div>\n</div>\n<div aria-labelledby='classroom-considerations-heading' class='rp-info-section'>\n<h2 class='title' id='classroom-considerations-heading'>Classroom Considerations</h2>\n<div class='classroom-considerations'><div class='fai fa-solid fa-bell'></div>Best For: Explaining a topic</div><div class='classroom-considerations'><div class='fai fa-solid fa-bell'></div>Video is ad-free</div> \n</div>\n<div aria-labelledby='educator-ratings-heading' class='rp-info-section'>\n<h2 class='title sr-only' id='educator-ratings-heading'>Educator Ratings</h2>\n<div id=\"educator-ratings-root\"></div><div id=\"all-educator-ratings-root\"></div><div id=\"educator-rating-form-root\"></div>\n</div>\n</div>\n<div class='rp-resource'>\n<div aria-label='Show resource details' class='rp-show-info' role='button' tabindex='0'>\n<i class='fai fa-solid fa-align-left'></i>\nShow resource details\n</div>\n<div aria-label='Video player' class='player' id='player-wrapper' role='region'>\n<div class='relative container mx-auto' id='lp-boclips-visitor-thumbnail'>\n<a class=\"block\" data-html=\"true\" data-placement=\"bottom\" data-trigger=\"click\" data-content=\"<div class=&quot;text-center py-2&quot;><a class=&quot;bold&quot; href=&quot;/auth/users/sign_in&quot;>Sign in</a> or <a class=&quot;bold text-danger&quot; data-posthog-event=&quot;Signup: LP Signup Activity&quot; data-posthog-location=&quot;body_link_boclips&quot; data-remote=&quot;true&quot; href=&quot;/subscription/new&quot;>Join Now</a></div>\" data-title=\"Get Full Access\" data-container=\"body\" rel=\"popover\" tabindex=\"0\" aria-label=\"Play video: Web Scraping Tutorial with Scrapy and Python for Beginners - Wait for Selector/Elements Using Page Co-routines\" href=\"/subscription/new\"><img class=\"resource-img img-thumbnail img-responsive z-10 lp-boclips-thumbnail w-full h-full lozad\" alt=\"Web Scraping Tutorial with Scrapy and Python for Beginners - Wait for Selector/Elements Using Page Co-routines\" title=\"Web Scraping Tutorial with Scrapy and Python for Beginners - Wait for Selector/Elements Using Page Co-routines\" onError=\"handleImageNotLoadedError(this)\" data-default-image=\"https://statictemp.lp.lexp.cloud/images/attachment_defaults/resource/large/missing.png\" data-src=\"https://cdnapisec.kaltura.com/p/1776261/thumbnail/entry_id/1_jb57i455/width/250/vid_slices/3/vid_slice/1\" width=\"315\" height=\"220\" src=\"data:image/png;base64,R0lGODlhAQABAAD/ACwAAAAAAQABAAACADs\" />\n<span aria-hidden='true' class='flex justify-center items-center bg-white rounded-full w-16 h-16 absolute top-1/2 left-1/2 -mt-8 -ml-8 cursor-pointer z-0 border-2 border-primary drop-shadow-md lp-boclips-thumbnail-playBtn'>\n<i class='fa-solid fa-play text-primary text-3xl ml-1 drop-shadow-xl'></i>\n</span>\n</a></div>\n</div>\n</div>\n</div>\n</div>\n</div>\n"}