Screaming Frog SEO Spider is a powerful tool for crawling websites. Still, it doesn’t have a built-in feature to specifically report URLs Without Specific DIV or URLs that do not contain a specific DIV element. However, you can achieve this goal using Custom Extraction with XPath or CSS Path. Follow the steps below to set up the extraction and export the results:
Launch Screaming Frog and enter the website URL you want to crawl.
Before starting the crawl, go to ‘Configuration’ > ‘Custom’ > ‘Extraction.’
Click ‘Add’ to create a new custom extraction.
Screaming Frog extraction method:
XPath: If you’re familiar with XPath, you can use it to target the specific DIV element. For example, if the DIV has an ID, you can use //div[@id=’your-target-div-id’].
CSS Path: If you prefer CSS selectors, use the appropriate selector to target the DIV element. For example, for a DIV with a specific class, use div.your-target-div-class.
Set the ‘Type’ of extraction to ‘Exists’. This will return ‘true’ if the element is found and ‘false’ if it’s not.
Click ‘OK’ to save the custom extraction.
Start the crawl by clicking the ‘Start’ button.
Once the crawl is completed, click on the ‘Custom’ tab and then click the ‘Extraction’ sub-tab. You will see a table with the crawled URLs and the results of the custom extraction (true/false).
To export the results, click ‘Export’ > ‘CSV’ and save the file to your computer.
Open the CSV file in your preferred spreadsheet application (e.g., Microsoft Excel, Google Sheets). Filter the results to show only the URLs where the extraction result is ‘false’. These are the URLs that do not contain the specific DIV element you were looking for.
Remember that Custom Extraction can be resource-intensive, so it might take longer to crawl larger websites. Additionally, make sure to double-check your XPath or CSS Path expressions to ensure accurate results.
