Screaming Frog Data | URL profiler Data Example

 

Crawling the site.

  • Take inventory of all content available for indexation
  • Crawling the site with Screaming Frog or xenu
  • Export and duplicate the crawled list
  • Separate all URLs with robots meta or X-Robots noindex tags
  • Separate other URLS based on server status errors

 

Understanding Sheet Data (Actionable metrics)

 

Gather

  • Export the “Internal All” file, from screaming frog
  • Combine this sheet with additional indexable URLs found outside of the crawl via GSC, GA, and elsewhere
  • Now use this sheet as a seed list in URL Profiler
  • Check image below for URL profiler settings.
  • Give Google Analytics and Google Webmaster Credentials to URL Profiler
  • Give the Google pagespeed insights API Key to URL Profiler (Info).
  • Use the same credentials of the Google Analytics and the webmaster
  • Google Indexation needs around 10 proxy servers, otherwise the IP address will be blocked and we won’t get correct results

 

 

 

What to track? What tool to use?
Indexed or not A non-indexed URL is often a sign of an un crawled or low-quality page.).
Content uniqueness Copyscape, Siteliner, and now URL Profiler can provide this data
Organic Search Traffic Typically 90 days
Conversion Tracking Revenue and/or conversions
Publish date discover stale content
Internal links ensuring the most important pages have the most internal links
External links (These can come from Moz, SEMRush, and a variety of other tools)
One Page Time Landing pages resulting in low time-on-site
Bounce Rate Landing pages resulting in Low Pages-Per-Visit
Page Speed Page speed and mobile-friendliness through google tool
Page HTTP Status Response code 2xx, 3xx, 4xx, 5xx
Page Canonical tags

General Content & Auditing

The site content is classified into some general categories. Then we follow certain procedures to improve these general pages. Once this is done we move to part two to target specific issues and improve them. 

Content Classification How to Decide this category Action How to decide Action & exact procedure
Trust Building Pages These are pages like
  • About Us Page
  • Contacts Us Page: Should have Contact Form, Address + Map, Phone, Email
  • Testimonial Page
  • Terms & Conditions Page || Privacy Policy Page
  • For ecomm site: Refund Policy Shipping Information Return policy
Always Improve

OR Create

Every site must have these pages.
  • Create if the site is missing these pages
  • Improve if they have less content or lack functionality
Editorial content If the site has a blog. All the blogs or articles come under this category.
  • Blogs
  • Articles
  • News
  • tutorials/how-to content
Improve, Keep, or Remove
  • Improve Outdated Blogs: The information in older pages may be valuable but parts of may need to be updated
  • Remove Irrelevant Blogs: Information relevant 2 years ago may not be relevant today
  • Keep Blogs that are current with relevant information.
Archive Pages These are auto generated pages by many CMSs. These could be category pages, tag pages, author archives etc Page Redesign As these are autogenerated they may have duplicate content. We almost always have redesign these pages so duplicate content is minimized.
Sales Pages All Product Pages where users can make purchase Improve
External Guest Posts These are articles you have written elsewhere in hopes of getting backlinks.
  • Remove guest posts that were published elsewhere
Internal Guest Posts These are articles that someone else has written on your site.
Stub Content (e.g., “No content is here yet, but if you sign in and leave some user-generated-content, then we’ll have content here for the next guy.” By the way, want our newsletter? Click an AD!)
Curated Content Comprised almost entirely of bits and pieces of content that exists elsewhere.
Media & Related Pages
  • Image Pages & Files
  • PDF PAges & Files – Google indexes PDFs just as it indexes regular pages. Are PDFs being linked from Indexed pages?
  • Videos Pages & Files
  • Audio Pages & Files

Special issues & Fixes

These is for content that has specific issues. 

Content Classification How to Decide this category Action How to decide Action & exact procedure
Prime Content Pages with high visits/entrances but low conversion, time-on-site, page views per session, etc. Improve & Analyse These pages are really good place to add A/B testing and take the optimization to next level.
Influencer Pages Pages with good link and social metrics. Analyse These pages are really good place to learn what is working on your website. Find out why these pages are getting more shares and likes and how to repeat this formula for other pages.
Thin Content pages Thin or Short content are pages Glossed over the topic, too few words, or all image-based content. Improve, Remove or Consolidate
  • If there are many thing content pages belonging to one topic consider consolidating such pages and implement canonicalization
  • Remove pages that don’t serve any purpose
  • Improve pages that serve some purpose. There should be > 1500 words per page.
Misleading SEO Titles or keywords targeting queries for which content doesn’t answer or deserve to rank. Generally not providing the information the visitor was expecting to find. Improve
Low Quality pages Poor grammar, written primarily for search engines (includes keyword stuffing), unhelpful, inaccurate Improve OR Remove
Irrelevant content
  • Content with no external links, no social shares, and very few or no entrances/visits
  • Pages with poor link, traffic, and social metrics related to low-quality content that isn’t worth updating
  • The strategy will depend on link equity and traffic as to whether it gets redirected or simply removed.
  • The strategy will depend on link equity and traffic as to whether it gets redirected or simply removed.
  • Too many indexable blog tag or blog category pages
  • Prune from site to remove duplicate content. & Out-of-date content that isn’t worth updating or consolidating T
  • Recommend allowing the URL to return 404 or 410 response code.
  • Remove all internal links, including from the sitemap.”
Internally duplicated
  • Plagiarized content
  • on other pages (e.g., categories, product variants, archives, technical issues, etc.). www.siteliner.com
Remove or Canonicalize
Externally duplicated
  • (e.g., manufacturer product descriptions, product descriptions duplicated in feeds used for other channels like Amazon, shopping comparison sites and eBay, plagiarized content, etc.) Use www.copyscape.com.
Remove or Canonicalize

Understanding the Actions

Consolidate Content
  • When you have overlapping topics that don’t provide much unique value of their own, but could make a great resource when combined.
  • Mark the page in the set with the best metrics as “Improve” and in the Details column, outline which pages are going to be consolidated into it. This is the canonical page.
  • Mark the pages that are to be consolidated into the canonical page as “Consolidate” and provide further instructions in the Details column, such as:
  • Use portions of this content to round out /canonicalpage/ and then 301 redirect this page into /canonicalpage/
  • Update all internal links
  • Campaign-based or seasonal pages that could be consolidated into a single “Evergreen” landing page (e.g., Best Sellers of 2012 and Best Sellers of 2013 —> Best Sellers).
Content left As-is
  • Important pages which have had their content stolen. Sort by entrances or visits (filtering out any that were already finished)
  • Pages with good traffic, conversions, time on site, etc. that also have good content.
  • These may or may not have any decent external links.

 

Writing up the report

As given below

 

As a result of our comprehensive content audit, we are recommending the following, which will be covered in more detail below:

Removal of about 624 pages from Google index by deletion or consolidation:


203 Pages were marked for Removal with a 404 error (no redirect needed)
110 Pages were marked for Removal with a 301 redirect to another page
311 Pages were marked for Consolidation of content into other pages

Followed by a redirect to the page into which they were consolidated

 

Rewriting or improving of 668 pages


605 Product Pages are to be rewritten due to use of manufacturer product descriptions (duplicate content), these being prioritized from first to last within the Content Audit.
63 “Other” pages to be rewritten due to low-quality or duplicate content.


Keeping 226 pages as-is


No rewriting or improvements needed
These changes reflect an immediate need to “improve or remove” content in order to avoid an obvious content-based penalty from Google (e.g. Panda) due to thin, low-quality and duplicate content, especially concerning Representative and Dealers pages with some added risk from Style pages.

 

jass

Share
Published by
jass

Recent Posts

Google Chrome Developer Tools (DevTools)

🔍 How to Open DevTools You can open DevTools in several ways: Right-click on any…

5 days ago

Why Ignoring the Model Context Protocol (MCP) Could Cripple California’s AI Future

Model Context Protocol (MCP) is revolutionizing how AI applications connect with external tools, APIs, and…

6 days ago

Funding opportunity through Small Business Innovation Research (SBIR) for New/Small Businesses

The Small Business Innovation Research (SBIR) program offers a vital pathway for startups to secure…

2 weeks ago

California Dreaming: 10 Business Ideas with Case Studies and SWOT Analysis

Looking to start a business in California? This guide explores 10 high-potential business ideas—from real…

2 weeks ago

USA resources

UNited States https://www.glassdoor.co.in/Overview/Working-at-Prefr-co-EI_IE7010540.11,19.htm https://www.ambitionbox.com/overview/prefr-dot-co-overview https://rocketreach.co/prefrco-management_b7c95a7ec0ee0ee2 https://www.signalhire.com/companies/prefr-co https://www.zoominfo.com/c/prefrco/557658591 United Kingdom https://www.yellowtom.co.uk/reviews/1417493 Canada https://www.pandia.com/ca/oshawa-on/graphic-design Singapore https://talenttribe.asia/companies/prefr-co India…

2 weeks ago

Blockchain Engineer

We are seeking a skilled and motivated Blockchain Engineer to join our team. In this…

2 weeks ago