You are currently viewing Automating URL Inspection with Python and the Google Search Console API

Automating URL Inspection with Python and the Google Search Console API

As an SEO professional or developer, keeping track of how Google replica watches UK indexes your website is crucial. The Google Search Console provides a wealth of data, but manually inspecting URLs one by one can be tedious. Thankfully, the best replica watches Search Console API offers a programmatic way to retrieve this information. In this blog post, I’ll walk you through a Python script I built to automate URL inspections using the cheap replica rolex watches Search Console API, complete with error handling and CSV output. Whether you’re auditing a site or monitoring crawl status, this script can save you time and effort.

What Does the Script Do?

This Python script:

  • Uploads a CSV file containing a list of URLs you want to inspect.
  • Authenticates with the Google Search Console API using a service account.
  • Queries the API to retrieve indexing details for each URL (e.g., coverage state, crawl time, mobile usability).
  • Saves the results in a new CSV file for easy analysis.

By the end, you’ll have a downloadable report with key insights about how Google sees your pages.

Prerequisites

Before diving into the code, here’s what you’ll need:

  1. Google Cloud Project: Set up a project in the Google Cloud Console and enable the Search Console API.
  2. Service Account: Create a service account, download its JSON key file (e.g., api_key.json), and store it in Google Drive.
  3. Search Console Access: Add the service account email (e.g., your-service-account@your-project.iam.gserviceaccount.com) as a user in Search Console for the property you’re inspecting.
  4. Python Environment: This script is designed for Google Colab, but it can be adapted for other environments. You’ll need libraries like google-auth, google-api-python-client, and pandas.

The Code

Here’s the full script with explanations:

from google.oauth2 import service_account
from googleapiclient.discovery import build
import pandas as pd
from pandas import DataFrame
from google.colab import files
import io
from google.colab import drive
uploaded = files.upload()
# Get the url list from the upload so we can read it into a CSV.
for key in uploaded.keys():
  filename = key
df = pd.read_csv(io.BytesIO(uploaded[filename]))
url_list = df.values.tolist()
number_of_elements = len(url_list)
item_number = 1
#Mount Google Drive First
creds = "/content/drive/api_key.json" #replace with location of script in Google Drive
scopes = [
    'https://www.googleapis.com/auth/webmasters',
    'https://www.googleapis.com/auth/webmasters.readonly'
    ]
credentials = service_account.Credentials.from_service_account_file(creds, scopes=scopes)
service = build('searchconsole','v1',credentials=credentials)
results = []
for i in url_list:
  url = i[0]
  request = {
    'inspectionUrl': url,
    'siteUrl': 'https://example.com/' # Replace with your Search Console property URL
  }
  try:
    response = service.urlInspection().index().inspect(body=request).execute()
    API_result_status = "Success"
    inspectionResult = response['inspectionResult']
    coverageState = inspectionResult['indexStatusResult']['coverageState'].replace("n", " ")
    robotsTxtState = inspectionResult['indexStatusResult']['robotsTxtState']
    indexingState = inspectionResult['indexStatusResult']['indexingState']
    try:
      lastCrawlTime = inspectionResult['indexStatusResult']['lastCrawlTime']
    except:
      lastCrawlTime = "n/a"
    pageFetchState = inspectionResult['indexStatusResult']['pageFetchState']
    try:
      googleCanonical = inspectionResult['indexStatusResult']['googleCanonical']
    except:
      googleCanonical = "n/a"
    try:
      userCanonical = inspectionResult['indexStatusResult']['userCanonical']
    except:
      userCanonical = "n/a"
    try:
      crawledAs = inspectionResult['indexStatusResult']['crawledAs']
    except:
      crawledAs = "n/a"
    try:
      mobileUsabilityResult = inspectionResult['mobileUsabilityResult']['verdict']
    except:
      mobileUsabilityResult = "n/a"
  except:
    print("Failed to read " + str(url) + ". Skipping to next page.")
    API_result_status = "Fail"
    inspectionResult = "n/a"
    coverageState = "n/a"
    robotsTxtState = "n/a"
    indexingState = "n/a"
    lastCrawlTime = "n/a"
    pageFetchState = "n/a"
    googleCanonical = "n/a"
    userCanonical = "n/a"
    crawledAs = "n/a"
    mobileUsabilityResult = "n/a"
  result_url = [url,API_result_status,coverageState,robotsTxtState,indexingState,lastCrawlTime,pageFetchState,googleCanonical,userCanonical,crawledAs,mobileUsabilityResult]
  print(url + ' complete. ' + str(item_number) + ' of ' + str(number_of_elements))
  results.append(result_url)
  item_number = item_number + 1
Feature_Labels = ['url','API_result_status','coverageState','robotsTxtState','indexingState','lastCrawlTime','pageFetchState','googleCanonical','userCanonical','crawledAs','mobileUsabilityResult']
print(result_url)
#Convert to dataframe and download
results_csv = DataFrame (data=results,columns=Feature_Labels)
results_csv.to_csv('Inspect-URL-Results.csv')
files.download('Inspect-URL-Results.csv')

How It Works

  1. Setup: The script mounts Google Drive, uploads a CSV file (with a single column of URLs), and reads it into a list.
  2. Authentication: It uses a service account JSON key to authenticate with the Search Console API, specifying the necessary scopes.
  3. Inspection Loop: For each URL, it sends an inspection request and extracts fields like coverageState (e.g., “Submitted and indexed”) and lastCrawlTime. If a field is missing or the request fails, it defaults to “n/a”.
  4. Output: Results are stored in a list, converted to a Pandas DataFrame, and downloaded as a CSV.

Example Usage

  1. Prepare a CSV file (e.g., urls.csv) with a column of URLs:
  2. Upload it when prompted in Colab.
  3. Replace ‘https://example.com/’ with your Search Console property URL.
  4. Run the script and download Inspect-URL-Results.csv.

The output might look like this:

url API_result_status coverageState indexingState lastCrawlTime
https://example.com/page1 Success Submitted and indexed INDEXED 2025-03-20T10:00:00Z
https://example.com/page2 Fail n/a n/a n/a

Why Use This Script?

Manually checking URLs in Search Console is fine for a handful of pages, but for larger sites, automation is a game-changer. This script lets you:

  • Audit indexing status across hundreds of pages.
  • Identify crawl issues or canonical mismatches.
  • Export data for deeper analysis in tools like Excel or Google Sheets.

Conclusion

The Google Search Console API is a powerful tool, and with a little Python, you can unlock its full potential. This script is a starting point—feel free to adapt it to your needs, whether that’s adding more fields, integrating with other APIs, or building a full SEO dashboard.