Quick tricks: Duplicate content

Today: Finding duplicate content on your website using Python and Google Colab

What is SEO?

SEO (Search Engine Optimisation): techniques to improve visibility.

• On-page: direct optimisation of your pages.
• Off-page: external strategies (backlinks).

Google weighs relevance, UX and technical performance.

How it works

Bots crawl and index, evaluating:

• Relevance based on keywords
• Authority from backlinks
• Technical factors (speed, responsiveness, SSL)

Good SEO architecture

• Important pages reachable in a few clicks.
• Strategic internal links.
• Clean, descriptive URLs.
• Proper H1/H2/H3 hierarchy.

Why duplicate content hurts

It confuses search engines and splits relevance between pages, causing rankings to drop.

Python + Colab to detect duplicate content

Libraries

• requests – downloads HTML
• BeautifulSoup – parses text
• TfidfVectorizer – converts text to vectors
• cosine_similarity – similarity score 0-1

Full code

import requests
from bs4 import BeautifulSoup
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import re

urls_filtradas = [url for url in enlaces_internos_unicos
                  if '/services' not in url and '/category' not in url]

def obtener_contenido_filtrado(url):
    try:
        respuesta = requests.get(url)
        sopa = BeautifulSoup(respuesta.content, 'html.parser')
        for elemento in sopa.select('nav, footer, header, aside'):
            elemento.decompose()
        parrafos = [p.get_text() for p in sopa.find_all('p')]
        texto_filtrado = []
        for parrafo in parrafos:
            if not re.search(r"categoría|archivos|navegación|menú", parrafo, re.IGNORECASE):
                texto_filtrado.append(parrafo)
        return ' '.join(texto_filtrado)
    except Exception as e:
        print(f"Error al obtener contenido de {url}: {e}")
        return ""

def detectar_duplicados(urls, umbral=0.95):
    textos = [obtener_contenido_filtrado(url) for url in urls]
    vectorizador = TfidfVectorizer().fit_transform(textos)
    sim_matrix = cosine_similarity(vectorizador)
    duplicados = {}
    for i in range(len(urls)):
        duplicidad_total = 0
        detalles_duplicados = []
        for j in range(len(urls)):
            if i != j:
                duplicidad_total += sim_matrix[i][j]
                if sim_matrix[i][j] > umbral:
                    detalles_duplicados.append((urls[j], sim_matrix[i][j]))
        duplicidad_total /= (len(urls) - 1)
        duplicados[urls[i]] = (duplicidad_total, detalles_duplicados)
    return duplicados

duplicados = detectar_duplicados(urls_filtradas)

for url, (duplicidad_total, detalles) in duplicados.items():
    print(f"\nURL: {url}\nTasa total de duplicidad: {duplicidad_total:.2f}")
    if detalles:
        for dup_url, similitud in detalles:
            print(f"  {dup_url} similitud {similitud:.2f}")

Installation

!pip install scikit-learn

How it works

1. Filter URLs.
2. Extract relevant paragraphs.
3. Remove regex patterns.
4. Vectorise using TF-IDF.
5. Calculate cosine similarity.
6. Identify matches with 0.95 threshold.

Links

• Previous post (URL crawling): /blog/trucos-rapidos-para-salir-del-paso-rastreo-de-url/
• Colab notebook: https://colab.research.google.com/drive/1sihrcK5Z5FqHHmonAXoQggQ05WfbRZyo

CTAs

• Newsletter sign-up
• "Book my session" – 30-minute networking