Logic to Clean and Preprocess a Website’s Content for RAG

Implementing a cleaning logic for data involves several steps. You’ll need to remove unnecessary elements (like HTML tags, special characters, or stop words), format the text consistently, and tokenize it for further processing. Here’s a step-by-step guide with sample code:

Step 1: Remove Unnecessary Elements

You can use regular expressions, natural language processing (NLP) libraries, and other text processing tools to clean your text data.

import pandas as pd
import re
from bs4 import BeautifulSoup
import nltk
from nltk.corpus import stopwords
nltk.download('stopwords')
nltk.download('punkt')

# Load the exported CSV file
data = pd.read_csv('path_to_exported_file.csv')

# Combine content into a single text column if needed
data['content'] = data['title'] + ' ' + data['content']

# Function to clean and preprocess text
def clean_text(text):
    # Remove HTML tags
    text = BeautifulSoup(text, "html.parser").get_text()
    
    # Remove special characters and digits
    text = re.sub(r'[^a-zA-Z\s]', '', text)
    
    # Convert to lowercase
    text = text.lower()
    
    # Tokenize text
    tokens = nltk.word_tokenize(text)
    
    # Remove stop words
    tokens = [word for word in tokens if word not in stopwords.words('english')]
    
    # Join tokens back into a string
    cleaned_text = ' '.join(tokens)
    
    return cleaned_text

# Apply cleaning function to content
data['cleaned_content'] = data['content'].apply(clean_text)

# Save the cleaned data
data.to_csv('cleaned_data.csv', index=False)

Step 2: Format the Text

Ensure the text is in a consistent format, such as converting all text to lowercase, removing extra whitespace, etc.

Example:

The ‘clean_text‘ function in the example above already covers most formatting tasks, such as converting to lowercase and removing extra whitespace by joining tokens with a single space.

Step 3: Tokenize the Text

Tokenization is the process of splitting text into smaller units (tokens), such as words or subwords. The example above demonstrates word tokenization using NLTK.

Detailed Explanation of Cleaning Logic:

Removing HTML Tags:

text = BeautifulSoup(text, "html.parser").get_text()

This removes any HTML tags from the text using BeautifulSoup.

Removing Special Characters and Digits:

text = re.sub(r'[^a-zA-Z\s]', '', text)

This regular expression replaces any character that is not a letter or whitespace with an empty string.

Converting to Lowercase:

text = text.lower()

This converts the entire text to lowercase to ensure consistency.

Tokenizing Text:

tokens = nltk.word_tokenize(text)

This splits the text into individual words (tokens).

Removing Stop Words:

tokens = [word for word in tokens if word not in stopwords.words('english')]

This removes common words (stop words) that do not contribute significant meaning to the text.

Joining Tokens:

cleaned_text = ' '.join(tokens)

This joins the tokens back into a single string with spaces in between.

Optional Enhancements:

Depending on your specific needs, you might want to add further cleaning steps, such as:

• Lemmatization/Stemming: Reduce words to their base or root form.
• Handling Contractions: Expand contractions (e.g., “don’t” to “do not”).
• Removing Short Words: Remove words shorter than a certain length (e.g., two characters).

Example of Lemmatization:

from nltk.stem import WordNetLemmatizer
nltk.download('wordnet')

lemmatizer = WordNetLemmatizer()

def lemmatize_text(tokens):
    return [lemmatizer.lemmatize(token) for token in tokens]

data['lemmatized_content'] = data['cleaned_content'].apply(lambda x: ' '.join(lemmatize_text(nltk.word_tokenize(x))))

Conclusion

The provided code and explanations should give you a solid foundation for cleaning and preprocessing your text data. You can adjust and expand the cleaning logic based on the specific requirements of your domain and the nature of your data.