Deduplicate Records With Fuzzy Matching
Designs a deduplication pipeline for records — customers, transactions, or products — that carry small formatting or spelling variations instead of exact duplicates, using the dataset description, key identifying fields, and known variation types as inputs. Combines exact-match removal with fuzzy techniques like Levenshtein or phonetic matching, assigns a confidence score to each match, proposes a merge strategy, and generates a human-reviewable report before anything is deleted.
Prompt
My dataset probably has duplicate records, but they're not exact matches — there are small variations between them. Dataset: {row_description} Key identifying fields: {key_fields} Known variation types: {variation_types} Design a deduplication pipeline that: 1. Removes exact duplicates first 2. Detects fuzzy duplicates using suitable matching techniques (Levenshtein distance, phonetic matching, etc.) 3. Assigns a match-confidence score (high/medium/low) 4. Defines a merge strategy (which record to keep, how to combine fields) 5. Generates a report of duplicate clusters for a human to review before anything is deleted Implement this as a Python script with pandas plus any needed libraries (fuzzywuzzy, recordlinkage, etc.).
