Remove Duplicate Lines

Remove duplicate lines from any list in seconds. Paste email addresses, keywords, URLs, product codes or names, one per line, and the tool keeps a single copy of each. You choose whether Apple and apple count as the same, whether stray spaces should be ignored, whether empty lines go, and whether to keep the first or last copy of each repeat. It shows how many duplicates were removed, lists every line that repeated and how often, and can sort the result A to Z. It all happens in your browser, so customer data never leaves your device.

How to use the Remove duplicates

1. 1 Paste your listPut one item per line in the box. You can paste a column straight from Excel or Google Sheets and each cell becomes a line.
2. 2 Set how lines are comparedLeave case-sensitive off to treat Apple and apple as the same line, and keep trimming on so trailing spaces do not hide duplicates.
3. 3 Decide which copy to keepKeep the first copy to preserve the original order, or the last copy if later lines are more up to date.
4. 4 Choose the orderKeep the original order, or sort the unique lines A to Z or Z to A using British English alphabetical rules.
5. 5 Copy or download the resultCheck the removed count and the list of repeated lines, then copy the unique lines or download them as a text file.

What counts as a duplicate line

Two lines are duplicates when they match exactly after the options you have chosen are applied. With the default settings, the tool ignores spaces at the start and end of each line and ignores differences in capital letters, so " Sarah@Example.co.uk" and "sarah@example.co.uk " are treated as the same entry. That matches what most people mean by a duplicate when they are cleaning a list by hand.

Spaces inside a line still matter. "Oak Road" and "Oak Road" with two spaces are different lines, as are "colour" and "color". If your list has inconsistent spacing or spelling inside entries, tidy those first with our find and replace tool, for example by replacing two spaces with one, and then remove the duplicates.

Lower-case comparison uses British English rules built into your browser, so accented letters such as É and é match each other when case-sensitive is off. Accented and unaccented letters are still different, so "café" and "cafe" are kept as separate lines.

Case-sensitive, trimming and empty lines

Each option changes what the tool treats as the same, so it is worth thinking about your data before you click.

• Case-sensitive off (the default) suits email addresses, domain names, keywords and most names. Email addresses are treated as case-insensitive by almost every mail provider in practice, so John@ and john@ are the same person.
• Case-sensitive on suits passwords, product codes, file names on Linux servers, and anything where capital letters carry meaning, such as the reference codes some booking systems generate.
• Trimming removes invisible spaces and tabs at the ends of lines. Spreadsheets and copied web pages often add them, and they are the most common reason two lines look identical but are not removed.
• Remove empty lines deletes blank lines completely and reports how many there were. Turn it off if blank lines separate groups you want to keep, in which case repeated blank lines collapse into one.

Keeping the first or the last copy

When a line appears more than once, only one copy survives. Keeping the first copy leaves each item where it first appeared, which preserves the original order of your list. This is the usual choice and the right one for most mailing lists, keyword lists and URL lists.

Keeping the last copy places each item where it last appeared instead. That is useful when your list was built up over time and later lines are more recent, such as a log of sign-ups exported in date order or an inventory list where the latest entry is the one to trust. The unique lines are the same either way; only their positions change.

If you also choose to sort the result, the first or last choice makes no visible difference unless case-sensitive is off. In that case it decides which spelling of the line is kept, for example "London" or "LONDON".

Common jobs it helps with

Duplicate lines creep into almost every list that is merged, exported or copied from more than one place. Typical clean-ups include combining newsletter sign-ups from a website form, an event and a shop till into one list before importing it into your email platform, so nobody gets the same campaign twice. Under UK GDPR you also need accurate records, and a list full of repeats makes it harder to honour unsubscribe and erasure requests properly.

SEO and marketing work produces long lists too. Keyword research from several tools, lists of URLs from a crawl and a sitemap, backlink exports and lists of pages to redirect all tend to overlap. Removing duplicates first makes the list shorter and easier to work through, and the repeated lines panel shows which items appeared in more than one source.

Developers and IT staff use it for log lines, IP address lists, firewall allow lists, package names and configuration values. Because the tool runs entirely in your browser, it is safe to use for internal hostnames, customer email addresses and other data you would not want to paste into an online service that uploads it.

Reading the results

The three figures at the top tell you what happened. Lines in is the number of lines you pasted, including empty ones. Unique lines is how many are left. Duplicates removed is how many extra copies were deleted, not counting blank lines, which are reported separately underneath when that option is on.

The list of lines that were duplicated shows each repeated entry and how many times it appeared in total, with the most repeated first. A line marked 3× appeared three times, so two copies were removed. This is often the most useful part of the tool, because it shows whether a problem is a handful of accidental double entries or one item copied dozens of times by an import error. If the removed count is much higher than you expected, scroll through the panel before you copy the result, as it may reveal that two different items were treated as one because of the options you chose.

Working with spreadsheets and large lists

To clean a single column from Excel or Google Sheets, select the cells, copy them and paste into the box. Each cell becomes one line. After removing duplicates, copy the result and paste it back into the sheet, where each line becomes a cell again. For whole rows with several columns, a spreadsheet's own Remove duplicates feature is a better fit, because it can compare chosen columns while keeping the rows together.

The tool happily handles tens of thousands of lines on a normal laptop. Very large lists of several hundred thousand lines will still work, but each change of option takes a moment longer on older phones and tablets. The Use as input button lets you run a second pass with different settings, such as removing duplicates case-sensitively first and then sorting the result. Nothing is saved between visits, so keep your original file until you are happy with the cleaned version.

Examples you can copy

Before:
sarah@example.co.uk
James@Example.co.uk
  sarah@example.co.uk

james@example.co.uk
priya@example.co.uk

After (keep first):
sarah@example.co.uk
James@Example.co.uk
priya@example.co.uk

2 duplicates removed, 1 empty line removed

Input:   red, blue, red, green, blue
First:   red, blue, green
Last:    red, green, blue

Common mistakes to avoid

Frequently asked questions