How to remove duplicate data from datasource using loadash.remove()

Get Help
1

How to remove duplicate data from datasource using loadash.remove()

2

Hello!
There are several ways to remove data from a data source. Could you provide a little more information so that I can answer your question:
How many records do you want to delete? Is there just one or a few/all columns that are duplicated? Do you want to delete duplicate entries multiple times or just once? Perhaps it’s better not to add data to the data source if it already exists, rather than deleting it later?

3

there is a cloumn with duplicate values i need to keep only unique values

4

There is a diff page to display unique column so

5

I’m not sure I understand what you are trying to achieve. As I understood from your question, you only want to store unique records in the data source. So, if you have 30 records in a data source, 10 of which have a duplicate ‘Email’ column value, do you want to overwrite the data source records so that you end up with only 20 records with unique email values? Or do you want to filter the records to see only unique records on the frontend, but leave the records of the data source untouched?

6

Yup only 20 records. Overwrite

7

Great! Here you go then. Please replace ‘dataSourseName’ and ‘columnNameWithDuplicates’ values with your data source name and duplicated column name.

let dataSourseName = 'Data sourse name';
let columnNameWithDuplicates = 'Email';

Fliplet.DataSources.connectByName(dataSourseName).then(function (connection) {
  let Connection = connection;
  //find all records in the DS
  return connection.find().then(function(records){

    //filter unique records by the column name
    let uniqRecords = _.uniqBy(records, `data.${columnNameWithDuplicates}`);

    //get duplicated records
    let duplicates = _.differenceBy(records, uniqRecords, 'id');

    //get duplicated records ids
    let idsToRemove = _.reduce(duplicates, function(result, value, key) {
      result.push(value.id);
      return result;
    }, []);
    return idsToRemove;
  }).then(function(idsToRemove){
    return Connection.commit({
      entries: [],
      delete: idsToRemove,
      append: true,
      extend: true
    });
  });
});