Docs/Example Request - Show Full RLData experiment + results in Python

[anthonymichaelclark]

Hello,

As an understanding check while reading the paper I attempted to recreate the experiment from Appendix 1. Appendix 1 says in part:

For this example, we considered the RLdata10000 dataset from Sariyar and Borg (2022). This is a
synthetic dataset containing 10,000 records with first name, last name, and date of birth attributes. There
is noise in these attributes and a 10% duplication rate. Ground truth identity is known for all records.
The disambiguation algorithm we consider matches records if any of the following conditions are met:
• records agree on first name, last name, and birth year,
• records agree on first name, birth day, and birth year, or
• records agree on last name, birth day, and birth year.
Note that this is not at all a good disambiguation algorithm. It has 52% precision and 83% recall.

I've attempted to reproduce those precision-recall metrics using that disambiguation algorithm in Python on the RLdata10000 dataset but haven't been able to. Any chance you'd be willing to share it as an example for future readers?

My (likely erroneous) implementation below in case it's helpful, which on my machine returns a precision of 0.5943 and a recall of 0.832:

import pandas as pd

pd.set_option('display.max_columns', None)

df = pd.read_csv('RLdata10000.csv')
comparisons = pd.merge(df, df, how="cross", suffixes=["_left", "_right"])

comparisons['true_match'] = (comparisons["ent_id_left"] == comparisons["ent_id_right"])

first_names_c1_match = comparisons['fname_c1_left'] == comparisons['fname_c1_right']
first_names_c2_match = (
    (comparisons['fname_c2_left'] == comparisons['fname_c2_right'])
    | (comparisons['fname_c2_left'].isna() & comparisons['fname_c2_right'].isna())
)
first_names_match = first_names_c1_match & first_names_c2_match


last_names_c1_match = comparisons['lname_c1_left'] == comparisons["lname_c1_right"]
last_names_c2_match = (
    (comparisons['lname_c2_left'] == comparisons['lname_c2_right'])
    | (comparisons['lname_c2_left'].isna() & comparisons['lname_c2_right'].isna())
)
last_names_match = last_names_c1_match & last_names_c2_match

birth_days_match = comparisons['bd_left'] == comparisons["bd_right"]
birth_years_match = comparisons['by_left'] == comparisons['by_right']

condition_1 = first_names_match & last_names_match & birth_years_match
condition_2 = first_names_match & birth_days_match & birth_years_match
condition_3 = last_names_match & birth_days_match & birth_years_match

comparisons['predicted_match'] = condition_1 | condition_2 | condition_3

comparisons = comparisons[comparisons['predicted_match'] | comparisons['true_match']]
comparisons = comparisons[comparisons['rec_id_left'] != comparisons['rec_id_right']]

comparisons['true_and_predicted_match'] = comparisons['true_match'] & comparisons['predicted_match']

num_true_matches = comparisons['true_match'].sum()
num_predicted_matches = comparisons['predicted_match'].sum()
num_true_and_predicted_matches = comparisons['true_and_predicted_match'].sum()

true_precision = num_true_and_predicted_matches/num_predicted_matches
true_recall = num_true_and_predicted_matches/num_true_matches
true_f1 = (2 * true_precision * true_recall) / (true_precision + true_recall)

print(f"There is a true precision of {round(true_precision, 4)}")
print(f"There is a true recall of {round(true_recall, 4)}")
print(f"There is a true f1 of {round(true_f1, 4)}")