EleutherAI/delphi

Add plotting logic for results

Closed this issue · 1 comments

It would be nice to be able to plot the accuracy / probability distributions, similarly to https://github.com/EleutherAI/clearnets/blob/bbf4ca75f0a20dd1950b8f47f405276204ddd448/clearnets/autointerp/autointerp_plot.py#L92C1-L92C14

This could probably be accomplished by adding something like the below code snippet to delphi/log/result_analysis.py.

def plot_line(df):
    out_path = Path("images")
    out_path.mkdir(parents=True, exist_ok=True)

    for score_type in df["score_type"].unique():
        # Create density curves for accuracies
        plot_data = []
        mask = (df["score_type"] == score_type)
        values = df[mask]["accuracy"]
        if len(values) > 0:
            kernel = stats.gaussian_kde(values)
            x_range = np.linspace(values.min(), values.max(), 200)
            density = kernel(x_range)
            plot_data.extend([{"x": x, "density": d} 
                            for x, d in zip(x_range, density)])

        fig = px.line(
            plot_data,
            x="x",
            y="density",
            title=f"Accuracy Distribution - {score_type}"
        )
        fig.write_image(out_path / f"autointerp_accuracies_{score_type}.pdf", format="pdf")

Resolved in https://github.com/EleutherAI/delphi/pull/64/files. Switched back to normal histograms because the kde thing is confusing when the number of data points is low.