Skip to main content

Table 3 Results of the probability calibration study for all targets. The performance of the models is evaluated on the test set. The model HPs were tuned to optimize the accuracy (ACC) on a validation set. CEs, BSs and AUC values are shown across targets for all model types. Results are averaged over 10 model repetitions, except for the deep ensemble models, for which 5 model repeats were computed

From: Achieving well-informed decision-making in drug discovery: a comprehensive calibration study using neural network-based structure-activity models

HP-Metric: ACC

Target

Model

ECE

ACE

BS

AUC

CYP3A4

MLP

0.1488 ± 0.0031

0.1435 ± 0.0036

0.1759 ± 0.0015

0.7798 ± 0.0055

MLP + P

0.037 ± 0.0053

0.0371 ± 0.004

0.1536 ± 0.0018

0.7798 ± 0.0055

MLP-D

0.0453 ± 0.0123

0.0457 ± 0.0096

0.1559 ± 0.0021

0.7781 ± 0.0049

MLP-E

0.1369 ± 0.0031

0.1333 ± 0.0032

0.1703 ± 0.0003

0.7877 ± 0.0008

MLP-E + P

0.0345 ± 0.0012

0.0342 ± 0.0012

0.1517 ± 0.0001

0.7877 ± 0.0008

HBLL

0.0373 ± 0.0075

0.046 ± 0.0051

0.158 ± 0.0023

0.7741 ± 0.0067

HBLL + P

0.0373 ± 0.0075

0.046 ± 0.0051

0.158 ± 0.0023

0.7741 ± 0.0067

MAO-A

MLP

0.2207 ± 0.0232

0.2136 ± 0.0252

0.2391 ± 0.0224

0.7082 ± 0.0287

MLP + P

0.0544 ± 0.0159

0.0604 ± 0.015

0.188 ± 0.0107

0.7082 ± 0.0287

MLP-D

0.1247 ± 0.0284

0.1275 ± 0.0256

0.2126 ± 0.0088

0.66 ± 0.0292

MLP-E

0.139 ± 0.0028

0.1345 ± 0.0023

0.2005 ± 0.0011

0.728 ± 0.0038

MLP-E + P

0.0568 ± 0.0044

0.0628 ± 0.0059

0.1826 ± 0.001

0.728 ± 0.0038

HBLL

0.0563 ± 0.0166

0.0638 ± 0.0137

0.1897 ± 0.0078

0.7039 ± 0.0292

HBLL + P

0.0563 ± 0.0166

0.0638 ± 0.0137

0.1897 ± 0.0078

0.7039 ± 0.0292

hERG

MLP

0.1028 ± 0.0991

0.1019 ± 0.0997

0.0792 ± 0.0324

0.6928 ± 0.0806

MLP + P

0.0091 ± 0.0016

0.0205 ± 0.0045

0.0591 ± 0.0024

0.6928 ± 0.0806

MLP-D

0.2654 ± 0.0859

0.2645 ± 0.0866

0.138 ± 0.0486

0.6281 ± 0.078

MLP-E

0.0851 ± 0.0135

0.0766 ± 0.0127

0.0626 ± 0.0023

0.741 ± 0.0064

MLP-E + P

0.0089 ± 0.0016

0.0217 ± 0.003

0.0566 ± 0.0003

0.741 ± 0.0064

HBLL

0.0302 ± 0.0128

0.0304 ± 0.0128

0.0595 ± 0.0036

0.6917 ± 0.0835

HBLL + P

0.0302 ± 0.0128

0.0304 ± 0.0128

0.0595 ± 0.0036

0.6917 ± 0.0835