Skip to main content

Table 2 Results of the probability calibration study for all targets. The performance of the models is evaluated on the test set. The model HPs were tuned to optimize the AUC score on a validation set. CEs, BSs and AUC values are shown across targets for all probability calibration methods. Results are averaged over 10 model repetitions, except for the deep ensemble models, for which 5 model repeats were computed

From: Achieving well-informed decision-making in drug discovery: a comprehensive calibration study using neural network-based structure-activity models

HP-Metric: AUC

Target

Model

ECE

ACE

BS

AUC

CYP3A4

MLP

0.1799 ± 0.0111

0.1709 ± 0.0121

0.1951 ± 0.0103

0.7682 ± 0.0087

MLP + P

0.0415 ± 0.0072

0.0438 ± 0.0068

0.1586 ± 0.0048

0.7682 ± 0.0087

MLP-D

0.2262 ± 0.0964

0.2262 ± 0.0948

0.2395 ± 0.0539

0.6968 ± 0.0276

MLP-E

0.1142 ± 0.0017

0.1054 ± 0.0024

0.1602 ± 0.0002

0.7957 ± 0.0015

MLP-E + P

0.0551 ± 0.0046

0.0542 ± 0.004

0.1503 ± 0.0002

0.7957 ± 0.0015

HBLL

0.0452 ± 0.0114

0.0448 ± 0.0103

0.1589 ± 0.0046

0.7684 ± 0.0083

HBLL + P

0.0452 ± 0.0114

0.0448 ± 0.0103

0.1589 ± 0.0046

0.7684 ± 0.0083

MAO-A

MLP

0.2379 ± 0.0185

0.2281 ± 0.017

0.252 ± 0.016

0.695 ± 0.0134

MLP + P

0.0574 ± 0.0125

0.0619 ± 0.0129

0.1926 ± 0.0058

0.695 ± 0.0134

MLP-D

0.0902 ± 0.0322

0.0942 ± 0.0337

0.2063 ± 0.0123

0.6562 ± 0.0196

MLP-E

0.1544 ± 0.0053

0.1437 ± 0.0029

0.2099 ± 0.0016

0.7187 ± 0.0028

MLP-E + P

0.0643 ± 0.0044

0.0578 ± 0.0101

0.1861 ± 0.0011

0.7187 ± 0.0028

HBLL

0.0607 ± 0.0148

0.0647 ± 0.0128

0.1959 ± 0.0047

0.6824 ± 0.0139

HBLL + P

0.0607 ± 0.0148

0.0647 ± 0.0128

0.1959 ± 0.0047

0.6824 ± 0.0139

hERG

MLP

0.0763 ± 0.0061

0.0721 ± 0.0085

0.079 ± 0.0052

0.761 ± 0.0139

MLP + P

0.0102 ± 0.0024

0.015 ± 0.0023

0.0576 ± 0.0005

0.761 ± 0.0139

MLP-D

0.3175 ± 0.1577

0.3172 ± 0.1578

0.1875 ± 0.1368

0.6205 ± 0.0526

MLP-E

0.0539 ± 0.0033

0.0535 ± 0.0031

0.0662 ± 0.0008

0.8088 ± 0.0032

MLP-E + P

0.0126 ± 0.0007

0.0199 ± 0.0026

0.0559 ± 0.0002

0.8088 ± 0.0032

HBLL

0.0126 ± 0.0007

0.0199 ± 0.0026

0.0559 ± 0.0002

0.8088 ± 0.0032

HBLL + P

0.0126 ± 0.0007

0.0199 ± 0.0026

0.0559 ± 0.0002

0.8088 ± 0.0032