Benchmarks
We test several HDBO methods in the PMO benchmark using the tools provided by the Therapeutics Data Commons . In future work we plan to incorporate objective functions involving protein engineering and other high-dimensional discrete optimization tasks.
Model rankings
PMO 128D
PMO 2D
PMO 2D
PMO 128D
HillClimbing
12.41 ± 1.33
14.33 ± 1.89
Hvarfner's VanillaBO
19.65 ± 1.81
13.63 ± 2.16
RandomLineBO
19.60 ± 4.59
7.13 ± 5.92
SAASBO
19.35 ± 8.38
20.54 ± 3.29
Turbo
14.58 ± 4.84
16.12 ± 4.08
Bounce
1.72 ± 2.47
1.62 ± 2.47
ProbRep
3.93 ± 3.57
3.96 ± 3.56
BAxUS
15.82 ± 4.28
Oracle
HillClimbing
Hvarfner's VanillaBO
RandomLineBO
SAASBO
BAxUS
Turbo
Bounce
ProbRep
albuterol_similarity
0.47 ± 0.00
0.41 ± 0.04
0.29 ± 0.10
0.43 ± 0.05
0.33 ± 0.09
0.35 ± 0.03
0.16 ± 0.01
0.21 ± 0.03
amlodipine_mpo
0.44 ± 0.00
0.42 ± 0.03
0.36 ± 0.04
0.41 ± 0.02
0.41 ± 0.02
0.42 ± 0.03
0.00 ± 0.00
0.00 ± 0.00
celecoxib_rediscovery
0.04 ± 0.01
0.05 ± 0.01
0.03 ± 0.00
0.18 ± 0.04
0.16 ± 0.01
0.16 ± 0.05
0.02 ± 0.01
0.02 ± 0.00
deco_hop
0.53 ± 0.00
0.53 ± 0.00
0.53 ± 0.00
0.53 ± 0.01
0.53 ± 0.02
0.53 ± 0.02
0.50 ± 0.00
0.51 ± 0.00
drd2_docking
0.03 ± 0.00
0.09 ± 0.06
0.02 ± 0.01
0.03 ± 0.00
0.03 ± 0.00
0.03 ± 0.00
0.01 ± 0.00
0.01 ± 0.00
fexofenadine_mpo
0.45 ± 0.01
0.40 ± 0.03
0.29 ± 0.02
0.64 ± 0.04
0.49 ± 0.30
0.54 ± 0.24
0.13 ± 0.13
0.20 ± 0.08
gsk3_beta
0.33 ± 0.10
0.27 ± 0.03
0.20 ± 0.07
0.22 ± 0.06
0.12 ± 0.04
0.13 ± 0.05
0.09 ± 0.08
0.12 ± 0.02
isomer_c7h8n2o2
0.67 ± 0.13
0.49 ± 0.04
0.06 ± 0.04
0.66 ± 0.18
0.49 ± 0.05
0.20 ± 0.17
0.11 ± 0.09
0.24 ± 0.11
isomer_c9h10n2o2pf2cl
0.52 ± 0.10
0.56 ± 0.05
0.32 ± 0.21
0.43 ± 0.19
0.36 ± 0.16
0.37 ± 0.22
0.01 ± 0.01
0.06 ± 0.03
jnk3
0.20 ± 0.02
0.16 ± 0.03
0.10 ± 0.03
0.08 ± 0.02
0.08 ± 0.02
0.09 ± 0.05
0.05 ± 0.04
0.06 ± 0.01
median_1
0.06 ± 0.01
0.05 ± 0.01
0.03 ± 0.01
0.16 ± 0.01
0.12 ± 0.01
0.14 ± 0.02
0.03 ± 0.01
0.02 ± 0.00
median_2
0.03 ± 0.01
0.03 ± 0.01
0.01 ± 0.00
0.12 ± 0.00
0.12 ± 0.02
0.12 ± 0.02
0.01 ± 0.00
0.01 ± 0.00
mestranol_similarity
0.20 ± 0.04
0.18 ± 0.03
0.06 ± 0.03
0.35 ± 0.05
0.34 ± 0.05
0.30 ± 0.02
0.01 ± 0.00
0.02 ± 0.00
osimetrinib_mpo
0.63 ± 0.02
0.60 ± 0.01
0.59 ± 0.01
0.59 ± 0.06
0.22 ± 0.35
0.33 ± 0.32
0.30 ± 0.31
0.59 ± 0.04
perindopril_mpo
0.02 ± 0.03
0.00 ± 0.00
0.02 ± 0.03
0.25 ± 0.08
0.24 ± 0.14
0.22 ± 0.13
0.00 ± 0.00
0.00 ± 0.00
ranolazine_mpo
0.37 ± 0.15
0.26 ± 0.15
0.07 ± 0.03
0.63 ± 0.01
0.54 ± 0.17
0.48 ± 0.19
0.00 ± 0.00
0.11 ± 0.02
rdkit_logp
6.32 ± 0.75
5.93 ± 1.19
5.60 ± 4.35
19.87 ± 1.21
17.84 ± 2.53
20.87 ± 1.85
3.12 ± 1.20
5.49 ± 3.01
rdkit_qed
0.64 ± 0.06
0.61 ± 0.02
0.42 ± 0.03
0.79 ± 0.13
0.80 ± 0.09
0.75 ± 0.13
0.52 ± 0.09
0.60 ± 0.05
sa_tdc
9.06 ± 0.23
8.84 ± 0.25
8.17 ± 0.83
7.14 ± 0.88
7.56 ± 0.05
5.55 ± 0.31
8.36 ± 0.46
8.59 ± 0.13
scaffold_hop
0.38 ± 0.01
0.38 ± 0.01
0.37 ± 0.00
0.38 ± 0.01
0.37 ± 0.01
0.37 ± 0.02
0.34 ± 0.01
0.34 ± 0.00
sitagliptin_mpo
0.12 ± 0.09
0.12 ± 0.13
0.05 ± 0.05
0.11 ± 0.12
0.09 ± 0.07
0.16 ± 0.06
0.00 ± 0.00
0.00 ± 0.00
thiothixene_rediscovery
0.04 ± 0.02
0.05 ± 0.01
0.02 ± 0.00
0.22 ± 0.02
0.19 ± 0.04
0.20 ± 0.03
0.02 ± 0.01
0.03 ± 0.01
troglitazone_rediscovery
0.05 ± 0.02
0.05 ± 0.01
0.02 ± 0.00
0.16 ± 0.01
0.13 ± 0.01
0.15 ± 0.01
0.02 ± 0.01
0.02 ± 0.00
valsartan_smarts
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
zaleplon_mpo
0.10 ± 0.09
0.09 ± 0.03
0.00 ± 0.00
0.33 ± 0.08
0.05 ± 0.05
0.13 ± 0.11
0.00 ± 0.00
0.00 ± 0.00
Sum (normalized per row)
14.33 ± 1.89
13.63 ± 2.16
7.13 ± 5.92
20.54 ± 3.29
15.82 ± 4.28
16.12 ± 4.08
1.62 ± 2.47
3.96 ± 3.56
Experiment setup
batch size: 1
nr. initialization points: 10
budget: 300
replications (nr. of seeds): 3
Oracle
HillClimbing
Hvarfner's VanillaBO
RandomLineBO
SAASBO
Turbo
Bounce
ProbRep
albuterol_similarity
0.31 ± 0.10
0.40 ± 0.02
0.36 ± 0.02
0.36 ± 0.07
0.40 ± 0.08
0.18 ± 0.02
0.21 ± 0.03
amlodipine_mpo
0.30 ± 0.04
0.34 ± 0.02
0.40 ± 0.05
0.39 ± 0.06
0.37 ± 0.03
0.00 ± 0.00
0.00 ± 0.00
celecoxib_rediscovery
0.09 ± 0.01
0.13 ± 0.01
0.14 ± 0.00
0.14 ± 0.01
0.13 ± 0.04
0.02 ± 0.01
0.02 ± 0.00
deco_hop
0.51 ± 0.00
0.53 ± 0.01
0.53 ± 0.01
0.52 ± 0.00
0.53 ± 0.01
0.50 ± 0.00
0.51 ± 0.00
drd2_docking
0.03 ± 0.00
0.03 ± 0.01
0.03 ± 0.00
0.02 ± 0.01
0.03 ± 0.00
0.01 ± 0.00
0.01 ± 0.00
fexofenadine_mpo
0.01 ± 0.01
0.60 ± 0.04
0.57 ± 0.02
0.59 ± 0.07
0.23 ± 0.30
0.13 ± 0.13
0.20 ± 0.08
gsk3_beta
0.07 ± 0.02
0.14 ± 0.02
0.20 ± 0.02
0.18 ± 0.05
0.09 ± 0.03
0.09 ± 0.08
0.12 ± 0.02
isomer_c7h8n2o2
0.45 ± 0.05
0.81 ± 0.07
0.79 ± 0.10
0.84 ± 0.09
0.25 ± 0.13
0.11 ± 0.09
0.24 ± 0.11
isomer_c9h10n2o2pf2cl
0.30 ± 0.14
0.49 ± 0.11
0.58 ± 0.02
0.52 ± 0.18
0.27 ± 0.08
0.01 ± 0.01
0.06 ± 0.03
jnk3
0.10 ± 0.03
0.10 ± 0.03
0.08 ± 0.02
0.08 ± 0.04
0.06 ± 0.02
0.05 ± 0.04
0.06 ± 0.01
median_1
0.13 ± 0.03
0.13 ± 0.02
0.16 ± 0.01
0.15 ± 0.04
0.17 ± 0.03
0.03 ± 0.01
0.02 ± 0.00
median_2
0.08 ± 0.01
0.12 ± 0.01
0.12 ± 0.02
0.12 ± 0.01
0.10 ± 0.01
0.01 ± 0.00
0.01 ± 0.00
mestranol_similarity
0.37 ± 0.01
0.34 ± 0.00
0.35 ± 0.03
0.36 ± 0.03
0.32 ± 0.09
0.01 ± 0.00
0.02 ± 0.00
osimetrinib_mpo
0.00 ± 0.00
0.65 ± 0.03
0.65 ± 0.06
0.64 ± 0.00
0.45 ± 0.39
0.30 ± 0.31
0.59 ± 0.04
perindopril_mpo
0.12 ± 0.01
0.14 ± 0.00
0.14 ± 0.01
0.23 ± 0.10
0.20 ± 0.08
0.00 ± 0.00
0.00 ± 0.00
ranolazine_mpo
0.58 ± 0.09
0.62 ± 0.13
0.69 ± 0.08
0.55 ± 0.07
0.29 ± 0.05
0.00 ± 0.00
0.11 ± 0.02
rdkit_logp
20.85 ± 0.21
20.73 ± 1.05
18.74 ± 1.83
16.94 ± 7.17
20.15 ± 3.07
3.12 ± 1.20
5.49 ± 3.01
rdkit_qed
0.82 ± 0.00
0.77 ± 0.02
0.77 ± 0.10
0.80 ± 0.10
0.63 ± 0.03
0.52 ± 0.09
0.60 ± 0.05
sa_tdc
7.64 ± 0.47
7.98 ± 0.02
6.86 ± 1.96
7.96 ± 0.05
7.16 ± 0.30
8.36 ± 0.46
8.59 ± 0.13
scaffold_hop
0.34 ± 0.00
0.37 ± 0.00
0.37 ± 0.00
0.37 ± 0.01
0.38 ± 0.01
0.34 ± 0.01
0.34 ± 0.00
sitagliptin_mpo
0.05 ± 0.03
0.08 ± 0.06
0.08 ± 0.06
0.15 ± 0.13
0.00 ± 0.01
0.00 ± 0.00
0.00 ± 0.00
thiothixene_rediscovery
0.15 ± 0.02
0.21 ± 0.04
0.18 ± 0.05
0.18 ± 0.03
0.15 ± 0.01
0.02 ± 0.01
0.03 ± 0.01
troglitazone_rediscovery
0.10 ± 0.01
0.15 ± 0.00
0.16 ± 0.01
0.14 ± 0.01
0.12 ± 0.02
0.02 ± 0.01
0.02 ± 0.00
valsartan_smarts
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
zaleplon_mpo
0.02 ± 0.02
0.15 ± 0.10
0.22 ± 0.11
0.12 ± 0.04
0.05 ± 0.04
0.00 ± 0.00
0.00 ± 0.00
Sum (normalized per row)
12.41 ± 1.33
19.65 ± 1.81
19.60 ± 4.59
19.35 ± 8.38
14.58 ± 4.84
1.72 ± 2.47
3.93 ± 3.57
Experiment setup
batch size: 1
nr. initialization points: 10
budget: 100
replications (nr. of seeds): 3