Benchmarks
We test several HDBO methods on discrete sequence optimization tasks. So far, we have optimized the PMO benchmark on SELFIES representations, and also thermal stability of red fluorescent proteins and their mutations using RaSP . Discrete solvers like GeneticAlgorithm
, Bounce
and ProbRep
work directly on sequence space, while others rely on latent representations learned using autoencoders. Higher scores are better for all tasks.
For details, check our repository on GitHub .
Model rankings
PMO 128D
PMO 2D
RaSP 32D
Ehrlich
PMO 2D
PMO 128
RaSP
Ehrlich
BAxUS
15.01 ± 4.28
0.79 +/- 0.39
Bounce
1.65 ± 2.47
1.39 ± 2.47
2.00 +/- 0.33
CMA-ES
17.27 ± 2.56
20.08 ± 1.57
0.59 ± 0.36
1.24 +/- 0.45
GeneticAlgorithm
13.73 ± 1.26
14.35 ± 1.19
1.50 +/- 0.31
HillClimbing
12.34 ± 1.33
13.80 ± 2.03
1.00 ± 0.22
0.56 +/- 0.64
Hvarfner's VanillaBO
19.54 ± 1.81
12.96 ± 2.14
0.84 ± 0.41
1.26 +/- 0.47
ProbRep
3.85 ± 3.57
3.64 ± 3.56
1.80 +/- 0.31
RandomLineBO
19.54 ± 4.59
6.80 ± 5.92
0.00 ± 0.00
1.04 +/- 0.60
SAASBO
19.24 ± 8.38
19.66 ± 3.29
0.67 ± 0.57
0.92 +/- 0.46
Turbo
14.52 ± 4.84
15.39 ± 4.08
0.17 ± 0.37
1.86 +/- 0.40
DirectedEvolution
2.23 +/- 0.27
albuterol_similarity
amlodipine_mpo
celecoxib_rediscovery
deco_hop
drd2_docking
fexofenadine_mpo
gsk3_beta
isomer_c7h8n2o2
isomer_c9h10n2o2pf2cl
jnk3
median_1
median_2
mestranol_similarity
osimetrinib_mpo
perindopril_mpo
ranolazine_mpo
rdkit_logp
rdkit_qed
sa_tdc
scaffold_hop
sitagliptin_mpo
thiothixene_rediscovery
troglitazone_rediscovery
valsartan_smarts
zaleplon_mpo
Sum (normalized per row)
HillClimbing
0.47 ± 0.00
0.44 ± 0.00
0.04 ± 0.01
0.53 ± 0.00
0.03 ± 0.00
0.45 ± 0.01
0.33 ± 0.10
0.67 ± 0.13
0.52 ± 0.10
0.20 ± 0.02
0.06 ± 0.01
0.03 ± 0.01
0.20 ± 0.04
0.63 ± 0.02
0.02 ± 0.03
0.37 ± 0.15
6.32 ± 0.75
0.64 ± 0.06
9.14 ± 0.37
0.38 ± 0.01
0.12 ± 0.09
0.04 ± 0.02
0.05 ± 0.02
0.00 ± 0.00
0.10 ± 0.09
13.80 ± 2.03
GeneticAlgorithm
0.32 ± 0.03
0.29 ± 0.06
0.08 ± 0.00
0.52 ± 0.00
0.03 ± 0.00
0.48 ± 0.03
0.19 ± 0.01
0.72 ± 0.09
0.56 ± 0.05
0.10 ± 0.01
0.18 ± 0.03
0.08 ± 0.01
0.26 ± 0.00
0.63 ± 0.01
0.10 ± 0.03
0.53 ± 0.02
13.28 ± 0.41
0.66 ± 0.07
8.71 ± 0.20
0.36 ± 0.00
0.05 ± 0.05
0.13 ± 0.03
0.11 ± 0.02
0.00 ± 0.00
0.05 ± 0.00
14.35 ± 1.19
CMA-ES
0.44 ± 0.03
0.41 ± 0.01
0.16 ± 0.01
0.54 ± 0.01
0.03 ± 0.00
0.66 ± 0.01
0.20 ± 0.04
0.73 ± 0.12
0.57 ± 0.14
0.10 ± 0.03
0.17 ± 0.01
0.12 ± 0.00
0.38 ± 0.02
0.51 ± 0.30
0.25 ± 0.10
0.59 ± 0.03
21.62 ± 0.12
0.90 ± 0.04
7.48 ± 0.31
0.39 ± 0.01
0.15 ± 0.15
0.20 ± 0.05
0.16 ± 0.01
0.00 ± 0.00
0.13 ± 0.04
20.08 ± 1.57
Hvarfner's VanillaBO
0.41 ± 0.04
0.42 ± 0.03
0.05 ± 0.01
0.53 ± 0.00
0.09 ± 0.06
0.40 ± 0.03
0.27 ± 0.03
0.49 ± 0.04
0.56 ± 0.05
0.16 ± 0.03
0.05 ± 0.01
0.03 ± 0.01
0.18 ± 0.03
0.60 ± 0.01
0.00 ± 0.00
0.18 ± 0.13
5.93 ± 1.19
0.61 ± 0.02
8.84 ± 0.25
0.38 ± 0.01
0.12 ± 0.13
0.05 ± 0.01
0.05 ± 0.01
0.00 ± 0.00
0.09 ± 0.03
12.96 ± 2.14
RandomLineBO
0.29 ± 0.10
0.36 ± 0.04
0.03 ± 0.00
0.53 ± 0.00
0.02 ± 0.01
0.29 ± 0.02
0.20 ± 0.07
0.06 ± 0.04
0.32 ± 0.21
0.10 ± 0.03
0.03 ± 0.01
0.01 ± 0.00
0.06 ± 0.03
0.59 ± 0.01
0.02 ± 0.03
0.07 ± 0.03
5.60 ± 4.35
0.42 ± 0.03
8.17 ± 0.83
0.37 ± 0.00
0.05 ± 0.05
0.02 ± 0.00
0.02 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
6.80 ± 5.92
SAASBO
0.43 ± 0.05
0.41 ± 0.02
0.18 ± 0.04
0.53 ± 0.01
0.03 ± 0.00
0.64 ± 0.04
0.22 ± 0.06
0.66 ± 0.18
0.43 ± 0.19
0.08 ± 0.02
0.16 ± 0.01
0.12 ± 0.00
0.35 ± 0.05
0.59 ± 0.06
0.25 ± 0.08
0.63 ± 0.01
19.87 ± 1.21
0.79 ± 0.13
7.14 ± 0.88
0.38 ± 0.01
0.11 ± 0.12
0.22 ± 0.02
0.16 ± 0.01
0.00 ± 0.00
0.33 ± 0.08
19.66 ± 3.29
BAxUS
0.33 ± 0.09
0.41 ± 0.02
0.16 ± 0.01
0.53 ± 0.02
0.03 ± 0.00
0.49 ± 0.30
0.12 ± 0.04
0.49 ± 0.05
0.36 ± 0.16
0.08 ± 0.02
0.12 ± 0.01
0.12 ± 0.02
0.34 ± 0.05
0.22 ± 0.35
0.24 ± 0.14
0.54 ± 0.17
17.84 ± 2.53
0.80 ± 0.09
7.56 ± 0.05
0.37 ± 0.01
0.09 ± 0.07
0.19 ± 0.04
0.13 ± 0.01
0.00 ± 0.00
0.05 ± 0.05
15.01 ± 4.28
Turbo
0.35 ± 0.03
0.42 ± 0.03
0.16 ± 0.05
0.53 ± 0.02
0.03 ± 0.00
0.54 ± 0.24
0.13 ± 0.05
0.20 ± 0.17
0.37 ± 0.22
0.09 ± 0.05
0.14 ± 0.02
0.12 ± 0.02
0.30 ± 0.02
0.33 ± 0.32
0.22 ± 0.13
0.48 ± 0.19
20.87 ± 1.85
0.75 ± 0.13
5.55 ± 0.31
0.37 ± 0.02
0.16 ± 0.06
0.20 ± 0.03
0.15 ± 0.01
0.00 ± 0.00
0.13 ± 0.11
15.39 ± 4.08
Bounce
0.16 ± 0.01
0.00 ± 0.00
0.02 ± 0.01
0.50 ± 0.00
0.01 ± 0.00
0.13 ± 0.13
0.09 ± 0.08
0.11 ± 0.09
0.01 ± 0.01
0.05 ± 0.04
0.03 ± 0.01
0.01 ± 0.00
0.01 ± 0.00
0.30 ± 0.31
0.00 ± 0.00
0.00 ± 0.00
3.12 ± 1.20
0.52 ± 0.09
8.36 ± 0.46
0.34 ± 0.01
0.00 ± 0.00
0.02 ± 0.01
0.02 ± 0.01
0.00 ± 0.00
0.00 ± 0.00
1.39 ± 2.47
ProbRep
0.21 ± 0.03
0.00 ± 0.00
0.02 ± 0.00
0.51 ± 0.00
0.01 ± 0.00
0.20 ± 0.08
0.12 ± 0.02
0.24 ± 0.11
0.06 ± 0.03
0.06 ± 0.01
0.02 ± 0.00
0.01 ± 0.00
0.02 ± 0.00
0.59 ± 0.04
0.00 ± 0.00
0.11 ± 0.02
5.49 ± 3.01
0.60 ± 0.05
8.59 ± 0.13
0.34 ± 0.00
0.00 ± 0.00
0.03 ± 0.01
0.02 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
3.64 ± 3.56
Experiment setup
batch size: 1
nr. initialization points: 10
budget: 300
replications (nr. of seeds): 3
albuterol_similarity
amlodipine_mpo
celecoxib_rediscovery
deco_hop
drd2_docking
fexofenadine_mpo
gsk3_beta
isomer_c7h8n2o2
isomer_c9h10n2o2pf2cl
jnk3
median_1
median_2
mestranol_similarity
osimetrinib_mpo
perindopril_mpo
ranolazine_mpo
rdkit_logp
rdkit_qed
sa_tdc
scaffold_hop
sitagliptin_mpo
thiothixene_rediscovery
troglitazone_rediscovery
valsartan_smarts
zaleplon_mpo
Sum (normalized per row)
HillClimbing
0.31 ± 0.10
0.30 ± 0.04
0.09 ± 0.01
0.51 ± 0.00
0.03 ± 0.00
0.01 ± 0.01
0.07 ± 0.02
0.45 ± 0.05
0.30 ± 0.14
0.10 ± 0.03
0.13 ± 0.03
0.08 ± 0.01
0.37 ± 0.01
0.00 ± 0.00
0.12 ± 0.01
0.58 ± 0.09
20.85 ± 0.21
0.82 ± 0.00
7.64 ± 0.47
0.34 ± 0.00
0.05 ± 0.03
0.15 ± 0.02
0.10 ± 0.01
0.00 ± 0.00
0.02 ± 0.02
12.34 ± 1.33
GeneticAlgorithm
0.26 ± 0.01
0.30 ± 0.05
0.06 ± 0.02
0.52 ± 0.01
0.03 ± 0.00
0.43 ± 0.02
0.19 ± 0.01
0.63 ± 0.08
0.46 ± 0.17
0.10 ± 0.02
0.13 ± 0.00
0.08 ± 0.02
0.28 ± 0.03
0.62 ± 0.01
0.10 ± 0.08
0.54 ± 0.06
14.76 ± 0.45
0.56 ± 0.02
8.74 ± 0.11
0.36 ± 0.01
0.02 ± 0.01
0.13 ± 0.03
0.11 ± 0.02
0.00 ± 0.00
0.02 ± 0.01
13.73 ± 1.26
CMA-ES
0.38 ± 0.05
0.36 ± 0.04
0.13 ± 0.04
0.52 ± 0.01
0.03 ± 0.00
0.55 ± 0.01
0.16 ± 0.01
0.42 ± 0.10
0.41 ± 0.20
0.06 ± 0.01
0.17 ± 0.02
0.11 ± 0.01
0.36 ± 0.05
0.34 ± 0.28
0.18 ± 0.04
0.57 ± 0.09
20.67 ± 0.66
0.82 ± 0.11
7.00 ± 0.68
0.37 ± 0.00
0.08 ± 0.07
0.18 ± 0.02
0.17 ± 0.02
0.00 ± 0.00
0.12 ± 0.05
17.27 ± 2.56
Hvarfner's VanillaBO
0.40 ± 0.02
0.34 ± 0.02
0.13 ± 0.01
0.53 ± 0.01
0.03 ± 0.01
0.60 ± 0.04
0.14 ± 0.02
0.81 ± 0.07
0.49 ± 0.11
0.10 ± 0.03
0.13 ± 0.02
0.12 ± 0.01
0.34 ± 0.00
0.65 ± 0.03
0.14 ± 0.00
0.62 ± 0.13
20.73 ± 1.05
0.77 ± 0.02
7.98 ± 0.02
0.37 ± 0.00
0.08 ± 0.06
0.21 ± 0.04
0.15 ± 0.00
0.00 ± 0.00
0.15 ± 0.10
19.54 ± 1.81
RandomLineBO
0.36 ± 0.02
0.40 ± 0.05
0.14 ± 0.00
0.53 ± 0.01
0.03 ± 0.00
0.57 ± 0.02
0.20 ± 0.02
0.79 ± 0.10
0.58 ± 0.02
0.08 ± 0.02
0.16 ± 0.01
0.12 ± 0.02
0.35 ± 0.03
0.65 ± 0.06
0.14 ± 0.01
0.69 ± 0.08
18.74 ± 1.83
0.77 ± 0.10
6.86 ± 1.96
0.37 ± 0.00
0.08 ± 0.06
0.18 ± 0.05
0.16 ± 0.01
0.00 ± 0.00
0.22 ± 0.11
19.54 ± 4.59
SAASBO
0.36 ± 0.07
0.39 ± 0.06
0.14 ± 0.01
0.52 ± 0.00
0.02 ± 0.01
0.59 ± 0.07
0.18 ± 0.05
0.84 ± 0.09
0.52 ± 0.18
0.08 ± 0.04
0.15 ± 0.04
0.12 ± 0.01
0.36 ± 0.03
0.64 ± 0.00
0.23 ± 0.10
0.55 ± 0.07
16.94 ± 7.17
0.80 ± 0.10
7.96 ± 0.05
0.37 ± 0.01
0.15 ± 0.13
0.18 ± 0.03
0.14 ± 0.01
0.00 ± 0.00
0.12 ± 0.04
19.24 ± 8.38
Turbo
0.40 ± 0.08
0.37 ± 0.03
0.13 ± 0.04
0.53 ± 0.01
0.03 ± 0.00
0.23 ± 0.30
0.09 ± 0.03
0.25 ± 0.13
0.27 ± 0.08
0.06 ± 0.02
0.17 ± 0.03
0.10 ± 0.01
0.32 ± 0.09
0.45 ± 0.39
0.20 ± 0.08
0.29 ± 0.05
20.15 ± 3.07
0.63 ± 0.03
7.16 ± 0.30
0.38 ± 0.01
0.00 ± 0.01
0.15 ± 0.01
0.12 ± 0.02
0.00 ± 0.00
0.05 ± 0.04
14.52 ± 4.84
Bounce
0.18 ± 0.02
0.00 ± 0.00
0.02 ± 0.01
0.50 ± 0.00
0.01 ± 0.00
0.13 ± 0.13
0.09 ± 0.08
0.11 ± 0.09
0.01 ± 0.01
0.05 ± 0.04
0.03 ± 0.01
0.01 ± 0.00
0.01 ± 0.00
0.30 ± 0.31
0.00 ± 0.00
0.00 ± 0.00
3.12 ± 1.20
0.52 ± 0.09
8.36 ± 0.46
0.34 ± 0.01
0.00 ± 0.00
0.02 ± 0.01
0.02 ± 0.01
0.00 ± 0.00
0.00 ± 0.00
1.65 ± 2.47
ProbRep
0.21 ± 0.03
0.00 ± 0.00
0.02 ± 0.00
0.51 ± 0.00
0.01 ± 0.00
0.20 ± 0.08
0.12 ± 0.02
0.24 ± 0.11
0.06 ± 0.03
0.06 ± 0.01
0.02 ± 0.00
0.01 ± 0.00
0.02 ± 0.00
0.59 ± 0.04
0.00 ± 0.00
0.11 ± 0.02
5.49 ± 3.01
0.60 ± 0.05
8.59 ± 0.13
0.34 ± 0.00
0.00 ± 0.00
0.03 ± 0.01
0.02 ± 0.00
0.00 ± 0.00
0.00 ± 0.00
3.85 ± 3.57
Experiment setup
batch size: 1
nr. initialization points: 10
budget: 100
replications (nr. of seeds): 3
RaSP
HillClimbing
2.02 ± 0.22
CMAES
1.36 ± 0.36
Hvarfner's VanillaBO
1.75 ± 0.41
RandomLineBO
0.39 ± 0.00
SAASBO
1.47 ± 0.57
Turbo
0.67 ± 0.37
Experiment setup
batch size: 1
nr. initialization points: 6
budget: 500
replications (nr. of seeds): 5
max runtime: 24h
PestControlEquiv
Ehrlich(L=5)
Ehrlich(L=15)
Ehrlich(L=64)
Sum (normalized per row)
DirectedEvolution
0.968 +/- 0.03
1.000 +/- 0.00
0.448 +/- 0.16
0.114 +/- 0.07
2.23 +/- 0.27
HillClimbing
0.640 +/- 0.12
0.500 +/- 0.25
0.392 +/- 0.20
0.089 +/- 0.07
0.56 +/- 0.64
CMAES
0.816 +/- 0.06
0.750 +/- 0.18
0.312 +/- 0.13
0.077 +/- 0.08
1.24 +/- 0.45
GeneticAlgorithm
0.712 +/- 0.02
0.950 +/- 0.11
0.336 +/- 0.10
0.083 +/- 0.08
1.50 +/- 0.31
Hvarfner's VanillaBO
0.928 +/- 0.08
0.650 +/- 0.14
0.328 +/- 0.18
0.079 +/- 0.08
1.26 +/- 0.47
RandomLineBO
0.624 +/- 0.11
0.700 +/- 0.27
0.472 +/- 0.14
0.084 +/- 0.08
1.04 +/- 0.60
SAASBO
0.792 +/- 0.05
0.600 +/- 0.14
0.328 +/- 0.21
0.075 +/- 0.06
0.92 +/- 0.46
Turbo
0.896 +/- 0.04
0.850 +/- 0.14
0.480 +/- 0.12
0.124 +/- 0.11
1.86 +/- 0.40
BAxUS
0.712 +/- 0.08
0.550 +/- 0.11
0.400 +/- 0.11
0.077 +/- 0.08
0.79 +/- 0.39
Bounce
1.000 +/- 0.00
0.900 +/- 0.14
0.416 +/- 0.13
0.076 +/- 0.06
2.00 +/- 0.33
ProbRep
0.896 +/- 0.04
0.950 +/- 0.11
0.328 +/- 0.08
0.076 +/- 0.08
1.80 +/- 0.31