Benchmarking HDBO
  • Home 
  • About 
  • Benchmarks 
  • Related work 
  •  

Benchmarks

We test several HDBO methods on discrete sequence optimization tasks. So far, we have optimized the PMO benchmark on SELFIES representations, and also thermal stability of red fluorescent proteins and their mutations using RaSP. Discrete solvers like GeneticAlgorithm, Bounce and ProbRep work directly on sequence space, while others rely on latent representations learned using autoencoders. Higher scores are better for all tasks.

For details, check our repository on GitHub.

   PMO 2D  PMO 128  RaSP  Ehrlich
BAxUS 15.01 ± 4.28 0.79 +/- 0.39
Bounce 1.65 ± 2.47 1.39 ± 2.47 2.00 +/- 0.33
CMA-ES 17.27 ± 2.56 20.08 ± 1.57 0.59 ± 0.36 1.24 +/- 0.45
GeneticAlgorithm 13.73 ± 1.26 14.35 ± 1.19 1.50 +/- 0.31
HillClimbing 12.34 ± 1.33 13.80 ± 2.03 1.00 ± 0.22 0.56 +/- 0.64
Hvarfner's VanillaBO 19.54 ± 1.81 12.96 ± 2.14 0.84 ± 0.41 1.26 +/- 0.47
ProbRep 3.85 ± 3.57 3.64 ± 3.56 1.80 +/- 0.31
RandomLineBO 19.54 ± 4.59 6.80 ± 5.92 0.00 ± 0.00 1.04 +/- 0.60
SAASBO 19.24 ± 8.38 19.66 ± 3.29 0.67 ± 0.57 0.92 +/- 0.46
Turbo 14.52 ± 4.84 15.39 ± 4.08 0.17 ± 0.37 1.86 +/- 0.40
DirectedEvolution 2.23 +/- 0.27
   albuterol_similarity  amlodipine_mpo  celecoxib_rediscovery  deco_hop  drd2_docking  fexofenadine_mpo  gsk3_beta  isomer_c7h8n2o2  isomer_c9h10n2o2pf2cl  jnk3  median_1  median_2  mestranol_similarity  osimetrinib_mpo  perindopril_mpo  ranolazine_mpo  rdkit_logp  rdkit_qed  sa_tdc  scaffold_hop  sitagliptin_mpo  thiothixene_rediscovery  troglitazone_rediscovery  valsartan_smarts  zaleplon_mpo  Sum (normalized per row)
HillClimbing 0.47 ± 0.00 0.44 ± 0.00 0.04 ± 0.01 0.53 ± 0.00 0.03 ± 0.00 0.45 ± 0.01 0.33 ± 0.10 0.67 ± 0.13 0.52 ± 0.10 0.20 ± 0.02 0.06 ± 0.01 0.03 ± 0.01 0.20 ± 0.04 0.63 ± 0.02 0.02 ± 0.03 0.37 ± 0.15 6.32 ± 0.75 0.64 ± 0.06 9.14 ± 0.37 0.38 ± 0.01 0.12 ± 0.09 0.04 ± 0.02 0.05 ± 0.02 0.00 ± 0.00 0.10 ± 0.09 13.80 ± 2.03
GeneticAlgorithm 0.32 ± 0.03 0.29 ± 0.06 0.08 ± 0.00 0.52 ± 0.00 0.03 ± 0.00 0.48 ± 0.03 0.19 ± 0.01 0.72 ± 0.09 0.56 ± 0.05 0.10 ± 0.01 0.18 ± 0.03 0.08 ± 0.01 0.26 ± 0.00 0.63 ± 0.01 0.10 ± 0.03 0.53 ± 0.02 13.28 ± 0.41 0.66 ± 0.07 8.71 ± 0.20 0.36 ± 0.00 0.05 ± 0.05 0.13 ± 0.03 0.11 ± 0.02 0.00 ± 0.00 0.05 ± 0.00 14.35 ± 1.19
CMA-ES 0.44 ± 0.03 0.41 ± 0.01 0.16 ± 0.01 0.54 ± 0.01 0.03 ± 0.00 0.66 ± 0.01 0.20 ± 0.04 0.73 ± 0.12 0.57 ± 0.14 0.10 ± 0.03 0.17 ± 0.01 0.12 ± 0.00 0.38 ± 0.02 0.51 ± 0.30 0.25 ± 0.10 0.59 ± 0.03 21.62 ± 0.12 0.90 ± 0.04 7.48 ± 0.31 0.39 ± 0.01 0.15 ± 0.15 0.20 ± 0.05 0.16 ± 0.01 0.00 ± 0.00 0.13 ± 0.04 20.08 ± 1.57
Hvarfner's VanillaBO 0.41 ± 0.04 0.42 ± 0.03 0.05 ± 0.01 0.53 ± 0.00 0.09 ± 0.06 0.40 ± 0.03 0.27 ± 0.03 0.49 ± 0.04 0.56 ± 0.05 0.16 ± 0.03 0.05 ± 0.01 0.03 ± 0.01 0.18 ± 0.03 0.60 ± 0.01 0.00 ± 0.00 0.18 ± 0.13 5.93 ± 1.19 0.61 ± 0.02 8.84 ± 0.25 0.38 ± 0.01 0.12 ± 0.13 0.05 ± 0.01 0.05 ± 0.01 0.00 ± 0.00 0.09 ± 0.03 12.96 ± 2.14
RandomLineBO 0.29 ± 0.10 0.36 ± 0.04 0.03 ± 0.00 0.53 ± 0.00 0.02 ± 0.01 0.29 ± 0.02 0.20 ± 0.07 0.06 ± 0.04 0.32 ± 0.21 0.10 ± 0.03 0.03 ± 0.01 0.01 ± 0.00 0.06 ± 0.03 0.59 ± 0.01 0.02 ± 0.03 0.07 ± 0.03 5.60 ± 4.35 0.42 ± 0.03 8.17 ± 0.83 0.37 ± 0.00 0.05 ± 0.05 0.02 ± 0.00 0.02 ± 0.00 0.00 ± 0.00 0.00 ± 0.00 6.80 ± 5.92
SAASBO 0.43 ± 0.05 0.41 ± 0.02 0.18 ± 0.04 0.53 ± 0.01 0.03 ± 0.00 0.64 ± 0.04 0.22 ± 0.06 0.66 ± 0.18 0.43 ± 0.19 0.08 ± 0.02 0.16 ± 0.01 0.12 ± 0.00 0.35 ± 0.05 0.59 ± 0.06 0.25 ± 0.08 0.63 ± 0.01 19.87 ± 1.21 0.79 ± 0.13 7.14 ± 0.88 0.38 ± 0.01 0.11 ± 0.12 0.22 ± 0.02 0.16 ± 0.01 0.00 ± 0.00 0.33 ± 0.08 19.66 ± 3.29
BAxUS 0.33 ± 0.09 0.41 ± 0.02 0.16 ± 0.01 0.53 ± 0.02 0.03 ± 0.00 0.49 ± 0.30 0.12 ± 0.04 0.49 ± 0.05 0.36 ± 0.16 0.08 ± 0.02 0.12 ± 0.01 0.12 ± 0.02 0.34 ± 0.05 0.22 ± 0.35 0.24 ± 0.14 0.54 ± 0.17 17.84 ± 2.53 0.80 ± 0.09 7.56 ± 0.05 0.37 ± 0.01 0.09 ± 0.07 0.19 ± 0.04 0.13 ± 0.01 0.00 ± 0.00 0.05 ± 0.05 15.01 ± 4.28
Turbo 0.35 ± 0.03 0.42 ± 0.03 0.16 ± 0.05 0.53 ± 0.02 0.03 ± 0.00 0.54 ± 0.24 0.13 ± 0.05 0.20 ± 0.17 0.37 ± 0.22 0.09 ± 0.05 0.14 ± 0.02 0.12 ± 0.02 0.30 ± 0.02 0.33 ± 0.32 0.22 ± 0.13 0.48 ± 0.19 20.87 ± 1.85 0.75 ± 0.13 5.55 ± 0.31 0.37 ± 0.02 0.16 ± 0.06 0.20 ± 0.03 0.15 ± 0.01 0.00 ± 0.00 0.13 ± 0.11 15.39 ± 4.08
Bounce 0.16 ± 0.01 0.00 ± 0.00 0.02 ± 0.01 0.50 ± 0.00 0.01 ± 0.00 0.13 ± 0.13 0.09 ± 0.08 0.11 ± 0.09 0.01 ± 0.01 0.05 ± 0.04 0.03 ± 0.01 0.01 ± 0.00 0.01 ± 0.00 0.30 ± 0.31 0.00 ± 0.00 0.00 ± 0.00 3.12 ± 1.20 0.52 ± 0.09 8.36 ± 0.46 0.34 ± 0.01 0.00 ± 0.00 0.02 ± 0.01 0.02 ± 0.01 0.00 ± 0.00 0.00 ± 0.00 1.39 ± 2.47
ProbRep 0.21 ± 0.03 0.00 ± 0.00 0.02 ± 0.00 0.51 ± 0.00 0.01 ± 0.00 0.20 ± 0.08 0.12 ± 0.02 0.24 ± 0.11 0.06 ± 0.03 0.06 ± 0.01 0.02 ± 0.00 0.01 ± 0.00 0.02 ± 0.00 0.59 ± 0.04 0.00 ± 0.00 0.11 ± 0.02 5.49 ± 3.01 0.60 ± 0.05 8.59 ± 0.13 0.34 ± 0.00 0.00 ± 0.00 0.03 ± 0.01 0.02 ± 0.00 0.00 ± 0.00 0.00 ± 0.00 3.64 ± 3.56
Experiment setup
  • batch size: 1
  • nr. initialization points: 10
  • budget: 300
  • replications (nr. of seeds): 3
   albuterol_similarity  amlodipine_mpo  celecoxib_rediscovery  deco_hop  drd2_docking  fexofenadine_mpo  gsk3_beta  isomer_c7h8n2o2  isomer_c9h10n2o2pf2cl  jnk3  median_1  median_2  mestranol_similarity  osimetrinib_mpo  perindopril_mpo  ranolazine_mpo  rdkit_logp  rdkit_qed  sa_tdc  scaffold_hop  sitagliptin_mpo  thiothixene_rediscovery  troglitazone_rediscovery  valsartan_smarts  zaleplon_mpo  Sum (normalized per row)
HillClimbing 0.31 ± 0.10 0.30 ± 0.04 0.09 ± 0.01 0.51 ± 0.00 0.03 ± 0.00 0.01 ± 0.01 0.07 ± 0.02 0.45 ± 0.05 0.30 ± 0.14 0.10 ± 0.03 0.13 ± 0.03 0.08 ± 0.01 0.37 ± 0.01 0.00 ± 0.00 0.12 ± 0.01 0.58 ± 0.09 20.85 ± 0.21 0.82 ± 0.00 7.64 ± 0.47 0.34 ± 0.00 0.05 ± 0.03 0.15 ± 0.02 0.10 ± 0.01 0.00 ± 0.00 0.02 ± 0.02 12.34 ± 1.33
GeneticAlgorithm 0.26 ± 0.01 0.30 ± 0.05 0.06 ± 0.02 0.52 ± 0.01 0.03 ± 0.00 0.43 ± 0.02 0.19 ± 0.01 0.63 ± 0.08 0.46 ± 0.17 0.10 ± 0.02 0.13 ± 0.00 0.08 ± 0.02 0.28 ± 0.03 0.62 ± 0.01 0.10 ± 0.08 0.54 ± 0.06 14.76 ± 0.45 0.56 ± 0.02 8.74 ± 0.11 0.36 ± 0.01 0.02 ± 0.01 0.13 ± 0.03 0.11 ± 0.02 0.00 ± 0.00 0.02 ± 0.01 13.73 ± 1.26
CMA-ES 0.38 ± 0.05 0.36 ± 0.04 0.13 ± 0.04 0.52 ± 0.01 0.03 ± 0.00 0.55 ± 0.01 0.16 ± 0.01 0.42 ± 0.10 0.41 ± 0.20 0.06 ± 0.01 0.17 ± 0.02 0.11 ± 0.01 0.36 ± 0.05 0.34 ± 0.28 0.18 ± 0.04 0.57 ± 0.09 20.67 ± 0.66 0.82 ± 0.11 7.00 ± 0.68 0.37 ± 0.00 0.08 ± 0.07 0.18 ± 0.02 0.17 ± 0.02 0.00 ± 0.00 0.12 ± 0.05 17.27 ± 2.56
Hvarfner's VanillaBO 0.40 ± 0.02 0.34 ± 0.02 0.13 ± 0.01 0.53 ± 0.01 0.03 ± 0.01 0.60 ± 0.04 0.14 ± 0.02 0.81 ± 0.07 0.49 ± 0.11 0.10 ± 0.03 0.13 ± 0.02 0.12 ± 0.01 0.34 ± 0.00 0.65 ± 0.03 0.14 ± 0.00 0.62 ± 0.13 20.73 ± 1.05 0.77 ± 0.02 7.98 ± 0.02 0.37 ± 0.00 0.08 ± 0.06 0.21 ± 0.04 0.15 ± 0.00 0.00 ± 0.00 0.15 ± 0.10 19.54 ± 1.81
RandomLineBO 0.36 ± 0.02 0.40 ± 0.05 0.14 ± 0.00 0.53 ± 0.01 0.03 ± 0.00 0.57 ± 0.02 0.20 ± 0.02 0.79 ± 0.10 0.58 ± 0.02 0.08 ± 0.02 0.16 ± 0.01 0.12 ± 0.02 0.35 ± 0.03 0.65 ± 0.06 0.14 ± 0.01 0.69 ± 0.08 18.74 ± 1.83 0.77 ± 0.10 6.86 ± 1.96 0.37 ± 0.00 0.08 ± 0.06 0.18 ± 0.05 0.16 ± 0.01 0.00 ± 0.00 0.22 ± 0.11 19.54 ± 4.59
SAASBO 0.36 ± 0.07 0.39 ± 0.06 0.14 ± 0.01 0.52 ± 0.00 0.02 ± 0.01 0.59 ± 0.07 0.18 ± 0.05 0.84 ± 0.09 0.52 ± 0.18 0.08 ± 0.04 0.15 ± 0.04 0.12 ± 0.01 0.36 ± 0.03 0.64 ± 0.00 0.23 ± 0.10 0.55 ± 0.07 16.94 ± 7.17 0.80 ± 0.10 7.96 ± 0.05 0.37 ± 0.01 0.15 ± 0.13 0.18 ± 0.03 0.14 ± 0.01 0.00 ± 0.00 0.12 ± 0.04 19.24 ± 8.38
Turbo 0.40 ± 0.08 0.37 ± 0.03 0.13 ± 0.04 0.53 ± 0.01 0.03 ± 0.00 0.23 ± 0.30 0.09 ± 0.03 0.25 ± 0.13 0.27 ± 0.08 0.06 ± 0.02 0.17 ± 0.03 0.10 ± 0.01 0.32 ± 0.09 0.45 ± 0.39 0.20 ± 0.08 0.29 ± 0.05 20.15 ± 3.07 0.63 ± 0.03 7.16 ± 0.30 0.38 ± 0.01 0.00 ± 0.01 0.15 ± 0.01 0.12 ± 0.02 0.00 ± 0.00 0.05 ± 0.04 14.52 ± 4.84
Bounce 0.18 ± 0.02 0.00 ± 0.00 0.02 ± 0.01 0.50 ± 0.00 0.01 ± 0.00 0.13 ± 0.13 0.09 ± 0.08 0.11 ± 0.09 0.01 ± 0.01 0.05 ± 0.04 0.03 ± 0.01 0.01 ± 0.00 0.01 ± 0.00 0.30 ± 0.31 0.00 ± 0.00 0.00 ± 0.00 3.12 ± 1.20 0.52 ± 0.09 8.36 ± 0.46 0.34 ± 0.01 0.00 ± 0.00 0.02 ± 0.01 0.02 ± 0.01 0.00 ± 0.00 0.00 ± 0.00 1.65 ± 2.47
ProbRep 0.21 ± 0.03 0.00 ± 0.00 0.02 ± 0.00 0.51 ± 0.00 0.01 ± 0.00 0.20 ± 0.08 0.12 ± 0.02 0.24 ± 0.11 0.06 ± 0.03 0.06 ± 0.01 0.02 ± 0.00 0.01 ± 0.00 0.02 ± 0.00 0.59 ± 0.04 0.00 ± 0.00 0.11 ± 0.02 5.49 ± 3.01 0.60 ± 0.05 8.59 ± 0.13 0.34 ± 0.00 0.00 ± 0.00 0.03 ± 0.01 0.02 ± 0.00 0.00 ± 0.00 0.00 ± 0.00 3.85 ± 3.57
Experiment setup
  • batch size: 1
  • nr. initialization points: 10
  • budget: 100
  • replications (nr. of seeds): 3
   RaSP
HillClimbing 2.02 ± 0.22
CMAES 1.36 ± 0.36
Hvarfner's VanillaBO 1.75 ± 0.41
RandomLineBO 0.39 ± 0.00
SAASBO 1.47 ± 0.57
Turbo 0.67 ± 0.37
Experiment setup
  • batch size: 1
  • nr. initialization points: 6
  • budget: 500
  • replications (nr. of seeds): 5
  • max runtime: 24h
   PestControlEquiv  Ehrlich(L=5)  Ehrlich(L=15)  Ehrlich(L=64)  Sum (normalized per row)
DirectedEvolution 0.968 +/- 0.03 1.000 +/- 0.00 0.448 +/- 0.16 0.114 +/- 0.07 2.23 +/- 0.27
HillClimbing 0.640 +/- 0.12 0.500 +/- 0.25 0.392 +/- 0.20 0.089 +/- 0.07 0.56 +/- 0.64
CMAES 0.816 +/- 0.06 0.750 +/- 0.18 0.312 +/- 0.13 0.077 +/- 0.08 1.24 +/- 0.45
GeneticAlgorithm 0.712 +/- 0.02 0.950 +/- 0.11 0.336 +/- 0.10 0.083 +/- 0.08 1.50 +/- 0.31
Hvarfner's VanillaBO 0.928 +/- 0.08 0.650 +/- 0.14 0.328 +/- 0.18 0.079 +/- 0.08 1.26 +/- 0.47
RandomLineBO 0.624 +/- 0.11 0.700 +/- 0.27 0.472 +/- 0.14 0.084 +/- 0.08 1.04 +/- 0.60
SAASBO 0.792 +/- 0.05 0.600 +/- 0.14 0.328 +/- 0.21 0.075 +/- 0.06 0.92 +/- 0.46
Turbo 0.896 +/- 0.04 0.850 +/- 0.14 0.480 +/- 0.12 0.124 +/- 0.11 1.86 +/- 0.40
BAxUS 0.712 +/- 0.08 0.550 +/- 0.11 0.400 +/- 0.11 0.077 +/- 0.08 0.79 +/- 0.39
Bounce 1.000 +/- 0.00 0.900 +/- 0.14 0.416 +/- 0.13 0.076 +/- 0.06 2.00 +/- 0.33
ProbRep 0.896 +/- 0.04 0.950 +/- 0.11 0.328 +/- 0.08 0.076 +/- 0.08 1.80 +/- 0.31
logo logo
MLLS

This project is brought to you by Center for Basic Machine Learning Research in Life Science

Copyright © 2024 HDBO Team.
Benchmarking HDBO
Code copied to clipboard