Solver Statistics

From Sokoban Wiki

Jump to: navigation, search

A Sokoban solver program is a program that tries to solve Sokoban levels. There are several solver programs available. (See Sokoban solvers.)

Solver programs may have different objectives:

  1. Find any solution
  2. Find push optimal solutions
  3. Find push optimal solutions with best moves
  4. Find move optimal solutions
  5. Find move optimal solutions with best pushes

Given the computational complexity of these tasks, a solver program is usually designed for just finding any solution. If a solution is to be optimized, an optimizer program can be used. (See Optimizers.)

The statistics presented here provide a picture of what the best freely available Sokoban solvers can do.
For copyrights, credits and redistribution of these statistics, please read the disclaimer.


Contents

General benchmarking information

For practical reasons, almost all tests have been performed with a 10 minutes time limit. The tests are, however, not a 10 minutes contest but a showcase for what the solvers can do, so other time limits are entirely feasible.

The hardware used for the tests is a standard PC with a modern operating system (64bit).

Tests are performed with default solver settings.

Solver Version Author Architecture CPU usage Source code available More info
Festival 3.1 Yaron Shoham 64bit 8 threads Festival 3.1 Website with papers about the solver
Sokolution 6.3 Florent Diedler 64bit all threads no Sokoban solver "scribbles" by Florent Diedler
Takaken 7.4 Ken'ichiro Takahashi (takaken) 64bit 2 threads no Description of the used algorithms
Curry 0.3 Yaron Shoham 64bit 1 thread Curry 0.3 Website with paper about the solver
YASS 2.150 Brian Damgaard 32bit 1 thread YASS 2.150 Sokoban solver "scribbles" by Brian Damgaard
JSoko 2.04 Matthias Meger 64bit 1 thread JSoko 2.04 Solver information
Rolling Stone 1.0 Andreas Junghanns 32bit 1 thread RS Rolling Stone papers

Contact: SokobanoWiki@gmail.com

XSokoban Test Suite

The XSokoban level set is the de facto standard test suite in the academic literature on Sokoban solver programming.

Sokoban Solver Statistics - XSokoban Test Suite - Solved Levels
Collection Author Levels Rolling Stone Takaken YASS JSoko Sokolution Curry Festival
XSokoban Thinking Rabbit, ... 90 54 90 89 67 90 90 90
Sokoban Solver Statistics - XSokoban Test Suite - Push-optimal Solutions
Collection Author Levels Rolling Stone Pereira et al. JSoko Sokolution
XSokoban Thinking Rabbit, ... 90 6 28 31 59

The statistic shows that Sokolution is by far the best solver when searching for push optimal solutions.

Large Test Suite

The large test suite comprises level packs selected for being a challenge for the solvers, but not too difficult. The statistics shows the results for the leading Sokoban solver programs.

This test suite is not subject to change, as it is intended to be a "Large Standard Test Suite" for Sokoban solver programs.

Sokoban Solver Statistics - Large Test Suite - Solved levels
Levels Festival Sokolution Takaken YASS
Total levels 3272 3205 3171 2953 2865

Click here to see the detailed results for the Large Test Suite.

Download the entire test suite here or the individual level sets from www.sokobano.de.

Open Test Suite

The statistics shows the results for the leading Sokoban solvers for various level collections. Please note that the grand totals are simple summations. The number of solved unique levels may differ slightly due to duplicate levels in the level packs.

This test suite may be changed by adding new level collections from time to time.

Sokoban Solver Statistics - Open Test Suite - Solved levels
Levels Festival Sokolution Takaken YASS
Total levels 4762 4634 4566 4207 3996

Click here to see the detailed results for the Open Test Suite.

GroupEffort Test Suite

The only reason for benchmarking the GroupEffort test suite is to demonstrate the limited veracity of the statement made in section "6 Conclusion" of the thesis, where the author writes that the GroupEffort solver "outperforms existing solvers".

The benchmarks show that this was only true for the cherry-picked small test set at the time the thesis was published, never for the presumably more representative large test set.

Sokoban Solver Statistics - GroupEffort Test Suite - Solved Levels
Collection Levels GroupEffort Festival Sokolution Takaken JSoko YASS
Small Set 200 196 (a)(c) 200 (b) 200 (a) 192 (b) 180 (b) 191 (b)
Large Set 2851 1566 (a)(c) 2766 (b) 2471 (a) 2462 (b) 2165 (b) 2260 (b)
Total 3051 1762 2966 2671 2654 2345 2471

(a): Time limit: 5 minutes     (b): Time limit: 10 minutes     (c): Best GroupEffort solver configuration

Disclaimer

Sokoban
=======

Sokoban (R) Registered Trademark of Falcon Co., Ltd., Japan Sokoban Copyright (C) 1982-2004 by Hiroyuki Imabayashi, Japan Sokoban Copyright (C) 1989, 1990, 2001-2004 by Falcon Co., Ltd., Japan
License =======
Redistribution and use of the Sokoban Solver statistics are permitted provided that the following conditions are met:
* Redistributions may not be sold, nor may they be used in a commercial product or activity. * Redistributions must reproduce the entire content of this disclaimer in the documentation and/or other materials provided with the distribution.
Credits (in no particular order) ================================
CosmoNotes, CosmoPoly, LOMA, MicroCosmos, MiniCosmos, NaboCosmos and PicoCosmos level sets Copyright (C) by Aymeric du Peloux. E-mail: aymeric.du-peloux@gadz.org Web site: unknown
dh1, dh2, bagatelle, cantrip, and maelstrom level sets Copyright (C) by David Holland E-mail: unknown Web site: unknown
Microban, Mas Microban, Sasquatch, Mas Sasquatch, Sasquatch III, Sasquatch IV, Sasquatch V, Sasquatch VI and Sasquatch VII level sets. Copyright (C) by David W. Skinner E-mail: sasquatch@bentonrea.com Web site: unknown
Grigr2001, Grigr2002 and GrigrSpecial level sets Copyright (C) by Evgeny Grigoriev (aka GRIGoRusha) E-mail: grigr@yandex.ru Web site: http://grigr.narod.ru/
SokEvo and SokHard level sets Copyright (C) 2002-2006 by Lee J Haywood E-mail: fruise@gmail.com Web Site: http://leehaywood.org/games/sokoban/
Sven level set Copyright (C) 2007 by Sven Egevad E-mail: sven.egevad@telia.com Web Site: unknown
Sokoban Perfect and Sokoban Revenge level sets Copyright (C) 1989, 1991 by Thinking Rabbit Web Site: https://sokoban.jp/
Y.M. Auto and Y.M. Handmade level sets Copyright (C) Yoshio Murase E-mail: yoshio@asahi.email.ne.jp Web Site: http://www.ne.jp/asahi/ai/yoshio/sokoban/index.html
JSoko Copyright (C) by Matthias Meger E-mail: JSoko@mail.de Web site: https://www.sokoban-online.de
Sokoban Automatic Solver (aka Takaken solver) Copyright (C) 2002 - 200X Ken'ichiro Takahashi (aka takaken) E-mail: takaken@ic-net.or.jp Web site: https://computerpuzzle.net/english/sokoban/ver740/index.html
YASS - Yet Another Sokoban Solver and Optimizer - For Small Levels Copyright (C) 2007 by Brian Damgaard Web site: https://sourceforge.net/projects/sokobanyasc/
Sokoban YASC Copyright (C) 2001-2020 by Brian Damgaard Web Site: https://sourceforge.net/projects/sokobanyasc/
YSokoban Copyright (C) 2007 by George Petrov E-mail: YGP@WriteMe.com Web Site: http://ygp.orgfree.com/sokoban.html
XSokoban Copyright (C) by Andrew Myers E-mail: andru@cs.cornell.edu Web Site: http://www.cs.cornell.edu/andru/xsokoban.html
Sokofan Copyright (C) 2005-2007 by Eric Leung E-mail: eleung2@hotpop.com Web Site: unknown
Sokolution Copyright (C) 2016-2019 by Florent DIEDLER E-mail: fdiedler2000@gmail.com Web site: https://sokolution.000webhostapp.com/
Festival Copyright (C) 2020 by Yaron Shoham Web site: https://festival-solver.site/
Trademarks ==========
Company names, brand names and product names are trademarks or registered trademarks of their respective holders.
Contact =======
E-mail: sokobanowiki@gmail.com

Personal tools