benchmarks

cédric (dev) said on 04/20/2018

if you want to do useful comparisons, you could try to do them on various colortype (paletted, grayscale, truecolor + alpha, etc.). the more files you test, the more it is significant. your benchmark should provide:

  • time *: essential to evaluate the efficiency
  • savings *: or how much bytes the tool could save
  • files *: original files, and eventually optimized
  • context *: lossless, lossy, usage context, etc.
  • options *: the exact command-line you used
  • memory: how much was used
  • machine specs: CPU/RAM, OS, etc.

you should avoid to do comparisons on different context (lossless vs lossy) but for demonstrations. if it is lossy, use some metrics — butteraugli, dssim, etc. — to do evaluations. if tools have various compression levels or specific fast processes — as removing sections/chunks, it makes sense to compare them in the same scope

cédric (dev) said on 06/04/2018

pingo 0.9 series, FX-4100 @ 3.6 Ghz - 8 Go RAM - Windows 7 64-bit
-s0 profile
version 380 files - paletted (saved) 104 files - RGBA (saved) 24 files - reductions (saved)
0.90 <-> 0.93 428.86 KB 1.63s 531.49KB 2.26s 293.35KB 0.25s
0.94 434.01 KB 1.92s 598.64KB 2.95s 296.79KB 0.30s
0.95 <-> 0.96 434.01 KB 2.05s 606.46KB 2.96s 298.32KB 0.33s
0.97 434.89 KB 1.95s 606.46KB 2.93s 298.62KB 0.28s
-s1 profile
version 380 files - paletted (saved) 104 files - RGBA (saved) 24 files - reductions (saved)
0.90 <-> 0.92 458.96 KB 2.68s 639.62 KB 3.62s 296.25 KB 0.40s
0.93 457.39 KB 2.50s 639.62 KB 3.40s 295.59 KB 0.36s
0.94 478.22 KB 2.99s 732.87 KB 4.43s 306.34 KB 0.41s
0.95 <-> 0.96 478.22 KB 3.18s 733.10 KB 4.22s 306.78 KB 0.38s
0.97 479.84 KB 2.89s 733.10 KB 4.15s 307.11 KB 0.42s
-s2 profile
version 380 files - paletted (saved) 104 files - RGBA (saved) 24 files - reductions (saved)
0.90 488.60 KB 5.72s 769.85 KB 6.20s 301.87 KB 0.67s
0.91 488.64 KB 5.55s 769.85 KB 6.52s 301.83 KB 0.66s
0.92 488.66 KB 5.41s 767.95 KB 5.60s 301.53 KB 0.64s
0.93 452.84 KB 4.70s 765.71 KB 4.92s 299.94 KB 0.49s
0.94 503.25 KB 5.31s 765.43 KB 5.12s 308.84 KB 0.58s
0.95 <-> 0.96 503.25 KB 5.31s 765.74 KB 4.98s 309.19 KB 0.53s
0.97 508.23 KB 5.63s 765.74 KB 4.95s 309.67 KB 0.59s
-s3 profile
version 380 files - paletted (saved) 104 files - RGBA (saved) 24 files - reductions (saved)
0.90 509.34 KB 12.67s 779.37 KB 11.14s 309.01 KB 1.19s
0.91 509.34 KB 12.46s 778.36 KB 10.61s 308.98 KB 1.13s
0.92 509.32 KB 11.58s 779.70 KB 10.85s 308.98 KB 1.08s
0.93 490.95 KB 5.94s 771.85 KB 10.16s 305.19 KB 0.83s
0.94 513.14 KB 6.24s 791.37 KB 8.18s 310.92 KB 0.90s
0.95 <-> 0.96 513.14 KB 6.52s 791.98 KB 8.07s 311.40 KB 0.89s
0.97 513.89 KB 5.94s 791.98 KB 7.77s 311.62 KB 0.95s
-s4 profile
version 380 files - paletted (saved) 104 files - RGBA (saved) 24 files - reductions (saved)
0.90 499.88 KB 12.15s 801.51 KB 13.74s 308.93 KB 1.72s
0.91 500.02 KB 12.14s 801.51 KB 14.29s 308.93 KB 1.72s
0.92 499.99 KB 11.89s 800.76 KB 14.07s 308.86 KB 1.72s
0.93 510.46 KB 9.52s 805.52 KB 14.48s 310.48 KB 1.25s
0.94 521.37 KB 8.93s 810.78 KB 10.67s 311.27 KB 1.04s
0.95 521.39 KB 9.00s 810.78 KB 10.78s 311.57 KB 1.02s
0.96 521.37 KB 8.72s 810.78 KB 10.26s 311.50 KB 0.97s
0.97 522.86 KB 8.58s 810.78 KB 10.37s 311.87 KB 1.03s
380 files - paletted : 72xpaletted - 40xpaletted - 76xpaletted- 192xpaletted
104 files - RGBA : 72xrgba - 10x256rgba - 10x512rgba - 12x512rgba
24 files - reductions : 24xreductions