1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
|
# allocbench - benchmark tool for POSIX memory allocators
allocbench is a POSIX memory allocator benchmarking framework and tooling.
It was used in Florian Fischer's [BA thesis](https://muhq.space/ba.html).
What can allocbench do for you ?
* Deterministically builds and patches allocators from git or source archives
* Couples allocators with various included benchmarks as well as your custom ones
* Supports you in analyzing your benchmark results by providing statistical and plot helper functions
* Comes with support for two different malloc tracers to help you understand benchmark and allocator behavior
* Contains numerous wildly used benchmarks in allocator research (espresso, cfrac, larson, ...)
* It is extended easily with our own allocators and or custom benchmarks
To obtain allocbench run
```shell
git clone https://muhq.space/software/allocbench.git
```
## Requirements
* python >= 3.8
* make, find, gcc (build dependencies)
* perf (`perf stat -d` is the default command to measure benchmark results)
* util-linux (`whereis` is used to find system installed allocators)
* git, tar to handle external artifacts
* numpy, scipy and matplotlib to summarize results and generate plots
## Usage
allocbench consists of three small utilities: `bench.py`, `summarize.py` and `merge.py`.
`bench.py` is used to prepare, analyze and run benchmarks.
usage: bench.py [-h] [--analyze] [-r RUNS] [-v]
[-b BENCHMARKS [BENCHMARKS ...]]
[-xb EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]]
[-a ALLOCATORS [ALLOCATORS ...]] [-rd RESULTDIR] [--license]
[--version]
benchmark memory allocators
optional arguments:
-h, --help show this help message and exit
--analyze analyze benchmark behavior using malt
-r RUNS, --runs RUNS how often the benchmarks run
-v, --verbose more output
-b BENCHMARKS [BENCHMARKS ...], --benchmarks BENCHMARKS [BENCHMARKS ...]
benchmarks to run
-xb EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...], --exclude-benchmarks EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]
explicitly excluded benchmarks
-a ALLOCATORS [ALLOCATORS ...], --allocators ALLOCATORS [ALLOCATORS ...]
allocators to test
-rd RESULTDIR, --resultdir RESULTDIR
directory where all results go
--license print license info and exit
--version print version info and exit
`./summarize.py` is used to summarize results created with bench.py.
It groups the included allocators into categories to produce readable and not extremely noisy plots.
usage: summarize.py [-h] [-t FILE_EXT] [--license] [--version]
[-b BENCHMARKS [BENCHMARKS ...]]
[-x EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]]
results
Summarize allocbench results in allocator sets
positional arguments:
results path to results
optional arguments:
-h, --help show this help message and exit
-t FILE_EXT, --file-ext FILE_EXT
file extension used for plots
--license print license info and exit
--version print version info and exit
-b BENCHMARKS [BENCHMARKS ...], --benchmarks BENCHMARKS [BENCHMARKS ...]
benchmarks to summarize
-x EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...], --exclude-benchmarks EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]
benchmarks to exclude
`./merge.py` can combine the results of different benchmark runs.
usage: merge.py [-h] [--license] [--version] [-b BENCHMARKS [BENCHMARKS ...]]
[-x EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]]
src dest
Merge to allocbench results
positional arguments:
src results which should be merged into dest
dest results in which src should be merged
optional arguments:
-h, --help show this help message and exit
--license print license info and exit
--version print version info and exit
-b BENCHMARKS [BENCHMARKS ...], --benchmarks BENCHMARKS [BENCHMARKS ...]
benchmarks to summarize
-x EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...], --exclude-benchmarks EXCLUDE_BENCHMARKS [EXCLUDE_BENCHMARKS ...]
benchmarks to exclude
### Examples
./bench.py -b loop
runs only the loop benchmark for all allocators found on the system and will put its
results in `$PWD/results/$HOSTNAME/<time>/loop`.
./bench.py -a "tcmalloc*"
builds all tcmalloc variants shipped with allocbench and runs all benchmarks.
./summarize.py <path/to/saved/results>
summarizes the previously created results.
## Benchmarks
You want to compare allocators with your own software or add a new benchmark,
have a look at [doc/Benchmarks.md](doc/Benchmarks.md).
## Allocators
By default all included allocators will be build and measured. For more precise control
about used allocators use the `-a` option.
For more details about what allocators are available, how they are used and how to
include new once have a look at [doc/Allocators.md](doc/Allocators.md).
## License
This program is released under GPLv3. You can find a copy of the license
in the LICENSE file in the programs root directory.
|
