Investigate performance issues of weak_mem

The weak_mem branch is much slower than master, even though the size of the development is not really much larger. There might be some low-hanging performance improvement that we could do.