ycsb load with mem databases fails, yielding INSERT_ERROR

[h-w-chen]

when rkv uses mem database as storage backend, go-ycsb load phase fails with INSERT_ERROR (go-ycsb workload specifies threadcount 4):

$ ./bin/go-ycsb load rkv -P workloads/workloada
***************** properties *****************
"dotransactions"="false"
"operationcount"="1000"
"scanproportion"="0"
"workload"="core"
"readallfields"="true"
"threadcount"="4"
"requestdistribution"="uniform"
"updateproportion"="0.5"
"recordcount"="1000"
"readproportion"="0.5"
"insertproportion"="0"
**********************************************
Run finished, takes 179.69837ms
INSERT - Takes(s): 0.2, Count: 55, OPS: 310.2, Avg(us): 1403, Min(us): 492, Max(us): 3987, 99th(us): 3213, 99.9th(us): 3987, 99.99th(us): 3987
INSERT_ERROR - Takes(s): 0.1, Count: 941, OPS: 6705.2, Avg(us): 523, Min(us): 187, Max(us): 9167, 99th(us): 3247, 99.9th(us): 7919, 99.99th(us): 9167

rkv log indicates concurrent map write causing the crash

The url is 172.31.9.140:6379 and the pool is &{0x6c8780 <nil> <nil> 80 12000 0s false 0s {0 0} false 0 {0 {0 0}} <nil> {0 <nil> <nil>} 0 0}
The url is 172.31.12.96:6380 and the pool is &{0x6c8780 <nil> <nil> 80 12000 0s false 0s {0 0} false 0 {0 {0 0}} <nil> {0 <nil> <nil>} 0 0}
fatal error: concurrent map writes

goroutine 315 [running]:
runtime.throw(0x7cdf55, 0x15)
        /home/howell/go/go1.16.9/src/runtime/panic.go:1117 +0x72 fp=0xc0003f3e30 sp=0xc0003f3e00 pc=0x437ab2
runtime.mapassign_faststr(0x7636c0, 0xc0003821e0, 0xc00002264c, 0x3, 0x0)
        /home/howell/go/go1.16.9/src/runtime/map_faststr.go:211 +0x3f1 fp=0xc0003f3e98 sp=0xc0003f3e30 pc=0x415eb1
github.com/regionless-storage-service/pkg/database.MemDatabase.Put(...)
        /home/howell/work/regionless-storage-service/pkg/database/mem.go:31
github.com/regionless-storage-service/pkg/database.(*MemDatabase).Put(0xc0003c6000, 0xc00002264c, 0x3, 0xc000468000, 0xdd7, 0x0, 0x0, 0x0, 0x83be20)
        <autogenerated>:1 +0x65 fp=0xc0003f3ed0 sp=0xc0003f3e98 pc=0x6c8ba5
github.com/regionless-storage-service/pkg/piping.(*ChainPiping).Write.func1(0xc000022640, 0x83be20, 0xc00051dd10, 0xc0000ca2a0, 0xc00002264c, 0x3, 0xc000468000, 0xdd7)
        /home/howell/work/regionless-storage-service/pkg/piping/chain_piping_manager.go:61 +0x18d fp=0xc0003f3fa0 sp=0xc0003f3ed0 pc=0x6d244d
runtime.goexit()
        /home/howell/go/go1.16.9/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0003f3fa8 sp=0xc0003f3fa0 pc=0x46d3e1
created by github.com/regionless-storage-service/pkg/piping.(*ChainPiping).Write
        /home/howell/work/regionless-storage-service/pkg/piping/chain_piping_manager.go:57 +0x32d

...

Other things worthy of noting:

• the delay code of mem database Put method was temporily commented out before the test;
• go-ycsb workload threadcount 4

[jshaofuturewei]

mem database is used for unit tests only. Please use redis to get real test data.