how to get cadence-bench test result?

[Johnite]

I want to know the performance of cadence-server , so, I read the cadence-bench's readme file. Now, I know how to use this tool, but I dont know how to get the result or report ? when the test finished , I got a result, but I am can not understand it well, this is my result:

input:
./cadence --address 192.168.1.10:7933 --domain sailing-cron-domain wf start --tl cadence-bench-tl-0 -wt basic-load-test-workflow -dt 30 -et 60 --if config/bench/basic.json
output:
Started Workflow Id: dcb369f0-01b2-4bae-aa73-aab01c1364c6, run Id: ab4bf5d9-2dc1-478e-b045-d246792ff02b

and I started the cadence-bench
./cadence-bench start
2021/09/03 11:17:47 Loading config; env=development,zone=,configDir=config/bench
2021/09/03 11:17:47 Loading configFiles=[config/bench/base.yaml config/bench/development.yaml]
{"level":"info","ts":"2021-09-03T11:17:48.037+0800","msg":"Started Workflow Worker","Domain":"sailing-cron-domain","TaskList":"cadence-bench-tl-1","WorkerID":"81814@cadence-bench-tl-1"}
{"level":"info","ts":"2021-09-03T11:17:48.037+0800","msg":"Started Workflow Worker","Domain":"sailing-cron-domain","TaskList":"cadence-bench-tl-0","WorkerID":"81814@cadence-bench-tl-0"}
{"level":"info","ts":"2021-09-03T11:17:48.050+0800","msg":"Started Activity Worker","Domain":"sailing-cron-domain","TaskList":"cadence-bench-tl-0","WorkerID":"81814@cadence-bench-tl-0"}
{"level":"info","ts":"2021-09-03T11:17:48.050+0800","msg":"Started Activity Worker","Domain":"sailing-cron-domain","TaskList":"cadence-bench-tl-1","WorkerID":"81814@cadence-bench-tl-1"}
^@^@^@^@^@^@{"level":"info","ts":"2021-09-03T11:19:00.184+0800","msg":"Start activity routineID","Domain":"sailing-cron-domain","TaskList":"cadence-bench-tl-0","WorkerID":"81814@cadence-bench-tl-0","ActivityID":"0","ActivityType":"basic-load-test-launch-activity","WorkflowType":"basic-load-test-workflow","WorkflowID":"dcb369f0-01b2-4bae-aa73-aab01c1364c6","RunID":"ab4bf5d9-2dc1-478e-b045-d246792ff02b","Test":"basic","RoutineID":1}
^@{"level":"info","ts":"2021-09-03T11:19:01.808+0800","msg":"Result of running launcher activity","Domain":"sailing-cron-domain","TaskList":"cadence-bench-tl-0","WorkerID":"81814@cadence-bench-tl-0","ActivityID":"0","ActivityType":"basic-load-test-launch-activity","WorkflowType":"basic-load-test-workflow","WorkflowID":"dcb369f0-01b2-4bae-aa73-aab01c1364c6","RunID":"ab4bf5d9-2dc1-478e-b045-d246792ff02b","Test":"basic","Result":"SuccessCount: 10, FailedCount: 0"}
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

this is my config:
config/bench/development.yaml

bench:
name: "cadence-bench"
#domains: ["cadence-bench", "cadence-bench-sync", "cadence-bench-batch"]
domains: ["sailing-cron-domain"]
numTaskLists: 2

cadence:
service: "cadence-frontend"
host: "192.168.1.10:7933"

config/bench/basic.json
{
"totalLaunchCount": 10,
"waitTimeBufferInSeconds": 60,
"routineCount": 1,
"chainSequence": 4,
"concurrentCount": 100,
"payloadSizeBytes": 1024,
"contextTimeoutInSeconds": 3,
"failureThreshold": 0.0
}

anyone help?

[Johnite]

我们在页面上可以看到一些情况如失败等：

我们可以看出来，有些任务是失败的，但是我想知道失败率是多少，或者一个报告总结也可以。

[Johnite]

另外，在cadence-bench，我们可以得到如下的一些log输出：
{"level":"error","ts":"2021-09-03T11:42:19.719+0800","msg":"basic test stress workflow echo activity execution failed","Domain":"sailing-cron-domain","TaskList":"cadence-bench-tl-0","WorkerID":"81814@yaoxiuqiang@cadence-bench-tl-0","WorkflowType":"github.com/uber/cadence/bench/load/basic.stressWorkflowExecute","WorkflowID":"bca28482-b506-4d6e-8308-57aed2f66822-1-16","RunID":"02533c56-8da9-4930-8b4a-6170d4a4cf54","error":"TimeoutType: SCHEDULE_TO_START","stacktrace":"github.com/uber/cadence/bench/load/basic.stressWorkflowExecute.func1\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/bench/load/basic/stressWorkflow.go:80\ngo.uber.org/cadence/internal.(*selectorImpl).Select.func8.1\n\t/Users/didi/go/pkg/mod/go.uber.org/cadence@v0.15.0/internal/internal_workflow.go:1080\ngo.uber.org/cadence/internal.(*selectorImpl).Select\n\t/Users/didi/go/pkg/mod/go.uber.org/cadence@v0.15.0/internal/internal_workflow.go:1110\ngithub.com/uber/cadence/bench/load/basic.stressWorkflowExecute\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/bench/load/basic/stressWorkflow.go:87\nreflect.Value.call\n\t/usr/local/go/src/reflect/value.go:475\nreflect.Value.Call\n\t/usr/local/go/src/reflect/value.go:336\ngo.uber.org/cadence/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow\n\t/Users/didi/go/pkg/mod/go.uber.org/cadence@v0.15.0/internal/workflow.go:371\ngo.uber.org/cadence/internal.(*workflowExecutor).Execute\n\t/Users/didi/go/pkg/mod/go.uber.org/cadence@v0.15.0/internal/internal_worker.go:690\ngo.uber.org/cadence/internal.(*syncWorkflowDefinition).Execute.func1\n\t/Users/didi/go/pkg/mod/go.uber.org/cadence@v0.15.0/internal/internal_workflow.go:463\ngo.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1\n\t/Users/didi/go/pkg/mod/go.uber.org/cadence@v0.15.0/internal/internal_workflow.go:874"}
{"level":"error","ts":"2021-09-03T11:42:20.155+0800","msg":"basic test stress workflow echo activity execution failed","Domain":"sailing-cron-domain","TaskList":"cadence-bench-tl-1","WorkerID":"81814@yaoxiuqiang@cadence-bench-tl-1","WorkflowType":"github.com/uber/cadence/bench/load/basic.stressWorkflowExecute","WorkflowID":"39c1a6e0-fe77-46d3-bfa9-18cc4dca65a6-1-24","RunID":"143e842f-0e9b-46f1-ba6b-dbcbcb7dda3f","error":"TimeoutType: SCHEDULE_TO_START","stacktrace":"github.com/uber/cadence/bench/load/basic.stressWorkflowExecute.func1\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/bench/load/basic/stressWorkflow.go:80\ngo.uber.org/cadence/internal.(*selectorImpl).Select\n\t/Users/didi/go/pkg/mod/go.uber.org/cadence@v0.15.0/internal/internal_workflow.go:1094\ngithub.com/uber/cadence/bench/load/basic.stressWorkflowExecute\n\t/Users/didi/go/src/go.uber.org/cadence-0.20.0/bench/load/basic/stressWorkflow.go:87\nreflect.Value.call\n\t/usr/local/go/src/reflect/value.go:475\nreflect.Value.Call\n\t/usr/local/go/src/reflect/value.go:336\ngo.uber.org/cadence/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow\n\t/Users/didi/go/pkg/mod/go.uber.org/cadence@v0.15.0/internal/workflow.go:371\ngo.uber.org/cadence/internal.(*workflowExecutor).Execute\n\t/Users/didi/go/pkg/mod/go.uber.org/cadence@v0.15.0/internal/internal_worker.go:690\ngo.uber.org/cadence/internal.(*syncWorkflowDefinition).Execute.func1\n\t/Users/didi/go/pkg/mod/go.uber.org/cadence@v0.15.0/internal/internal_workflow.go:463\ngo.uber.org/cadence/internal.(*dispatcherImpl).newNamedCoroutine.func1\n\t/Users/didi/go/pkg/mod/go.uber.org/cadence@v0.15.0/internal/internal_workflow.go:874"}

我们可以看出这些输出表明了一些失败情况，但是确实是零散的。

[longquanzheng]

The launch workflow HAS the report. Please read the document / README 😐

[longquanzheng]

Also make sure you use the latest code in the master branch. There was a bug in the bench launch workflow which would get a wrong number for failed count

[longquanzheng]

Result":"SuccessCount: 10, FailedCount: 0"}

This is the result. It tells you the test has passed. You setup is able to start 10 basic workflows in parallel and completed within the timeout (5 mins by default)

You can also use workflow observe command or workflow show command to get the results . The sample command is in the README

[Johnite]

I Have this result：
./cadence --address 192.168.1.10:7933 --do sailing-cron-domain wf ob -w 4e9a7c0f-d60a-44f2-8623-1ff25bcdb47a
Progress:
1, 2021-09-03T12:02:47+08:00, WorkflowExecutionStarted
2, 2021-09-03T12:02:47+08:00, DecisionTaskScheduled
3, 2021-09-03T12:02:48+08:00, DecisionTaskStarted
4, 2021-09-03T12:02:48+08:00, DecisionTaskCompleted
5, 2021-09-03T12:02:48+08:00, ActivityTaskScheduled
6, 2021-09-03T12:02:48+08:00, ActivityTaskStarted
7, 2021-09-03T12:02:49+08:00, ActivityTaskCompleted
8, 2021-09-03T12:02:49+08:00, DecisionTaskScheduled
9, 2021-09-03T12:02:49+08:00, DecisionTaskStarted
10, 2021-09-03T12:02:49+08:00, DecisionTaskCompleted
11, 2021-09-03T12:02:49+08:00, TimerStarted
12, 2021-09-03T12:03:47+08:00, WorkflowExecutionTimedOut

Result:
Run Time: 44 seconds
Status: TIMEOUT
Timeout Type: START_TO_CLOSE

cadence-bench log：
{"level":"info","ts":"2021-09-03T12:02:48.067+0800","msg":"Start activity routineID","Domain":"sailing-cron-domain","TaskList":"cadence-bench-tl-0","WorkerID":"81814@yaoxiuqiang@cadence-bench-tl-0","ActivityID":"0","ActivityType":"basic-load-test-launch-activity","WorkflowType":"basic-load-test-workflow","WorkflowID":"4e9a7c0f-d60a-44f2-8623-1ff25bcdb47a","RunID":"5ac535d2-4336-4359-8066-e56761335dc6","Test":"basic","RoutineID":1}
{"level":"info","ts":"2021-09-03T12:02:49.806+0800","msg":"Result of running launcher activity","Domain":"sailing-cron-domain","TaskList":"cadence-bench-tl-0","WorkerID":"81814@yaoxiuqiang@cadence-bench-tl-0","ActivityID":"0","ActivityType":"basic-load-test-launch-activity","WorkflowType":"basic-load-test-workflow","WorkflowID":"4e9a7c0f-d60a-44f2-8623-1ff25bcdb47a","RunID":"5ac535d2-4336-4359-8066-e56761335dc6","Test":"basic","Result":"SuccessCount: 10, FailedCount: 0"}

[Johnite]

总是得到这样的结果，但是我不太清楚timeout的原因

[longquanzheng]

This means the launch workflow timeout. How did you start the launch workflow? Did you pass 3600 (1 hour) as execution timeout like in the README?

“ --et 3600”

[Johnite]

3600 秒我觉得太长，想测一下60-600之间。

[longquanzheng]

Do you have advanced visibility? If not make sure you set useBasicVisibilityValidation to true

[Johnite]

我在测试过程中会调高concurrentCount的值如下所示：
{
"totalLaunchCount": 100,
"waitTimeBufferInSeconds": 60,
"routineCount": 1,
"chainSequence": 4,
"concurrentCount": 2000,
"payloadSizeBytes": 1024,
"contextTimeoutInSeconds": 3,
"failureThreshold": 0.0
}
~
但是我会在测试过程中遇到一些error信息，“BadRequestError{Message: Operation not support. Please use on ElasticSearch}"，我想问的是，当我们设置较高的concurrentCount的值的时候是必须要有ES这个组件才可以吗？

[longquanzheng]

You need to set useBasicVisibilityValidation to true….please read the README