Check failed: action.input != nullptr Unable to alloc from host_io_pool for infer action input
I tried sending 8 outstanding requests to the worker in a closed-loop controller. The controller predicts a latency time for requests, and if the request exceeds that time we cancel the next request.
The worker starts sending the error "CopyInputTask failed to allocate memory from io_pool" after around 400 requests, but then the system recovers and continues serving incoming requests. This pattern is repeated throughout the experiment. However, in the case of using a small value for the predicted latency, which means that more requests are canceled, the system terminates with the error message "Check failed: action.input != nullptr Unable to alloc from host_io_pool for infer action input"
I tried examining the remaining value for the io_pools, I noticed the pattern that it keeps decreasing when requests are completed successfully. However, after the system runs out of memory the remaining value increase again.