Mace fixes

• Bugfix in batching logic for infer_only_scheduler
• A quick python script for summarizing action log
• Fixed a controller memory leak in the controller networking code
• Tweaked worker mempool

Controllers need to be careful about freeing input and output memory of InferRequest (client), InferResponse (client), Infer (worker), and InferResult (worker) in the following ways:

• InferRequest from client - the controller network code will allocate memory for input. the controller needs to delete input when done
• InferResponse to client - the controller network code assumes that your controller allocated memory for output, and will delete output after sending has completed
• Infer to worker - the controller network code does not allocate or free any memory. you can safely set Infer::input = InferRequest::input .
• InferResult from worker - the controller network code will allocate memory for output. you can safely set InferResponse::output = InferResult::output
• Batching if you batch InferRequest then your controller will have to allocate memory for the batch. This means you need to pay attention to the following:
• Batching Infer - you will have to allocate memory for input. You should wait until you receive an InferResult before freeing the input of the original Infer.
• Batching InferResponse - you will have to allocate memory for each request's output. As described above, this will be automatically deleted by the controller network code once the response has been sent to the client
• Batching InferResult - after copying the batched output to each individual InferResponse, you will need to free the InferResult::output.

bin/process_action_log.py

+
+import math
+import subprocess
+import itertools
+import os
+import json
+import time
+import argparse
+import numpy as np
+from scipy import signal
+import random
+from collections import Counter
+
+parser = argparse.ArgumentParser(description='Process an actions log')
+parser.add_argument("inputfile", metavar="INPUTFILE", type=str, help="Name of an actions log file")
+# parser.add_argument("outputfile", metavar="OUTPUTFILE", type=str, help="Outputfile for summary")
+
+class ModelInstance:
+	def __init__(self, worker_id, gpu_id, model_id, batch_size):
+		self.worker_id = worker_id
+		self.gpu_id = gpu_id
+		self.model_id = model_id
+		self.batch_size = batch_size
+		self.controller_durations = []
+		self.worker_durations = []
+		self.samples = []
+
+	def add_sample(self, t, controller_action_duration, worker_exec_duration):
+		self.controller_durations.append(controller_action_duration)
+		self.worker_durations.append(worker_exec_duration)
+		self.samples.append((t, controller_action_duration, worker_exec_duration))
+
+	def summary(self):
+		return "%d\t%d\t%d\t%d\t%d\t%d" % (
+			self.worker_id, 
+			self.gpu_id, 
+			self.model_id, 
+			self.batch_size, 
+			len(self.controller_durations), 
+			len(self.worker_durations))
+
+def process(lines):
+	models = {}
+	for line in lines:
+		try:
+			splits = [int(v) for v in line.split("\t")]
+		except:
+			continue
+
+		t = splits[0]
+		action_id = splits[1]
+		action_type = splits[2]
+		status = splits[3]
+		worker_id = splits[4]
+		gpu_id = splits[5]
+		model_id = splits[6]
+		batch_size = splits[7]
+		controller_action_duration = splits[8]
+		worker_exec_duration = splits[9]
+
+		key = (worker_id, gpu_id, model_id, batch_size)
+
+		if key not in models:
+			models[key] = ModelInstance(worker_id, gpu_id, model_id, batch_size)
+		models[key].add_sample(t, controller_action_duration, worker_exec_duration)
+
+	return models;
+
+def printsummary(models):
+	for key, model in models.items():
+		print(model.summary())
+
+
+
+def simplesummary(args):
+	with open(args.inputfile, "r") as f:
+		lines = f.readlines()
+
+	headers = lines[0].split("\t")
+	models = process(lines[1:])
+	printsummary(models)
+
+
+if __name__ == '__main__':
+    args = parser.parse_args()
+    exit(simplesummary(args))