What is the best way to monitor GPU usage in Mam?


Issue: What is the best way to monitor GPU usage in Mam?


Solution:

 What you do is add a usage record to MAM of type Gpus.Moab passes this to MAM in a xml tag <Gpus>1</Gpus>.

Example xml and docs.


2016-03-10 15:31:57.727 3283 INFO MAM::Message::receiveChunk 0 Read message payload (798, <Envelope component="ClusterScheduler" count="1" name="moab" version="9.0.1"><Signature><DigestValue>0GmtUgpxfHAZOzgiKXaKxXNkqfU=</DigestValue><SignatureValue>UoUyyGmVQsN1Gg6OhxkdvWoWy3s=</SignatureValue></Signature><Body timestamp="1457641917"><Request action="Reserve" actor="root"><Object>UsageRecord</Object><Option name="Replace">True</Option><Option name="AccountingMode">strict-allocation</Option><Option name="Duration">30</Option><Data><UsageRecord><Stage>Start</Stage><Type>Job</Type><Instance>Moab.4</Instance><User>jbooth</User><Group>jbooth</Group><Class>batch</Class><Machine>pbs</Machine><Nodes>1</Nodes><NodeType>DEFAULT</NodeType><Processors consumptionRate="1.000000">1</Processors><Gpus>1</Gpus><SubmitTime>1457641917</SubmitTime></UsageRecord></Data></Request></Body></Envelope>).


http://docs.adaptivecomputing.com/9-0-1/MAM/help.htm#topics/moabAccountingManager/14-managingUsageRecords/usageRecordPropertyAutoGeneration.htm

http://docs.adaptivecomputing.com/9-0-1/MWM/help.htm#topics/moabWorkloadManager/topics/fairness/allocationmanagement.html#accountingPropertiesReportedToMam

http://docs.adaptivecomputing.com/9-0-1/MAM/help.htm#topics/moabAccountingManager/14-managingUsageRecords/customizingTheUsageRecordObject.htm%3FTocPath%3D14%2520Managing%2520Usage%2520Records%7C_____9


First Create the usage record for Gpus.

bash-4.2$ mam-shell Attribute Create Object=UsageRecord Name=Gpus DataType=Integer
Successfully created 1 attribute
bash-4.2$

MAM will begin tracking jobs that use GPUs after the usage record is added.

bash-4.2$ mam-list-usagerecords --show=Type,Instance,Charge,User,Gpus
Type Instance Charge User Gpus
---- -------- ------ ------ ----
Job Moab.1 0 jbooth
Job Moab.2 0 jbooth
Job Moab.3 0 jbooth
Job Moab.4 0 jbooth
Job Moab.5 0 jbooth 1
Job Moab.6 0 jbooth 1

Tags: gpu usage
Last update:
2016-05-16 18:02
Author:
Jason Booth
Revision:
1.0
Average rating:0 (0 Votes)

You cannot comment on this entry

Chuck Norris has counted to infinity. Twice.

Records in this category

Tags