Results were truncated -- use chunking to enable large messages.


Issue:  When doing a query in MAM that will return a large number of records you can run into an internal limit on the number of records that are returned.  This will usually happen if you're querying the usage records or transactions.  When you hit this limit you will get the following message: "Results were truncated -- use chunking to enable large messages"

 

Solution:  MAM has the ability to get the records from the database in chunks so it doesn't overwhelm the mam-server process.  To take advantage of this feature you need to verify that the following parameter is set in your mam-server.conf file:

response.chunksize = 100000

 

The chunking of the data only happens when you do the query with the mam-shell rather than the mam-list-usagerecords or mam-list-transactions commands.  You also have to ask for some kind of ordering of the data for the database to be able to return the data without duplicating records.  You can do have the flexibility of choosing which column you sort by.  Here is an example of a query you can do to get usage records for a year.

mam-shell UsageRecord Query CreationTime\>="2016-01-01" CreationTime\<"2017-01-01" Show:="\"Sort(Id),Instance,Charge,Class,Machine,Nodes,Processors\"" > /tmp/usagerecords.out

 

This sorts the records by the usage record id, which may not be desirable in all circumstances.  You could choose to put the "Sort" on the Instance attribute, which equates to the job id, or any other sequential attribute.  You can see a list of the attributes you can specify in the 'Show' section by running this:

mam-shell Attribute Query Object==UsageRecord

 

You can do similar queries for transactions as well.  To see the attributes you can specify to query the transactions you can run this:

mam-shell Attribute Query Object==Transaction

Tags: chunk, chunksize, goldsh, large, query, shell, sort, transaction, truncate, usagerecord
Last update:
2017-03-17 22:27
Author:
Ben Roberts
Revision:
1.0
Average rating:0 (0 Votes)

You cannot comment on this entry

Chuck Norris has counted to infinity. Twice.

Records in this category

Tags