Aggregate pipleline Error: getMore: cursor didn't exist on server, possible restart or timeout

mongodb aggregate
allowdiskuse mongodb
mongodb aggregate explain
mongodb aggregation cursor
mongodb cursor timeout
mongodb query timeout
mongodb timeout
mongodb aggregate timeout

I am facing this issue with mongodb. My code is something like this

for(loop) {
    var cursorQuery = db.beacon_0000.aggregate([
        {
            $match: {
                ...
            }
        },
        {
            $project: {
                ...
            }
        },
        {
            $group: {
                ...
            }
        },
        {
            $sort: {
                ...
            }    
        }
    ], {allowDiskUse: true} );
     ...
    while(cursorQuery.hasNext()) {
        var cursor = cursorQuery.next();
        ...
    }
}

I run the above query via command and mongo shell as

$ mongo dbName file.js

After a while I get the cursor didn't exist on server error at line cursorQuery.hasNext().

In find query if I get this error, I can resolve by adding addOption(DBQuery.Option.noTimeout) However this option does not seem to be available with aggregate

Please let me know how can I resolve or workaround this issue.

Just to provide additional update: When say I use

var cursor = db.collection..aggregate([ ...], {allowDiskUse: true} ).addOption(DBQuery.Option.noTimeout)

I get this error E QUERY TypeError: Object # has no method 'addOption'

However when say I use

var cursor = db.collection..find({...}, {...}).addOption(DBQuery.Option.noTimeout)

It works fine.

Checking the aggregate doc https://docs.mongodb.com/v3.0/reference/method/db.collection.aggregate/ It says: Returns:A cursor to the documents produced by the final stage of the aggregation pipeline operation

And then checking cursor doc https://docs.mongodb.com/v3.0/reference/method/cursor.addOption/#cursor.addOption

There is no suggestion that aggregate cursor is different from find cursor and former does not support DBQuery.Option.noTimeout.

So is there a bug at mongodb for this. Any way to fix it or have a workaround.

Note mongodb version is 3.0

[SERVER-13358] long aggregation queries get a cursor timeout error, "ok" : 0. } : aggregate failed. Error: command failed: {. "errmsg" : "exception: getMore: cursor didn't exist on server, possible restart or timeout?",. Looking at mongod logs, there are no lines mentioning "killing cursor" nor "aggregate". This is quite problematic since it makes it unusable for long aggregations. I will try to disable cursor timeout in the query to see if it makes a difference.

You have sort of answered this yourself.


Adding the option addOption(DBQuery.Option.noTimeout) will indeed fix the issue when using find because it stops the cursor from timing out and therefore it will exist when you try .hasNext()

However the cursor for aggregation does not have that option so you can't stop it from timing out unfortunately.

Aggregation Freamwork, The Error I'm getting is: "cursor id xxxxxxx didn't exist on server". the driver to issue getmore requests more frequently during cursor iteration. of a question of the index (if any) that is used to start your aggregation pipeline. getMore: long: The cursor id. collection: string: The name of the collection over which the cursor is operating. batchSize: positive integer: Optional. The number of documents to return in the batch. maxTimeMS: non-negative integer: Optional. Specifies a time limit in milliseconds for processing operations on a cursor.

you can actually use : maxTimeMS as it described in documentation:

Optional. Specifies a time limit in milliseconds for processing operations on a cursor. If you do not specify a value for maxTimeMS, operations will not time out. A value of 0 explicitly specifies the default unbounded behavior.

There is an option in mongodb documentation that you can set it no a non-negative number for the time that you want your cursor to be alive. you can see the more detail on : documentation

Aggregation Pipeline Limits, aggregate() method returns a cursor. Memory Restrictions¶. Pipeline stages have a limit of 100 MiB (100 * 1024 * 1024 bytes) of RAM. If  The .forEach() has no place here since an "aggregation cursor" is actually just a node "stream" interface, and therefore only the "data" event is actually doing anything. . However if that is how your code is actually set up then this would suggest that you have a MongoDB 2.4 or lower server instance, which of course does not support a "cursor/stream" respo

db.collection.aggregate(), See Return Information on Aggregation Pipeline Operation for an example. Not available in If an error occurs, the aggregate() helper throws an exception. For cursors created inside a session, you cannot call getMore outside the session​. Hello, I' am returning an aggregation on meteor publish and i receiving the following error: Exception from sub TySCANsTYoXHQCKyq Error: Publish function returned an array of non-Cursors How do i enable that the aggregate returns an arra

MongoCollection::aggregateCursor - Manual, (PECL mongo >=1.5.0). MongoCollection::aggregateCursor — Execute an aggregation pipeline command and retrieve results through a cursor aggregateCursor with mongoDB 3.6 can cause error: "the command cursor did not return a correctly structured response" to fix it on 64 bit systems, temporarily turn on: In the mongo shell, if the cursor returned from the db.collection.aggregate() is not assigned to a variable using the var keyword, then the mongo shell automatically iterates the cursor up to 20 times. See Iterate a Cursor in the mongo Shell for handling cursors in the mongo shell.

Azure Cosmos DB's API for MongoDB (3.2 version) supported , delete; find; findAndModify; getLastError; getMore; insert; update Cosmos DB supports aggregation pipeline for MongoDB 3.2 in public preview. cursor.sort(), cursor.sort({ "Elevation": -1 }), Documents without sort key do not get Unique indexes ensure that a specific field doesn't have duplicate values  0 Aggregate pipleline Error: getMore: cursor didn't exist on server, possible restart or timeout Aug 1 '16 0 GTM history change pushing to dataLayer but not triggering collect call to www.google-analytics.com Sep 3 '19

Comments
  • cursorTimeoutMillis changes the default timeout period server-wide. If this is not possible, for the time being the solution is recoding aggregates so they run faster.
  • So what are my options here. Would setting a batchSize to a large number help here. say db.collection.aggregate().batchSize(10000) Anything else you can suggest to prevent cursor from timing out for aggregate pipelines.
  • Someone that makes the aggregation take less time is what you want. Considering you can't prevent the cursor from timing out you need to use it within the default timeout length. It does look like they are trying to fix this though. See: jira.mongodb.org/browse/SERVER-6036