Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support built-in bulk operations #92

Open
michaelklishin opened this issue Nov 14, 2014 · 15 comments
Open

Support built-in bulk operations #92

michaelklishin opened this issue Nov 14, 2014 · 15 comments
Assignees

Comments

@michaelklishin
Copy link
Owner

See http://docs.mongodb.org/manual/core/bulk-write-operations/ and https://groups.google.com/forum/#!forum/clojure-mongodb.

@vesse
Copy link

vesse commented Nov 19, 2014

Is there a workaround available meanwhile? Would need to run initializeUnorderedBulkOp, add operations to the returned bulk, and then execute it.

@michaelklishin
Copy link
Owner Author

Monger accepts DB instances, so you can always use Java interop and monger.conversion.

@alolis
Copy link

alolis commented Nov 19, 2014

@michaelklishin, I am using monger 1.8.0 with Mongo 2.6.5 and I tried to follow your work around until this is properly implemented but I am getting a BulkWriteException on duplicate key, although I shouldn't since I am using unordered writes.

(monger.core/connect! my-connection-map)
(monger.core/use-db! my-database-string)
(def db (monger.core/get-db))
(def collection (.getCollection db "items"))
(def bulk-write (.initializeUnorderedBulkOperation collection))

(.insert bulk-write (monger.conversion/to-db-object {:_id "a" :foo "bar"}))
(.insert bulk-write (monger.conversion/to-db-object {:_id "a" :foo "bar"})) ;; Dup key 
(.insert bulk-write (monger.conversion/to-db-object {:_id "b" :foo "bar"}))

(.execute bulk-write) ;; BulkWriteException here. First document gets inserted in database, third one does not, although it should.

I did have a look inside monger.core source but I couldn't find a problem. I also tried the bulk operation via the Mongo shell and it worked as expected so I am thinking it might be something in the mongo java driver (maybe just an update will do?) or in the way you are initializing the java driver (although I didn't see anything weird). You will probably have a better idea than I do at the moment.

@michaelklishin
Copy link
Owner Author

Why do you expect bulk writes with duplicate keys to work? I'm not sure how they can, regardless of the client or database.

@alolis
Copy link

alolis commented Nov 19, 2014

The documentation states that this is possible (and indeed it is because I tried it in the mongo shell)

http://docs.mongodb.org/manual/reference/method/db.collection.initializeUnorderedBulkOp/

Error Handling
If an error occurs during the processing of one of the write operations, MongoDB will continue to process remaining write operations in the list.

@vesse
Copy link

vesse commented Nov 20, 2014

Here's a simple upsert example just in case someone's interested (since it didn't go as I expected from the shell examples):

(let [col  (.getCollection db "collection_name")
      bulk (.initializeUnorderedBulkOperation col)]
  (.update (.upsert (.find bulk (monger.conversion/to-db-object {:_id (:_id object)})))
           (monger.conversion/to-db-object {$set (dissoc object :_id)}))
  (.execute bulk))

@alolis
Copy link

alolis commented Nov 20, 2014

Well, it seems that the problem I was describing is in the mongo-java-driver version (v2.12.1) which monger 1.8.0 is using.

https://jira.mongodb.org/browse/JAVA-1541

EDIT: I confirm that monger v1.8.0 with mongo-java-driver v2.13.0-rc0 works as expected.

@michaelklishin
Copy link
Owner Author

@alolis please submit a PR that upgrades the dependency.

@alolis
Copy link

alolis commented Nov 26, 2014

@michaelklishin , sorry for the delay. Yeah sure, I can do it first thing tomorrow morning but please verify the commit in order to be 100% sure.

Is this the one?
https://github.com/michaelklishin/monger/tree/v1.8.0

@sql1
Copy link

sql1 commented Apr 3, 2016

I have also good in using Mongodb and I like all functionality. Here I would like to mention a link that Will help you. All the things is discussed properly. Feel free to visit

@robert-m-johnson
Copy link

I'd be interested in having a look at this. Did you have anything in mind for the API?

@michaelklishin
Copy link
Owner Author

@robert-m-johnson thank you. The API can vary from passing in a sequence/collection to a full-blown mini-DSL à la what Neocons and Elastisch have. I'm open to suggestions and would need to take a look at what the modern Java driver provides.

@robert-m-johnson
Copy link

robert-m-johnson commented Jun 22, 2016

The newer versions of the driver have you build up a sequence of WriteModel objects, like the following example taken from http://stackoverflow.com/questions/31470702/

MongoCollection<Document> collection = db.getCollection("sample");

List<WriteModel<Document>> updates = Arrays.<WriteModel<Document>>asList(
    new UpdateOneModel<Document>(
        new Document(),                   // find part
        new Document("$set",1),           // update part
        new UpdateOptions().upsert(true)  // options like upsert
    )
);

BulkWriteResult bulkWriteResult = collection.bulkWrite(updates);

It makes the most sense to me to go with passing in a sequence since it would both be idiomatic Clojure and similar to the Java API, but I guess this could be augmented with a mini-DSL if that wouldn't be too confusing.

My questions are

  1. How do we represent the type of the type of each individual write operation?
  2. How do we indicate whether the bulk write is ordered or not?

My initial thoughts on these are to simply use a symbol to represent a type, and a function argument could indicate ordered or unordered. A vector would represent each write operation. So the Monger version of the above Java example could look something like

(mc/bulk-write db "sample" [[:update-one {} {"$set" 1} {:upsert true}]] :ordered)

@mcferren
Copy link

mcferren commented Oct 3, 2017

Would be very grateful for this mc/bulk-write feature

@viebel
Copy link

viebel commented Sep 1, 2019

Any progress on adding bulk-write to monger?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants