i have a somewhat unique requirement in that i need to perform some batch processing within a process that also has normal business logic. ive searched the forums countless times for other scenarios using the StatelessSession but i dont see anything particularly helpful..thus im asking in a new thread. i apologize in advance for the long comment but im trying to explain the scenario properly.

so basically the process flow is as follows:

1. start of process
2. do some work
3. do a lot of batch inserts/updates/deletes
4. do some more work
5. end of process

in step 2 i load and persist some entities with relationships and collections.
in step 3 i am persisting millions of POJO value objects.
in step 4 i am working again with entities with relationships and collections

i want the whole process to execute atomically within a transaction (i.e. either all steps 1-5 succeed or they fail)

in order to perform steps 2 and 4 i want to use a regular session and for step 3 i would like to use a StatelessSession

what im trying to do is piggyback a StatelessSession on a Session using the following technique

unfortunately, this doesnt seem to work. the only way i can get it to work is if i invoke beginTransaction() on the statelessSession before step 3 and then call commit() on the statelessSession after step 3. essentially beginning and committing two transactions on the same jdbc connection. im confused as to why this is needed? when i wrap step 3 in its own transaction using the StatelessSession interface, i get the debug output listed below.....its as if two seperate JDBC connections are being used. is a nested subtransaction actually occurring (i.e. where a committed inner transaction is only permanent if the outer transaction commits)? for me a single flat transaction would be best (which is why i tried to piggyback the connection).

is there some way to properly accomplish what im trying to do in a fully atomic way?


in case anyone is wondering, the process is executed by a single thread, im using hibernate 3, mysql innodb and the following properties:
hibernate.jdbc.batch_size=80
hibernate.cache.use_second_level_cache=false
hibernate.transaction.factory_class=org.hibernate.transaction.JDBCTransactionFactory
hibernate.current_session_context_class=thread
hibernate.connection.autocommit=false

thanks in advance

does anyone have any clue why this isnt working?

from what ive gathered, in mysql if you begin a new transaction on a connection that already has a running transaction then it automatically commits the changes up until the point where the 2nd tx begins. so in other words, attempting to call begin transaction on the StatelessSession that was instantiated using the Session's connection is a bad idea.

what i dont understand is why the changes made by the StatelessSession on the piggybacked session arent being executed/commited.

i thought the connection retrieved by Session.connection() is maybe some sort of "fake" object. so i checked out the source and it is indeed a proxy wrapper connection (BorrowedConnectionProxy) but that seems to wrap the real connection.

maybe someone could better explain what one needs to do to properly use a connection hijacked from a Session...?

ok now im just lost.

if i hijack a connection from a Session object and then perform a manual insert using jdbc and the raw connection, when i commit the Session/transaction (using the Session api), then the changes are persisted fine.

what is so special about creating a StatelessSession using an existing connection? is this a bug?

10 bucks via paypal to anyone that can pull this off?

ok i finally figured this out. there appears to be a bug in the way StatelessSession behaves.

i finally managed to get this to work by setting the JDBC batch size to 1. originally i was using a batch size of 80.

here is how i discovered the problem.



debugging through the statelessSession.insert() call, i realized that if the number of records inserted is less than the JDBC batch size, then the stateless session never actually executes the batch. i think the flaw in the behaviour is due to the fact that Hibernate assumes one will use their Transaction API to initiate and commit the transaction. in that case, the transaction.commit() invocation ends up pushing any "leftover" inserts to the DB. i say "leftover" because this will usually be the case at the very end of a large batch insert when:
( totalNumberOfRecordsToInsert mod jdbcBatchSize ) != 0

by setting the JDBC batch size to 1, each call to statelessSession.insert() actually results in an insert in the DB. of course in this case we lose the benefits of batching entirely.

unfortunately there is no way to force the stateless session to push those "leftover" inserts to the DB as the normal Session offers through the flush() method.

can a hibernate expert perhaps comment on whether this behaviour is intended? at this time the only solution i can see is to cast the StatelessSession to a SessionImplementor, retrieve the batcher via getBatcher() and call executeBatch() when processing the final group of records.

Thanks for the update.

Although this thread would seem like you're having a conversation with yourself, I've found it interesting. :)

Hi Bennini,

Did you find a proper solution for this flush issue? if not can you post your hacky flush code please?


Regards,
Ced.

I've encountered the same problem and I've created a jira entry about this : http://opensource.atlassian.com/project ... e/HHH-4042

Hi Bennini,

I refered to your other threads as well regarding bulk insert. As you rightly said its better use stateless session with hibernate when object is simple and without any collection.

But thats pretty useless to me. what are the faster options that one has when object contains collection maps?

Thanks

A session with flush(), jdbc batch_size > 1, disabled second level cache and disabled interceptors... to start with.

Question is, if we are committing 100 records in one commit using the same statelessSession/session, shouldn't it be intelligent enough to automatically commit 100 records in jdbc batch, why even specify the batch_size somewhere in the configuration, may be they should expose it as a runtime rather than a global constant.

Committing 1000 records using statelessSession with a batch size of 500, will commit 500 in a batch and 500 again rather than 500 separate inserts?

http://opensource.atlassian.com/project ... e/HHH-5935

execute(new StatelessHibernateCallback<T>() {
public T doInStatelessSession(StatelessSession statelessSession) {
for(Entity entity : entitiesBatch)
{
statelessSession.insert(entity);
}
return null;
}
});

inserts entity by entity which is completely ABSURD. Should'nt it insert all of them or none, rather it is inserting one by one by default. I have been using this to insert millions of records assuming it is batching, which is a shame.