Updates are not seen inside transaction created using beginTransaction() (until commit)
#3295
Comments
|
I am not 100% sure but here is my thinking (see comments) fun `using createTransaction()`() {
// Here you create a new entity and save it to db using auto-commit mode
val object1 = EaObject(id = 111L, name = "object1").also { database.save(it) }
// Here you create a transaction that is not managed by ebean. It is meant to be
// passed around manually and ebean has many methods that have a transaction parameter.
// The code in your use block does not use these methods and thus everything you do
// below is still done in auto-commit mode.
// So you get a SELECT, an UPDATE and a SELECT again.
// I think if you would have used the Ebean methods that take a transaction the test
// would fail as well, since object1 has never been loaded within the transaction and
// the update will become a stateless update.
// See comments in your other test below.
database.createTransaction().use {
printVisibly("before update", objectName(111L)) // "object1" as expected
object1.name = "updated"
database.update(object1)
printVisibly("before commit", objectName(111L)) // "updated" as expected
}
}
fun `using beginTransaction()`() {
// this creates a new entity and saves it using auto-commit mode
val object1 = EaObject(id = 111L, name = "object1").also { database.save(it) }
// here you start a transaction and each transaction has its own persistence context / cache
database.beginTransaction().use {
// you load an entity from database by id and output its name.
// The entity you load here is not the same instance as object1. It is a new instance
// and that new instance is stored in the persistence context / cache of this
// transaction.
printVisibly("before update", objectName(111L)) // "object1" as expected
// Here you modify the entity created and saved before the transaction has started.
// This entity instance has never been loaded during the existence of the transaction
// and thus is not known to persistence context / cache.
// Because it has an ID, ebean will do a stateless update in the DB, which means
// it will just generate an update query by using object1 as a template
object1.name = "updated"
database.update(object1)
// I think this causes the stateless update query to commit to db
it.commitAndContinue() // this doesn't help
// Here you load again an entity by the same id as above. The persistence context
// already knows this entity because you have already loaded it once. So no select
// query is done at all as seen in your console output.
// In the code above you changed object1 which was never loaded by this transaction.
printVisibly("before commit", objectName(111L)) // "object1", but should be "updated"
// This commits the tx created by beginTransaction which basically also
// terminates/clears the persistence context / cache
it.commit() // this does, but it shouldn't be needed
// Now you are in auto commit mode again and a new select is done.
printVisibly("after commit", objectName(111L)) // "updated" as expected
}
} |
|
hi @jnehlmeier and thanks for your reply. this all makes perfects sense (and I didn't know that database.beginTransaction().use { transaction ->
val object1 = EaObject(id = 111L, name = "object1").also { database.save(it) }
printVisibly("before update", objectName(111L)) // "object1" as expected
object1.name = "updated"
database.update(object1)
transaction.commitAndContinue() // this doesn't help
printVisibly("before commit", objectName(111L)) // "object1", but should be "updated"
transaction.commit() // this does, but it shouldn't be needed
printVisibly("after commit", objectName(111L)) // "updated" as expected
}the problem still persists. Actually I first noticed this problem in a test that is completely run inside a transaction - I have a helper base class for this which does |
|
By the way, as per your comment
I tried to rewrite the first test using explicit transaction passing: @Test
fun `using createTransaction()`() {
database.createTransaction().use { transaction ->
val object1 = EaObject(id = 111L, name = "object1").also { database.save(it, transaction) }
printVisibly("before update", objectName(111L, transaction)) // "object1" as expected
object1.name = "updated"
database.update(object1, transaction)
transaction.commitAndContinue() // this doesn't help
printVisibly("before commit", objectName(111L, transaction)) // "object1", but should be "updated"
transaction.commit() // this does, but it shouldn't be needed
// BEWARE: this shouldn't use `transaction` as it is already committed and deactivated
printVisibly("after commit", objectName(111L, transaction)) // "updated" as expected
}
}
@Suppress("SameParameterValue")
private fun objectName(objectId: Long, transaction: Transaction? = null) =
if (transaction == null) database.find(EaObject::class.java, objectId)!!.name
else database.find(EaObject::class.java, objectId, transaction)!!.nameand made two observations:
but what actually happens is that the call doesn't make any new queryies (see output below) and still prints the old value, which is actually worse than the case with 11:08:52.117 [Test worker] DEBUG io.ebean.SQL - txn[1001] insert into T_OBJECT (OBJECT_ID, NAME, ALIAS, STEREOTYPE, NOTE, TPOS, EA_GUID, OBJECT_TYPE, PARENTID, PACKAGE_ID, BACKCOLOR, PDATA1, CLASSIFIER, STATUS, Version, CREATEDDATE, MODIFIEDDATE, AUTHOR) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?); -- bind(111,object1,null,null,null,null,{278ebe3c-5512-4abe-b7c7-eb1147c5dd89},null,null,0,-1,null,0,null,1.0,Sun Dec 17 11:08:52 CET 2023,Sun Dec 17 11:08:52 CET 2023,test)
11:08:52.118 [Test worker] DEBUG io.ebean.SQL - txn[1001] select t0.OBJECT_ID, t0.NAME, t0.ALIAS, t0.STEREOTYPE, t0.NOTE, t0.TPOS, t0.EA_GUID, t0.OBJECT_TYPE, t0.PARENTID, t0.PACKAGE_ID, t0.BACKCOLOR, t0.PDATA1, t0.CLASSIFIER, t0.STATUS, t0.Version, (case when t0.object_type = 'Package' then 1 else 2 end), t0.CREATEDDATE, t0.MODIFIEDDATE, t0.AUTHOR from T_OBJECT t0 where t0.OBJECT_ID = ?; --bind(111, ) --micros(530)
========================================
>>> before update: object1 <<<
========================================
11:08:52.119 [Test worker] DEBUG io.ebean.SQL - txn[1001] update T_OBJECT set NAME=?, MODIFIEDDATE=?, AUTHOR=? where OBJECT_ID=?; -- bind(updated,Sun Dec 17 11:08:52 CET 2023,test,111)
11:08:52.121 [Test worker] DEBUG io.ebean.TXN - txn[1001] Commit
========================================
>>> before commit: object1 <<<
========================================
11:08:52.122 [Test worker] DEBUG io.ebean.TXN - txn[1001] Commit
========================================
>>> after commit: object1 <<<
========================================
11:08:52.123 [Test worker] DEBUG io.ebean.SQL - txn[1002] select t0.OBJECT_ID from T_OBJECT t0; --bind() --micros(347)
11:08:52.124 [Test worker] DEBUG io.ebean.SQL - txn[1002] delete from T_OBJECTPROPERTIES where (OBJECT_ID) in (?); -- bind(Array[1]={111}) rows(0)
11:08:52.125 [Test worker] DEBUG io.ebean.SQL - txn[1002] select t0.CONNECTOR_ID from T_CONNECTOR t0 where (START_OBJECT_ID) in (?); --bind(Array[1]={111}) --micros(347)
11:08:52.126 [Test worker] DEBUG io.ebean.SQL - txn[1002] select t0.CONNECTOR_ID from T_CONNECTOR t0 where (END_OBJECT_ID) in (?); --bind(Array[1]={111}) --micros(368)
11:08:52.127 [Test worker] DEBUG io.ebean.SQL - txn[1002] delete from T_OBJECT where OBJECT_ID in (?); -- bind(Array[1]={111}) rows(1)
11:08:52.127 [Test worker] DEBUG io.ebean.TXN - txn[1002] Commit |
|
Refer to https://ebean.io/docs/query/option#persistenceContextScope ... use Use: |
|
hi @rbygrave and thanks, this is nice as a work-around, but from this reply, should I understand that this is an expected behavior? I would think that this shouldn't be needed by default, that results of updates ran inside the same transaction should just be seen as usual. Or is there something special about this case so that this is needed? |
|
Probably first is to understand that the persistence context is there for "consistent object graph building" which means that when a query is executed it uses the persistence context to "de-duplicate" instances that have already been loaded / "de-duplicate on ToMany and ToOne relationships". By default the persistence context is scoped to the transaction ... just like JPA. (known as "transaction scoped persistence context"). Also the "persistence context" is also and perhaps more commonly known as the L1 cache but note that it's primary job is for "consistent object graph building" and imo it's just more that this also works like a cache - hence L1 cache / Level 1 cache. With Ebean we have the option of changing this on a per-query basis or indeed globally. On a per-query basis we use There are folks that also change this globally for an application via |
If I read it correctly there are 2 instances used here for the logical same row (the first printVisibly loads the second instance and puts that into the persistence context but then the code does not use THAT instance to do the update and instead uses the first instance that was used for the insert). So the instance that is first queried [and loaded into the transaction scoped persistence context] is not subsequently used for the update etc [so its held in memory with id 111L with always that old original value]. |
|
Ooh, that's pretty subtle. Okay, I hope I understand now, but please just let me confirm, so that we're on the same page.
did I get this right? What I still don't understand is, how do stateless updates fit into this. I also tried to replace the name set and QEaObject(database).where().id.eq(111L).asUpdate().set(QEaObject._alias.name, "updated").update()which didn't work either, but does that mean that all cached objects changed by stateless updates (or any updates that don't use the currently-cached object) should be considered stale afterwards and have to be reloaded with this In the bigger picture, although this still seems to me like a considerable overhead to keep in mind all the context interactions and their invalidation consequences, I understand there are advantages for the persistence context behaving this way and I guess with a bit of practice one can get used to this and not make too many mistakes like this. What seems most scary about this to me is the case (which is definitely not rare at least in our project) in which the code that does the update (or another action that will invalidate some previously loaded instances) and the code affected by it may be in completely different parts of the codebase and not know about each other's internal behavior. I can easily imagine a piece of code which 1) starts a transaction loads some records, does something with them, then 2) calls some function which may do a stateless update or find the same records again and update them (and the calling side doesn't and shouldn't know about that, because of separation of concerns, boundaries, etc.), thus invalidating the cache, then returns, after which 3) the calling side wants to do some more actions, but will get stale objects, similarly to my example. What would be your suggestions to avoid problems like this. I don't think it's practical to add the Lastly, I found some documentation on the L2 cache on ebean website, but except for this I can't find mentions of the L1 cache (or the "caching" behavior of the persistence context in general, only the graph-building one concerning related object fetching) and its differences from L2. Is there some page that I missed? As always, thank you so much for both your replies and the great work you guys do maintaining this library, I do get confused by some of these internals from time to time, but barring these moments, I'm so glad we decided to make this switch and it was one of the best decisions we've made. |
|
Oh, I just got to this again and I found out, that the definition of |
|
Hmm, I'm going to try another angle / explanation so lets see how this goes. ORM queries are "graph building" - building a graph of connected objects [from sql result sets] Example:
Q: How many instances of Customer 42 are held in-memory by the application? Its of the job of the persistence context to de-dup [when reading the resultSet and building objects into the graph] and ensure there is only 1 instance of Customer 42 [and by default this is scoped to the transaction]. Now, conversely if the answer is that there are multiple instances that represent Customer 42 ... it can get interesting if the application looks to mutate Customer 42 and save it etc. So generally we prefer to build consistent object graphs where we know there is only 1 instance of Customer 42 held in-memory [by the transaction]. To a large extent imo this is the main definition and requirement of an ORM - to build consistent graphs. The "scope" of this is also known as "Unit of Work" or "Session" - we work with a graph of connected entity beans on a "unit of work" basis which by default is exactly the same scope of a transaction. "Transaction scoped Unit of Work" ... and in that unit of work we expect 1 instance of Customer 42.
This is not "refreshing" the persistence context but instead creating a wholly independent graph of objects. It is a wholly independent persistence context that contains all the objects built by the query PLUS any objects built subsequently via lazy loading on any of those objects.
This is a feature that is expected to be used to avoid loading the bean at all. That is, if we are loading the bean then we really expect code to mutate the bean and save() or update() the bean and NOT use stateless update. Stateless update is about performing an update without loading the bean / avoid the select to fetch the bean in the first place. It is not expected that we load the bean but then use stateless update to update the row that the bean represents.
Bulk updates are good and we should use them BUT ebean doesn't know what rows are updated - there is no mechanism for ebean to determine which rows were modified and then remove the associated entries from the persistence context. If we build entity beans before a bulk update - yes those beans could be "stale" [because the related rows were modified by the bulk sql update statement]. Currently, the expectation would be to refresh() them or not use them or use My gut is telling me that the confusion around this is when we mix using entity beans with bulk updates &/or stateless updates.
This does not clear() the persistence context. What this does is flush any batched insert, update, delete statements [that have been batched up in order to use JDBC batch]. Questions that you might have asked probably ... Q: When a stateless update is executed, does it clear that entry from the persistence context? Q: But in the last example ... huh? Q: So a bean that is inserted doesn't go into the persistence context? Q: So there is no PersistenceContext.clear() method ... that I might want to use after say a bulk update is executed? |
|
Wow, thank you for such a detailed explanation.
Yeah, this I understood
Ok, here I got my terminology completely wrong, I basically thought they're the same thing. What I meant here were what you call "bulk" updates. Now that I actually know what stateless updates are, I'll try to use them more, because they should fix at least some of my problems with stale objects in the persistence context.
Yeah, I just tried a few mins ago and saw that this does not help. I also tried to set
What I meant here was the alternative (commented out) variant of the test that used "bulk" update.
:( I still kinda think we do need sth like this in our case. The problem, as I tried to explain with completely wrong terminology, is like this class ClassOne(
private val database: Database,
private val classTwo: ClassTwo,
) {
fun doSomeBigChange() {
database.beginTransaction().use { tr ->
// load a bunch of objects
// do some stuff with them
// - ClassTwo is a complete black box from ClassOne's perspective, the only thing ClassOne
// knows about it is that it needs to be called at this point, inside the same transaction
// - this might do bulk updates that might affect some of the previously loaded objects
// or it might find them again (and then do "normal" updates or maybe deletes),
// which will replace "our" previously loaded objects in the context
// (if I understood correctly what happens in my case of printVisibly())
classTwo.doSomeMoreStuff()
// - so to be sure, I'd like to clear the context here
// do some more stuff with the previously loaded objects
tr.commit()
}
}
}Actually, another option that I think would solve our problem, would be to be able to specify the |
Yes. Another alternative that I need to think through is to have the behaviour that the persistence context is cleared [automatically by ebean] after any bulk update or bulk delete statement is executed. e.g when there is Then the persistence context could be automatically cleared for note: the persistence context is a map of maps first keyed by entity class type and then keyed by When a |
Well, that seems ideal to me, it would definitely fix our problems and should prevent other people getting confused when making similar kind of mistakes. But if that's hard to do, being able to specify the scope for each transaction seems to me like an easy win. Thank you again, you've been as always extremely helpful. |
Using SqlUpdate or an ORM Update query clear the appropriate part of the PersistenceContext. The effect is that ORM queries executed after a bulk update will effectively load a fresh copy of the data from the database and will not reuse an instance from the persistence context if the bean in question had already been loaded.
…ontext Also make improvements to the test asserts in TestPersistenceContextQueryScope
…-update #3295 Clear PersistenceContext on execution of bulk updates or deletes
…d mixed queries 14.0.0 via #3295 introduced this bug. #3295 introduced a behaviour where bulk updates clear the persistence context. Now the lazy loading of a BeanCollection works with an assumption that the "parent bean" is in the persistence context (and this assumption is broken with that change for this test case). That is, the bulk update is clearing the persistence context - removing the parent bean BEFORE the lazy loading is invoked ... and that doesn't then work because the parent bean isn't in the persistence context. This fix means that when lazy loading many's, the parent beans are putIfAbsent into the persistence context.
…d mixed queries 14.0.0 via #3295 introduced this bug. #3295 introduced a behaviour where bulk updates clear the persistence context. Now the lazy loading of a BeanCollection works with an assumption that the "parent bean" is in the persistence context (and this assumption is broken with that change for this test case). That is, the bulk update is clearing the persistence context - removing the parent bean BEFORE the lazy loading is invoked ... and that doesn't then work because the parent bean isn't in the persistence context. This fix means that when lazy loading many's, the parent beans are putIfAbsent into the persistence context.
Expected behavior
When I run update query (using
Database.update()orQuery.update()) and then read the value from db inside the same transaction created usingbeginTransaction()before callingTransaction.commit(), I would like to see the update value.Actual behavior
I get the old value.
Steps to reproduce
Context
Just updated to ebean version 13.25.1, but the behavior persists. Is this a bug, or am I doing sth wrong? Should I reproduce this in the minimal ebean test repository?
The text was updated successfully, but these errors were encountered: