[This is a lightly edited version of a note I sent to Pieter on 11/7/2006. We decided to put our email exchange up on this web site]
Hi Pieter,
We took a quick look at your benchmark, and I think that there is some serious problem with your Hibernate code. Are you executing a query for each of the 1000 items you are returning? Also, are you using something less than the Small 007 database size? This is not good. If the answer to either of these questions is "true", then I think that you should take your article off ODBMS until we can fix it up some.
My student, Ali Ibrahim, will also contact you. We are willing to share our 007 implementation if you are going to go open source. I don't think that there is any problem with it, since we wrote it ourselves. We have an implementation that is more tuned for 007. We did see some traversals (T1) that took up to 248 milliseconds in Hibernate (22 seconds non-optimized). But taking 300 seconds to do find 1000 items doesn’t make sense.
Overall I am very interested in this topic. However, the bigger issue is that 007 is, unfortunately, not a very good benchmark for real-world applications. It was based on CAD applications, which are not the most common things people write.
William
William Cook