Product News from the Core Team

Storing PowerShell objects into a Db4o database

Adriano Verona — Tue, 30 Dec 2008 18:59:00 GMT

Any (Windows user) CLI enthusiast should already be aware about PowerShell, MS latest bet in this area.

I must confess that even being aware about it I haven't had the time to use it other than for really simple tests. You know,

most of the time we need to get something done instead of learning a new set of shell commands, syntax, philosophy, etc.

But this changed recently due to some cool guys asking about how they could use Db4o to store PowerShell objects.Now

I have the motivation (and some allocated time) to pursue this goal, cool!

I do suggest that you give PowerShell a serious try (this way you'll gain some familiarity with it) before you continue to read this post.

Among other places, you can find a lot of information regarding PowerShell here, here and here (for programing purposes). I am not a expert in the subject, so I am going to explain only the basics to get the code in this post to work.

Have you gotten yourself comfortable with PowerShell already? Ok, grab the source code for this post and have fun :) (svn co https://source.db4o.com/db4o/trunk/sandbox/Adriano/Blogs/PowerShell/)

My first step was to download PowerShell 2 (CTP 3) and install it. Then I just took a sample from the powershell community, compiled

and installed the new cmdlet.

Well, "just" is an euphemism of course. It took me some time to understand what was required, so, in order to make it simple to you, I'll give a short explanation about what's required:

In order to compile the code you need to set references to PowerShell assemblies located under "%ProgramFiles%\Reference Assemblies\Microsoft\WindowsPowerShell\v1.0" (or, in PowerShell syntax: "$env:ProgramFiles\Reference Assemblies\Microsoft\WindowsPowerShell\v1.0")
Register your cmdlet (i.e, make PowerShell aware of our new cmdlet). In a "PowerShell" type (this requires administrator privileges):

PS C:\> $env:windir\Microsoft.Net\Framework\V2.0.50727\InstallUtil.exe PATH-TO-YOUR-CMDLET-DLL<enter>
Finally, add your cmdlet to the current shell (take a look in export-console cmdlet if you don't want to repeat this everytime you open a shell):

PS C:\> Add-PSSnapin Db4o.PowerShellSnapin

Now we are ready to run our cmdlet.

PS C:\Users\adriano> select-db4o-object

Once I get past this step I just tried to store / query a simple POCO (defined in the same project) using Db4o and it just worked ®.

Since it worked flawlessly I started to dig into how to define objects dynamically inside PowerShell as this could pose problems for Db4o. After some research

I found the add-member cmdlet, which is used to dynamically add members to objects. So, the next thing I've tried was:

PS C:\> $obj = ""
PS C:\> $obj = $obj | add-member -membertype noteproperty Name Adriano -passthru
PS C:\> $obj = $obj | add-member -membertype noteproperty Country Brazil -passthru
PS C:\> $obj = $obj | add-member -membertype scriptmethod Print {$this.Name + ": " + $this.Country} -passthru

This creates a new object with properties Name and Country and a Print() method .

Then I've tried to persist this object into Db4o and got an exception back saying that some collection object could not be stored in Db4o because it had no usable public constructors (a constructor that doesn't throw when called with default values (null for references, 0 for int, etc.). As explained here, some classes (for instance collections) require their constructor to be called in order to get a valid object

so this was a show stopper for me. Again, after some investigation I realized that I could write a custom typehandler and persist only PSObject's "interesting" bits. At least for the basic

scenario it worked fine:

PS C:\> add-db4o-object -DatabasePath test.odb -Item $obj
PS C:\> $t = select-db4o-object -DatabasePath test.odb
PS C:\> $t.Print()
Adriano: Brazil

(note that the database file [in this sample, test.odb] will always be created / looked up in the user home folder, i.e, $env:homepath) (of course you can change this behavior passing a full path to Db4o)

Basically this typehandler just acts as a translator, creating (and storing) an instance of PSObjectWrapper (a more Db4o friendly class) for each PSObject. When reading it just goes trough the inverse path: it reads a PSObjectWrapper from the database, instantiate a new PSObject and set its properties based on the just read object; finally it returns this new object as if it was read from the database.

What is still missing in this cmdlet is a way to query for objects in the database and to handle multiple references to an object correctly but I'll leave this as an exercise for the reader ;)

Another possible approach would be to emit a "strong" type for every PSObject in a object graph; this would make querying simpler and faster but would not fix the issue with multiple references for objects.

Let me know if you have ideas, suggestions or questions.

Best

Adriano

7.4 - Release Notes

Tetyana Loskutova — Sat, 29 Nov 2008 11:45:00 GMT

Now is the time for the newer and better Stable version - 7.4. As usual it is available in download center and recommended for product shipment. Note, that the production release did not change and is also 7.4.

The most prominent db4o features introduced since 6.4 release are:

And here is the list of Jira issues resolved since the previous Stable release (6.4):

COR-1451 - garbage collection breaks transparent persistence

COR-1441 - Memory Leak in ClientTransactionPool

COR-1439 - schema evolution not working correctly for client server mode.

COR-1412 - IStoredField.Get() returns null when attempting to retrieve a stored field that's been removed from class

COR-1396 - Updating from 6.4 to 7.4, some queries throw NullPointerExceptions

COR-1394 - Defrag does not check super classes for field indices when deciding on address mapping for an object slot

COR-1388 - Public API to query for number of objects of a given type

COR-1362 - Unable to get refactored field values due to wrong state in field meta data

COR-1353 - NRE with LINQ in Visual Basic

COR-1341 - Version handling for ClassAspects

COR-1337 - Custom Typehandler implementation as Class Aspects

COR-1335 - patch for adding maven deployment to build

COR-1334 - Deleting an object from one client and updating from another the object is lost

COR-1321 - [Java] Use ListTypeHandler for ArrayList

COR-1319 - NPE on ExtObjectContainer#storedClasses, if class is not present

COR-1318 - db4o 7.4 .NET 2 Development release download link is broken

COR-1313 - Remove flushFileBuffers(false) configuration setting.

COR-1311 - partial activated item when querying with a hashmap

COR-1301 - TracerBullet: Implement generic Map Typehandler

COR-1299 - Rename FirstClassHandler#readArrayHandler() and use context as parameter

COR-1298 - TypeHandler API: #cascadeActivation() should have an ActivationContext as parameter.

COR-1297 - Spike: Class derived from ArrayList using custom Typehandler

COR-1284 - Defragment against unknown classes runs into NPE on btree index lookup

COR-1282 - [decaf] functional test for resolving db4oj/db4ojdk1.2 conversion issues

COR-1281 - Running out of disk space causes unrecoverable data corruption

COR-1280 - Transparent Update Won't Work On Iterative Updates

COR-1279 - Delegate in untyped arrays cause ObjectNotStorableException

COR-1265 - ArrayHandler: Defragment old databases

COR-1264 - ArrayHandler: Updating from old databases to new format

COR-1262 - Pass configuration on to reflector on startup

COR-1261 - Move constructor handling into reflector

COR-1252 - Db4objects.Db4o.Tools-CF-2008 projects missing in Db4objects.Db4o.Tests-CF-2008.sln db4o-7.3.38.10577-net35.msi

COR-1248 - dll missing in db4o-7.3\bin\compact-3.5 for db4o-7.3.38.10577-net35.msi

COR-1238 - Trunk Version throws NullPointerException when using UniqueFieldValueConstraint and VersionNumbers

COR-1236 - Linq OrderBy operator fails when using TA enabled assemblies

COR-1234 - Port COR-1126 (explicit AND joins to implicit) to 6.4 branch

COR-1231 - Proper support for generic lists of nullable types

COR-1225 - For db4o-6.4.36.10458-net2.msi testcases not executed for Db4objects.Db4o.Tests-CF-2005.sln

COR-1223 - for db4o-6.4.36.10458-net2.msi few testcase fails for Db4objects.Db4o.Tests-2005.sln

COR-1219 - Linq Exception whne using TA enabled assemblies

COR-1218 - [Release 6.4] com.db4o.db4ounit.common.concurrency.query.ConcurrentQueryTestCase failed

COR-1217 - TracerBullet: Defragment for Java ArrayList using the new Typehandler

COR-1216 - TracerBullet: Deletion for Java ArrayList using the new Typehandler

COR-1214 - Db server closed when upgraded from 5.5 version to 6.4.14.8331 with btree based freespace management system

COR-1204 - Define release 7.3 for April

COR-1187 - Lazy queries returned by ObjectSet.iterator don't respect the db4o lock

COR-1186 - Fully remove second write indirection logic from Marshalling

COR-1179 - Linq 'orderby' throwing exceptions

COR-1174 - Mark factory methods using global configuration as deprecated

COR-1173 - OSGi class loading configuration fails when global configuration is used

COR-1170 - Version update for System.Collections.Generic.List

COR-1169 - Reading and Writing of System.Collections.Generic.List

COR-1167 - Send only the Db4oDatabase id intestead of the full signature when sending the UUID of deleted objects to the committed callback

COR-1161 - Files missing in \db4o-7.2\bin\net-3.5 of db4o-7.2.30.10211-net35.msi

COR-1160 - Files missing in \bin\compact-3.5 of db4o-7.2.30.10211-net35.msi

COR-1157 - Files missing in \bin\compact-2.0 of db4o-7.2.30.10211

COR-1152 - [Linq] Query caching yielding wrong results

COR-1147 - Solution doesn't compile

COR-1146 - db4o.chm documentation is not generated

COR-1145 - Issues in the .NET 35 LINQ build

COR-1141 - Produce plan for Generic Collections and Fast Collections

COR-1140 - Blog about optimized LINQ for db4o

COR-1128 - TracerBullet: Query Java ArrayList using the new Typehandler

COR-1126 - Optimize constraint graph to remove ANDs between constraints if not needed

COR-1123 - TracerBullet: Store and Retrieve Java ArrayList using the new Typehandler

COR-1117 - Callback API gets reference of a new ObjectContainer

COR-1114 - Adapt TA/TP .NET bytecode instrumentation to new Activatable#bind() semantics

COR-1113 - blog about RollbackStrategy with deactivate example

COR-1110 - Adapt TA/TP Java bytecode instrumentation to new Activatable#bind() semantics

COR-1109 - Benchmark unoptimized linq to optimized linq

COR-1108 - integrate linq on the build

COR-1107 - Transparent Persistence (manual implementation) does not work in VB.NET

COR-1105 - [TA] java.lang.IllegalStateException if unbind is called

COR-1097 - Corrupted class index in some delete/re-add scenarios

COR-1094 - ArrayList4.CopyTo doesn't activate

COR-1086 - SODA query returns wrong query result for .NET interfaces

COR-1085 - Implement databaseGrowthSize configuration to reduce fragmentation

COR-1083 - Code review/refactoring for legacy handler defrag

COR-1081 - [NQ] Native query does no longer work with current svn version: InvalidIteratorException

COR-1079 - Configuration interface for creating custom Typehandlers

COR-1074 - unbind Activatable objects when the ObjectContainer is closed

COR-1073 - 'closing' event to notify interested parties that an ObjectContainer is being closed

COR-1072 - Activatable objects should be bound to their Activator upon Store

COR-1071 - introduce TransparentUpdateSupport configuration option

COR-1070 - introduce RollbackStrategy interface for Transparent Update

COR-1066 - [TA] FileEnhancer doesn't work anymore

COR-1065 - Transparent Update for ArrayList4

COR-1064 - Transparent Update for ArrayMap4

COR-1063 - Clean up FreeSpaceManager interface

COR-1047 - Ability to reply to client messages using MessageRecipient

COR-1043 - [.JAVA] Transparent Update Instrumentation

COR-1042 - [.NET] Transparent Update Instrumentation

COR-1041 - Transparent Update Infrastructure

COR-1040 - Remove 'useIndexSystem' freespace system

COR-1038 - Improve ObjectContainer interface

COR-1033 - extract TypeHandler4 from ClassMetadata

COR-1027 - Generic Collection Design and Store/Retrieve "Tracer bullet"

COR-1025 - Defragment with StoredClassFilter corrupts the database

COR-1020 - Finalize Typehandler4 interface, implement compare(obj, obj)

COR-1019 - [TA] Setter Methods doesn't get instrumented for transparent activation

COR-1013 - [.NET CF 2.0] Make sure ArrayList4 ta tests passes on .NET CF 2.0

COR-997 - Pass all test cases on Android

COR-990 - [.NET] Build time NQ optimization

COR-989 - [.NET] TA instrumentation should skip non storable field types

COR-988 - Refactor Db4oAdmin

COR-987 - [.NET] Respecting the filter for the field's DeclaringType

COR-986 - .NET MSBuild task - implement options by-name, by-filter and by-attribute

COR-983 - [.NET] Instrumenting Db4objects.Db4o.Tests leads to peverify errors

COR-981 - Enhancing DTrace to be able to log exceptional conditions to files

COR-980 - [.NET] TA instrumentation for type hierarchies that contains generic base classes and generic fields

COR-975 - #deactivate and #refresh test cases fail

COR-961 - Simplest Possible Ant Script on Instrumentation

COR-960 - [TA] add TAFixtureConfiguration to cruise control build

COR-958 - New ExtObjectContainer#activate(obj) without parameters that respects activation settings

COR-956 - Diagnostic events for query engine are misleading when it fails to load Db4objects.Db4o.Tools.dll assembly

COR-955 - [.Net] Diagnostic events for query engine are misleading when it is unable to optimize a NQ.

COR-953 - [TA] Make sure TA mode respects cascading() event cancellation

COR-951 - Ensure correct Bloat handling of JDK5 class file format specifics

COR-948 - TA tests for second classes types

COR-947 - Finalize Typehandler4 interface: Refactor defrag() method

COR-946 - Finalize Typehandler4 interface: Refactor delete() method.

COR-945 - Finalize Typehandler4 interface, use design as spec'd in forums

COR-944 - Run db4o tests in TA mode

COR-941 - AssemblyInfo.cs in deployed sources shows the wrong version number

COR-940 - Integrate TA with RSSOwl

COR-939 - Updates to Map cause database to grow unexpectedly

COR-936 - TA/Java: Ensure Jars are accepted as instrumentation source

COR-932 - TA tests for multi-dimensional arrays

COR-931 - TA tests for String, Date, and Array

COR-929 - TA tests for different data types

COR-927 - Move TA code to core, move TA collection code to JDK5

COR-926 - TA/Java: Sample project for instrumentation usage

COR-910 - TACol: Concurrency tests for ArrayMap4

COR-909 - TACol: Concurrency tests for ArrayList4

COR-900 - activateDepth(0) doesn't work in c/s mode

COR-896 - [.NET] Review DateTime Handler Upgrade test case

COR-892 - Simplify Typehandler4, remove read(MarshallerFamily) and delete all old reading code

COR-888 - [.NET] Run TypeHandler upgrade tests against libraries providing the legacy API

COR-882 - Defrag causes data loss in 6.3

COR-879 - JAVA: Improve mechanism to specify classes to be persisted/instrumented

COR-878 - Simplify Typehandler4, remove readQuery(), readSubCandidate()

COR-874 - [.NET] NewTypehandler: Database file creation for upgrade tests

COR-873 - Simplify Typehandler4, remove classReflector(), getID(), linkLength()

COR-872 - java packages should include the full version name (e.g.: db4o-java5-6.4.8.2078.zip, etc...)

COR-870 - add svn.revision to Db4oVersion

COR-869 - Typehandler updates for .NET structs

COR-868 - Typehandler updates for .NET primitive types

COR-867 - TypeHandler updates for arrays and multi-dimensional arrays

COR-866 - Typehandler updates for String, Date and double

COR-865 - TypeHandler updates for boolean, byte, char

COR-860 - NewTypehandler: Database file creation for upgrade tests

COR-859 - TypeHandler upgrade tests for short long float

COR-856 - TypeHandler upgrade tests for int

COR-855 - TypeHandler Version sets to replace MarshallerFamily

COR-843 - Complete new TypeHandler reading for all Types

COR-840 - New Simple TypeHandler

COR-839 - TA/JAVA: Provide Ant task for build time TA instrumentation

COR-838 - Factor out common Bloat bytecode abstraction library from NQ, TA, CLDC,...

COR-829 - .NET: Support for multiple assemblies

COR-827 - .NET: Change instrumentation from method to field access (self encapsulate field)

COR-826 - Java: Change instrumentation from method to field access (self encapsulate field)

COR-824 - Java: add TA test suite to cruise control

COR-823 - .NET: add TA test suite to cruise control

COR-822 - TAStrategy: Break cascading when TA aware class is found

COR-821 - TAStrategy: Implement configurations

COR-820 - TAStrategy: Configuration API Design

COR-819 - Activation Strategies for Transparent Activation

COR-818 - TACol: .NET Dictionary

COR-817 - TACol: .NET ArrayList

COR-816 - TACol: Java Map

COR-815 - TACol: Java ArrayList

COR-805 - TypeHandlerRepository: Dynamic Typehandler generation

COR-804 - TypeHandlerRepository: Create basic hardwired repository

COR-803 - Versioned Typehandler4 Repository

COR-800 - NET: preserving debugging information

COR-797 - Java: Proper handling of inheritance hierarchies

COR-795 - dont instrument compiler generated classes

COR-781 - FastCollections: New TypeHandler4 writing for .NET types

COR-780 - FastCollections: New TypeHandler reading primitive types Java

COR-775 - Defragmenting an encrypted DB produces unencrypted DB

COR-770 - Defrag on 5.2 database after upgrade results in data loss -> Legacy Handler Defrag

COR-765 - Java client does not work with .NET server and vice versa

COR-755 - .NET: Proper handling of inheritance hierarchies

COR-618 - Investigate handling of non-private field access for Transparent Activation

COR-591 - Extract Java bytecode instrumentation tool

COR-547 - Storing String value in a field declared as Comparable gives unexpected results

COR-546 - ObjectInfo.getUUID for deleted object throws NPE in a committed callback

COR-536 - Don't catch Exception if unnecessary

COR-505 - Add configuration item for accessing java databases from .net

COR-242 - C#, Generic IList property that is a string[] does not get stored

COR-240 - Fields of type java.lang.Number cannot be stored

COR-199 - StackOverflow occurs when storing a plain object instance as a field.

COR-147 - sensible exception handling

COR-102 - Performance and memory progress needs to be constantly visible

New db4o library structure in db4o 7.7

Carl Rosenberger — Tue, 18 Nov 2008 13:31:00 GMT

The db4o core engine used to get built to one big fat .jar/.dll.

From our users on low resource devices we constantly hear requests for a smaller library size. For people using db4o on handhelds and phones like Android, 1 MB seems to be a critical size for an input component. We decided to respond:
We have made the db4o distribution more modular.

For db4o 7.7 we have factored out ClientServer functionality and some optional functionality into two separate Jars/DLLs. The optional package contains primitive Cluster querying, support for working with .NET and Java against the same database, IoAdapters to Debug and for Symbian, a SelfReflector package and all the old tools like Logger, MemoryStats, QueryStats and Statistics.

The names of the new libraries should be fairly self-explaining, but here is a little overview. Assuming you want to work with Java on JDK5, you will need the following Jars, if you rely on the previous functionality of db4o-[version]-java5.jar:
db4o-[version]-core-java5.jar
db4o-[version]-cs-java5.jar
db4o-[version]-optional-java5.jar

Please make sure you don't blindly use all db4o libraries in the /lib/ folder of the download. You would mix db4o versions for different Java versions in that case.

The "java5" in the above names means that the libraries are compiled for Java JDK 5 or higher. You will also find versions for "java1.1" and "java1.2" in the same lib folder. For all Jars that have a "java" in their name, please choose only the one for the platform that you need. All Jars without a "java" in their name are generic and should work for all Java platforms that we support.

On .NET you will have to add the following three DLLs to your project references, to get the same functionality that previously
was completely in Db4objects.Db4o.dll:
Db4objects.Db4o.dll
Db4objects.Db4o.CS.dll
Db4objects.Db4o.Optional.dll

Our embedded users are probably happy with the change. We hope we have not created a big hurdle for all others.

Enjoy!

Changes in Db4o Configuration API

Adriano Verona — Tue, 18 Nov 2008 11:00:00 GMT

Hi.

Recently we introduced some changes to our configuration APIs striving for simplicity.

You can read more about these changes here.

Best

Adriano

Fast Reflection for .Net

Adriano Verona — Mon, 10 Nov 2008 12:30:00 GMT

Hi,

Recently we decided to evaluate how much impact (in performance) reflection represented in db4o.

In .Net side we already knew (due to previous experience) that reflection is not as fast as we would like to.

At the same time, profiling proved that reflection was not a bottleneck in Java side.

So we settled up for trying to replace the reflection layer (in .Net) used to set / get field values at store / retrieval time.

What we did was basically emit a setter / getter method dynamically (using DynamicMethod) for each field. These setters/getters are able to directly access object's fields (even private ones),completly avoiding reflection (to be more precise, reflection is used to some extent to generate the setter/getter methods). Once generated we keep these setter/getter methods cached to avoid regenerating them needlessly. In other words we are trading space (memory) for performance.

One aspect of current implementation is that the getter/setters cache grows indefinitely. If we do get reports about performance improvements, and so we decide to use this mechanism (either by default or through a configuration) we can introduce MRU (or MFU) and cache expiration strategies to keep memory usage as low as possible).

Even doing some benchmarks we don't have enough data to take conclusions. For instance, our benchmarks pointed to a 8 ~ 10% improvement but when we tried to run our full test suite with this new "reflection" layer we got worst results (more time to run) (but surely our test suite is not a good candidate as a benchmark for this setting; it has lots of different classes and do lots of disk accesses to emit assemblies for regression tests and the like).

The version in trunk still uses the traditional reflection layer by default. You can configure your application to use the fast reflection layer with:

IEmbeddedConfiguration config = Db4oEmbedded.NewConfiguration();
config.Common.ReflectWith(new FastNetReflector());
IObjectContainer db = Db4oEmbedded.OpenFile(config, "database.odb");

Alternatively you can recompile the Db4objects.Db4o assembly so it uses the new reflection by default. All you have to do is to define a USE_FAST_REFLECTION symbol and recompile db4o assembly.

So, if you think this "reflection less" getters/setters could improve performance for you and you are willing to help db4o community, please, use it (either by configuring the reflector or by recompiling db4o with USE_FAST_REFLECTION) and deploy your application to an environment as close to a live one as possible and gather some data regarding performance and let us known.

Best

Adriano / Rodrigo

New fast collection implementation for huge sets: BigSet

Carl Rosenberger — Tue, 28 Oct 2008 12:01:00 GMT

Isn't it fun that you don't have to write any "INNER JOIN" SQL syntax in db4o whenever you want to describe that a set of objects is linked from another object? Instead db4o allows you to work with collections and to simply traverse the graph of objects contained in the collection.

The multiple different collection types that come with Java/.NET have different performance characteristics. Some may allow faster inserts. Some may allow faster access to elements.

Two recommendations to be able to tune your application with the right collection as it evolves:

(1) Use interface declararations only when you add references to collection objects to your application classes.

Good:
class Company{
Collection employees;
}

Bad:
class Company{
ArrayList employees;
}

(2) Use a single collection factory in one place in your code so you can easily exchange the concrete collection implementation to try out alternatives:

company.employees = CollectionFactory.newCollection();

CollectionFactory would be a class that you would write and where you could simply exchange one line of code to create a db4o ArrayList4 with Transparent Persistence support instead of a normal ArrayList.

db4o comes with some built-in collections that will work better for specific usescases. In the latest build we have added a new species to this list: BigSet.

This is a highly specialized class to hold large and huge sets of references to other first class objects. Internally the implementation is based on the same fast BTrees that are also in use for db4o class and field indexes.

Our benchmark results from comparing BigSet to a normal JDK ArrayList are very promising: Especially when you make small changes to huge sets, the advantage can be multiple orders of magnitude.

For now we have only completed a BigSet for embedded use in single user mode. A Client/Server implementation with a BigSet proxy is being thought about.

BigSet is very easy to use:
CollectionFactory.forObjectContainer(objectContainer).<Item>newBigSet();

Please let us know how fast it works for you! Enjoy!

New pluggable StringEncoding interface

Carl Rosenberger — Thu, 09 Oct 2008 13:21:00 GMT

When db4o stores and loads strings it has to convert them to and from bytes. Until recently db4o only used to supply 2 encodings to do this work:
One for 16-bit Unicode, the other for ISO 8859-1. To decide for one or the other you would call Configuration#unicode() and pass true for Unicode and false for ISO 8859-1.

To supply more encodings and to allow the creation of own string encodings we deprecated the old configuration interface and supply a new one. Now you can either use one of the built-in encodings from the StringEncodings class or pass your own encoding. Examples:

Configuration config = Db4o.newConfiguration();
config.stringEncoding(StringEncodings.utf8());
Db4o.openFile(config, "myDB.db4o");

or

Configuration config = Db4o.newConfiguration();
config.stringEncoding(new StringEncoding() {
	public byte[] encode(String str) {
		// TODO: implement
		return null;
	}
	public String decode(byte[] bytes, int start, int length) {
		// TODO: implement
		return null;
	}
});
Db4o.openFile(config, "myDB.db4o");

Please note that you should configure the string encoding before you create the database file, in the first call to #openFile() or #openServer(). If you use one of the built-in encodings, db4o will remember the encoding for all subsequent open calls. If you use a StringEncoding class that you have written yourself, you have to supply the same StringEncoding and configure it for all subsequent open calls, otherwise db4o will not be able to read and write strings correctly. Since db4o also uses the string encodings for metadata like classnames and fieldnames, you are likely to notice any misconfiguration very quickly by not being able to read any objects at all.

The StringEncoding used has to stay the same for the lifetime of a database.

Although we keep the Unicode string encoding as the standard setting, we encourage the use of the new built-in UTF-8 string encoding for smaller and faster database files.

We also hope that the pluggable interface will be useful for you, especially if you want db4o to work with Chinese or Japanese character sets.

Enjoy!

LINQ for the Compact Framework 3.5

Rodrigo B. de Oliveira — Wed, 24 Sep 2008 11:40:00 GMT

Db4o for .NET 3.5 ships with an optimizing LINQ provider. This provider is able to optimize the execution of a query by analyzing the LINQ expression tree emitted by the C# (or VB.NET) compiler to produce an equivalent SODA query which can be executed by db4o at full speed.

The Compact Framework 3.5 however includes no support for expression trees. No expression trees, no optimizing LINQ provider. A saddening state of affairs for Compact Framework db4o users.

As it turns out, expression trees are provided by the System.Linq.Expressions namespace which is not available in the standard Compact Framework 3.5 distribution.

Mono to the rescue

The Mono project provides a free software implementation of the .net framework including LINQ and the System.Linq.Expressions namespace.

Hypothesis: will the Compact Framework compiler emit expression trees if it sees a reference to a compatible assembly that provides a System.Linq.Expressions namespace?

Yes it will. LINQ for Compact Framework seemed to be possible.

There was a remaining issue. When the db4o LINQ provider cannot optimize a query down to SODA it can simply execute it as a LINQ for Objects query. This is accomplished by invoking the Compile method on the expression tree. The returned delegate can be used to evaluate the expression whenever needed. But the Compile method relies on System.Reflection.Emit which is again not supported by the Compact Framework. Hmpf.

The final piece of the puzzle appeared when Mainsoft contributed an expression tree interpreter to Mono. Expression tree evaluation that didn't rely on System.Reflection.Emit. Yummy.

So thanks to the hard and loving work of the Mono and Mainsoft guys we are able to ship to you the first optimizing LINQ provider for the Compact Framework.

A nice hack indeed.

Collected Generics, Volume 1

Patrick Roemer — Fri, 12 Sep 2008 11:49:00 GMT

Finally we are starting to harvest the fruits of our work on pluggable type handlers. As a first step, please welcome .NET generic collections to the exclusive club of types natively supported by db4o in the upcoming 7.6 build.

In principle, .NET generic collections could already be persisted in previous versions, but there was no dedicated handling in place for them. db4o would not recognize them as collections and would simply handle them as any other object - inspect their internal layout and persist the field values found. This approach came with some pitfalls. Most notably, db4o would not recognize the collection's internal state objects as parts of a single, atomic unit - in C/S mode, concurrent updates from multiple clients against the same collections could have resulted in inconsistent internal collection state in the worst case. Furthermore, issues could arise with specific collection implementations: Dictionaries, for example, would persist hash codes alongside keys/values and could stumble when attempting to reload this data in a different runtime version (as GetHashCode() implementations may change yielding different hash values). Plus, last but not least, dedicated collection handling opens the path for performance optimizations.

With the current type handler infrastructure in place, providing custom handling for .NET generic collections as its first "real world" application went pretty smooth - actually the bulk of the effort was spent on the implementation of combinatorial regression testing of permutations of collection, field and element types - take a look at GenericCollectionTypeHandlerTestSuite, if you're interested. As of now, dedicated type handler support is available for the following collection flavors:

List<T>
LinkedList<T>
Stack<T>
Queue<T>
Dictionary<K,V>
SortedList<K,V>
SortedDictionary<K,V>

These collections now should blend in seamlessly with the behavior of non-generic collections. More to come - but if you're missing a dedicated type handler for a specific flavor, of course you're also invited to contribute. ;)

Latest production build: 7.4.58.11547

Carl Rosenberger — Tue, 02 Sep 2008 10:51:00 GMT

here is a short info to resolve a race condition between our newsletter sendout and the very latest production build:

We got informed about a serious (blocker) bug with a deadlock in committed callbacks after our newsletter had already been released to marketing: COR-1377

This bug received the highest priority and it has has been fixed immediately.

A new production build with the build number 7.4.58.11547 is online in the meanwhile. If you downloaded 7.4 right after the newsletter and if you use committed callbacks or UUIDs, we recommend to check the build number you are using (call Db4o#version() / Db4oFactory#Version() ) and to download the very latest build once again if it is lower than 7.4.58.

To our knowledge this bug has been present in all 7.2 builds.

Performance Contest Results

Carl Rosenberger — Wed, 27 Aug 2008 10:16:00 GMT

Two months ago we invited the db4o community to take part in the first db4o performance contest. Now the contest has been completed. Here are the results.

Three entries are eligible for prizes:

June 15
Andrew Zhang submitted a patch that improves the speed to sort queries.

June 19
Andrew Zhang submitted a UTF-8 string encoder.

July 17
Erik Putrycz submitted a patch to reuse query results in subsequent similar queries.

Andrew and Erik wrote own Poleposition circuits for their patches to be able to measure performance improvements. They have been incorporated into the db4opolepos project in our SVN as circuits "Hockenheim" and "Hungaroring".

The three patched jars that Andrew and Eric supplied have been renamed to db4o-7.2-sortedquery.jar, db4o-7.2-utf8.jar, and db4o-7.4-cachedqueries.jar to make it easy to distinguish the results. These jars have also been checked into the db4opolepos project. Anyone can easily reproduce the performance race by running org.polpos.AllRacesRunner from this project.

The results from running this benchmark on our hardware are attached to this posting as PerformanceContest.pdf.

Our impression from reviewing the results and the patches:

- The effect of the tuned quicksort algorithm on query sorting is absolutely amazing. Improvements are indeed in the range of 20x to 300x as you can see in Hockenheim#query_ascending and #query_descending on pages 52 and 53 of the attached Poleposition results. The patch is very straightforward and very nice, so it can immediately go into production code as is. A perfect submission.

- The UTF-8 string encoder reduces the required slot length for stored strings. Shorter slots result in faster writes as you can see in Nurburgring#write on page 24 of the attached Poleposition run. A performance improvement of about 20% is measurable. Incorporation of the patch code into the db4o core would require some more work: Probably we would add UTF-8 as a third configurable string encoding. Possibly we could make it the default. In any case we would have to continue to support the existing Unicode string encoding. The submission shows very nicely how easy a UTF-8 string encoder is. We should make one available soon.

- The idea to cache query results is great. A working implementation is certainly an order of magnitude more difficult to write than the other two submitted patches. Somehow the Hungaroring benchmark shows a slowdown instead of an improvement. After Erik's comments that our Trunk code got slower for concurrent queries, we have also compared running the patched Jar against Trunk and Trunk still turns out to be faster. Erik's submission also does not exactly follow the rules of the contest:
The patch is not applied to the 7.2 workspace, so a direct comparison run to other submissions is not completely fair. In any case we really liked the idea and we have to give credit for trying.

So how do we judge the three submissions and who gets first, second and third prize?

Andrew's submissions both are measurable and they are both in compliance with the rules that we have set up. Erik's submission is a great idea, but it is not 100% complete and we have not yet seen the improvement clearly documented in a Poleposition run.

The core team reviewers suggest to award prizes as follows:
1st USD 3000 Andrew Zhang
2nd USD 2000 Andrew Zhang
3rd USD 1000 Erik Putrycz

If anyone in our community has a different opinion, please post it to our performance forum.

Andrew and Erik: Thank you very much for your submissions. You have deserved the prizes!

For all other developers that tried to improve performance but did not complete a submission on time: Thank you for trying!
If you have suggestions how to improve the Performance Contest, please tell us about them. Maybe it would make sense to let the contest run for a longer period of time than 2 months? What do you think?

We are very happy to have received usable patches in this first db4o performance contest. Hopefully we will get even more contributions the next time.

It is great to work with such a nice community!

Wanted: TypeHandlers

Carl Rosenberger — Fri, 22 Aug 2008 16:29:00 GMT

You were playing with the idea to contribute to db4o? So far you did not find a nice, easy and short task to take on? Here is your chance: Write a TypeHandler for db4o and get your code included in the db4o core distribution tomorrow.

db4o comes with a big promise: Just call ObjectContainer#store() and any object and all attached objects will be automagically stored. The challenge to do this is high: Every possible class out there needs to be "understood" by db4o and an efficient byte array representation needs to be created on storage media. Indexing of "parts" of objects and querying is part of the task. With evolving languages and an increasing number of class constructs that need to be supported we decided to make it easier to write the code for marshalling and unmarshalling: Enter TypeHandlers.

TypeHandlers define how objects of a specific class are stored, loaded and queried by db4o.

By default db4o stores and restores all non-transient fields of an object. By writing TypeHandlers it is possible to alter the behaviour. A TypeHandler could decide to store only some members of a class or it could decide to store a completely different and more efficient representation.

The possible benefits of using a Typehandler:
- Storing and restoring objects can be faster.
- The database file can be smaller.
- Objects that consist of sub-objects (Example: collections) can be guarded against inconsistencies in concurrent multi-user environments by preventing "the object" from being split, but using one single contiguous slot as the marshalled representation. ( We will soon also demonstrate a stateful collection implementation over multiple pages, also powered by a Typehandler.)
- Fields that are intended to be transient do not need to be stored.
- Typehandlers can alter behaviour even if the sources of the persistent classes are not available or can not be altered.
- Querying support.

TypeHandlers are expected to do the complete marshalling of an object by writing/reading directly to/from the marshalling buffer.

About a year ago the TypeHandler4 interface consisted of more than 40 methods. Today only 5 methods remain. To make sure that the TypeHandler interface is ready for heavy duty use, we have already used it ourselves to implement some collection typehandlers. See:
com.db4o.typehandlers.CollectionTypeHandler (Java)
com.db4o.typehandlers.MapTypeHandler (Java)
Db4objects.Db4o.Typehandlers.ListTypeHandler (.NET)
Db4objects.Db4o.Typehandlers.MapTypeHandler (.NET)

In the latest development build (7.5.x.y) these TypeHandlers are already turned on to do real work storing, loading and querying some selected collections. To see how internal TypeHandlers are configured, take a look at:
com.db4o.internal.TypeHandlerConfigurationJDK_1_2 (Java)
Db4objects.Db4o.Internal.TypeHandlerConfigurationDotNet (.NET)

You will notice that many TypeHandlers are still missing if the full featureset of JDK and .NET collections is to be supported. Here is where we would like to ask you for your help:
Could you write a TypeHandler for us for a collection class that you think we need to support?

Adapting and registering one of the already existing typehandlers for your favourite collection will probably take you less than an hour of work. You can simply follow the pattern that is already there.

However writing the TypeHandler code is not the complete story. Since we want to be absolutely sure that a new TypeHandler works correctly, the bigger part of the implementation work is required for testing. We want to make sure that objects stored with an older version of db4o can be read and stored even after your new TypeHandler is installed.

For this purpose we developed the "Db4oMigrationTestSuite" test.
On Java you can run this class directly. You will find it in our SVN in the db4ojdk1.2 project.
( If you have not worked with the db4o sources before, here is a guide to get started. )

If you develop for .NET you would check out the db4o.net project from our SVN. You would typically work with the Db4o-2008.sln solution in the top level folder. To run the tests, the Db4objects.Db4o.Tests-2008 project would be your startup project. To save time by not running all the tests every time you can edit Db4objects.Db4o.Tests.AllTests in the Db4objects.Db4o.Tests-2008 project: If you want to run migration tests only, you can comment out all the entries in the #TestCases() method and add the following instead:
typeof(Db4objects.Db4o.Tests.Common.Migration.Db4oNETMigrationTestSuite)

Both for Java and .NET:
If you want the Db4oMigrationTestSuite to truely run against all relevant old db4o engines, you should check out the db4o.archives project from our SVN also.

Once you have testing set up and running, you are ready to go to write your own test for updating your favourite collection.

For an example how an update test could look like, please see:
com.db4o.db4ounit.common.handlers.ArrayListUpdateTestCase (Java)
Db4objects.Db4o.Tests.CLI2.Handlers.GenericListVersionUpdateTestCase (.NET)

Only when you have the test in place it makes start to implement your own TypeHandler as a more efficient method to store the collection. Test first, remember?

Should you have any problems to get started, please ask for help in the db4o product developer forum. It is a good idea to announce any work that you are about to do in the forum in any case. That will ensure that the work for a specific collection is only done once.

Now that you are all set to start:
Good luck for writing your TypeHandler!

All the new Typehandlers we have built so far are now available in the TRUNK of our SVN and with our continous build.

They help to make storage of collections more efficient:
- Less state is pushed around than with our previous translator approach.
- Maps no longer store an internal Entry object for each entry.
- All .NET collections, including all .NET generic collections will soon be handled efficiently with Typehandlers. Current issues with .NET generic collections in concurrent multi-user setups will soon be history.
- Old database files can be read with new Typehandlers installed. Future marshalling format changes are easily possible by writing and installing yet further new Typehandlers.
- We have a very nice hook into collection treatment that we will be able to use for much faster collections.

Enjoy!

Synchronization issues in networking C/S fixed

Patrick Roemer — Sat, 02 Aug 2008 13:32:00 GMT

We have fixed some thread synchronization issues in the C/S protocol and put new versions (rev. x.y.54.11278) for all branches online in the download center. Users with concurrent networking C/S in connection with lazy queries are recommended to upgrade. For more information, please refer to the corresponding tracker entry.

Debugging TA Enhanced Code and the Observer Effect

Rodrigo B. de Oliveira — Tue, 29 Jul 2008 00:03:00 GMT

Most of those who pair programmed with me know that I'm not very fond of debugging code by stepping through breakpoints inside an IDE. I'd much rather construct hypotheses on why the code is failing and systematically test them with carefully placed assertions and print statements.

One reason for that is the observer effect of debugging: the execution environment can be a little different, timings are usually changed, object inspectors might end up calling methods with unwanted side-effects and so on.

The last point is a particularly important one to keep in mind when dealing with code that's been enhanced for Transparent Persistence.

Let's say we have built the latest and greatest library management application and now want to put a web front end on top of it. We've been following web development trends and so we know AJAX is a must these days. We just need some simple way to transfer the objects from the server to the browser javascript runtime: JSON to the rescue. Good thing we can reuse some reflection based JSON serialization code we had laying around.

We write a quick spike to see how well it works:

    Dim book = New Book("The God Delusion")
    book.AddAuthor(New Author("Richard Dawkins"))
    
    Console.WriteLine(New JSONSerializer().Serialize(book))

Which give us the expected output:

	{ _title: "The God Delusion", _authors: [{ _name: "Richard Dawkins" }] }

We try one more spike to convince us we can get JSON out of a db4o query result:

    Using db = OpenDatabase()
        For Each book In db.Query(of Book)()
            Console.WriteLine(New JSONSerializer().Serialize(book))
        Next
    End Using

Unfortunately this gives us something completely unexpected and it's not the Spanish Inquisition:

	{ _title: null, _authors: null }

We place a breakpoint inside the loop so we can inspect the book instance:

It sure looks right and oddly enough we get the right output this time:

	{ _title: "The God Delusion", _authors: [{ _name: "Richard Dawkins" }] }

What's going on?

Transparent Activation is going on or rather NOT going on:

Our Book and Author classes were enhanced for Transparent Activation, Db4oTool injected the code required to implement IActivatable as well as the code that transparently activates the object before any field access;
JSONSerializer uses reflection to access objects fields;
Reflection completely bypasses any code introduced by Db4oTool which explains the "{ _title: null, _authors: null }" output, the object was never activated;
When inspecting an object in the debugger watch window its properties are accessed;
Accessing a property of our enhanced Book object causes its *transparent* activation which also explains why we got the right output aftewards;

Ok, now that we have a logical explanation for what's happening how do we get our application to work as expected?

Since JSONSerializer bypasses activation we have to ensure the object graph is completely activated before JSONSerializer gets to see it:

    Using db = OpenDatabase()
        For Each book In db.Query(of Book)()
            db.Activate(book, Integer.MaxValue)
            Console.WriteLine(New JSONSerializer().Serialize(book))
        Next
    End Using

And that's it.

Hopefully that will save you some frustration next time you need to debug Transparent Persistence enhanced code or to have it interacting with reflection based libraries.

The complete code can be found here.

Have fun!

Performance Contest - for one more month

Carl Rosenberger — Thu, 17 Jul 2008 10:38:00 GMT

Do you like profiling? Do you like tuning code? If you do, how about taking part in the db4o performance contest? We offer USD 6000 in prizes for the best contributions to make db4o faster.

We have made it very easy for you to get started by providing an Eclipse workspace set up with the db4o sources and the Poleposition benchmark.

Links to get you started:
Performance Contest page
"Getting Started" video
Performance Contest Forum

The contest has been running for a month already and we are seeing first contributions coming in.

Andrew has improved sorted queries and provides a UTF-8 string encoder.
Erik is working on a query cache to make queries reusable.

Can you beat their patches? I am sure you can.
The contest is still open for contributions for one more month until August 15.