An Introduction to FsPickler
The following provides an overview of the basic functionalities offered by the library.
The core serialization API
The basic API is accessible through instances of type FsPicklerSerializer
that can be initialized as follows:
1: 2: 3: 4: 5: 6: |
|
Json serialization formats can be accessed by referencing the FsPickler.Json
project.
If evaluating from F# interactive, make sure to add an explicit reference to Json.Net.
1: 2: 3: 4: 5: 6: |
|
A simple serialization/deserialization roundtrip can be performed as follows:
1: 2: |
|
FsPickler instances can be used to produce binary pickles, that is byte arrays containing the serialized values:
1: 2: |
|
Text-based serialization
The Xml and Json serializers are instances of type FsPicklerTextSerializer
that offers functionality for text-based serialization:
1: 2: 3: 4: 5: |
|
Sequence serialization
FsPickler offers support for on-demand sequence serialization/deserialization:
1: 2: 3: 4: 5: |
|
Picklers and Pickler combinators
A pickler is essentially the type
1: 2: 3: 4: 5: |
|
which defines the serialization/deserialization rules for a given type. Picklers are strongly typed and perform serialization without reflection or intermediate boxings.
There are two kinds of picklers:
-
Primitive or atomic picklers that are self-contained definitions for simple values like primitives, strings or timespans.
-
Composite picklers which are derived from composition of simpler types. They are generated using pickler combinators, functions taking a collection of picklers as inputs yielding a composite result.
FsPickler is essentially an automated pickler generation framework: picklers are generated at runtime and on demand using a combination of reflection and dynamic IL generation. Picklers are cached for future use, hence the cost of generation has a constant price.
Moreover, the library provides an experimental combinator module that allows direct manipulation of picklers in a more functional style:
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: |
|
The module includes all primitive combinators as described in Andrew Kennedy's
Pickler Combinators
likewrap
and alt
. Fixpoint combinators for declaring recursive picklers are also available:
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: |
|
When it comes to generic types, picklers can be created through user-defined combinators:
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: |
|
or it could be done using automatic resolution of type parameters:
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: |
|
SerializationInfo Picklers
It is possible to define picklers that serialise objects using SerializationInfo. For example, consider the record:
1:
|
|
We can define a SerializationInfo based pickler using the following combinator:
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: |
|
Experimental N-way Sum and Product Combinators
N-way sum and product combinators provide an alternative pretty syntax for defining picklers over arbitrary discriminated unions and records. Unfortunately at the moment the performance of resulting picklers is sub-optimal, this might improve in the future.
The types involved in the examples are not fit for human consumption, but thankfully F# infers them automatically. The implementation is total and purely functional, see this gist for some Coq code used to model these combinators.
Records / Product Types
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: |
|
Unions / Sum Types
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: |
|
Generating custom picklers in type definitions
FsPickler can be instructed to use custom pickler definitions for given types using the following design pattern:
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: |
|
This tells FsPickler to generate a pickler for the given type using
that particular factory method. It should be noted that the Read
/Write
operations are not commutative, hence care should be taken so that ordering is matched.
The IPicklerResolver
argument provides a handle to the pickler generator and can be used recursively:
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: |
|
Custom pickler registrations
Consider a type declaration that has not been made serializable:
1: 2: 3: |
|
Attempting to generate a pickler for the particular type
1:
|
|
would result in an error:
1: 2: 3: 4: |
|
This problem can be overcome by creating a custom pickler cache that accepts user-supplied pickler registrations. Suppose we have a custom pickler definition:
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: |
|
We can then create a custom pickler cache like so:
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: |
|
In some cases, it might be sufficient to declare that FsPickler should treat
the type as if it wasn't lacking a [<Serializable>]
annotation:
1:
|
|
Additional tools
This section describes some of the additional tools offered by the library:
Object Cloning
FsPickler 1.2 adds support for fast cloning of serializable objects. This is done in a node-per-node basis, without the need for serialization formats and intermediate buffers.
1:
|
|
Structural Hashcodes
FsPickler offers experimental support for structural, non-cryptographic, hashcode generation:
1: 2: 3: 4: 5: 6: 7: 8: |
|
This will generate a 128-bit structural hashcode based on the MurMurHash algorithm.
Implementation is memory efficient, since the hashing algorithm is integrated with
the underlying stream implementation. It is possible for users to define their own hashing
algorithms by inheriting the special HashStream
class.
Hashing functionality offered by FsPickler is an ideal replacement to .GetHashCode()
for
large objects or complex object graphs, but it is not recommended for small values or primitives.
If a hashcode is not required, the size of an object alone can be computed as follows:
1: 2: 3: |
|
Typed Serialization
It is possible to create typed picklings of objects:
1:
|
|
this will produce a serialization annotated with the type of the original object. They can then be easily deserialized like so:
1:
|
|
Object Sifting
FsPickler 1.2 comes with a 'sifting' functionality which allows serialization/cloning while omitting specified instances from an object graph. For example, consider the object graph
1: 2: 3: |
|
The size of the object becomes evident when running
1: 2: 3: |
|
Supposing we knew that the size of the graph was being bloated by large arrays, we can use FsPickler to optimize serialization by sifting away occurences from the object graph.
1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: |
|
This will return a sifted clone of the original object as well as a collection of all objects that were sifted from the original input. The sifted copy encapsulated in a wrapper type so that it cannot be consumed while in its partial state. The original input graph will not be mutated in any way. We can verify that the size of the sifted object has been reduced:
1: 2: 3: |
|
Sifted objects can be put back together by calling
1:
|
|
Object Graph Visitors
FsPickler is capable of efficiently traversing arbitrary object graphs (as long as they are serializable) by exploiting its pickler infrastructure. This can be done by calling the method:
1:
|
|
which takes as input a serializable object graph and a so-called object visitor:
1: 2: 3: 4: |
|
A few applications of this are provided by the core library:
1: 2: 3: 4: 5: 6: |
|
Disabling Subtype Resolution
For security reasons, it might often be desirable to disable subtype resolution when serializing classes:
1:
|
|
This essentially disables the serialization of any object whose declaring type is specified on the serialization itself. Attempting to serialize or deserialize any such object will result in an exception.
Note that enabling this option prevents serialization of the following types:
System.Object
or any abstract class (excluding F# DUs).- Any delegate instance or F# function.
- Any
ISerializable
class.
As a further precaution, it is also possible to disable implicit assembly loading when deserializing objects:
1:
|
|
Defining Custom Pickle Formats
It is possible to create user-defined pickle formats for FsPickler. One simply needs to implement the interface
1: 2: 3: 4: 5: 6: 7: |
|
which can then be bolted on a class that inherits either of the
FsPicklerSerializer
or
FsPicklerTextSerializer
.
from Microsoft.FSharp.Core.Operators
inherit MarshalByRefObject
member BeginRead : buffer:byte[] * offset:int * count:int * callback:AsyncCallback * state:obj -> IAsyncResult
member BeginWrite : buffer:byte[] * offset:int * count:int * callback:AsyncCallback * state:obj -> IAsyncResult
member CanRead : bool
member CanSeek : bool
member CanTimeout : bool
member CanWrite : bool
member Close : unit -> unit
member CopyTo : destination:Stream -> unit + 1 overload
member CopyToAsync : destination:Stream -> Task + 3 overloads
member Dispose : unit -> unit
...
type FsPicklerSerializer =
new : formatProvider:IPickleFormatProvider * ?typeConverter:ITypeNameConverter * ?picklerResolver:IPicklerResolver -> FsPicklerSerializer
member ComputeHash : value:'T * ?hashFactory:IHashStreamFactory -> HashResult
member ComputeSize : value:'T * ?pickler:Pickler<'T> -> int64
member CreateObjectSizeCounter : ?encoding:Encoding * ?resetInterval:int64 -> ObjectSizeCounter
member Deserialize : stream:Stream * ?pickler:Pickler<'T> * ?streamingContext:StreamingContext * ?encoding:Encoding * ?leaveOpen:bool -> 'T
member DeserializeSequence : stream:Stream * ?pickler:Pickler<'T> * ?streamingContext:StreamingContext * ?encoding:Encoding * ?leaveOpen:bool -> seq<'T>
member DeserializeSequenceUntyped : stream:Stream * pickler:Pickler * ?streamingContext:StreamingContext * ?encoding:Encoding * ?leaveOpen:bool -> IEnumerable
member DeserializeSifted : stream:Stream * sifted:(int64 * obj) [] * ?pickler:Pickler<'T> * ?streamingContext:StreamingContext * ?encoding:Encoding * ?leaveOpen:bool -> 'T
member DeserializeUntyped : stream:Stream * pickler:Pickler * ?streamingContext:StreamingContext * ?encoding:Encoding * ?leaveOpen:bool -> obj
member Pickle : value:'T * ?pickler:Pickler<'T> * ?streamingContext:StreamingContext * ?encoding:Encoding -> byte []
...
--------------------
new : formatProvider:MBrace.FsPickler.IPickleFormatProvider * ?typeConverter:MBrace.FsPickler.ITypeNameConverter * ?picklerResolver:MBrace.FsPickler.IPicklerResolver -> MBrace.FsPickler.FsPicklerSerializer
inherit MarshalByRefObject
member Close : unit -> unit
member Dispose : unit -> unit
member DisposeAsync : unit -> ValueTask
member Encoding : Encoding
member Flush : unit -> unit
member FlushAsync : unit -> Task
member FormatProvider : IFormatProvider
member NewLine : string with get, set
member Write : value:char -> unit + 18 overloads
member WriteAsync : value:char -> Task + 5 overloads
...
inherit MarshalByRefObject
member Close : unit -> unit
member Dispose : unit -> unit
member Peek : unit -> int
member Read : unit -> int + 2 overloads
member ReadAsync : buffer:Memory<char> * ?cancellationToken:CancellationToken -> ValueTask<int> + 1 overload
member ReadBlock : buffer:Span<char> -> int + 1 overload
member ReadBlockAsync : buffer:Memory<char> * ?cancellationToken:CancellationToken -> ValueTask<int> + 1 overload
member ReadLine : unit -> string
member ReadLineAsync : unit -> Task<string>
member ReadToEnd : unit -> string
...
private new : unit -> FsPickler
static member Clone : value:'T * ?pickler:Pickler<'T> * ?streamingContext:StreamingContext -> 'T
static member ComputeHash : value:'T * ?hashFactory:IHashStreamFactory -> HashResult
static member ComputeSize : value:'T * ?pickler:Pickler<'T> -> int64
static member CreateBinarySerializer : ?forceLittleEndian:bool * ?typeConverter:ITypeNameConverter * ?picklerResolver:IPicklerResolver -> BinarySerializer
static member CreateObjectSizeCounter : ?encoding:Encoding * ?resetInterval:int64 -> ObjectSizeCounter
static member CreateXmlSerializer : ?typeConverter:ITypeNameConverter * ?indent:bool * ?picklerResolver:IPicklerResolver -> XmlSerializer
static member EnsureSerializable : graph:'T * ?failOnCloneableOnlyTypes:bool -> unit
static member GatherObjectsInGraph : graph:obj -> obj []
static member GatherTypesInObjectGraph : graph:obj -> Type []
...
static member CreateJsonSerializer : ?indent:bool * ?omitHeader:bool * ?typeConverter:ITypeNameConverter * ?picklerResolver:IPicklerResolver -> JsonSerializer
member FsPicklerSerializer.Serialize : stream:System.IO.Stream * value:'T * ?pickler:Pickler<'T> * ?streamingContext:System.Runtime.Serialization.StreamingContext * ?encoding:System.Text.Encoding * ?leaveOpen:bool -> unit
member FsPicklerSerializer.Deserialize : stream:System.IO.Stream * ?pickler:Pickler<'T> * ?streamingContext:System.Runtime.Serialization.StreamingContext * ?encoding:System.Text.Encoding * ?leaveOpen:bool -> 'T
val float : value:'T -> float (requires member op_Explicit)
--------------------
type float = System.Double
--------------------
type float<'Measure> = float
type Expr =
override Equals : obj:obj -> bool
member GetFreeVars : unit -> seq<Var>
member Substitute : substitution:(Var -> Expr option) -> Expr
member ToString : full:bool -> string
member CustomAttributes : Expr list
member Type : Type
static member AddressOf : target:Expr -> Expr
static member AddressSet : target:Expr * value:Expr -> Expr
static member Application : functionExpr:Expr * argument:Expr -> Expr
static member Applications : functionExpr:Expr * arguments:Expr list list -> Expr
...
--------------------
type Expr<'T> =
inherit Expr
member Raw : Expr
val int : value:'T -> int (requires member op_Explicit)
--------------------
type int = int32
--------------------
type int<'Measure> = int
val seq : seq<string>
--------------------
type seq<'T> = System.Collections.Generic.IEnumerable<'T>
from Microsoft.FSharp.Collections
val string : value:'T -> string
--------------------
type string = System.String
type Pickler =
private new : Type -> Pickler
abstract member private Cast : unit -> Pickler<'S>
abstract member private Unpack : IPicklerUnpacker<'U> -> 'U
abstract member private UntypedAccept : state:VisitState -> value:obj -> unit
abstract member private UntypedClone : state:CloneState -> value:obj -> obj
abstract member private UntypedRead : state:ReadState -> tag:string -> obj
abstract member private UntypedWrite : state:WriteState -> tag:string -> value:obj -> unit
abstract member ImplementationType : Type
abstract member IsCacheByRef : bool
abstract member IsCloneableOnly : bool
...
--------------------
type Pickler<'T> =
{ Write: WriteState -> 'T -> unit
Read: ReadState -> 'T }
private new : formatter:IPickleFormatWriter * resolver:IPicklerResolver * reflectionCache:ReflectionCache * isHashComputation:bool * disableSubtypeResolution:bool * ?streamingContext:StreamingContext * ?sifter:IObjectSifter -> WriteState
member private GetObjectId : obj:obj * firstTime:byref<bool> -> int64
member private Reset : unit -> unit
member private CyclicObjectSet : HashSet<int64>
member DisableSubtypeResolution : bool
member private Formatter : IPickleFormatWriter
member IsHashComputation : bool
member private ObjectCount : int64
member private ObjectStack : Stack<int64>
member private PicklerResolver : IPicklerResolver
...
private new : formatter:IPickleFormatReader * resolver:IPicklerResolver * reflectionCache:ReflectionCache * disableSubtypeResolution:bool * disableAssemblyLoading:bool * ?streamingContext:StreamingContext * ?sifted:(int64 * obj) [] -> ReadState
member private EarlyRegisterArray : array:Array -> unit
member private NextObjectId : unit -> int64
member private Reset : unit -> unit
member DisableAssemblyLoading : bool
member DisableSubtypeResolution : bool
member private Formatter : IPickleFormatReader
member private IsUnSifting : bool
member private ObjectCache : Dictionary<int64,obj>
member private ObjectCount : int64
...
val int : Pickler<int>
--------------------
type int = int32
--------------------
type int<'Measure> = int
module Pickler
from MBrace.FsPickler.Combinators
--------------------
type Pickler =
private new : Type -> Pickler
abstract member private Cast : unit -> Pickler<'S>
abstract member private Unpack : IPicklerUnpacker<'U> -> 'U
abstract member private UntypedAccept : state:VisitState -> value:obj -> unit
abstract member private UntypedClone : state:CloneState -> value:obj -> obj
abstract member private UntypedRead : state:ReadState -> tag:string -> obj
abstract member private UntypedWrite : state:WriteState -> tag:string -> value:obj -> unit
abstract member ImplementationType : Type
abstract member IsCacheByRef : bool
abstract member IsCloneableOnly : bool
...
--------------------
type Pickler<'T> =
inherit Pickler
private new : unit -> Pickler<'T>
abstract member Accept : state:VisitState -> value:'T -> unit
abstract member Clone : state:CloneState -> value:'T -> 'T
abstract member Read : state:ReadState -> tag:string -> 'T
abstract member Write : state:WriteState -> tag:string -> value:'T -> unit
override private Unpack : IPicklerUnpacker<'R> -> 'R
override private UntypedAccept : state:VisitState -> value:obj -> unit
override private UntypedClone : state:CloneState -> obj -> obj
override private UntypedRead : state:ReadState -> tag:string -> obj
...
val string : Pickler<string>
--------------------
type string = System.String
from MBrace.FsPickler.Combinators
module Json
from MBrace.FsPickler.Combinators
--------------------
namespace MBrace.FsPickler.Json
| Zero
| Succ of Peano
| Leaf
| Node of 'T * Tree<'T> list
from MBrace.FsPickler.Combinators
{ FirstName: string
MiddleName: string option
Surname: string }
{ Address: string
Age: int
Name: string }
Person.Name: string
--------------------
type Name =
{ FirstName: string
MiddleName: string option
Surname: string }
| Case1
| Case2 of int
| Case3 of string * int
union case CustomPicklerRegistration.CustomPickler: factory: IPicklerResolver -> Pickler -> CustomPicklerRegistration
--------------------
type CustomPicklerAttribute =
inherit Attribute
new : unit -> CustomPicklerAttribute
--------------------
new : unit -> CustomPicklerAttribute
type CustomClass<'T,'S> =
new : x:'T * y:'S -> CustomClass<'T,'S>
member X : 'T
member Y : 'S
static member CreatePickler : resolver:IPicklerResolver -> Pickler<CustomClass<'T,'S>>
--------------------
new : x:'T * y:'S -> CustomClass<'T,'S>
interface
abstract member IsSerializable : unit -> bool
abstract member IsSerializable : Type -> bool
abstract member Resolve : unit -> Pickler<'T>
abstract member Resolve : Type -> Pickler
end
abstract member IPicklerResolver.Resolve : System.Type -> Pickler
type RecursiveClass =
new : ?nested:RecursiveClass -> RecursiveClass
member Value : RecursiveClass option
static member CreatePickler : resolver:IPicklerResolver -> Pickler<RecursiveClass>
--------------------
new : ?nested:RecursiveClass -> RecursiveClass
static member FsPickler.GeneratePickler : t:System.Type -> Pickler
type AutoSerializableAttribute =
inherit Attribute
new : value:bool -> AutoSerializableAttribute
member Value : bool
--------------------
new : value:bool -> AutoSerializableAttribute
type NonSerializable =
new : value:int -> NonSerializable
member Value : int
--------------------
new : value:int -> NonSerializable
type CustomPicklerRegistry =
interface ICustomPicklerRegistry
new : unit -> CustomPicklerRegistry
member DeclareSerializable : unit -> unit
member DeclareSerializable : [<ParamArray>] typesToSerialize:Type [] -> unit
member DeclareSerializable : isSerializable:(Type -> bool) -> unit
member RegisterFactory : factory:(IPicklerResolver -> Pickler<'T>) -> unit
member RegisterPickler : pickler:Pickler -> unit
member RegisterPicklers : [<ParamArray>] picklers:Pickler [] -> unit
member IsGenerationStarted : bool
member PicklerFactories : Type []
...
--------------------
new : unit -> CustomPicklerRegistry
interface IPicklerResolver
private new : registry:ICustomPicklerRegistry -> PicklerCache
member GeneratePickler : unit -> Pickler<'T>
member GeneratePickler : t:Type -> Pickler
member IsPicklerGenerated : t:Type -> bool
member IsSerializableType : unit -> bool
member IsSerializableType : t:Type -> bool
member Registry : ICustomPicklerRegistry
static member FromCustomPicklerRegistry : registry:ICustomPicklerRegistry -> PicklerCache
static member Instance : PicklerCache
member CustomPicklerRegistry.DeclareSerializable : [<System.ParamArray>] typesToSerialize:System.Type [] -> unit
member CustomPicklerRegistry.DeclareSerializable : isSerializable:(System.Type -> bool) -> unit
static member FsPickler.Sift : value:'T * sifter:IObjectSifter * ?pickler:Pickler<'T> * ?streamingContext:System.Runtime.Serialization.StreamingContext -> Sifted<'T> * (int64 * obj) []
interface
abstract member Visit : Pickler<'T> * 'T -> bool
end
interface
abstract member CreateReader : Stream * Encoding * leaveOpen:bool -> IPickleFormatReader
abstract member CreateWriter : Stream * Encoding * leaveOpen:bool -> IPickleFormatWriter
abstract member DefaultEncoding : Encoding
abstract member Name : string
end
member BodyName : string
member Clone : unit -> obj
member CodePage : int
member DecoderFallback : DecoderFallback with get, set
member EncoderFallback : EncoderFallback with get, set
member EncodingName : string
member Equals : value:obj -> bool
member GetByteCount : chars:char[] -> int + 5 overloads
member GetBytes : chars:char[] -> byte[] + 7 overloads
member GetCharCount : bytes:byte[] -> int + 3 overloads
...
interface
inherit IDisposable
abstract member BeginReadObject : tag:string -> ObjectFlags
abstract member BeginReadRoot : tag:string -> unit
abstract member EndReadObject : unit -> unit
abstract member EndReadRoot : unit -> unit
abstract member ReadBigInteger : tag:string -> bigint
abstract member ReadBoolean : tag:string -> bool
abstract member ReadByte : tag:string -> byte
abstract member ReadBytes : tag:string -> byte []
abstract member ReadCachedObjectId : unit -> int64
...
end
interface
inherit IDisposable
abstract member BeginWriteObject : tag:string -> objectFlags:ObjectFlags -> unit
abstract member BeginWriteRoot : tag:string -> unit
abstract member EndWriteObject : unit -> unit
abstract member EndWriteRoot : unit -> unit
abstract member Flush : unit -> unit
abstract member WriteBigInteger : tag:string -> value:bigint -> unit
abstract member WriteBoolean : tag:string -> value:bool -> unit
abstract member WriteByte : tag:string -> value:byte -> unit
abstract member WriteBytes : tag:string -> value:byte [] -> unit
...
end