flatbuffers alternatives and similar packages
Based on the "Data" category.
Alternatively, view flatbuffers alternatives based on common mentions on social networks and blogs.
-
compendium-client
DISCONTINUED. Mu (μ) is a purely functional framework for building micro services. -
streaming
An optimized general monad transformer for streaming applications, with a simple prelude of functions
CodeRabbit: AI Code Reviews for Developers
Do you think we are missing an alternative of flatbuffers or a related project?
README
Haskell Flatbuffers
An implementation of the flatbuffers protocol in Haskell.
Getting started
Start off by writing a flatbuffers schema with the data structures you want to serialize/deserialize.
namespace Data.Game; table Monster { name: string; hp: int; locations: [string] (required); }
Create a Haskell module named after the namespace in the schema.
module Data.Game where
Use
mkFlatBuffers
to generate constructors and accessors for the data types in your schema.{-# LANGUAGE TemplateHaskell #-} module Data.Game where import FlatBuffers $(mkFlatBuffers "schemas/game.fbs" defaultOptions)
The following declarations will be generated for you.
data Monster -- Constructor monster :: Maybe Text -> Maybe Int32 -> WriteVector Text -> WriteTable Monster -- Accessors monsterName :: Table Monster -> Either ReadError (Maybe Text) monsterHp :: Table Monster -> Either ReadError Int32 monsterLocations :: Table Monster -> Either ReadError (Vector Text)
We can now construct a flatbuffer using encode
and read it using decode
:
{-# LANGUAGE OverloadedStrings #-}
import Data.ByteString.Lazy (ByteString)
import FlatBuffers
import qualified FlatBuffers.Vector as Vector
-- Writing
byteString = encode $
monster
(Just "Poring")
(Just 50)
(Vector.fromList 2 ["Prontera Field", "Payon Forest"])
-- Reading
readMonster :: ByteString -> Either ReadError String
readMonster byteString = do
someMonster <- decode byteString
name <- monsterName someMonster
hp <- monsterHp someMonster
locations <- monsterLocations someMonster >>= Vector.toList
Right ("Monster: " <> show name <> " (" <> show hp <> " HP) can be found in " <> show locations)
For the rest of this document, we'll assume these imports/extensions are enabled:
{-# LANGUAGE OverloadedStrings #-}
import Data.ByteString.Lazy (ByteString)
import Data.Text (Text)
import qualified Data.Text as Text
import FlatBuffers
import qualified FlatBuffers.Vector as Vector
Codegen
You can check exactly which declarations were generated by browsing your module in ghci:
λ> :m Data.Game FlatBuffers FlatBuffers.Vector
λ> :browse Data.Game
data Monster
monster :: Maybe Int32 -> WriteTable Monster
monsterHp :: Table Monster -> Either ReadError Int32
Or by launching a local hoogle server with Stack:
> stack hoogle --rebuild --server
There are lots of examples in the test/Examples folder and the THSpec
module.
In particular, test/Examples/schema.fbs
and test/Examples/vector_of_unions.fbs
contain a variety of data structures and Examples.HandWritten
demonstrates what the code generated by mkFlatBuffers
would look like.
Enums
enum Color: short {
Red, Green, Blue
}
Given the enum declarationa above, the following code will be generated:
data Color
= ColorRed
| ColorGreen
| ColorBlue
deriving (Eq, Show, Read, Ord, Bounded)
toColor :: Int16 -> Maybe Color
fromColor :: Color -> Int16
colorName :: Color -> Text
Usage:
table Monster {
color: Color;
}
data Monster
monster :: Maybe Int16 -> WriteTable Monster
monsterColor :: Table Monster -> Either ReadError Int16
-- Writing
byteString = encode $
monster (Just (fromColor ColorBlue))
-- Reading
readMonster :: ByteString -> Either ReadError Text
readMonster byteString = do
someMonster <- decode byteString
i <- monsterColor someMonster
case toColor i of
Just color -> Right ("This monster is " <> colorName color)
Nothing -> Left ("Unknown color: " <> show i) -- Forwards compatibility
Bit flags / Bitmasks
enum Colors: uint16 (bit_flags) {
Red, Green, Blue
}
Given the enum declarationa above, the following code will be generated:
colorsRed, colorsGreen, colorsBlue :: Word16
colorsRed = 1
colorsGreen = 2
colorsBlue = 4
allColors :: [Word16]
colorsNames :: Word16 -> [Text]
Usage:
table Monster {
colors: Colors = "Red Blue";
}
data Monster
monster :: Maybe Word16 -> WriteTable Monster
monsterColors :: Table Monster -> Either ReadError Word16
import Control.Monad.Except (MonadError, MonadIO, liftEither, liftIO)
import Data.Bits ((.|.), (.&.))
import qualified Data.Text.IO as Text
-- Writing
byteString = encode $
monster (Just (colorsBlue .|. colorsGreen))
-- Reading
readMonster :: (MonadIO m, MonadError ReadError m) => ByteString -> m ()
readMonster byteString = do
someMonster <- liftEither $ decode byteString
colors <- liftEither $ monsterColors someMonster
let isRed = colors .&. colorsRed /= 0
liftIO $ putStrLn $ "Is this monster red? " <> if isRed then "Yes" else "No"
liftIO $ Text.putStrLn $ "Monster colors: " <> Text.intercalate ", " (colorsNames colors)
Structs
struct Coord {
x: long;
y: long;
}
Given the struct declaration above, the following code will be generated:
data Coord
instance IsStruct Coord
-- Constructor
coord :: Int64 -> Int64 -> WriteStruct Coord
-- Accessors
coordX :: Struct Coord -> Either ReadError Int64
coordY :: Struct Coord -> Either ReadError Int64
Usage:
table Monster {
position: Coord (required);
}
data Monster
monster :: WriteStruct Coord -> WriteTable Monster
monsterPosition :: Table Monster -> Either ReadError (Struct Coord)
-- Writing
byteString = encode $
monster (coord 123 456)
-- Reading
readMonster :: ByteString -> Either ReadError String
readMonster byteString = do
someMonster <- decode byteString
pos <- monsterPosition someMonster
x <- coordX pos
y <- coordY pos
Right ("Monster is located at " <> show x <> ", " <> show y)
Unions
table Sword { power: int; }
table Axe { power: int; }
union Weapon { Sword, Axe }
Given the union declaration above, the following code will be generated:
-- Accessors
data Weapon
= WeaponSword !(Table Sword)
| WeaponAxe !(Table Axe)
-- Constructors
weaponSword :: WriteTable Sword -> WriteUnion Weapon
weaponAxe :: WriteTable Axe -> WriteUnion Weapon
Usage:
table Character {
weapon: Weapon;
}
data Character
character :: WriteUnion Weapon -> WriteTable Character
characterWeapon :: Table Character -> Either ReadError (Union Weapon)
-- Writing
byteString = encode $
character
(weaponSword (sword (Just 1000)))
-- Reading
readCharacter :: ByteString -> Either ReadError String
readCharacter byteString = do
someCharacter <- decode byteString
weapon <- characterWeapon someCharacter
case weapon of
Union (WeaponSword sword) -> do
power <- swordPower sword
Right ("Weilding a sword with " <> show power <> " Power.")
Union (WeaponAxe axe) -> do
power <- axePower axe
Right ("Weilding an axe with " <> show power <> " Power.")
UnionNone -> Right "Character has no weapon"
UnionUnknown byte -> Left "Unknown weapon" -- Forwards compatibility
Note that, like in the official FlatBuffers implementation, unions are always optional.
Adding the required
attribute to a union field has no effect.
To create a character with no weapon, use none :: WriteUnion a
byteString = encode $
character none
File Identifiers
From "File identification and extension":
Typically, a FlatBuffer binary buffer is not self-describing, i.e. it needs you to know its schema to parse it correctly. But if you want to use a FlatBuffer as a file format, it would be convenient to be able to have a "magic number" in there, like most file formats have, to be able to do a sanity check to see if you're reading the kind of file you're expecting.
Now, you can always prefix a FlatBuffer with your own file header, but FlatBuffers has a built-in way to add an identifier to a FlatBuffer that takes up minimal space, and keeps the buffer compatible with buffers that don't have such an identifier.
table Monster { name: string; }
root_type Monster;
file_identifier "MONS";
data Monster
instance HasFileIdentifier Monster
-- Usual constructor and accessors...
We can now construct a flatbuffer using encodeWithFileIdentifier
and use checkFileIdentifier
to check if it's safe to decode it to a specific type:
{-# LANGUAGE TypeApplications #-}
-- Writing
byteString = encodeWithFileIdentifier $
monster (Just "Poring")
-- Reading
readName :: ByteString -> Either ReadError (Maybe Text)
readName byteString = do
if checkFileIdentifier @Monster byteString then do
someMonster <- decode byteString
monsterName someMonster
else if checkFileIdentifier @Character byteString then do
someCharacter <- decode byteString
characterName someCharacter
else
Left "Unexpected flatbuffer identifier"
TODO
Features
- [ ] gRPC support
- [ ] Size-prefixed buffers (needed for streaming multiple messages)
- [ ] Fixed length arrays in structs
- [ ] unions of strings / structs
- [ ]
key
attribute (See "Storing dictionaries in a FlatBuffer" section) - [ ]
nested_flatbuffer
attribute - [ ]
hash
attribute - [ ] DSL that allows sharing of data (e.g. reuse an offset to a string/table)
- [ ]
shared
attribute
Other
- [ ] TH: sort table fields by size + support
original_order
attribute - [ ] Enrich
Vector
API: drop, take, null, folds, sum, elem, for_, traverse_, ideally support most of operations inData.Foldable
- [ ] Improve error messages during
SemanticAnalysis
stage, provide source code location - [ ] Try alternative bytestring builders:
fast-builder
,blaze-builder
- [ ] Try alternative bytestring parsers:
cereal