Monday, November 29, 2010

SecDB - dependency-graph database

SecDB is an in-memory object database, just like gemfire. But Gemfire doesn't have graphs; only standalone hash tables. Gemfire is credated by OODB (object-oriented database) veterans.

SecDB runs (in-memory) 24/7 with mirror sites and disk-persistence. No early morning cache loading like many trading systems do.

SecDB is, presumably, a master/slave database accessed over the network, just like a regular RDBMS.

This is internal low-level design so not many people would confirm, so here's my personal speculation. For maximum flexibility SecDB probably uses dynamic object types in the python sense, not the static types in java. (Perl and javascript OO is also based on the generic hash). GS says SecDB is object-oriented, but I guess it's not strongly typed. Perhaps less like struct-with-methods but more like python's dictionary-based objects.

1) Here's the Best way to start understanding secDB -
    Assumption 1 - we can build most FI trading apps using java.

SecDB is a schemaless graph database, like a huge xml. Each node represents a domain object, such as Trade, Position/Deal, Price, Product, a series of market data...

    Assumption 2 - operations are mostly CRUD i.e. create-read-update-delete.
    Assumption 2a - Most CRUD business logic are doable in java without RDBMS, using graph-walk.

SecDB can therefore implement the same logic using graph-walk.

2) Technical comparison against java + rdbms.
Data access is index lookup in RDBMS but SecDB requires deep graph-walk. Occassionally, in-memory indexing can improve performance. A gemfire index is a auxiliary hash table, so secDB may be similar.

3) functional comparison against java + rdbms.
Strength -- time-to-market with a new "instrument". Actually, "Product" is the buzzword used in GS. Using rdbms, we need table changes, application changes, impact analygis and regression test -- rigidity.

Weakness -- fast-changing data. SecDB is great for relatively static data. To update a tree node repeatedly, we probably need to walk the tree every time. But is disk-based rdbms any faster?
Weakness -- equity -- latency-sensitive. (But somehow JPM and BofA are using graph-DB for FX risk mgmt.) To address performance hit of graph-walk, GS throws in massive amount of hardware. GSS risk uses a compute-farm aka grid.

Strength -- PnL Rollup is naturally a tree algorithm.
Strength -- what-if scenario analysis exploits spreadsheet-style chain reaction
Strength -- risk stress testing or scenario testing.     Risk is the most important user of SecDB.
Strength -- UAT datatabase is the prod DB but in RO mode
Strength -- real time pricing an instrument that depends on many other "objects". You can give a what-if tweak to one of those objects and see the spreadsheet-style ripple effect. I think it's because all these objects are in memory and listening to each other.

No comments:

Total Pageviews

my favorite topics (labels)

_fuxi (302) _misLabel (13) _orig? (3) _rm (2) _vague (2) clarified (58) cpp (39) cpp_const (22) cpp_real (76) cpp/java/c# (101) cppBig4 (54) cppSmartPtr (35) cppSTL (33) cppSTL_itr (27) cppSTL_real (26) cppTemplate (28) creditMkt (14) db (65) db_sybase (43) deepUnder (31) dotnet (20) ECN (27) econ/bank` (36) fin/sys_misc (43) finGreek (34) finReal (45) finRisk (30) finTechDesign (46) finTechMisc (32) finVol (66) FixedIncom (28) fMath (7) fMathOption (33) fMathStoch (67) forex (39) gr8IV_Q (46) GTD_skill (15) GUI_event (30) inMemDB (42) intuit_math (41) intuitFinance (57) javaMisc (68) javaServerSide (13) lambda/delegate (22) marketData (28) math (10) mathStat (55) memIssue (8) memMgmt (66) metaProgram` (6) OO_Design (84) original_content (749) polymorphic/vptr (40) productive (21) ptr/ref (48) py (28) reflect (8) script`/unix (82) socket/stream (39) subquery/join (30) subvert (13) swing/wpf (9) sysProgram` (16) thread (164) thread_CAS (15) thread_cpp (28) Thread* (22) timeSaver (80) transactional (23) tune (24) tuneDB (40) tuneLatency (30) z_ajax (9) z_algoDataStruct (41) z_arch (26) z_arch_job (27) z_automateTest (17) z_autoTrad` (19) z_bestPractice (39) z_bold (83) z_bondMath (35) z_book (18) z_boost (19) z_byRef^Val (32) z_c#GUI (43) z_c#misc (80) z_cast/convert (28) z_container (67) z_cStr/arr (39) z_Favorite* (8) z_FIX (15) z_forex (48) z_fwd_Deal (18) z_gz=job (33) z_gzBig20 (13) z_gzMgr (13) z_gzPain (20) z_gzThreat (19) z_hib (19) z_IDE (52) z_ikm (5) z_IR_misc (36) z_IRS (26) z_javaWeb (28) z_jdbc (10) z_jobFinTech (46) z_jobHunt (20) z_jobRealXp (10) z_jobStrength (15) z_jobUS^asia (27) z_letter (42) z_linq (10) z_memberHid` (11) z_MOM (54) z_nestedClass (5) z_oq (24) z_PCP (12) z_pearl (1) z_php (20) z_prodSupport (7) z_py (31) z_quant (14) z_regex (8) z_rv (38) z_skillist (48) z_slic`Problem (6) z_SOA (14) z_spring (25) z_src_code (8) z_swingMisc (50) z_swingTable (26) z_unpublish (2) z_VBA/Excel (8) z_windoz (17) z_wpfCommand (9)

About Me

New York (Time Square), NY, United States
http://www.linkedin.com/in/tanbin