Sunday, October 16, 2011

IV questions/answers on trading system testing

See post on biz-scenario [S] ^ implementation error [I].

More and more interviewers seem to ask increasingly serious questions about trading system testing. I feel that's part of our professional growth towards senior positions (though many Wall St lead developers I know aren't big fans). In some cases, the higher you are paid, the deeper testing experience is expected of you.

Testing skill is a complement to design skill, both of which become increasingly important as we move up. These Interview questions tend to be open-ended. Interviewer gives us a chance to sell and convince them that we do have real experience and personal insight. Here are some pointers for myself. Your inputs welcome.
--At the Tools level, we can easily talk about Fitnesse, Mockito, JUnit, DbUnit etc [I]. Easy to read up about these. Easy to tell real stories. Nothing unique about trading systems.
--Another level of easy topics - regression testing, integrated testing. Again nothing unique about trading system. Since i have personal experience [S] i usually spend a lot of time at this level.
* scenario testing -- essential in larger systems. Without scenarios, if you ask a rookie to list all the combination of various inputs, the list is endless and useless. Instead, I use assertions to reduce the combination.
* matrix testing -- i often draw up some spreadsheet with rows and columns of input values.
* production data -- essential if we were to involve users. The earlier we involve users, the better.
* I use JUnit to drive integrated tests with real DB, real MOM, real cache servers, real web servers

--At another level, special to trading systems are load testing, performance testing, thread testing, MOM testing. MOM and esp. thread testing uses special techniques. But in reality these are seldom adopted. An interesting question I like to ask is "at what specific (number please) load level will my system crash?" It's good if we can predict that magic number. If we can't then we might get caught with our pants down, which is quite common. I feel trading system testing can be divided into logic testing and performance testing. In reality, most small enhancements need only logic testing.
--Perhaps the highest level is UAT [S]. I would mention Fitnesse. I would make it clear that in GS, users or BA are called upon to generate test cases.
--At a methodology level, a tougher question is something like "How do you ensure adequate testing before release?" or "How do you decide how much testing is enough?". Since I never used test coverage [I] tools, I won't mention it. Frankly, I don't believe 80% test coverage means the code is more reliable than 0% test coverage
-- The most productive GS team members do no automated testing at all. They rely on experience to know what scenarios are worth testing, and then test those manually, without automation. When we realize bugs are released to production, in reality we almost always find our test scenarios incomplete. Automated tests won't help us. But I can't really say these things in interviews -- emperor's new dress.

No comments:

Total Pageviews

my favorite topics (labels)

_fuxi (302) _misLabel (13) _orig? (3) _rm (2) _vague (2) clarified (58) cpp (39) cpp_const (22) cpp_real (76) cpp/java/c# (101) cppBig4 (54) cppSmartPtr (35) cppSTL (33) cppSTL_itr (27) cppSTL_real (26) cppTemplate (28) creditMkt (14) db (65) db_sybase (43) deepUnder (31) dotnet (20) ECN (27) econ/bank` (36) fin/sys_misc (43) finGreek (34) finReal (45) finRisk (30) finTechDesign (46) finTechMisc (32) finVol (66) FixedIncom (28) fMath (7) fMathOption (33) fMathStoch (67) forex (39) gr8IV_Q (46) GTD_skill (15) GUI_event (30) inMemDB (42) intuit_math (41) intuitFinance (57) javaMisc (68) javaServerSide (13) lambda/delegate (22) marketData (28) math (10) mathStat (55) memIssue (8) memMgmt (66) metaProgram` (6) OO_Design (84) original_content (749) polymorphic/vptr (40) productive (21) ptr/ref (48) py (28) reflect (8) script`/unix (82) socket/stream (39) subquery/join (30) subvert (13) swing/wpf (9) sysProgram` (16) thread (164) thread_CAS (15) thread_cpp (28) Thread* (22) timeSaver (80) transactional (23) tune (24) tuneDB (40) tuneLatency (30) z_ajax (9) z_algoDataStruct (41) z_arch (26) z_arch_job (27) z_automateTest (17) z_autoTrad` (19) z_bestPractice (39) z_bold (83) z_bondMath (35) z_book (18) z_boost (19) z_byRef^Val (32) z_c#GUI (43) z_c#misc (80) z_cast/convert (28) z_container (67) z_cStr/arr (39) z_Favorite* (8) z_FIX (15) z_forex (48) z_fwd_Deal (18) z_gz=job (33) z_gzBig20 (13) z_gzMgr (13) z_gzPain (20) z_gzThreat (19) z_hib (19) z_IDE (52) z_ikm (5) z_IR_misc (36) z_IRS (26) z_javaWeb (28) z_jdbc (10) z_jobFinTech (46) z_jobHunt (20) z_jobRealXp (10) z_jobStrength (15) z_jobUS^asia (27) z_letter (42) z_linq (10) z_memberHid` (11) z_MOM (54) z_nestedClass (5) z_oq (24) z_PCP (12) z_pearl (1) z_php (20) z_prodSupport (7) z_py (31) z_quant (14) z_regex (8) z_rv (38) z_skillist (48) z_slic`Problem (6) z_SOA (14) z_spring (25) z_src_code (8) z_swingMisc (50) z_swingTable (26) z_unpublish (2) z_VBA/Excel (8) z_windoz (17) z_wpfCommand (9)

About Me

New York (Time Square), NY, United States