Saturday, February 26, 2011

avoid convenience high-level wrappers; program against low-level APIs

(another blog post.)

Many examples of the same observation -- convenient high-level wrappers insulate/shield a developer from the (all-important) underlying API. Consequently, he can't demonstrate his experience and mastery in that particular "real" technology -- he only knows how to use the convenience wrappers.

example -- servlet -- in one of my GS trading desks, all web apps were written atop an in-house framework "Panels", which was invented after servlet but before JSP, so it offers many features similar or in addition to JSP. After years of programming in Panels, you would remember nothing at all about the underlying servlet API, because that is successfully encapsulated. Only the Panels API authors work against the underlying servlet API. Excellent example of encapsulation, standardization, layering and division of labor.

Note the paradox -- guys who wrote those wrappers are considered (by colleagues and job interviewers) as stronger knowledge experts than the users of the wrappers. So, everyone agrees knowledge of the underlying technology is more valuable.

example -- Gemfire -- one of my trading engines was heavy on Gemfire. The first things we created was a bunch of wrapper classes to standardize how to access gemfire. Everyone must use the wrappers. If you are NOT the developer of those wrappers, then you won't know much about gemfire API even after 12 months of development. Interviewers will see that you have close to zero gemfire experience -- unfair!

example -- sockets -- i feel almost everywhere sockets are used, they are used via wrappers. Boost, ACE, java, c#, python, perl all offer wrappers over sockets. However, socket API is critical and a favorite interview topic. We had better go beyond those wrappers and experiment with underlying sockets. I recently had to describe my network programming experience. I talked about using perl telnet module, which is a wrapper over wrapper over sockets. I feel interviewer was looking for tcp/udp programming experience. But for that system requirement, perl telnet module was the right choice. That right choice is poor choice for knowledge-gain.

example -- thread-pool -- One of my ML colleagues told me (proudly) that he once wrote his own thread pool. I asked why not use the JDK and he said his is simple and flexible -- infinitely customizable. That experience definitely gave him confidence to talk about thread pools in job interviews.

If you ask me when a custom thread pool is needed, I can only hypothesize that the JDK pools are general-purpose, not optimized for the fastest systems. Fastest systems are often FPGA or hardware based but too costly. As a middle ground, I'd say you can create your customized threading tools including a thread pool.

example -- Boost::thread -- In C++ threading interviews, I often say I used a popular threading toolkit -- Boost::thread. However, interviewers repeatedly asked about how boost::thread authors implement certain features. If you use boost in your project, you won't bother to find out how those tools are internally implemented. That's the whole point of using boost. Boost is a cross-platform toolkit (more than a wrapper) and relies on things like win32 threads or pthreads. Many interviewers asked about pthreads. If you use a high-level threading toolkit, you avoid the low-level pthreads details..

example -- Hibernate -- If you rely on hibernate and HQL to generate native SQL queries and manage transaction and identities, then you lose touch with the intricate multitable join, index-selection, transaction and identity column issues. Won't do you good for interviews.

example -- RV -- one of my tibco RV projects used a simple and nice wrapper package, so all our java classes don't need to deal with tibrv objects (like transport, subject, event, rvd....). However, interviewers will definitely ask about those tibrv objects and how they interact -- non-trivial.

example -- JMS -- in GS, the JMS infrastructure (Tibco EMS) was wrapped under an industrial-strength GS firmwide "service". No applications are allowed to access EMS directly. As a result, I feel many (experienced) developers don't know JMS api details. For example, they don't use onMessage() much and never browse a queue without removing messages from it. We were so busy just to get things done, put out the fires and go home before midnight, so who has time to read what he never needs to know (unless he plans to quit this company)?

example -- spring jms -- I struggled against this tool in a project. I feel it is a thin wrapper over JMS but still obscures a lot of important JMS features. I remember stepping through the code to track down the call that started the underlying JMS server and client sessions, and i found it hidden deeply and not well-documented. If you use spring jms you would still need some JMS api but not in its entirety.

example -- xml parsing -- After many projects, I still prefer SAX and DOM parsers rather than the high-level wrappers. Interviewers love DOM/SAX.

example -- wait/notify -- i used wait/notify many times, but very few developers do the same because most would choose a high-level concurrency toolkit. (As far as I know, all threading toolkits must call wait/notify internally.) As a result, they don't know wait/notify very well but luckly, wait/notify are not complicated as the other APIs mentioned earlier.

example -- FIX -- one of my FIX projects uses a rather elaborate wrapper package to hide the complexity of FIX protocol. Interviewers will ask about FIX, but I don't know any from that project because all FIX details were hidden from me.

example -- raw arrays -- many c++ authors say you should use vector and String (powerful, standard wrappers) to replace the low-level raw arrays. However, in the real world, many legacy apps still use raw arrays. If you avoid raw arrays, you have no skill dealing with them in interviews or projects. Raw arrays integrate tightly with pointers and can be extremely extremely tricky. Half of C/C++ complexities step from pointers.

example -- DB tuning -- when Oracle 10 came out, my DBA friend told me the toughest DBA job is now easier because oracle 10 features excellent self-tuning. I still doubt that because query tuning is always a developer's job and the server software has limited insight. Even instance tuning can be very complicated so DBA hands-on knowledge is probably irreplaceable. If you naively believe in Oracle marketing and rely on self-tuning, you will soon find yourself falling behind other DBAs who look hard into tuning issues.

No comments:

Total Pageviews

my favorite topics (labels)

_fuxi (302) _misLabel (13) _orig? (3) _rm (2) _vague (2) clarified (58) cpp (39) cpp_const (22) cpp_real (76) cpp/java/c# (101) cppBig4 (54) cppSmartPtr (35) cppSTL (33) cppSTL_itr (27) cppSTL_real (26) cppTemplate (28) creditMkt (14) db (65) db_sybase (43) deepUnder (31) dotnet (20) ECN (27) econ/bank` (36) fin/sys_misc (43) finGreek (34) finReal (45) finRisk (30) finTechDesign (46) finTechMisc (32) finVol (66) FixedIncom (28) fMath (7) fMathOption (33) fMathStoch (67) forex (39) gr8IV_Q (46) GTD_skill (15) GUI_event (30) inMemDB (42) intuit_math (41) intuitFinance (57) javaMisc (68) javaServerSide (13) lambda/delegate (22) marketData (28) math (10) mathStat (55) memIssue (8) memMgmt (66) metaProgram` (6) OO_Design (84) original_content (749) polymorphic/vptr (40) productive (21) ptr/ref (48) py (28) reflect (8) script`/unix (82) socket/stream (39) subquery/join (30) subvert (13) swing/wpf (9) sysProgram` (16) thread (164) thread_CAS (15) thread_cpp (28) Thread* (22) timeSaver (80) transactional (23) tune (24) tuneDB (40) tuneLatency (30) z_ajax (9) z_algoDataStruct (41) z_arch (26) z_arch_job (27) z_automateTest (17) z_autoTrad` (19) z_bestPractice (39) z_bold (83) z_bondMath (35) z_book (18) z_boost (19) z_byRef^Val (32) z_c#GUI (43) z_c#misc (80) z_cast/convert (28) z_container (67) z_cStr/arr (39) z_Favorite* (8) z_FIX (15) z_forex (48) z_fwd_Deal (18) z_gz=job (33) z_gzBig20 (13) z_gzMgr (13) z_gzPain (20) z_gzThreat (19) z_hib (19) z_IDE (52) z_ikm (5) z_IR_misc (36) z_IRS (26) z_javaWeb (28) z_jdbc (10) z_jobFinTech (46) z_jobHunt (20) z_jobRealXp (10) z_jobStrength (15) z_jobUS^asia (27) z_letter (42) z_linq (10) z_memberHid` (11) z_MOM (54) z_nestedClass (5) z_oq (24) z_PCP (12) z_pearl (1) z_php (20) z_prodSupport (7) z_py (31) z_quant (14) z_regex (8) z_rv (38) z_skillist (48) z_slic`Problem (6) z_SOA (14) z_spring (25) z_src_code (8) z_swingMisc (50) z_swingTable (26) z_unpublish (2) z_VBA/Excel (8) z_windoz (17) z_wpfCommand (9)

About Me

New York (Time Square), NY, United States