Monday, June 9, 2014

linear correlation - minefield

If 2 "thingies" A and B have a (linear) correlation of 0.3 (rather low), we can easily interpret it incorrectly.

* If A/B have a physical non-linear relationship, they are not independent but corr will be low. Corr coeff measures Linear relationship only. Stat risk has a lot to say about this.
* if sample size is small, then the calculated corr may not reflect the true population corr
** if we take many, many large samples, the true corr would emerge.
* As [[[Prem Mann]] points out, there may not be a causal relationship between A and B even if corr is high.

The most common confusion in my mind is the context. The 0.3 corr is typically calculated from a sample, but when folks say A and B are weakly correlated, they usually refer to the population. They say things like when A increases , we are likely to see B increase too. We automatically assume some causality.

Due to the factors mentioned above, an observed low corr often doesn't mean A/B's independence in the population (a healthy blood pressure reading doesn't prove complete health). However, an observed strong corr often represents a good evidence of real corr within the population, provided the sample is statistically significant. Here's a classic example. Your friend stares at your head from behind. In each experiment, she either stares (1) or looks away (0). You guess.

A = the actual 1/0
B = your guessed 1/0

For a small sample of 10, you may see strong correlation between A and B. You feel you could sense the stare from behind. In a large sample, corr is probably 0.

Understanding the concept of corr is simpler within natural processes -- we could repeat experiments to infer the population corr. However in most economic problems the A/B thingies are influenced by human decisions. So it's harder to "manage". The population corr may change over time, or change with gender, or change with nationality. For example, if we take 2 samples, then sample 1 may be from a population with mostly young Asian girls, and sample2 may be from a different population. Then unknown to us, the first population's corr could differ from the 2nd population's.

No comments:

Total Pageviews

my favorite topics (labels)

_fuxi (302) _misLabel (13) _orig? (3) _rm (2) _vague (2) clarified (58) cpp (39) cpp_const (22) cpp_real (76) cpp/java/c# (101) cppBig4 (54) cppSmartPtr (35) cppSTL (33) cppSTL_itr (27) cppSTL_real (26) cppTemplate (28) creditMkt (14) db (65) db_sybase (43) deepUnder (31) dotnet (20) ECN (27) econ/bank` (36) fin/sys_misc (43) finGreek (34) finReal (45) finRisk (30) finTechDesign (46) finTechMisc (32) finVol (66) FixedIncom (28) fMath (7) fMathOption (33) fMathStoch (67) forex (39) gr8IV_Q (46) GTD_skill (15) GUI_event (30) inMemDB (42) intuit_math (41) intuitFinance (57) javaMisc (68) javaServerSide (13) lambda/delegate (22) marketData (28) math (10) mathStat (55) memIssue (8) memMgmt (66) metaProgram` (6) OO_Design (84) original_content (749) polymorphic/vptr (40) productive (21) ptr/ref (48) py (28) reflect (8) script`/unix (82) socket/stream (39) subquery/join (30) subvert (13) swing/wpf (9) sysProgram` (16) thread (164) thread_CAS (15) thread_cpp (28) Thread* (22) timeSaver (80) transactional (23) tune (24) tuneDB (40) tuneLatency (30) z_ajax (9) z_algoDataStruct (41) z_arch (26) z_arch_job (27) z_automateTest (17) z_autoTrad` (19) z_bestPractice (39) z_bold (83) z_bondMath (35) z_book (18) z_boost (19) z_byRef^Val (32) z_c#GUI (43) z_c#misc (80) z_cast/convert (28) z_container (67) z_cStr/arr (39) z_Favorite* (8) z_FIX (15) z_forex (48) z_fwd_Deal (18) z_gz=job (33) z_gzBig20 (13) z_gzMgr (13) z_gzPain (20) z_gzThreat (19) z_hib (19) z_IDE (52) z_ikm (5) z_IR_misc (36) z_IRS (26) z_javaWeb (28) z_jdbc (10) z_jobFinTech (46) z_jobHunt (20) z_jobRealXp (10) z_jobStrength (15) z_jobUS^asia (27) z_letter (42) z_linq (10) z_memberHid` (11) z_MOM (54) z_nestedClass (5) z_oq (24) z_PCP (12) z_pearl (1) z_php (20) z_prodSupport (7) z_py (31) z_quant (14) z_regex (8) z_rv (38) z_skillist (48) z_slic`Problem (6) z_SOA (14) z_spring (25) z_src_code (8) z_swingMisc (50) z_swingTable (26) z_unpublish (2) z_VBA/Excel (8) z_windoz (17) z_wpfCommand (9)

About Me

New York (Time Square), NY, United States
http://www.linkedin.com/in/tanbin