Tuesday, September 7, 2010

Runtime.gc() calls do make a difference

Hi YH,

You said calling Runtime.gc() is counter-productive. My experiment shows otherwise. Below is a heavy java job that uses about 200MB memory. "JJ_1001001=233564416" means after processing 1001001 rows, 233MB memory was in use. As you can see, usage grows with data volume.

[2010-09-07 14:03:07,132 INFO .Main:39] - {before_JJ=17944080, JJ_1=149162416, JJ_100101=161278496, JJ_200201=177440888, JJ_300301=179908768, JJ_400401=182329728, JJ_500501=198315560, JJ_600601=200645064, JJ_700701=216799944, JJ_800801=218142816, JJ_900901=231948296, JJ_1001001=233564416, before_destroySingletons=143855904}

-------- Now I added calls to Runtime.gc() after every 100,000 rows. Memory usage is now kept constant at around 120MB.

[2010-09-07 13:40:12,347 INFO .Main:39] - {before_JJ=17944080, JJ_1=120664072, JJ_100101=120664248, JJ_200201=120664072, JJ_300301=120664504, JJ_400401=120664176, JJ_500501=120664760, JJ_600601=120664584, JJ_700701=120664864, JJ_800801=120664840, JJ_900901=120665336, JJ_1001001=120665256, before_destroySingletons=131827256}

I run the job again. Results are similar, so GC behavior is not 100% predictable --

[2010-09-07 13:57:02,060 INFO .Main:39] - {before_JJ=17944080, JJ_1=120664224, JJ_100101=120663944, JJ_200201=120664528, JJ_300301=120664048, JJ_400401=120664632, JJ_500501=120664608, JJ_600601=120664888, JJ_700701=120664712, JJ_800801=120664992, JJ_900901=120665280, JJ_1001001=120665248, before_destroySingletons=131828080}

Here's how to request gc():

public static void runGC() {
// for whatever reason it helps to call Runtime.gc()
// using several method calls:
for (int r = 0; r < 4; ++r)
_runGC();
}

private static void _runGC() {
long usedMem1 = usedMemory(), usedMem2 = Long.MAX_VALUE;
for (int i = 0; (usedMem1 < usedMem2) && (i < 1000); ++i) {
s_runtime.runFinalization();
s_runtime.gc();
Thread.yield();
usedMem2 = usedMem1;
usedMem1 = usedMemory();
}
}

No comments:

Total Pageviews

my favorite topics (labels)

_fuxi (302) _misLabel (13) _orig? (3) _rm (2) _vague (2) clarified (58) cpp (39) cpp_const (22) cpp_real (76) cpp/java/c# (101) cppBig4 (54) cppSmartPtr (35) cppSTL (33) cppSTL_itr (27) cppSTL_real (26) cppTemplate (28) creditMkt (14) db (65) db_sybase (43) deepUnder (31) dotnet (20) ECN (27) econ/bank` (36) fin/sys_misc (43) finGreek (34) finReal (45) finRisk (30) finTechDesign (46) finTechMisc (32) finVol (66) FixedIncom (28) fMath (7) fMathOption (33) fMathStoch (67) forex (39) gr8IV_Q (46) GTD_skill (15) GUI_event (30) inMemDB (42) intuit_math (41) intuitFinance (57) javaMisc (68) javaServerSide (13) lambda/delegate (22) marketData (28) math (10) mathStat (55) memIssue (8) memMgmt (66) metaProgram` (6) OO_Design (84) original_content (749) polymorphic/vptr (40) productive (21) ptr/ref (48) py (28) reflect (8) script`/unix (82) socket/stream (39) subquery/join (30) subvert (13) swing/wpf (9) sysProgram` (16) thread (164) thread_CAS (15) thread_cpp (28) Thread* (22) timeSaver (80) transactional (23) tune (24) tuneDB (40) tuneLatency (30) z_ajax (9) z_algoDataStruct (41) z_arch (26) z_arch_job (27) z_automateTest (17) z_autoTrad` (19) z_bestPractice (39) z_bold (83) z_bondMath (35) z_book (18) z_boost (19) z_byRef^Val (32) z_c#GUI (43) z_c#misc (80) z_cast/convert (28) z_container (67) z_cStr/arr (39) z_Favorite* (8) z_FIX (15) z_forex (48) z_fwd_Deal (18) z_gz=job (33) z_gzBig20 (13) z_gzMgr (13) z_gzPain (20) z_gzThreat (19) z_hib (19) z_IDE (52) z_ikm (5) z_IR_misc (36) z_IRS (26) z_javaWeb (28) z_jdbc (10) z_jobFinTech (46) z_jobHunt (20) z_jobRealXp (10) z_jobStrength (15) z_jobUS^asia (27) z_letter (42) z_linq (10) z_memberHid` (11) z_MOM (54) z_nestedClass (5) z_oq (24) z_PCP (12) z_pearl (1) z_php (20) z_prodSupport (7) z_py (31) z_quant (14) z_regex (8) z_rv (38) z_skillist (48) z_slic`Problem (6) z_SOA (14) z_spring (25) z_src_code (8) z_swingMisc (50) z_swingTable (26) z_unpublish (2) z_VBA/Excel (8) z_windoz (17) z_wpfCommand (9)

About Me

New York (Time Square), NY, United States
http://www.linkedin.com/in/tanbin