Memory leak in the randomisation analyses

[GoogleCodeExporter]

The system consumes all the available memory over a series of iterations
(how fast depends on the size of the data set).  

Eventually it gets to the point that there is not enough memory to save the
data set or run any further analyses.  This occurs on WinXP, but is likely
to occur on all systems.

A workaround is to run the analysis for some number of iterations, save,
close and restart the application (this clears the memory), and then resume
the analysis.  Repeat solution as needed. 

The load_and_randomise_wrapper.pl script in the Biodiverse/bin folder has
been developed as a means of automating the close and restart problem but
will likely need modification for some problems.

Original issue reported on code.google.com by shawnlaffan on 25 Sep 2009 at 6:43

[GoogleCodeExporter]

There is no memory leak (at least not as identified using Devel::Cycle).  It is 
more
to do with the way Perl manages memory and can be reduced by more judicious use 
of
data structures.  This process is ongoing.  

Original comment by shawnlaffan on 18 Oct 2009 at 12:06

• Changed state: Started
• Added labels: Priority-Low
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Actually, there is a memory leak but it is not due to a reference cycle in the
objects.  It seems to occur after a data set is loaded from a bds file and a
randomisation is added, so something is keeping a reference outside the 
object's scope.

More hunting is needed.

Original comment by shawnlaffan on 25 Apr 2010 at 7:25

• Added labels: Priority-Medium
• Removed labels: Priority-Low

[GoogleCodeExporter]

Need to try Test::Weaken.
http://search.cpan.org/~jkegl/Test-Weaken-3.004000/lib/Test/Weaken.pm

Original comment by shawnlaffan on 20 Jun 2010 at 6:20

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

A workaround has been implemented for use until (and even after) we solve this 
issue.
See http://code.google.com/p/biodiverse/wiki/LoadAndRandomise

Original comment by shawnlaffan on 7 Jul 2010 at 3:20

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r949.

More leak hunting, but have found nothing.

Original comment by shawnlaffan on 12 Jun 2012 at 11:27

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1112.

Switching to Class::Inspector from Devel::Symdump.  
Might make some difference, but need to benchmark.  
Next is to test cloning as the source.  

Original comment by shawnlaffan on 28 Aug 2012 at 7:10

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1113.

Add script to run leaktraces.

Original comment by shawnlaffan on 28 Aug 2012 at 7:11

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1148.

Some musings.

Original comment by shawnlaffan on 22 Sep 2012 at 11:34

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Looks like this issue does not manifest under Strawberry Perl 5.16 (either of 
x32 and x64).  

Either perl has fixed itself, or something in the dependency chain was the 
cause and has been fixed (or both).

Leaving open for now, as we also need to test on Linux.  

Original comment by shawnlaffan on 10 Dec 2012 at 5:18

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Possibly it was fixed by this:  
http://repo.or.cz/w/Text-CSV_XS.git/commit/0cee5f2cbe2e83a

Original comment by shawnlaffan on 24 Dec 2012 at 9:19

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

No, it is still there.  
Strawberry 5.16.3 x64.

Original comment by shawnlaffan on 22 Jun 2013 at 4:42

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1653.

Add config parameter to control the progress dialogue.  
After turning it off it looks like the leakage is coming from either 
Biodiverse::GUI::ProgressDialog or from Gtk.  This is because there is no 
increased use of memory with no progress dialogs being shown.  

Original comment by shawnlaffan on 15 Jul 2013 at 3:20

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Looks like Gtk are on it.  

https://bugzilla.gnome.org/show_bug.cgi?id=685959 (marked as fixed) and 
https://bugzilla.gnome.org/show_bug.cgi?id=671538

Original comment by shawnlaffan on 15 Jul 2013 at 3:28

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1659.

Make sure the config parameter is boolean.  

Original comment by shawnlaffan on 15 Jul 2013 at 5:15

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1662.

Store the text and progress widgets directly.  
It doesn't fix the issue, but should at least speed things up a touch since we 
don't repeatedly call dlgxml->get_widget.

Original comment by shawnlaffan on 15 Jul 2013 at 9:00

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1663.

Add test script to hunt leaks.  
Memory usage increases as the number of progress dialogs increases.
It is stable if only one progress dialog is used, no matter for how long.  
The next approach is to not use the Glade file progress dialog, and generate 
new ones directly.  There is much leakage under the current approach.  

Original comment by shawnlaffan on 15 Jul 2013 at 10:21

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

It seems a reasonable workaround would be to reuse the same dialog and
show/hide it. (Cache the Gtk2::GladeXML instance in a global in new and hide 
the dialog
instead of destroying it in destroy).
Under Linux the dialog seems to grow vertically slightly each time, but
looks fine on Windows.
In both cases the memory use is steady.

Original comment by [email protected] on 15 Jul 2013 at 11:51

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

The issue with hiding is that we often need more than one progress dialog at a 
time.  Updating a single dialog will hide the higher level progress.  

Keeping a single dialog overall is a good idea, though, and is already logged 
as issue #295 .  In this approach we have a single window and add/remove 
sub-widgets as needed.  The dialog can be hidden when there are no more widgets 
to update.  

WRT recreating each time for now, the widget is not that complex to create - it 
is just a vbox in which one packs text and progress bar widgets.  

Original comment by shawnlaffan on 15 Jul 2013 at 10:40

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1666.

Ahab almost has his whale. 
Replace the GladeXML progress dialog with one rolled by hand.  
Memory usage no longer increases anywhere near the same rate as before.
A remaining issue is that closing the dialog does not cancel the operation.  
This needs to be looked into.  

Original comment by shawnlaffan on 16 Jul 2013 at 12:19

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1667.

Add signal handler so we destroy the progress widgets when the window is 
closed.  This then triggers the analysis cancellation.

Original comment by shawnlaffan on 16 Jul 2013 at 12:47

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1669.

Merge changes into the trunk. 

Original comment by shawnlaffan on 16 Jul 2013 at 1:16

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1670.

Remove some debugging code that causes croakage.

Original comment by shawnlaffan on 16 Jul 2013 at 1:21

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1671.

Merge latest change into trunk.  

Original comment by shawnlaffan on 16 Jul 2013 at 1:22

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Add milestone for 0.18

Original comment by shawnlaffan on 16 Jul 2013 at 10:21

• Added labels: Milestone-Release0.18
• Removed labels: ****

[GoogleCodeExporter]

At long last this issue can be marked as fixed.

Original comment by shawnlaffan on 18 Jul 2013 at 3:47

• Changed state: Fixed
• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1688.

Add a test file (even though the issue is closed).  

Original comment by shawnlaffan on 18 Jul 2013 at 4:03

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

This issue was updated by revision r1769.

Delete branch associated with this issue.  

Original comment by shawnlaffan on 19 Aug 2013 at 10:30

• Added labels: ****
• Removed labels: ****