473,465 Members | 1,991 Online
Bytes | Software Development & Data Engineering Community
Create Post

Home Posts Topics Members FAQ

Some thoughts on garbage collection

Hi all

I don't know whether this will be of interest, but I have just carried
out an exercise that I found useful, so I thought I would share it.

I am writing a multi-user business app, with a multi-threaded server
program to handle client connections. The server should 'serve
forever'. Each client connection is handled within its own thread, and
can run a variety of processes. Each process is a class, and is run by
instantiating it. As each client logs off, or the connection is broken,
the thread is terminated. I wanted to confirm positively that all
objects created by the thread are garbage collected. I was concerned
that if I left a superfluous reference dangling somewhere I might end
up with the equivalent of a memory leak.

I know from previous experience that you cannot confirm deletion of a
class object by printing a message from a __del__() method, if there
are any cyclic references to the object, as it will then not be garbage
collected at all. Therefore I use the 'delwatcher' class that was
explained by Tim Evans in a post on this subject a couple of years ago
-

class DelWatcher:
def __init__(self,obj):
self.objrepr = repr(obj)
def __del__(self):
print '%s deleted' % self.objrepr

class a:
def __init__(self):
self._delwatcher = DelWatcher(self)

You can now see that an object of class a is deleted, even if it has
cyclic references.

I used this technique, and I could see some objects being deleted at
the correct point. Others, however, did not get deleted until the
server program terminated. After some investigation, I found that I was
not generating enough 'dirty' objects to trigger the garbage collector
into running. I fixed that by putting the following lines at the top of
the server program -

import gc
gc.set_threshold(10) # on my system, the default is 700

Now I can see all objects being deleted at or close to the expected
point, and therefore I am confident that I do not have any leaks.
Obviously delwatcher and set_threshold are temporary measures to prove
a point - I have now removed them.

Is this a sensible approach, or are there easier ways to achieve this?

Frank Millman

Jan 23 '06 #1
5 1876
"Frank Millman" <fr***@chagford.com> writes:
Is this a sensible approach, or are there easier ways to achieve this?


In general you're supposed to just let gc do its thing. Doing your
own storage management defeats the purpose of gc. At most I'd say
check for leaks by running some native extension to scan all the
in-memory objects to see if anything didn't get gc'd.
Jan 23 '06 #2

Paul Rubin wrote:
"Frank Millman" <fr***@chagford.com> writes:
Is this a sensible approach, or are there easier ways to achieve this?
In general you're supposed to just let gc do its thing. Doing your
own storage management defeats the purpose of gc.


In principle I agree. My concern was that I might have inadvertently
done something wrong (e.g. left a reference dangling) which would
prevent gc from removing all objects which I wanted to be removed.
At most I'd say
check for leaks by running some native extension to scan all the
in-memory objects to see if anything didn't get gc'd.


If I knew what you meant, I would agree with you :-)

As all I really know is Python, the method I used was the best way I
could think of to accomplish what I wanted.

Frank

Jan 23 '06 #3
Frank Millman wrote:
In principle I agree. My concern was that I might have inadvertently
done something wrong (e.g. left a reference dangling) which would
prevent gc from removing all objects which I wanted to be removed.


Depends on what it really is that you want to know. If you want
to know whether gc can release all garbage objects, you should
look at gc.garbage, after a gc.collect call immediately before
the end of the program.

The issue here is that objects implementing __del__ in a cycle will
never get collected (but added to gc.garbage); this is something
you need to be aware of.

If you want to find out if there are objects which you hold onto
too long, you can look at len(gc.get_objects()) from time to
time. This won't be all objects, but just the container objects.
If you see the number growing over time, you have a leak.

You could then also categorize this by type, e.g.

frequency = {}
for o in gc.get_objects():
o = o.__class__.__name__
frequency[o] = frequency.get(o, 0) + 1
print sorted(frequency.iteritems(), key=operator.itemgetter(1),
reverse=1)

If you are interested in total object counts, you need to run
the debug version of Python.

HTH,
Martin
Jan 23 '06 #4

Martin v. Löwis wrote:
Frank Millman wrote:
In principle I agree. My concern was that I might have inadvertently
done something wrong (e.g. left a reference dangling) which would
prevent gc from removing all objects which I wanted to be removed.
Depends on what it really is that you want to know.

If you want to find out if there are objects which you hold onto
too long, you can look at len(gc.get_objects()) from time to
time. This won't be all objects, but just the container objects.
If you see the number growing over time, you have a leak.


Thank you - this is what I wanted to know.

You could then also categorize this by type, e.g.

frequency = {}
for o in gc.get_objects():
o = o.__class__.__name__
frequency[o] = frequency.get(o, 0) + 1
print sorted(frequency.iteritems(), key=operator.itemgetter(1),
reverse=1)


Very useful. Thanks.

Frank

Jan 24 '06 #5
In message <11*********************@z14g2000cwz.googlegroups. com>, Frank
Millman <fr***@chagford.com> writes
You could then also categorize this by type, e.g.


If you want a nice GUI and no requirement to modify your code Python
Memory Validator could be useful. http://www.softwareverify.com

Stephen
--
Stephen Kellett
Object Media Limited http://www.objmedia.demon.co.uk/software.html
Computer Consultancy, Software Development
Windows C++, Java, Assembler, Performance Analysis, Troubleshooting
Jan 24 '06 #6

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
by: Bob | last post by:
Are there any known applications out there used to test the performance of the .NET garbage collector over a long period of time? Basically I need an application that creates objects, uses them, and...
6
by: Ganesh | last post by:
Is there a utility by microsoft (or anyone) to force garbage collection in a process without have access to the process code. regards Ganesh
55
by: jacob navia | last post by:
Tired of chasing free(tm) bugs? Get serious about C and use lcc-win32. The garbage collector designed by Boehm is the best of its class. Very simple: #define malloc GC_malloc #define free(a)...
18
by: Matt | last post by:
I had these questions in an interview today: 1. Define a class using C. 2. Write a C functions that does garbage collection: frees all allocated memories. 3. Define semaphore. 4. What is...
18
by: Rein Petersen | last post by:
Is there any way to adjust thread priority for the garbage collector? Would be nice if I could tune the thread priority rules for the garbage collector too... From my readings of the process,...
2
by: C P | last post by:
I'm coming from Delphi where I have to explicitly create and destroy instances of objects. I've been working through a C#/ASP.NET book, and many of the examples repeat the same SqlConnection,...
2
by: roger.dunham | last post by:
I am trying to identify whether a .NET 1.1 application that I have written has a memory leak. I thought I understood how .NET memory management worked, but it appears that there is more to it...
84
by: jacob navia | last post by:
As many people know, I think that garbage collection is a good solution for many memory allocation problems. I am aware however, that nothing is "the silver bullet", not even the GC. A recent...
158
by: pushpakulkar | last post by:
Hi all, Is garbage collection possible in C++. It doesn't come as part of language support. Is there any specific reason for the same due to the way the language is designed. Or it is...
1
by: Sonnysonu | last post by:
This is the data of csv file 1 2 3 1 2 3 1 2 3 1 2 3 2 3 2 3 3 the lengths should be different i have to store the data by column-wise with in the specific length. suppose the i have to...
0
by: Hystou | last post by:
There are some requirements for setting up RAID: 1. The motherboard and BIOS support RAID configuration. 2. The motherboard has 2 or more available SATA protocol SSD/HDD slots (including MSATA, M.2...
0
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers,...
1
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows...
0
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing,...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new...
0
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and...
0
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
0
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated ...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.