Saving recursive objects to disc. cPickle wan't work.

Hi,

* Alex Polite <m4@polite.se> [26 May 2004 14:22:21 GMT]:

I need to put recursive data structures on disc and found out that
cPickle doesn't like recursion.

Hm, the documentation of Python 2.3.2, Section 3.14.1 "Relationship to
other Python modules" states that recursive objects, which are defined
as "objects that contain references to themselves" can't be handled by
marshal, but that pickle and cPickle should be fine: "pickle stores
such objects only once, and ensures that all other references point to
the master copy".

What exactly are your problems?

Regards
Lutz
--
pub 1024D/6EBDA359 1999-09-20 Lutz Horn <lu*******@web.de>
Key fingerprint = 438D 31FC 9300 CED0 1CDE A19D CD0F 9CA2 6EBD A359
sub 2048g/EA8CFEDE 2000-03-09

On ons, maj 26, 2004 at 04:34:00 +0200, Lutz Horn wrote:

Hm, the documentation of Python 2.3.2, Section 3.14.1 "Relationship to
other Python modules" states that recursive objects, which are defined
as "objects that contain references to themselves" can't be handled by
marshal, but that pickle and cPickle should be fine: "pickle stores
such objects only once, and ensures that all other references point to
the master copy".

What exactly are your problems?

That sound promising.

I'm building a library to generate markov models from text, much like
dadadodo.

You can get the code here.

http://snippsnapp.polite.se/wiki.pl?PyDodo
When I run my testsuit that tries to pickle a generated markov model I
get this:

python test/basicsuite.py

1000
E
================================================== ====================
ERROR: test_pickle (__main__.BasicTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/basicsuite.py", line 73, in test_pickle
print sys.settrecursionlimit(4000)
AttributeError: 'module' object has no attribute 'settrecursionlimit'

----------------------------------------------------------------------
Ran 1 test in 0.002s

If I up the recusionlimit the testsuit will segfault.

alex

--
Alex Polite
http://polite.se

Alex Polite wrote:

....

If I up the recusionlimit the testsuit will segfault.

Then you need a Python which does not have a stack problem.

Use Stackless Python. Just don't care of what else it does,
just use it, raise the recursionlimit to whatever,
and both pickle and cPickle will run without any limit but
main memory.

Well, almost true.
The current official version is unlimited on pickle.py,
becuase the recursive calls in pickle don't involve
recursive calls in the C interpreter.
Stack protection for cPickle is in my new developer version,
which is coming soon.
It has Stack spilling for cPickle and the interpreter, so
even in the rare cases where deep recursions cannot be avoided,
the Stack is always saved and restored before overflow.
And, well, I have full thread support since Monday :-))

ciao - chris

--
Christian Tismer :^) <mailto:ti****@stackless.com>
Mission Impossible 5oftware : Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/
14109 Berlin : PGP key -> http://wwwkeys.pgp.net/
work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/

On ons, maj 26, 2004 at 02:22:21 +0000, Alex Polite wrote:

I need to put recursive data structures on disc and found out that cPickle
doesn't like recursion.

What are my options?

alex

Christian Tismer had the kindness to look at my code and point out
that I might want to use pickle instead of cPickle, at least if I
wanted to benefit from using stackless. Chaning from cPickle to pickle
allowed to run the code under stackless as well as under standard
python.

thanks Christian.

alex

--
Alex Polite
http://polite.se

Alex Polite wrote:

On ons, maj 26, 2004 at 02:22:21 +0000, Alex Polite wrote:

I need to put recursive data structures on disc and found out that cPickle
doesn't like recursion.

What are my options?

alex

Christian Tismer had the kindness to look at my code and point out
that I might want to use pickle instead of cPickle, at least if I
wanted to benefit from using stackless. Chaning from cPickle to pickle
allowed to run the code under stackless as well as under standard
python.

thanks Christian.

Although I'm happy that things work even without Stackless,
this implies that there is an incompatibility between
pickle and cPickle.
If objects are treated identically by both, that normal Python
must use even more stack space for recursive objects that
cPickle, so I'd expect it crashes earlier.

But it doesn't crash. cPickle must have a bug.

ciao - chris
--
Christian Tismer :^) <mailto:ti****@stackless.com>
Mission Impossible 5oftware : Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/
14109 Berlin : PGP key -> http://wwwkeys.pgp.net/
work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776
PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04
whom do you want to sponsor today? http://www.stackless.com/