473,327 Members | 2,094 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,327 software developers and data experts.

My function to recursively list all files in directory tree..

Hi all,


I have spent hours trying to figure out where I have went wrong with my code for my recursive function to list all the files in a directory, and all of the files in all
of its subdirectories.

This is the code:


Expand|Select|Wrap|Line Numbers
  1.  
  2. import os
  3. import sys
  4. import dircache
  5. import stat
  6. import string
  7.  
  8.  
  9.  
  10.  
  11. def dirrecur(currsub):
  12.  
  13.  
  14.         thispath = os.getcwd()
  15.  
  16.         print thispath
  17.         print "\n"
  18.  
  19.  
  20.         if(len(currsub) == 0):   #if len = 0, no subdirectories were found.
  21.  
  22.                 os.chdir('..')   #go back up one step.
  23.  
  24.         else:
  25.                 dlist = []
  26.  
  27.                 for z in range(0,len(currsub)):
  28.  
  29.  
  30.                         os.chdir(currsub[z])
  31.  
  32.                         print currsub[z]
  33.                         print "\n"
  34.  
  35.                         direntries = dircache.opendir('.') #open this directory
  36.  
  37.                         for j in range(0,len(direntries)):
  38.  
  39.                                 statresult = os.stat(direntries[j])
  40.  
  41.                                 if (stat.S_ISDIR(statresult.st_mode) != 0):
  42.  
  43.  
  44.                                         dlist.append(direntries[j])     #dlist is built here.
  45.                                         print direntries[j]
  46.  
  47.                                 else:
  48.  
  49.                                         print direntries[j]
  50.  
  51.                          dirrecur(dlist)   #recur again with full list of subdirectories.
  52.  
  53.  
  54.  
  55. #MAIN
  56.  
  57.  
  58. abspath = os.getcwd()
  59.  
  60. direntries = dircache.opendir(abspath)
  61.  
  62. dirlist = []
  63.  
  64.  
  65.  
  66. for i in range(0,len(direntries)):
  67.  
  68.  
  69.         statresult = os.stat(direntries[i])
  70.  
  71.         if (stat.S_ISDIR(statresult.st_mode) != 0): #if it's a directory file.
  72.  
  73.                 dirlist.append(direntries[i])
  74.                 print direntries[i]                 #print it out too.
  75.  
  76.         else:
  77.  
  78.                 print direntries[i]
  79.  
  80.  
  81.  
  82. dirrecur(dirlist)
  83.  
  84.  
  85.  


It goes down to level 3 of the directory tree, but at one point I get the
error 2, the no such file or directory message. I have checked the
file permissions of the directory that it gets stuck on, and I have no idea why
it is gettintg stuck on that directory and not one of the directories before it.

I would be extremely grateful if someone could lend another set of eyes to my
code and see where I'm going wrong at, and why it goes to level 3 so to speak
of the tree and then fails.

I'm stumped!


Thanks so much in advance. Any and all input would be greatly appreciated!
Jul 16 '07 #1
9 19597
bartonc
6,596 Expert 4TB
Perhaps you are trying too hard. os.walk() will do the work for you:
walk( top[, topdown=True [, onerror=None]])

walk() generates the file names in a directory tree, by walking the tree either top down or bottom up. For each directory in the tree rooted at directory top (including top itself), it yields a 3-tuple (dirpath, dirnames, filenames).
dirpath is a string, the path to the directory. dirnames is a list of the names of the subdirectories in dirpath (excluding '.' and '..'). filenames is a list of the names of the non-directory files in dirpath. Note that the names in the lists contain no path components. To get a full path (which begins with top) to a file or directory in dirpath, do os.path.join(dirpath, name).

If optional argument topdown is true or not specified, the triple for a directory is generated before the triples for any of its subdirectories (directories are generated top down). If topdown is false, the triple for a directory is generated after the triples for all of its subdirectories (directories are generated bottom up).

When topdown is true, the caller can modify the dirnames list in-place (perhaps using del or slice assignment), and walk() will only recurse into the subdirectories whose names remain in dirnames; this can be used to prune the search, impose a specific order of visiting, or even to inform walk() about directories the caller creates or renames before it resumes walk() again. Modifying dirnames when topdown is false is ineffective, because in bottom-up mode the directories in dirnames are generated before dirpath itself is generated.

By default errors from the os.listdir() call are ignored. If optional argument onerror is specified, it should be a function; it will be called with one argument, an OSError instance. It can report the error to continue with the walk, or raise the exception to abort the walk. Note that the filename is available as the filename attribute of the exception object.
There are others as well.
Jul 16 '07 #2
bartonc
6,596 Expert 4TB
Perhaps you are trying too hard. os.walk() will do the work for you:There are others as well.
Unless you really want the experience of writing a directory walker; in which case, I'll have a look at what you got after work.
Jul 16 '07 #3
hi bartonc,


Thanks a million for taking a look!!


I would definitely like to learn how to write my own directory walker, and
it is nice to know that if I learn how to use os.walk() that I can get the output
I want, however I really would like to know where my code is going haywire at,
and I am really confused why it goes down to the third level of the directory
tree and then decides to crap out.


Thanks again so much.
Jul 16 '07 #4
bvdet
2,851 Expert Mod 2GB
Expand|Select|Wrap|Line Numbers
  1.  
  2. import os
  3. import sys
  4. import dircache
  5. import stat
  6. import string
  7.  
  8.  
  9.  
  10.  
  11. def dirrecur(currsub):
  12.  
  13.  
  14.         thispath = os.getcwd()
  15.  
  16.         print thispath
  17.         print "\n"
  18.  
  19.  
  20.         if(len(currsub) == 0):   #if len = 0, no subdirectories were found.
  21.  
  22.                 os.chdir('..')   #go back up one step.
  23.  
  24.         else:
  25.                 dlist = []
  26.  
  27.                 for z in range(0,len(currsub)):
  28.  
  29.  
  30.                         os.chdir(currsub[z])
  31.  
  32.                         print currsub[z]
  33.                         print "\n"
  34.  
  35.                         direntries = dircache.opendir('.') #open this directory
  36.  
  37.                         for j in range(0,len(direntries)):
  38.  
  39.                                 statresult = os.stat(direntries[j])
  40.  
  41.                                 if (stat.S_ISDIR(statresult.st_mode) != 0):
  42.  
  43.  
  44.                                         dlist.append(direntries[j])     #dlist is built here.
  45.                                         print direntries[j]
  46.  
  47.                                 else:
  48.  
  49.                                         print direntries[j]
  50.  
  51.                          dirrecur(dlist)   #recur again with full list of subdirectories.
  52.  
  53.  
  54.  
  55. #MAIN
  56.  
  57.  
  58. abspath = os.getcwd()
  59.  
  60. direntries = dircache.opendir(abspath)
  61.  
  62. dirlist = []
  63.  
  64.  
  65.  
  66. for i in range(0,len(direntries)):
  67.  
  68.  
  69.         statresult = os.stat(direntries[i])
  70.  
  71.         if (stat.S_ISDIR(statresult.st_mode) != 0): #if it's a directory file.
  72.  
  73.                 dirlist.append(direntries[i])
  74.                 print direntries[i]                 #print it out too.
  75.  
  76.         else:
  77.  
  78.                 print direntries[i]
  79.  
  80.  
  81.  
  82. dirrecur(dirlist)
  83.  
  84.  
  85.  
I think it would be simpler to use os.listdir(). Check out the thread HERE.

<Mod EDIT: Nice recovery below, BV. Admin is still trying to work the bugs out of code tags inside quotes. The weird thing is that it doesn't always happen. I've removed the quote here, rather than the code tags.>
Jul 16 '07 #5
bvdet
2,851 Expert Mod 2GB
Hi all,


I have spent hours trying to figure out where I have went wrong with my code for my recursive function to list all the files in a directory, and all of the files in all
of its subdirectories.





It goes down to level 3 of the directory tree, but at one point I get the
error 2, the no such file or directory message. I have checked the
file permissions of the directory that it gets stuck on, and I have no idea why
it is gettintg stuck on that directory and not one of the directories before it.

I would be extremely grateful if someone could lend another set of eyes to my
code and see where I'm going wrong at, and why it goes to level 3 so to speak
of the tree and then fails.

I'm stumped!


Thanks so much in advance. Any and all input would be greatly appreciated!
I think it would be simpler to use os.listdir(). Check out the thread HERE.
BV
Jul 16 '07 #6
If it helps any what I am looking for in my output,

I want the output to resemble the UNIX program ls -R as closely as possible.


I am running solaris 2.10.


I notice that when I use os.walk(), that each directory is showing files that
even ls -a won't show on those directories.

Are those files placed there by the system administrator?
Jul 16 '07 #7
bartonc
6,596 Expert 4TB
If it helps any what I am looking for in my output,

I want the output to resemble the UNIX program ls -R as closely as possible.


I am running solaris 2.10.


I notice that when I use os.walk(), that each directory is showing files that
even ls -a won't show on those directories.

Are those files placed there by the system administrator?
I'm afraid that you are dealing with a bunch of Windows users, here.
Not going to be much help on your output format, but, with regard to that, I do see a need to pass the recursion depth into the next iteration. Perhaps starting with a very simple, clean example of a recursive directory walker would help. I really like the simplicity of copytree() from the shutil module:
Expand|Select|Wrap|Line Numbers
  1.  
  2. def copytree(src, dst, symlinks=False):
  3.     """Recursively copy a directory tree using copy2().
  4.  
  5.     The destination directory must not already exist.
  6.     If exception(s) occur, an Error is raised with a list of reasons.
  7.  
  8.     If the optional symlinks flag is true, symbolic links in the
  9.     source tree result in symbolic links in the destination tree; if
  10.     it is false, the contents of the files pointed to by symbolic
  11.     links are copied.
  12.  
  13.     XXX Consider this example code rather than the ultimate tool.
  14.  
  15.     """
  16.     names = os.listdir(src)
  17.     os.mkdir(dst)
  18.     errors = []
  19.     for name in names:
  20.         srcname = os.path.join(src, name)
  21.         dstname = os.path.join(dst, name)
  22.         try:
  23.             if symlinks and os.path.islink(srcname):
  24.                 linkto = os.readlink(srcname)
  25.                 os.symlink(linkto, dstname)
  26.             elif os.path.isdir(srcname):
  27.                 copytree(srcname, dstname, symlinks)
  28.             else:
  29.                 copy2(srcname, dstname)
  30.             # XXX What about devices, sockets etc.?
  31.         except (IOError, os.error), why:
  32.             errors.append((srcname, dstname, why))
  33.         # catch the Error from the recursive copytree so that we can
  34.         # continue with other files
  35.         except Error, err:
  36.             errors.extend(err.args[0])
  37.     if errors:
  38.         raise Error, errors
So, what's that? Maybe 3 lines to get the name sorted out and 3 more to decide where or not to do the recursion.
Jul 17 '07 #8
ghostdog74
511 Expert 256MB
If it helps any what I am looking for in my output,

I want the output to resemble the UNIX program ls -R as closely as possible.


I am running solaris 2.10.


I notice that when I use os.walk(), that each directory is showing files that
even ls -a won't show on those directories.

Are those files placed there by the system administrator?
what does not show in ls -a that is shown in os.walk? here's a little walker for you
Expand|Select|Wrap|Line Numbers
  1. from os import listdir, sep
  2. from os.path import isdir
  3. def skywalker(dir):
  4.     for file in listdir(dir):
  5.         path = dir + sep + file        
  6.         if isdir(path):
  7.             print "path: ",path
  8.             skywalker(path)
  9.         else:
  10.             print '->' + file
  11. skywalker("/home/test")
  12.  
Jul 17 '07 #9
bvdet
2,851 Expert Mod 2GB
I have been using this function:
Expand|Select|Wrap|Line Numbers
  1. import os
  2.  
  3. def dirEntries(dir_name, subdir, *args):
  4.     '''Return a list of file names found in directory 'dir_name'
  5.     If 'subdir' is True, recursively access subdirectories under 'dir_name'.
  6.     Additional arguments, if any, are file extensions to match filenames. Matched
  7.         file names are added to the list.
  8.     If there are no additional arguments, all files found in the directory are
  9.         added to the list.
  10.     Example usage: fileList = dir_list(r'H:\TEMP', False, 'txt', 'py')
  11.         Only files with 'txt' and 'py' extensions will be added to the list.
  12.     Example usage: fileList = dir_list(r'H:\TEMP', True)
  13.         All files and all the files in subdirectories under H:\TEMP will be added
  14.         to the list.
  15.     '''
  16.     fileList = []
  17.     for file in os.listdir(dir_name):
  18.         dirfile = os.path.join(dir_name, file)
  19.         if os.path.isfile(dirfile):
  20.             if len(args) == 0:
  21.                 fileList.append(dirfile)
  22.             else:
  23.                 if os.path.splitext(dirfile)[1][1:] in args:
  24.                     fileList.append(dirfile)
  25.         # recursively access file names in subdirectories
  26.         elif os.path.isdir(dirfile) and subdir:
  27.             print "Accessing directory:", dirfile
  28.             fileList += dirEntries(dirfile, subdir, *args)
  29.     return fileList
Jul 17 '07 #10

Sign in to post your reply or Sign up for a free account.

Similar topics

4
by: Ruby Tuesday | last post by:
Is there a fast way to read files/directory recursively? Instead of inspecting each file(s)/dir(s), is there a way to know that its a file or a directory from its hidden attribut both for windows...
9
by: Penn Markham | last post by:
Hello all, I am writing a script where I need to use the system() function to call htpasswd. I can do this just fine on the command line...works great (see attached file, test.php). When my...
1
by: Antonio Lopez Arredondo | last post by:
hi all !!! I need to copy a folder and its subfolders to another location; which class should I use ? could only find the System.IO.Directory.MOVE but don't know how to COPY. thanks in...
1
by: Fatemeh | last post by:
Hi I want to write a program (FileCopier) that it allow the user to check files (or entire directories) in the left tree view (source). If the user presses the Copy button, the files checked on...
3
by: dibblm | last post by:
Below is current code used. I can only list one directory then move to next. I want to search one more directory further and can't seem to find how to get one deeper. What I want to accomplish is...
3
by: Chris | last post by:
I am clearly missing something simple here.... I have a function (fileTree) that recursively reads all the files in a directory passed to it, and returns an array of the files, I have included...
1
by: Aek | last post by:
What is the best way to recursively change the permissions of the directory we are installing into? Is there a nice way to do this in C# ..NET? We are using an MSI installer and will need to add...
3
by: empiresolutions | last post by:
I'm trying to build my first PHP Class. After days of tweaking, im lost. I am used to working with functions and arrays, but wrapping them in classes is confusing me. The following code is to...
2
by: sebastien.abeille | last post by:
Hello, I would like to create a minimalist file browser using pyGTK. Having read lot of tutorials, it seems to me that that in my case, the best solution is to have a gtk.TreeStore containing...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
isladogs
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 6 Mar 2024 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, we are pleased to welcome back...
1
by: PapaRatzi | last post by:
Hello, I am teaching myself MS Access forms design and Visual Basic. I've created a table to capture a list of Top 30 singles and forms to capture new entries. The final step is a form (unbound)...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
by: Faith0G | last post by:
I am starting a new it consulting business and it's been a while since I setup a new website. Is wordpress still the best web based software for hosting a 5 page website? The webpages will be...
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.