Module Command
[hide private]

Module Command

source code

Command.py implementation of shell like file manipulation functions. (c) 2009 Michel J. Anders

Version: 0.01a

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

Many of the file manipulation functions are already available in the module shutil but they almost always 'single src argument' whereas this implementation enhances a number of commands with the possibility to pass more than one src file.

Also implemented are a number of building blocks and actual functions to mimick shell-like behaviour in the use of pipes. E.g. it is possible to write:

   cat('*.data','*.txt') | grep('oyster') > stdout

or even this:

   ls -a | grep('foobar') -v --ignorecase | sort -r

The above is made possible with a couple of tricks:

  1. overloading the __sub__() and __neg__() operators (for long and short options)
  2. preassigning instances to variables (so you can use ls intead of ls() this was already used in this recipe
  3. defining __call__() (so e.g. ls('aaa') yields a new ls instance
  4. redefining __repr__() (so the result a pipeline is shown automatically when working interactive)
  5. and of course redefining __or__() (best seen in this recipe.

Note that this is syntactical sugar only: the 'functions' above are implemented as classes and do not spawn subprocesses (although functions are provided that DO spawn subprocesses).

A less technical introduction can be found here.

Classes [hide private]
  callout
Call a subprocess with args and make its output available to a pipeline.
  callpipe
Call a subprocess with args and make its input and output available to a pipeline.
  cat
Copies the contents of all files given as argument and makes it available to the next component in the pipeline.
  echo
Make elements of a list available to a pipeline.
  egrep
copy all lines offered to it in a pipeline and that match one of its regular expression arguments and makes it available to the next component in the pipeline.
  grep
copy all lines offered to it in a pipeline and that contain one of its arguments and makes it available to the next component in the pipeline.
  infile
Open a single named file and make it available to a shell-like pipeline.
  inputfilterbuffer
Filter incoming data.
  linebuffer
A container you can write() lines to and which provides an iterator to iterate over the buffered contents.
  linecountclass
An example of a blockprocessor.
  lsclass
Generate a list of matching file and directory names and make this available to the next component in a pipeline.
  option
An abstraction of an option.
  outfile
Open a single named file and make it available to a shell-like pipeline.
  outputfilterbuffer
a.k.a.
  pwdclass
make the name of the current directory available to the pipeline.
  pype
A mixin enabling a class that implements the linebuffer interface to participate in a shell-like pipeline.
  replace
Copy lines from src to a destination file while replacing occurences of pattern.
  sortclass
Sort input lines and produce the sorted lines as output.
  stacker
A mixin that accumulates options and provides a factory for instances of the same class.
  statclass
find filesystem attributes for a list of files.
  tee
Copy input to output and to any files specified as arguments
  uniqclass
Return unique lines only, optionally prefixed with a linecount.
Functions [hide private]
 
_xglob(arg, showhidden=False, hidenonexistent=False)
An augmented version of glob.glob()
source code
 
bunch(*args, **kw)
Process a number of files one by one and write the results to a file.
source code
 
call(*args, **kw)
Call a subprocess with args and ignore output.
source code
 
cd(path)
change the current directory to path.
source code
 
chmod(mode, *args)
Change the read, write and execute bits for directories or files.
source code
 
copy3(src, dst)
Copy data and all stat info if src is newer ("cp -pu src dst").
source code
 
copytree2(src, dst, symlinks=False, ignore_pattern=None, update=False)
Recursively copy a directory tree using copy2().
source code
 
cp(*args, **kw)
Copy files and/or directories.
source code
 
expand(*args, **kw)
Flatten a list of filenames, directory names a glob patterns to a single flat list of filenames.
source code
 
filter(*args, **kw)
Copy the contents of one or more source.files to destination while filtering each line through a filter.
source code
 
flatten(x)
return a (possibly nested) list of iterables as a single flat listen.
source code
 
lopen(*args)
Expand a (possibly nested) list of strings and globs and open all resulting filenames.
source code
 
mkdir(*args, **kw)
Create directories.
source code
 
mux(f, *args, **kw)
multiplex a function to any number of sourcefiles.
source code
 
muxnodst(f, *args, **kw)
multiplex a function to any number of sourcefiles.
source code
 
mv(*args, **kw)
Move or rename files and/or directories.
source code
 
rm(*args, **kw)
Remove files and/or directories.
source code
 
sshupdate(user, host, *args, **kw)
Update files and/or directories over a secure connection to a destination on host if they are newer than their counterpart in the destination.
source code
 
substitute(*args, **kw)
Copy the contents of one or more source.files to destination while scaning each line for variable definitions and variable references.
source code
 
touch(*args, **kw)
set the mtime (modification time) of files to the current time, creating empty files for files that do not exist.
source code
 
wget(*args, **kw)
Copy the contents of one or more urls to a single local file.
source code
Variables [hide private]
  __nonpipeable__ = ['touch', 'cp', 'sshupdate', 'rm', 'mkdir', ...
  __options__ = ['a', 'i', 'c', 'v', 'n']
  __package__ = None
  __pipeable__ = ['cat', 'grep', 'egrep', 'sort', 'tee', 'uniq',...
  __utils__ = ['mux', 'muxnodst', 'bunch', 'expand', 'lopen', 'f...
  a = <Command.option object at 0x00EA4C50>
  c = <Command.option object at 0x00EA4C70>
  i = <Command.option object at 0x00EA4BF0>
  linecount = ??
  ls = Command...
  n = <Command.option object at 0x00EA4C30>
  pwd = C:\Documents and Settings\Michel\Mijn documenten\Python\...
  sort =
  stat =
  stdin = <epydoc.docintrospecter._DevNull instance at 0x00D04E68>
  stdout = <epydoc.docintrospecter._DevNull instance at 0x00D04E68>
  tolist = <Command.linebuffer object at 0x00E28A30>
  uniq =
  v = <Command.option object at 0x00EA4C10>
  wc = ??
Function Details [hide private]

_xglob(arg, showhidden=False, hidenonexistent=False)

source code 

An augmented version of glob.glob()

Parameters:
  • showhidden - if True, return matching files that start with '.' as well
  • hidenonexistent - if True return arg unchanged if nothing matches

    The latter is usefull to define two different behaviours:

    e.q. touch('aaa') creates a file, if 'aaa' doesn't exist but ls('aaa') returns nothing if 'aaa' does not exist.

Returns:
a (possibly empty) list of filenames that match arg

bunch(*args, **kw)

source code 

Process a number of files one by one and write the results to a file.

Note that this is different from the mux() funtion:

   mux()    operates on src,dst filename-pairs
   bunch()  operates on the contents of several src files and writes to a dst file

A processor is passed by the keyword parameter and should be a class that implements the blockprocess interface. By default a simple copy processor is used.

The blockprocess interface:

   __init__() initializes a blockprocessor,
   process(iterator) iterates over iterator (typically a file object or a list of lines)
   end() ends the processing of input
   results() a generator that returns results.

cd(path)

source code 

change the current directory to path.

Parameters:
  • path - the relative or absolute path to the new directory.

chmod(mode, *args)

source code 

Change the read, write and execute bits for directories or files.

Parameters:
  • mode - a comma seperated list of mode assigments.

    Mode assigments have the following forms item=modes. Item is one or more of:

       u    : means user
       g    : means group
       o    : means other
       a    : shortcut for ugo
    

    Modes is one or more of:

       r    : read
       w    : writw
       x    : execute (means accessible for directories)
    

    Example:

       chmod('ug=rw,o=r', 'filea.txt', 'fileb.txt')
    

    Means make filea.txt and fileb.txt readable for others and also writeable for the user and the group.

copy3(src, dst)

source code 

Copy data and all stat info if src is newer ("cp -pu src dst").

The destination may be a directory.

The source is also considered newer if the destination does not exist.

Returns:
1 if file is copied, 0 otherwise

copytree2(src, dst, symlinks=False, ignore_pattern=None, update=False)

source code 

Recursively copy a directory tree using copy2().

The destination directory must not already exist, unless update=True.

If exception(s) occur, an Error is raised with a list of reasons.

If the optional symlinks flag is true, symbolic links in the source tree result in symbolic links in the destination tree; if it is false, the contents of the files pointed to by symbolic links are copied.

The optional ignore_pattern argument is a callable. If given, it is called with the `src` parameter, which is the directory being visited by copytree(), and `names` which is the list of `src` contents, as returned by os.listdir():

   callable(src, names) -> ignored_names

Since copytree() is called recursively, the callable will be called once for each directory that is copied. It returns a list of names relative to the `src` directory that should not be copied.

The optional update flag ensures that files are only copied if the source is newer than the destination or if the destination does not exist.

Note that this if a rather straightforward reimplementation of the shutil.copytree() function present in python 2.x and 3.x including most of the documentation.

Returns:
the number of files copied.

cp(*args, **kw)

source code 

Copy files and/or directories.

Parameters:
  • args (str) - one or more filenames
  • ignore - if True all errors are ignored.
  • recursive - if True contents of directories are copied as well.
  • preserve - if True modification times of copied objects is retained.
  • update - copy only if src newer than dst.
  • ignore_pattern - see shutil2.copy3 or shutil.copytree
Returns:
undocumented.

expand(*args, **kw)

source code 

Flatten a list of filenames, directory names a glob patterns to a single flat list of filenames.

Example:

expand('filea',['fileb','filec'],'*.txt')

might result in (depending on the content of the globbed dir):

['filea','fileb','filec','a.txt','b.txt']

args that contain no glob chars ('?' or '*') are not globbed.

Parameters:
  • args - a (possibly nested) list of strings or glob patterns.
  • directories - if True names of directories are included in the result.
Returns:
a list of strings (i.e. filenames although they need pot exist.)

filter(*args, **kw)

source code 

Copy the contents of one or more source.files to destination while filtering each line through a filter.

process=copy unchanged by default, see 'substitute()' for a more meaningfull application.

Note: source.filenames prefixed with a minus sign '-' are processed but not written to the destination.

Parameters:
  • args (str) - files, last one is destination.
  • ignore - if True all errors are ignored.
Returns:
nothing.

flatten(x)

source code 

return a (possibly nested) list of iterables as a single flat listen. Strings are *not* flattened. For examples see expand().

lopen(*args)

source code 

Expand a (possibly nested) list of strings and globs and open all resulting filenames. Return a single iterator that will iterate over all lines in all opend files.

Example:

   for line in lopen('*.txt'):
       print line,

will print all lines of all text files in the current directory.

mkdir(*args, **kw)

source code 

Create directories.

Parameters:
  • args (str) - directory name
  • recursive - if True missing intermedate directories are created as well.
  • ignore - if True all errors are ignored.
Returns:
a (possibly empty) list of errors.

mux(f, *args, **kw)

source code 

multiplex a function to any number of sourcefiles.

mux() calls function f() with a src and a dst argument and any keyword arguments are passed along. After expanding 'args' (see expand), dst is the last element of the resulting list and src is from the rest of the list, but one by one. So, say you have a function A(src,dst) a call like:

   mux(A,'*.txt','*.doc','summary.dat')

is equivalent to:

   A('a.txt','summary.dat')
   A('b.txt','summary.dat')
   A('c.doc','summary.dat')

depending on the actual files that match of course.

Parameters:
  • f - a function (callable) that takes two string arguments.
  • args - a (possibly nested) list of strings or glob patterns.
Returns:
nothing.

muxnodst(f, *args, **kw)

source code 

multiplex a function to any number of sourcefiles.

muxnodst() calls function f() with a single argument and any keyword arguments are passed along. After expanding 'args' (see expand), each argument is taken is from the resulting list, one by one. So, say you have a function A(src) a call like:

   mux(A,'*.txt','*.doc')

is equivalent to:

   A('a.txt')
   A('b.txt')
   A('c.doc')

depending on the actual files that match of course.

Parameters:
  • f - a function (callable) that takes one string argument.
  • args - a (possibly nested) list of strings or glob patterns.
Returns:
nothing.

mv(*args, **kw)

source code 

Move or rename files and/or directories.

Parameters:
  • args (str) - file/directory names
  • ignore - if True all errors are ignored.
Returns:
a (possibly empty) list of errors.

rm(*args, **kw)

source code 

Remove files and/or directories.

Parameters:
  • args (str) - file/directory name
  • recursive - if True directory contents are removed as well.
  • ignore - if True all errors are ignored.
Returns:
a (possibly empty) list of errors.

sshupdate(user, host, *args, **kw)

source code 

Update files and/or directories over a secure connection to a destination on host if they are newer than their counterpart in the destination.

Newer is defined as having a newer modification time or when the destination does not exist.

If a password is required a prompt is issued. If non-interrupted operation is required your ssh client configuration should be set up to work with certificates. See 'paramiko' and 'Braid.SshUpdate' for details.

Parameters:
  • user (str) - username to log in on sshserver.
  • host (str) - hostname or ip-address of sshserver
  • args (str) - one or more filenames
  • ignore - if True all errors are ignored.
Returns:
nothing.

substitute(*args, **kw)

source code 

Copy the contents of one or more source.files to destination while scaning each line for variable definitions and variable references.

Note: source.filenames prefixed with a minus sign '-' are processed but not written to the destination. This provides for simple inclusion of files with variable definitions.

Variables are defined in the following way:

$NAME=value where NAME is a combination of letters and digits and value a sequence of. characters, possibly enclosed in quotes.

variables may be referenced in tuo ways:

$NAME followed by whitespace, or

${NAME} which can appear anywhere.

Example:

   $COLOUR=white
   $COMMENT="off white with a yellow tinge"
   Let's paint the wall ${COLOUR}, ($COMMENT).

For a more exact syntax refer to 'Braid.Varsub'.

Parameters:
  • args (str) - files, last one is destination.
  • ignore - if True all errors are ignored.
Returns:
nothing.

touch(*args, **kw)

source code 

set the mtime (modification time) of files to the current time, creating empty files for files that do not exist.

Parameters:
  • args (str) - one or more filenames
  • ignore - if True all errors are ignored.
Returns:
a (possibly empty) list of errors.

wget(*args, **kw)

source code 

Copy the contents of one or more urls to a single local file.

TODO: pype

Parameters:
  • args (str) - urls, last one is destination file
  • ignore - if True all errors are ignored.
Returns:
nothing.

Variables Details [hide private]

__nonpipeable__

Value:
['touch',
 'cp',
 'sshupdate',
 'rm',
 'mkdir',
 'mv',
 'substitute',
 'call',
...

__pipeable__

Value:
['cat',
 'grep',
 'egrep',
 'sort',
 'tee',
 'uniq',
 'infile',
 'outfile',
...

__utils__

Value:
['mux',
 'muxnodst',
 'bunch',
 'expand',
 'lopen',
 'flatten',
 'copy3',
 'copytree2',
...

ls

Value:
Command.py
Command.pyc
epydoc-command
gpl.txt
SshUpdate.py
SshUpdate.pyc
using-command.html
using-command.html~
...

pwd

Value:
C:\Documents and Settings\Michel\Mijn documenten\Python\Command-standa\
lone