Command.py
Introduction
Sometimes it is hard to forget ingrained habits: if you are a long time user of command line shells, be it bash, csh, ksh or even a dos-box, you learn to to think in pipelines. Many a problem can be solved by breaking it down in simple tasks which input and output can be chained together to produce the result.
When I was working on a make like utility (to be documented elsewhere) that had its actions encoded as embedded python I quickly started to miss the pipeline metaphor that is exquisitely suited to the file oriented tasks so often encountered in situations that call for a make like utility. So what was I to do?Luckily python allows us to overload operators and one of its operators (bitwise or) happens to use the pipe symbol '|'. So if I would use that, I would be able to write something like:
item1 | item2 | item3If I could arrange for those items to stand for meaningfull tasks like
grep, sort or
cat, I would indeed be able
to use the pipeline idiom in python.
I am certainly not the first to think of this reuse of an operator (see for example these recipes) but for such an idea to be really usefull it should come with many shell-like commands already available and with the possibility to extend it in an easy manner. Enter Command.py
A simple example
Command.py is a module that implements the pipeline metaphor and comes equiped with a few dozen of shell-like commands that are ready to use. An example to give you a taste of what is possible:
from Command import *
cat('myfiles/*.txt') | egrep(r'ing$') | \
sort | uniq -c > outfile('stats.txt')
As you can see you can use glob patterns
as well and the '>' operator is overloaded
to a natural looking function. And this is
not all: when working in a shell you
expect a number of easy to use commands
to manipulate files and directories:
from Command import *
mkdir('oldfiles')
cp('*.old','oldfiles')
rm('*~','aaa','bbb')
There is even more as you will see in the
following sections.
Still, I am fully aware that all this does not make python a valid replacement for any shell (except mayby the overly simple dos-box) but it does make implementing some tasks easier, especially for those people who have used shells often.
Still, Command.py should be regarded as a work in progress, because there are still things to be done. I am especially not very charmed by having to put arguments to command between parentheses but I see at moment no other possibility (other than parsing the line, but that defeats the purpose. Whether there will be any devlopment at all depends on how well it interacts with my other project, the make like utility.
Pipe enabled commands
These are commands that may be put in a pipeline. If you click on the command itself you are referred to a page with much more detailed information. Note: where we speak of stdin or stdout we refer to the input and output of a command as used in the pipeline, not the real file descriptors.
| command | example | description |
|---|---|---|
| cat | cat('filea','fileb') | copy the contents of files to stdout |
| tee | tee('filec') | copy stdin to stdout and to files |
| grep | grep('aaa') | copy lines from stdin to stdout that contain a pattern |
| egrep | egrep('a*b$') | copy lines from stdin to stdout that match a regular expression |
| replace | replace('aaa','bbb') | copy lines from stdin to stdout while replacing a pattern |
| sort | ls | sort | filter out multiple occurences of lines |
| uniq | sort | uniq | copy sorted input to stdout |
| infile | infile('filea')|sort | open a single file for reading |
| outfile | ls > outfile('fileb') | open a single file for writing |
| linecount | infile('filea')|linecount | print statistics of stdin to stdout |
| ls | ls('*.txt') | list matching filenames one per line |
| stat | ls | stat | show file properties |
| echo | echo('one', 'two') | print flattened arguments to stdout, one per line |
File oriented commands
These are commands to manipulate files or directories. These can not be put in a pipeline.
| command | example | description |
|---|---|---|
| touch | touch('filea') | set modification time of file |
| cp | cp('filea','fileb') | cp files and directories |
| sshupdate | sshupdate('user','hostname', 'filea','/remotedir') | cp files and directories over ssh |
| rm | rm('filea') | delete files and directories |
| mkdir | mkdir('path/to/newdir') | create directories |
| mv | mv('files','fileb') | rename or movc files and directories |
| chmod | chmod('ug=rw,o=r','filea') | change permissions on files and directories |
Special commands
These commands go beyond basic functionality. Some might be used in a pipeline others are file oriented.
| command | example | description |
|---|---|---|
| filter | filter(process=myfunc, '*.txt', 'out.txt') | filter file thru user defined filter |
| substitute | substitute('*.css', 'style.css') | Interpolate variables |
| callpipetd> | ...| callpipe('prog') |... | call external program |
| callout | callout('ps waux') | call external program |
| wget | wget('http://www.nu.nl', 'nu.html') | fetch files from web
Utilities: roll your own
These functions and classes are the building blocks to implement new pipeline enabled or file oriented commands. Each one is described in fair detail in the epydoc generated documentation.
| mux muxnodst bunch | |
| expand lopen | |
| flatten | |
| copy3 copytree2 | |
| linebuffer | |
| inputfilterbuffer outputfilterbuffer | |
| pype |
Requirements & installation
You can download Command.py here. It is tested with python 2.6.x and released under a GPL license. The current version is not yet packaged, so to install it, just move the *.py files from the zip to your site-packages directory of your Python installation. Detailed documentation is available as well.
If you want to use the substitute() command you will need
Varsub.py. It is included in the zip and documented
on a separate page.
the sshupdate() command needs Paramiko otherwise it is
not avaible.