linux proxy problem revisited

Some time back i posted how to setup a local forwarding proxy in linux so that u do not need to set your proxy password in each and every program that asks, and that anyone and everyone cannot see your passwords by simply typing "echo $http_proxy"
But again it was not so automatic, you still needed to set the proxy to localhost "http://127.0.0.1:3128/"

Now this post is to make that also redundant. No program connecting to http (port 80) (not https) needs to have the proxy set. This is done by a transparent proxy (intercepting proxy), which 3proxy is by default without requiring any specific configuration (squid needs some option to be set in its config file)

Firstly set up the local forwarding proxy as mentioned in my previous post http://powerpan.blogspot.com/2010/06/linux-proxy-problem.html
Now you need to set an iptables (default linux firewall) policy which will redirect all outgoing traffic to port 80 to the local proxy at port 3128
You also need some way to exclude  the proxy itself from being redirected. So do the following:

Create a specific user for 3proxy:
Create a user on your computer (say 3proxy) with a specific uid (say 480)
# useradd -u 480 3proxy
Now in the 3proxy config file set it to change its user to 3proxy. Just before the "proxy" line in /etc/3proxy.cfg add the following line:
setuid 480
and restart the 3proxy service:
# service 3proxy restart

Redirect outgoing http traffic to local proxy
Here's an iptables rule that forwards outgoing traffic on port 80 (excluding those from user 3proxy) to the local proxy.
# iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner 3proxy --dport 80 -j REDIRECT --to-port 3128

Thats all. Now you are done. To make it persistent across reboots add it to some startup file (/etc/profile.d/).

If you are on Fedora there's a better way:
Create a file (/etc/iptables-transparent-proxy) with the following line:
-A OUTPUT -p tcp -m owner ! --uid-owner 3proxy -m tcp --dport 80 -j REDIRECT --to-ports 3128
Now open system -> config -> firewall and in the Custom Rules (bottommost filter on the left) add a new rule with protocol:ipv4, table:nat and file:/etc/iptables-transparent-proxy and you are done.

To test it, open a terminal, unset http_proxy and wget google.com. The index.html file should be downloaded


NOTE:

• This automatic forwarding does not work for https sites which are specifically designed to prevent such things (man-in-the-middle attacks)

• Outgoing http traffic to any port other than 80 is not redirected

cython functions coverage revsited

A few days back i showed how to get coverage of cython functions using Ned Batchelder's coverage.py
In that post i had posted a patch to coveragepy to enable cython function coverage.

However i realize that many of my friends have never applied a patch before, dont have admin rights on few machines few other problems which may hinder their using this new feature, so here's a good new for you.

I've rewritten the patch into a single file pyx_coverage.py . You can directly use this file instead of 'coverage' command to get cython function coverage, no need to patch anything. You still need to have Ned's coveragepy installed though.

All commands/options/configuration files for coveragepy are applicable here too.

To find coverage of cython files (pyx extension) you need to do following:
1. compile cython code to 'c' with directive profile=True
2. keep source pyx files in same locations as the compiled .so files
    i.e. use 'python setup.py build_ext --inplace' or 'python setup.py develop'
3. run coverage (this file) with the option timid enabled (can also set in in .coveragerc)
    i.e. 'python pyx_coverage.py run --timid my_module.py'

You can use nose test collector as follows:
$ python pyx_coverage.py run /path/to/nosetests /path/to/source

replacing the /paths as appropriate

Download the file from here: https://sites.google.com/site/pankaj86/files/pyx_coverage.py

Reader's Bonus: If you can help me write a python c extension for this treat assured.
Hint: See Ned's coverage.tracer python c extension.

cython functions coverage using coverage.py

For all the coders out there, if you have not been writing unit tests for your code then god bless you, but if you do write tests here's another tool you must use: code coverage.
In python, on of the most popular code coverage tools is Ned Batchelder's coverage.py. It reports statement coverage of all your tests and also can report coverage in beautiful html pages.Its a very nice tool and also integrates well with testing frameworks such as nose to automate your testing and coverage reporting tasks.

But, for all those who use cython, you must surely be aware of the difficulties it brings along while testing, you can never be sure if "all is well". coverage.py doesn't report coverage of cython modules, as those are compiled into native functions.
To mitigate this problem to some extent, i wrote a simple patch to enable coverage.py to report function coverage (not statement coverage) of cython pyx files too, hurray. So now after applying the patch to coverage, all you need to do is:

1. compile cython code with profiling enabled (cython --directive profile=True)
2. run your tests under coverage as you would normally do taking care to add the timid option (coverage --timid test.py)
3. ???
4. profit

So now that you can profit, i'd be greatly thankful if someone comes up and writes this patch into the tracer.c file in the coverage source (it's very simple, get the source using $ hg clone http://bitbucket.org/ned/coveragepy)