rtconsole -- Runtime Console
Have you ever wanted to see what your python application was doing while it was running? What about inspecting the state of any of its variables at runtime, and perhaps changing them in real time? And why not do it from a really nice interface like an IPython console while you are at it?
With the advent of cloud computing, more and more applications are adopting a SaaS framework. Loosely coupled applications talk to each other in real time, and it becomes increasingly difficult in such a world to easily “isolate all the inputs” and to make reproducible test cases without investing a large amount of time to log all intermediate states of your application. The inputs may come from other web services which could frankly be ANYTHING.
This is further complicated in the domain of data analysis where it’s not so much that it’s throwing out an error as much as the results are just a little suspect. In that context, it might be much more intuitive to get immediate feedback by checking/changing some inputs and see how it behaves than to create a full blown test case for each incremental change.
Even if you have full control of the application, a typical workflow could be something like this:
- stopping the application
- writing logging statements for something suspect
- rerunning the application and wait for something weird to happen
- piece together the information from the log, and building a precise test case
- realizing that the issue isn’t there, and you need to log some other part of the application
Instead it could be more like:
- start the application with rtconsole enabled
- inspect the state and change some variables to see how the application responds
- once you have a good idea of where the problem is, just write some commands to dump the states you care about right there and then
That’s the driving motivation behind rtconsole.
Copy rtconsole.py into your project and run the
test_console.py script, simplified version reproduced below:
import time from rtconsole import start_console start_console(locals()) time.sleep(5) # takes some time for it to start t = 0 hello = lambda x: x**2 while True: t += 1 s = hello(t) time.sleep(1) print t # this will print to the ipython console
Assuming no other ipython sessions are running, you can now access the script via the IPython console:
:~/Sites/env/default/project/rtconsole mirage007$ipython console --existing Python 2.7.5 (default, Sep 7 2013, 18:00:40) Type "copyright", "credits" or "license" for more information. IPython 2.2.0 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. In : t --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-1-b7269fa25085> in <module>() ----> 1 t NameError: name 't' is not defined In : t hello Out: 2 In : t Out: 7
We can see that
t wasn’t initialized when we first asked for it, but eventually gets created and incremented
In : s Out: 64 In : (t,s) Out: (13, 169) In : hello Out: <function __main__.<lambda>> In : hello? Type: function String form: <function <lambda> at 0x1117b87d0> File: /Users/mirage007/Sites/env/default/project/rtconsole/test_console.py Definition: hello(x) Docstring: <no docstring> In : hello?? Type: function String form: <function <lambda> at 0x1117b87d0> File: /Users/mirage007/Sites/env/default/project/rtconsole/test_console.py Definition: hello(x) Source: hello = lambda x: x**2
Docstrings and code inspection works
In : hello = lambda x: x+5 In : hello?? Type: function String form: <function <lambda> at 0x1105cc410> File: /Users/mirage007/Sites/env/default/project/rtconsole/<ipython-input-9-aa74116e9749> Definition: hello(x) Source: hello = lambda x: x+5
It even works when the function has been defined on the fly.
In : (t,s) Out: (41, 46)
s is now computed using the new hello function
To close out of the Kernel, press
Some key points:
- The print statement was redirected to the IPython console.
- We can also change it as it runs, and the app just keeps running with the new code
- We still have all the benefit from using IPython for introspection.
How this works
rtconsole patches the IPython kernel so that it can be run in a python thread. The IPython zmq kernel cannot do that as of IPython 2.2 because it uses the python
signal module, which according to documentation, can only be called on the main thread. Not being aware of this while also not being so familiar with the internals of IPython, I found out the hard way while trying to hook an rtconsole and an IPython client together. The failure of a
signal call would crash the application in such a way that no exception was thrown.
- I also haven’t found a way to redefine functions when the method uses predefined variables i.e.
hello = lambda x: t+5would crash
test_console.pywith the error
"global name 't' is not defined"
- When the application
test_console.pyterminates, the history manager throws an error, i haven’t found this to impact anything but definitely would like to see if anyone has insights to address these issues.
With the domain of data analysis and cloud computing becoming increasingly available and easy to use, the lines between developer and user is becoming increasingly blurred. This tool aims to speedup the debugging process and make python applications less of a black box. Please let me know if this post was interesting or if you have ideas on how to make this tool more useful.
blog comments powered by Disqus