Call a script over the network

I have a Python script on a box that I want to call from a Java server, over the network. I know that I can start a server on the Python side and then connect to it from the Java site, but I am trying to see if there is a way to avoid the overhead of running a server on the Python side. Is there any other way of doing it?

Well, put simply, without a server, the Python side won’t be able to receive incoming network messages. (Or, put differently, if the Python side is able to receive network messages, then there must be a server involved.)

There are a few options you have here.

Incorporate a simple custom server into the Python application. This server can be something you write yourself, based on network sockets and your own custom protocol. It takes some work, but you can tailor the protocol to your exact needs and leave out all the overhead that comes with more generic protocols, which is why I would only choose this route if performance is critical and network overhead is going to be an important factor. If the Python server is connected to the (public) internet, firewall it to prevent access to your custom server from the outside. Also, because your application will be long-running, you need to make sure you have good error handling, so that it can recover whenever anything goes wrong. Wrap it in a shell script loop (run server – sleep – repeat) if you want to be sure.

Incorporate an HTTP server into the Python application. There are a few libraries you can use for this, so the programming and testing overhead is going to be much less than for the custom protocol. Since it’s HTTP though, you’ll have a bit more overhead, and you’ll be bound to the request/response paradigm, which is probably fine considering that the Java server initiates the connection. Just like the custom protocol, this solution needs protection – either firewall it, or have it listen on localhost only and put it behind a reverse proxy.

Implement the CGI protocol and run behind a “real” web server. CGI is really easy to implement, but you’ll need a web server to drive it. Also, because CGI restarts your program for each request, you’ll have to design it to avoid long startup times. On the upside, you won’t have to fiddle with HTTP libraries, network sockets, and worry about many of the related security issues – just keep the host web server secure and don’t do anything stupid inside your CGI, and you’ll be fine.

Use ssh. This requires absolutely no extra effort on the Python part, and probably (I’m not a Java expert, so I don’t really know) not much effort on the Java side either; you will have to create and set up a suitable ssh key on the Java side though, and give it access on the Python side. You have to make it an ssh key without a passphrase, because otherwise the server won’t be able to use it without human intervention; this in turn means you probably want a dedicated user for just this purpose on the Python side, and lock that user down to the absolute bare requirements.

Use some sort of message-queue library. I’ve used ZeroMQ in the past, which is available for all popular programming languages, but there are others. You’ll still be implementing a custom protocol, but the transport part of it is handled for you, quite efficiently at that. This approach is especially suitable when you need reliable and efficient communication channels, and can’t afford restarting your script for each request. Here again, you need to catch any and all crashes to make sure it recovers.

Communicate through a shared persistent storage device. This can be a database, a shared filesystem, memcache, or maybe a server that exposes some sort of persistent storage API. This requires polling on the Python part, so it might not be the most efficient solution, plus you need to look out for race conditions and other timing-related problems. It is a great solution though if you have large quantities of data to process per message, and (provided your processing is designed to be atomic) it will survive close to every conceivable crash scenario without data loss.

A completely orthogonal choice is what your (message-level) API is going to look like. For the SSH case, the matter is already clear – you’re just going to pass some command-line options to your script, not much choice there. For the rest, however, your message body can use whatever format you like – JSON, XML, plain text, SOAP, you name it. Which one makes the most sense depends on the kind of data you need to feed your script.

Run lighttpd and put a hook on there to create a file. Have a cron job check if that file exists every interval. If it does, do action(s), then delete it.

Running a small server (develop it yourself) written in Java on the python box shouldn’t have too much overhead, if you don’t want to do this at all then there is no real native way in Java to do it.

Some options though:

If you have a web server running on the box already, create a CGI script that runs the desired program. When the java box makes a HTTP connection to the URL, the python script is run.
Use a remote shell and connect to it, e.g. System.exec("rsh remotebox progname");
Use a secure shell (like above) e.g. System.exec("ssh remotebox progname");

We used to run scripts on various boxes by having a chron job (or Scheduled Task on Windows) that checks the date on a file in a shared folder every minute. If the date changed, run the script. Otherwise, just exit and be run again in a minute. Really low-tech.

A simple way, you can use java Runtime and ssh. Sample as below:

String cmd = "sshpass -p <pwd> ssh <user>@<hostname> python <pythonScript> [parameters]";
Runtime.getRuntime().exec(cmd);

Tested works.

Filed under: softwareengineering - @ 01:25

Thẻ: java, python, rpc

Thiết kế website giá rẻ

Danh mục

Call a script over the network