SexyRPC |
[SourceForge
Project Page]
[Downloads] [Mailing Lists] |
sexyRPC - S-Expression RPC Library
sexyRPC 0.1.0
The latest version of this software and documentation will always be found at http://sexyrpc.sourceforge.net/.
sexyRPC is an S-expression Remote Procedure Call library. sexyRPC is similar to the xmlrpc spec ( see http://www.xmlrpc.com/ ) created by Dave Winer. Unlike xmlrpc, method calls and responses are written in S-expressions instead of XML. S-expressions are much more compact than XML and therefore greatly reduce the network impact of performing remote procedure calls.
____ ____ ______
________ _ ____ __/ __ \/ __ \/ ____/
/ ___/ _ \| |/_/ / / / /_/ / /_/ / /
(__ ) __/> </ /_/ / _, _/ ____/ /___
/____/\___/_/|_|\__, /_/ |_/_/ \____/
S-expression /____/Remote Procedure Call
v0.1.0
sexyRPC is a Remote Procedure Calling protocol that uses S-expressions to send method requests and receive responses.
S-expressions are resursively defined as being comprised of either atoms or s-expressions.
Here is an s-expression example
(a (b c))
This example contains an atom ``a'' followed by an s-expression that contains two atoms ``b'' and ``c''. S-expressions are very compact and simple.
Atoms which do not contain any spaces do not need quotes.
(one two three)
However, if an atom has an whitespace it must be quoted.
("atom one" "atom two" "atom three")
sexyRPC uses special atoms (see tables below) to formulate requests for procedure calls and generate responses.
(? get_city_name (i(94709)) )
In this example we are requesting method get_city_name and passing it a single integer 94709. Parameter formats are explained below.
The following is a boxcarred request example...
(? get_city_name (i(94709)) get_city_zipcode (s("Berkeley, CA")) )
would call two remote methods passing the first an integer and the second a string.
If you are calling a function with no arguments, you must specify an empty s-expression for the parameters as such..
(? my_method ())
This s-expression would call the method my_method without any parameters.
Atom Type Examples
------------------------------------------------------------------
i four-byte signed integer i(56)
h eight-byte signed integer h(9439439420)
s string s("hello world!")
d double-precision float d(3.14) d(-5.89)
t date/time in iso8601 format t(20030211T14:54:22)
b base64-encoded binary b(JfdkjfDdjkjDKjf=)
The type must be specified in order to be valid sexyRPC. There is no default type. The h (hyper) type (h) is only supported on 64-bit architectures (32-bit with long long?)
For example...
(? get_city_zipcode (Berkeley) )
is invalid sexyRPC since it has a typeless parameter. (Question: should we have a typeless parameter default to string as in xmlrpc? That would mean that the code is a bit more complex and slow and only saves 3 bytes. Hmm.)
Atom Type Examples
-----------------------------------------------------------------------
r structure (struct) w/name value pairs r(name s("Bob") age i(56))
a array with all elements of same type a(i(10 9 8 7 6 5 4 3 2 1))
m mixed array with mixed type elements m(i(10) s(nine) d(8.0))
? request structure (? add(i(5 8)))
. response structure (. (i(13)))
! error message !(345 "That is an error")
The following is a valid sexyRPC request
(?)
which should be replied to with a
(.)
This is a good way to check if you are speaking to a sexyRPC service.
For example...
r (name s("Mike Howard") age i(45) weight i(135) )
is a structure with three elements: name, age and weight. Each of the elements is followed by a value.
Structures can be recursive.
For example...
r (name r(first s(Mike) last s(Howard)) age i(45) r(weight i(135) units s(lbs.)))
Here is an example of an integer array...
a(i(10 9 8 7 6 5 4 3 2 1))
and here is an example of a mixed array...
m(i(10 9 8 7 6 5 4 3 2 1) s("Blast Off!"))
As you can see the integer array consists of integer elements only while the mixed array has both integer and string elements.
Arrays may also be resursive as in...
m(s(one) m(s(foo bar baz) i(5)))
or
a(a(i(1 2)) a(i(2 3)) a(i(3 4)))
or
m(a(i(1 2)) m(i(2) s(three)) a(i(4 5)))
(? get_city_zipcode(s(Berkeley)))
..might be..
(. (i(94709)))
If you boxcarred your request...
(? get_city_zipcode(s("Berkeley, CA"))
get_city_zipcode(s("St. Louis,MO")))
..the response might be..
(. (i(94709))
(i(63108)))
It is important to note that a response is not required. (Do we need to explicitly state the necessity of a response? Can a client demand (no) response? UDP messaging for example. How do I handle void methods? If we boxcar I need to reply with an empty response. I think.).
It is possible that you can receive a void return value. That signifies that the remote procedure succeeded (no errors) but returns no value.
For example...
(? remove_tmp_files() get_number_of_processes_running())
Asks for two methods to be run with no parameters. You might get a return like...
(. () (i(4)))
This s-expression shows that the first method returned void: (). If the remote_tmp_files method failed it would have returned an ! error message. A better return value might be the number of files removed. The second method however returned the number of running processes.
An error response to...
(? get_city_zipcode(s("Berkeley, CA")))
..might be..
(. (!(42 "There are multiple zip codes for that city")))
The ! atom is a special atom to denote a remote process failure. The ! s-expression has a two atoms: an error number (errno) followed by an error string. It is completely up to the application programmer what error number and string to use.
If a method name has spaces in it, you must place quotes around it.
For example...
(? "get city zipcode"(s("Berkeley, CA")))
Secondly, a method name can not begin with a sys. prefix since all internal system methods use that prefix: sys.listMethods and sys.methodHelp.
The sys.listMethods method will return an array of strings for each method in the sys. This method takes no parameters and returns a list of strings.
For example,
(? sys.listMethods())
..might return..
(. (a(s("find user" "list processes" "check alerts"))))
which tells the client there are three methods that can be called on this server: find user, list processes and check alerts.
The sys.methodHelp method takes a single string parameter (the name of a method) and returns a help string about the method.
For example..
(? sys.methodHelp(s("find user")))
..might return..
(. (s("This method takes a username and returns the local UID for that person")))
There will also be a few system methods for authentication using keys (for example). The fact that sexyRPC can be boxcarred means that you can have one or more authentication methods in a single request expression.
For example,
(?
(sys.Authenticate(b(jkerjEkJdfoJdsj))
("find user"( s("joe") ))
(sys.Authenticate(b(dasKEJcfjkdfjEs))
("find user"( s("karen") )))
would run the first find user method with the credentials of the first authenticated key (if it authenticates of course). The second find user method may then be run with different credentials. The s-expressions would also need to be sent encrypted to protect the authentication information.
For a good example of how to use the API, take a look at the example code
in the examples directory of the sexyRPC distribution.
These functions allow developers to create sexyRPC servers, register methods, answer requests and destroy the server.
For example,
server = SRPC_ServerCreate();
if(server == NULL)
{
fprintf(stderr,"Unable to create server\n");
exit(1);
}
SRPC_ServerRegisterMethodWithHelp() this function will not set any help information for the method
which can be queried with the sys.methodHelp method.
This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.
For example,
void
method_callback(SRPC_Server server, SRPC response, SRPC params, void *userData)
{
/* Process parameters and create response */
}
server = SRPC_ServerCreate();
rval = SRPC_ServerRegisterMethod( server, "my_method", method_callback);
if(rval == SRPC_FAILURE)
{
fprintf(stderr,"Failed to register my_method with server\n");
exit(1);
}
This example, creates a sexyRPC server and registers a single method,
my_method. When a client requests my_method the sexyRPC server
will run method_callback, passing it the parameters that the client
presented. The callback will then set the correct response to send to
the client.
There is no limit on how many methods can be registered with a sexyRPC server besides physical limitations such as the amount of memory available.
SRPC_ServerRegisterMethod() but also registers a help
string explaining the method. This help string can be queried by a client
using the sys.methodHelp method. It is highly recommended that you provide
help information to clients about your methods although it is not required.
This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.
For example,
server = ServerCreate();
SRPC_ServerRegisterMethodWithHelp( server, "add", add_callback,
"This method accepts two integers, adds them and returns an integer");
These functions are used exclusively by sexyRPC clients for encoding requests to be send to a sexyRPC server.
SRPC_EndMethodCall() and
the value functions but that is not strictly necessary.
Serializing the request at this point would return (?) which is the
most basic sexyRPC request. A sexyRPC server will response to such as
request with (.).
For example,
SRPC request; char *buf; unsigned int buflen; char *string_param = "My string parameter";
request = SRPC_RequestCreate(); SRPC_StartMethodCall( request, "remote_method"); SRPC_String( request, &string_param, NULL); SRPC_EndMethodCall( request );
buf = SRPC_Serialize( client_msg, &buflen ); /* Send the buffer which buflen bytes long */ SRPC_RequestDestroy( request ); free(buf);
In this example, we are building a request for the method, remote_method,
to be run passing it a single string parameter. Once the request is built,
we serialize it and then send it. The char *buf will point to the start
of the message and unsigned int buflen will contain the length of the
serialized message.
As a note, this function does not necessarily malloc any memory. sexyRPC
has memory management to reduce the number of malloc() and free() calls
that are made.
For an example, see SRPC_RequestCreate() .
callname, to a sexyRPC request.
Any parameters to be passed to the remote method should follow directly.
Parameters are optional. Method requests should always be closed with
the SRPC_EndMethodCall() function.
This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.
See SRPC_RequestCreate() for an example.
SRPC_StartMethodCall()
and SRPC_EndMethodCall().
The function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.
See SRPC_RequestCreate() for an example.
The value functions are used by both clients and servers to encode and decode data for requests and responses. The encoding and decoding of sexyRPC data is symmetrical which allows developers to use the same function to encode or decode sexyRPC data.
Encoding
If the SRPC passed in is being encoded, then the string pointed to
by string will be
added to the sexyRPC message. If the len parameter is set to
a non-NULL value, it will be set to the length of the string encoded.
This value should be identical to strlen(*string).
Decoding
If the SRPC passed in is being decoded, then string will be
set to point to the string received. If the len parameter is set to
a non-NULL value, it will be set the length of the string received.
For an example, see SRPC_RequestCreate() .
This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.
Encoding
If the SRPC passed in is being encoded, the integer pointed to by
i will be added to the sexyRPC message.
Decoding
If the SRPC passed in is being decoded, the integer pointed to by
i will be set to the integer that is decoded.
This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.
For example,
int i = 12345; SRPC srpc;
srpc = SRPC_RequestCreate(); /* We are encoding a message here */ SRPC_StartMethodCall( srpc, "test" ); SRPC_Integer( srpc, &i ); SRPC_EndMethodCall( srpc );
would serialize to the following
(? test(i(12345)))
Encode
If the SPRC passed in is being encoded, then the double pointed to
by d will be added to the sexyRPC message.
Decode
If the SRPC passed in is being decoded, the double pointed to by
d will be set to the value of the double received.
This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure
For example,
int d = 3.14; SRPC srpc;
srpc = SRPC_RequestCreate(); /* We are encoding a message here */ SRPC_StartMethodCall( srpc, "test" ); SRPC_Double( srpc, &d ); SRPC_EndMethodCall( srpc );
would serialize to the following request
(? test(d(3.14)))
Encode
If the SRPC passed in is being encoded, then the timestamp pointed
to by t will added to the sexyRPC stream in iso8601 format.
Decode
If the SRPC passed in is being decoded, then the timestamp pointed
to be t will be set to the timestamp received.
This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.
For example,
time_t t; SRPC srpc;
t = time(NULL); srpc = SRPC_RequestCreate(); /* We are encoding a message here */ SRPC_StartMethodCall( srpc, "test" ); SRPC_Time( srpc, &t ); SRPC_EndMethodCall( srpc );
might serialize to the following request
(? test(t(20031121T13:59:07)))
Encode
If the SRPC passed in is being encoded, then len bytes will be encoded
starting at the address pointed to by ptr. If op_len is set to a non-NULL
value, op_len will be set to the size of the base64 encoded data.
Decode
If the SRPC passed in is being decoded, then len should be set to the
size of the buffer available at address ptr. If len is not large enough
how the data received, SRPC_FAILURE will be returned. If op_len is set
to a non-NULL value, it will be set to the size of the data received.
This function will return SRPC_SUCCESS on success and SRPC_FAILURE on failure.
It should be noted the SRPC_Binary() is currently memory-limited i.e. you
can only send a receive data as large as the available memory. In the
future, this limitation may be removed. If this limitation is a problem
for your particular project, please email the author.
All sexyRPC function in the API operate on SRPC structures. Serialization
functions allow you serialize SRPC to a buffer and visa versa. These
buffers are what should be sent/receive over the communication channels
(pipes, files, sockets, shared memory, etc).
SPRC structure and convert it into a
buffer. If len is set to a non-NULL, it will be set the size of the
buffer that is returned.
This function will return a pointer to the start of the buffer on success and NULL on failure.
For example,
SRPC request; char *buf; unsigned int buflen;
request = SRPC_RequestCreate(); /* Requesting a single method "test" and passing it no parameters */ SRPC_StartMethodCall(request, "test"); SRPC_EndMethodCall(request); buf = SRPC_Serialize( request, &buflen);
This code would encode a method request and then serialize it to be buffer
to be sent. At this point, buf will be NULL if there was an error
serializing otherwise it will point to the start of the buffer of length
buflen. In this case the buffer will be (? test()) and buflen will
be 10.
len, into a SRPC
structure for decoding.
This function will return NULL on error and a SRPC structure on success.
For example,
SRPC request; char *message = "(? test())";
request = SRPC_Unserialize( message, 10 );
Some utility functions
For example,
char *version = SRPC_VersionString(); fprintf(stderr,"Version is %s\n", version);
would output
Version is 0.1.0
There are example snippets throughout the this documentation. It is
also recommend that you look in the examples directory of the
sexyRPC distribution.
sexyRPC has been tested on Linux (x86/ia64), FreeBSD, NetBSD, Solaris, MacOS X and Windows/Cygwin. It should compiles and run just about anywhere you have an ANSI C compiler.
sexyRPC was built to satisfy the requirement for the third generation ganglia system http://ganglia.sf.net/ .
Info about how to get help
XMLRPC (see http://www.xmlrpc.com/ ) was a great idea in distributed computing. The web page states that it is ``designed to be as simple as possible, while allowing complex data structures to be transmitted, processed and returned.''. In that regard, XMLRPC is a success. It is simple to understand and does allow complex data structures to be processed. XMLRPC uses XML and HTTP in a straight-forward way. XMLRPC XML is very descriptive and it is easy to understand exactly what an RPC request is asking or what the response it.
The biggest limitation of XMLRPC is the size of the messages that it generates. XML messages are not nearly as efficient as other text-based message formats. The response to the large size of XML is usually, ``Just run it through zlib''. I wasn't happy with that answer. It really isn't a solution to the problem. You've just pushed the problem from the network to your CPU. Why waste CPU cycles trying to work around XML?
S-expression are much more compact than XML and can be parsed much more rapidly. Here are some examples. The XMLRPC examples are taken directly from the XMLRPC web site.
Here is the first example from the XMLRPC spec which shows a XMLRPC request...
POST /RPC2 HTTP/1.0 User-Agent: Frontier/5.1.2 (WinNT) Host: betty.userland.com Content-Type: text/xml Content-length: 181
<?xml version="1.0"?>
<methodCall>
<methodName>examples.getStateName</methodName>
<params>
<param>
<value><i4>41</i4></value>
</param>
</params>
</methodCall>
Even when we ignore the size of the HTTP POST headers, this XML message is 181 bytes is size. The reason this message would compress so well is because XML is very redundant! The cost of compressing this single message (with say zlib) is small but quickly becomes unacceptable when you have an application which is making a large volume of requests.
Here is the same request in sexyRPC...
(? examples.getStateName(i(41)))
We have reduced the message size from 181 bytes to 32 bytes (an 83% reduction!). The CPU cost of parsing this S-expression is dramatically lower than parsing XML as well.
Here is an XMLRPC response example...
HTTP/1.1 200 OK Connection: close Content-Length: 158 Content-Type: text/xml Date: Fri, 17 Jul 1998 19:55:08 GMT Server: UserLand Frontier/5.1.2-WinNT
<?xml version="1.0"?>
<methodResponse>
<params>
<param>
<value><string>South Dakota</string></value>
</param>
</params>
</methodResponse>
Ignoring the POST HTTP headers, this message is 158 bytes. Here is the same response in sexyRPC
(.(s("South Dakota")))
This response is only 22 bytes (an 87% reduction!).
Here is an XMLRPC fault example
HTTP/1.1 200 OK Connection: close Content-Length: 426 Content-Type: text/xml Date: Fri, 17 Jul 1998 19:55:02 GMT Server: UserLand Frontier/5.1.2-WinNT
<?xml version="1.0"?>
<methodResponse>
<fault>
<value>
<struct>
<member>
<name>faultCode</name>
<value><int>4</int></value>
</member>
<member>
<name>faultString</name>
<value><string>Too many parameters.</string></value>
</member>
</struct>
</value>
</fault>
</methodResponse>
This fault response is 426 bytes! Here is the same fault in sexyRPC
(.(!(4 "Too many parameters.")))
We have reduced the fault response from 426 bytes to 32 bytes. A 93% reduction!
The XMLRPC spec was not designed to allow for boxcarring requests together. For network efficiency, it makes sense to be able to send multiple requests and to get multiple responses. sexyRPC allows requested to be easily boxcarred.
Immitation is the sincerest form of flattery. The truth is that sexyRPC is XMLRPC repackaged with an emphasis places on network and computational efficiency. To achieve that efficiency, XML was replaced with S-expressions. sexyRPC would not exist without XMLRPC.
Matt Sottile's small, fast s-expression library, http://sexpr.sourceforge.net/
UserLand XML-RPC, http://www.xmlrpc.com/
XML-RPC Specification, http://www.xmlrpc.com/spec
xdr(3), rpc(3).
Matt Massie <massie@CS.Berkeley.EDU>