SexyRPC


[SourceForge Project Page]

[Downloads]
[Mailing Lists]

Copyright © 2003 University of California, Berkeley
Author: Matt Massie <massie@CS.Berkeley.EDU>



NAME

sexyRPC - S-Expression RPC Library


VERSION

sexyRPC 0.1.0

The latest version of this software and documentation will always be found at http://sexyrpc.sourceforge.net/.


SYNOPSIS

sexyRPC is an S-expression Remote Procedure Call library. sexyRPC is similar to the xmlrpc spec ( see http://www.xmlrpc.com/ ) created by Dave Winer. Unlike xmlrpc, method calls and responses are written in S-expressions instead of XML. S-expressions are much more compact than XML and therefore greatly reduce the network impact of performing remote procedure calls.

Server-Side Functions
SRPC_ServerCreate, SRPC_ServerDestroy, SRPC_ServerRegisterMethod, SRPC_ServerRegisterMethodWithHelp, SRPC_ServerResponse, SRPC_ResponseDestroy.

Client-Side Functions
SRPC_RequestCreate, SRPC_RequestDestroy, SRPC_StartMethodCall, SRPC_EndMethodCall.

Value Functions
SRPC_String, SRPC_Integer, SRPC_Double, SRPC_Time, SRPC_Binary.

Serialization Functions
SRPC_Serialize, SRPC_Unserialize.

Misc. Functions
SRPC_VersionString.


DESCRIPTION

                          ____  ____  ______
    ________  _  ____  __/ __ \/ __ \/ ____/
   / ___/ _ \| |/_/ / / / /_/ / /_/ / /     
  (__  )  __/>  </ /_/ / _, _/ ____/ /___
 /____/\___/_/|_|\__, /_/ |_/_/    \____/
 S-expression   /____/Remote Procedure Call
                v0.1.0

sexyRPC is a Remote Procedure Calling protocol that uses S-expressions to send method requests and receive responses.

S-expressions are resursively defined as being comprised of either atoms or s-expressions.

Here is an s-expression example

  (a (b c))

This example contains an atom ``a'' followed by an s-expression that contains two atoms ``b'' and ``c''. S-expressions are very compact and simple.

Atoms which do not contain any spaces do not need quotes.

  (one two three)

However, if an atom has an whitespace it must be quoted.

  ("atom one" "atom two" "atom three")

sexyRPC uses special atoms (see tables below) to formulate requests for procedure calls and generate responses.

Request Example
 (? get_city_name (i(94709)) )

In this example we are requesting method get_city_name and passing it a single integer 94709. Parameter formats are explained below.

Request Details
The entire request must be contained with a single s-expression list with the first atom being ``?'' followed by a single method (as the example above has) or you can boxcar a list of methods together for network efficiency.

The following is a boxcarred request example...

  (? get_city_name (i(94709)) get_city_zipcode (s("Berkeley, CA")) )

would call two remote methods passing the first an integer and the second a string.

If you are calling a function with no arguments, you must specify an empty s-expression for the parameters as such..

  (? my_method ())

This s-expression would call the method my_method without any parameters.

Scalars
The following table lists the scalar types.
  Atom  Type                                Examples
  ------------------------------------------------------------------
  i     four-byte signed integer            i(56)
  h     eight-byte signed integer           h(9439439420)
  s     string                              s("hello world!")
  d     double-precision float              d(3.14) d(-5.89)
  t     date/time in iso8601 format         t(20030211T14:54:22)
  b     base64-encoded binary               b(JfdkjfDdjkjDKjf=)

The type must be specified in order to be valid sexyRPC. There is no default type. The h (hyper) type (h) is only supported on 64-bit architectures (32-bit with long long?)

For example...

  (?  get_city_zipcode (Berkeley) )

is invalid sexyRPC since it has a typeless parameter. (Question: should we have a typeless parameter default to string as in xmlrpc? That would mean that the code is a bit more complex and slow and only saves 3 bytes. Hmm.)

Non-Scalar Types
sexyRPC has the following non-scalar types as well
  Atom  Type                                   Examples
  -----------------------------------------------------------------------
  r     structure (struct) w/name value pairs  r(name s("Bob") age i(56))
  a     array with all elements of same type   a(i(10 9 8 7 6 5 4 3 2 1))
  m     mixed array with mixed type elements   m(i(10) s(nine) d(8.0))
  ?     request structure                      (? add(i(5 8)))
  .     response structure                     (. (i(13)))
  !     error message                          !(345 "That is an error")

The following is a valid sexyRPC request


  (?)

which should be replied to with a

  (.)

This is a good way to check if you are speaking to a sexyRPC service.

Structures
A value can also be a structure (struct). A structure contains a list of members that have a name and value. The list alternates between element names and element values.

For example...

  r (name s("Mike Howard") age i(45) weight i(135) )

is a structure with three elements: name, age and weight. Each of the elements is followed by a value.

Structures can be recursive.

For example...

  r (name r(first s(Mike) last s(Howard)) age i(45) r(weight i(135) units s(lbs.)))

Arrays
Arrays can be homogenous or mixed.

Here is an example of an integer array...

  a(i(10 9 8 7 6 5 4 3 2 1))

and here is an example of a mixed array...

  m(i(10 9 8 7 6 5 4 3 2 1) s("Blast Off!"))

As you can see the integer array consists of integer elements only while the mixed array has both integer and string elements.

Arrays may also be resursive as in...

  m(s(one) m(s(foo bar baz) i(5)))

or

  a(a(i(1 2)) a(i(2 3)) a(i(3 4)))

or

  m(a(i(1 2)) m(i(2) s(three)) a(i(4 5)))

Response Examples
A response to the following call example...
  (? get_city_zipcode(s(Berkeley)))

..might be..

  (. (i(94709)))

If you boxcarred your request...

  (? get_city_zipcode(s("Berkeley, CA"))
     get_city_zipcode(s("St. Louis,MO")))

..the response might be..

  (. (i(94709))
     (i(63108)))

It is important to note that a response is not required. (Do we need to explicitly state the necessity of a response? Can a client demand (no) response? UDP messaging for example. How do I handle void methods? If we boxcar I need to reply with an empty response. I think.).

It is possible that you can receive a void return value. That signifies that the remote procedure succeeded (no errors) but returns no value.

For example...

  (? remove_tmp_files() get_number_of_processes_running())

Asks for two methods to be run with no parameters. You might get a return like...

  (. () (i(4)))

This s-expression shows that the first method returned void: (). If the remote_tmp_files method failed it would have returned an ! error message. A better return value might be the number of files removed. The second method however returned the number of running processes.

Fault Example
The response examples above should generate an error from the remote process since both Berkeley, CA and St. Louis, MO have many zip codes. It is impossible to get the zip code without more detailed address information.

An error response to...

  (? get_city_zipcode(s("Berkeley, CA")))

..might be..

  (. (!(42 "There are multiple zip codes for that city")))

The ! atom is a special atom to denote a remote process failure. The ! s-expression has a two atoms: an error number (errno) followed by an error string. It is completely up to the application programmer what error number and string to use.

Introspection
Method names are completely arbitrary and can take on whatever values that developers like with two exceptions.

If a method name has spaces in it, you must place quotes around it.

For example...

  (? "get city zipcode"(s("Berkeley, CA")))

Secondly, a method name can not begin with a sys. prefix since all internal system methods use that prefix: sys.listMethods and sys.methodHelp.

o
sys.listMethods

The sys.listMethods method will return an array of strings for each method in the sys. This method takes no parameters and returns a list of strings.

For example,

  (? sys.listMethods())

..might return..

  (. (a(s("find user" "list processes" "check alerts"))))

which tells the client there are three methods that can be called on this server: find user, list processes and check alerts.

o
sys.methodHelp

The sys.methodHelp method takes a single string parameter (the name of a method) and returns a help string about the method.

For example..

  (? sys.methodHelp(s("find user")))

..might return..

  (. (s("This method takes a username and returns the local UID for that person")))

Authentication
(This section is not cooked yet)

There will also be a few system methods for authentication using keys (for example). The fact that sexyRPC can be boxcarred means that you can have one or more authentication methods in a single request expression.

For example,

  (?
    (sys.Authenticate(b(jkerjEkJdfoJdsj))
    ("find user"( s("joe") ))
    (sys.Authenticate(b(dasKEJcfjkdfjEs))
    ("find user"( s("karen") )))

would run the first find user method with the credentials of the first authenticated key (if it authenticates of course). The second find user method may then be run with different credentials. The s-expressions would also need to be sent encrypted to protect the authentication information.


APPLICATION PROGRAMMING INTERFACE (API)

For a good example of how to use the API, take a look at the example code in the examples directory of the sexyRPC distribution.

Server-Side Functions

These functions allow developers to create sexyRPC servers, register methods, answer requests and destroy the server.

SRPC_Server SRPC_ServerCreate(void)
This function will create a sexyRPC server ready to register methods and answer client requests. It returns NULL on failure.

For example,

  server = SRPC_ServerCreate();
  if(server == NULL)
    {
      fprintf(stderr,"Unable to create server\n");
      exit(1);
    }

int SRPC_ServerRegisterMethod( SRPC_Server server, char *name, SRPC_Callback cb)
This function registers a method on a sexyRPC server. Unlike SRPC_ServerRegisterMethodWithHelp() this function will not set any help information for the method which can be queried with the sys.methodHelp method.

This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.

For example,

  void
  method_callback(SRPC_Server server, SRPC response, SRPC params, void *userData) 
  {
    /* Process parameters and create response */
  }
  server = SRPC_ServerCreate();
  rval = SRPC_ServerRegisterMethod( server, "my_method", method_callback);
  if(rval == SRPC_FAILURE)
    {
      fprintf(stderr,"Failed to register my_method with server\n");
      exit(1);
    }

This example, creates a sexyRPC server and registers a single method, my_method. When a client requests my_method the sexyRPC server will run method_callback, passing it the parameters that the client presented. The callback will then set the correct response to send to the client.

There is no limit on how many methods can be registered with a sexyRPC server besides physical limitations such as the amount of memory available.

int SRPC_ServerRegisterMethodWithHelp( SRPC_Server server, char *name, SRPC_Callback cb, char *helpstr)
This function is identical to SRPC_ServerRegisterMethod() but also registers a help string explaining the method. This help string can be queried by a client using the sys.methodHelp method. It is highly recommended that you provide help information to clients about your methods although it is not required.

This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.

For example,

  server = ServerCreate();
  SRPC_ServerRegisterMethodWithHelp( server, "add", add_callback,
       "This method accepts two integers, adds them and returns an integer");

void SRPC_ServerDestroy( SRPC_Server server )
This function frees all structures associated with a sexyRPC server.

Client-Side Functions

These functions are used exclusively by sexyRPC clients for encoding requests to be send to a sexyRPC server.

SRPC SRPC_RequestCreate( void )
This function will create a sexyRPC request. This request should have method calls added using SRPC_StartMethodCall(), SRPC_EndMethodCall() and the value functions but that is not strictly necessary.

Serializing the request at this point would return (?) which is the most basic sexyRPC request. A sexyRPC server will response to such as request with (.).

For example,

  SRPC request;
  char *buf;
  unsigned int buflen;
  char *string_param = "My string parameter";
  request = SRPC_RequestCreate();
  SRPC_StartMethodCall( request, "remote_method");
  SRPC_String( request, &string_param, NULL);
  SRPC_EndMethodCall( request );
  buf = SRPC_Serialize( client_msg, &buflen );  
  /* Send the buffer which buflen bytes long */
  SRPC_RequestDestroy( request );
  free(buf);

In this example, we are building a request for the method, remote_method, to be run passing it a single string parameter. Once the request is built, we serialize it and then send it. The char *buf will point to the start of the message and unsigned int buflen will contain the length of the serialized message.

As a note, this function does not necessarily malloc any memory. sexyRPC has memory management to reduce the number of malloc() and free() calls that are made.

void SRPC_RequestDestroy( SRPC srpc )
This function will free all memory allocated for a sexyRPC request structure

For an example, see SRPC_RequestCreate() .

int SRPC_StartMethodCall( SRPC srpc, char *callname )
This function adds a new method call, callname, to a sexyRPC request. Any parameters to be passed to the remote method should follow directly. Parameters are optional. Method requests should always be closed with the SRPC_EndMethodCall() function.

This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.

See SRPC_RequestCreate() for an example.

int SRPC_EndMethodCall( SRPC srpc )
This function closes a method call request. Any parameters that should sent to the remote function should be added between SRPC_StartMethodCall() and SRPC_EndMethodCall().

The function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.

See SRPC_RequestCreate() for an example.

Value Functions

The value functions are used by both clients and servers to encode and decode data for requests and responses. The encoding and decoding of sexyRPC data is symmetrical which allows developers to use the same function to encode or decode sexyRPC data.

int SRPC_String( SRPC srpc, char **string, unsigned int *len )
This function will encode or decode a string from a sexyRPC message.

Encoding

If the SRPC passed in is being encoded, then the string pointed to by string will be added to the sexyRPC message. If the len parameter is set to a non-NULL value, it will be set to the length of the string encoded. This value should be identical to strlen(*string).

Decoding

If the SRPC passed in is being decoded, then string will be set to point to the string received. If the len parameter is set to a non-NULL value, it will be set the length of the string received.

For an example, see SRPC_RequestCreate() .

This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.

int SRPC_Integer( SRPC srpc, int *i )
This function will encode or decode an integer from a sexyRPC message.

Encoding

If the SRPC passed in is being encoded, the integer pointed to by i will be added to the sexyRPC message.

Decoding

If the SRPC passed in is being decoded, the integer pointed to by i will be set to the integer that is decoded.

This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.

For example,

  int i = 12345;
  SRPC srpc;
  srpc = SRPC_RequestCreate();
  /* We are encoding a message here */
  SRPC_StartMethodCall( srpc, "test" );
  SRPC_Integer( srpc, &i );
  SRPC_EndMethodCall( srpc );

would serialize to the following

  (? test(i(12345)))

int SRPC_Double( SRPC srpc, double *d )
This function will encode/decode a double-precision float in a sexyRPC message.

Encode

If the SPRC passed in is being encoded, then the double pointed to by d will be added to the sexyRPC message.

Decode

If the SRPC passed in is being decoded, the double pointed to by d will be set to the value of the double received.

This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure

For example,

  int d = 3.14;
  SRPC srpc;
  srpc = SRPC_RequestCreate();
  /* We are encoding a message here */
  SRPC_StartMethodCall( srpc, "test" );
  SRPC_Double( srpc, &d );
  SRPC_EndMethodCall( srpc );

would serialize to the following request

  (? test(d(3.14)))

int SRPC_Time( SRPC srpc, time_t *t )
This function will encode/decode a timestamp in a sexyRPC message

Encode

If the SRPC passed in is being encoded, then the timestamp pointed to by t will added to the sexyRPC stream in iso8601 format.

Decode

If the SRPC passed in is being decoded, then the timestamp pointed to be t will be set to the timestamp received.

This function returns SRPC_SUCCESS on success and SRPC_FAILURE on failure.

For example,

  time_t t;
  SRPC srpc;
  t = time(NULL);
  srpc = SRPC_RequestCreate();
  /* We are encoding a message here */
  SRPC_StartMethodCall( srpc, "test" );
  SRPC_Time( srpc, &t );
  SRPC_EndMethodCall( srpc );

might serialize to the following request

  (? test(t(20031121T13:59:07)))

int SRPC_Binary( SRPC srpc, void *ptr, unsigned int len, unsigned int *op_len )
This function will encode/decode binary data in a sexyRPC message.

Encode

If the SRPC passed in is being encoded, then len bytes will be encoded starting at the address pointed to by ptr. If op_len is set to a non-NULL value, op_len will be set to the size of the base64 encoded data.

Decode

If the SRPC passed in is being decoded, then len should be set to the size of the buffer available at address ptr. If len is not large enough how the data received, SRPC_FAILURE will be returned. If op_len is set to a non-NULL value, it will be set to the size of the data received.

This function will return SRPC_SUCCESS on success and SRPC_FAILURE on failure.

It should be noted the SRPC_Binary() is currently memory-limited i.e. you can only send a receive data as large as the available memory. In the future, this limitation may be removed. If this limitation is a problem for your particular project, please email the author.

Serialization Functions

All sexyRPC function in the API operate on SRPC structures. Serialization functions allow you serialize SRPC to a buffer and visa versa. These buffers are what should be sent/receive over the communication channels (pipes, files, sockets, shared memory, etc).

char * SRPC_Serialize( SRPC msg, unsigned int *len )
This function will take a SPRC structure and convert it into a buffer. If len is set to a non-NULL, it will be set the size of the buffer that is returned.

This function will return a pointer to the start of the buffer on success and NULL on failure.

For example,

  SRPC request;
  char *buf;
  unsigned int buflen;
  request = SRPC_RequestCreate();
  /* Requesting a single method "test" and passing it no parameters */
  SRPC_StartMethodCall(request, "test");
  SRPC_EndMethodCall(request);
  buf = SRPC_Serialize( request, &buflen);

This code would encode a method request and then serialize it to be buffer to be sent. At this point, buf will be NULL if there was an error serializing otherwise it will point to the start of the buffer of length buflen. In this case the buffer will be (? test()) and buflen will be 10.

SRPC SRPC_Unserialize( char *buf, int len )
This function will convert a buffer of length, len, into a SRPC structure for decoding.

This function will return NULL on error and a SRPC structure on success.

For example,

  SRPC request;
  char *message = "(? test())";
  request = SRPC_Unserialize( message, 10 );

Misc. Functions

Some utility functions

char * SRPC_VersionString(void)
This function returns the version string of the sexyRPC library.

For example,

  char *version = SRPC_VersionString();
  fprintf(stderr,"Version is %s\n", version);

would output

  Version is 0.1.0


EXAMPLES

There are example snippets throughout the this documentation. It is also recommend that you look in the examples directory of the sexyRPC distribution.


IMPLEMENTATION NOTES

sexyRPC has been tested on Linux (x86/ia64), FreeBSD, NetBSD, Solaris, MacOS X and Windows/Cygwin. It should compiles and run just about anywhere you have an ANSI C compiler.


HISTORY

sexyRPC was built to satisfy the requirement for the third generation ganglia system http://ganglia.sf.net/ .


BUG REPORTS AND SUPPORT

Info about how to get help


Comparing XMLRPC and sexyRPC

XMLRPC (see http://www.xmlrpc.com/ ) was a great idea in distributed computing. The web page states that it is ``designed to be as simple as possible, while allowing complex data structures to be transmitted, processed and returned.''. In that regard, XMLRPC is a success. It is simple to understand and does allow complex data structures to be processed. XMLRPC uses XML and HTTP in a straight-forward way. XMLRPC XML is very descriptive and it is easy to understand exactly what an RPC request is asking or what the response it.

The biggest limitation of XMLRPC is the size of the messages that it generates. XML messages are not nearly as efficient as other text-based message formats. The response to the large size of XML is usually, ``Just run it through zlib''. I wasn't happy with that answer. It really isn't a solution to the problem. You've just pushed the problem from the network to your CPU. Why waste CPU cycles trying to work around XML?

S-expression are much more compact than XML and can be parsed much more rapidly. Here are some examples. The XMLRPC examples are taken directly from the XMLRPC web site.

Here is the first example from the XMLRPC spec which shows a XMLRPC request...

  POST /RPC2 HTTP/1.0
  User-Agent: Frontier/5.1.2 (WinNT)
  Host: betty.userland.com
  Content-Type: text/xml
  Content-length: 181
  <?xml version="1.0"?>
  <methodCall> 
    <methodName>examples.getStateName</methodName> 
      <params> 
        <param> 
          <value><i4>41</i4></value> 
        </param> 
      </params> 
  </methodCall>

Even when we ignore the size of the HTTP POST headers, this XML message is 181 bytes is size. The reason this message would compress so well is because XML is very redundant! The cost of compressing this single message (with say zlib) is small but quickly becomes unacceptable when you have an application which is making a large volume of requests.

Here is the same request in sexyRPC...

  (? examples.getStateName(i(41)))

We have reduced the message size from 181 bytes to 32 bytes (an 83% reduction!). The CPU cost of parsing this S-expression is dramatically lower than parsing XML as well.

Here is an XMLRPC response example...

  HTTP/1.1 200 OK
  Connection: close
  Content-Length: 158
  Content-Type: text/xml
  Date: Fri, 17 Jul 1998 19:55:08 GMT
  Server: UserLand Frontier/5.1.2-WinNT
  <?xml version="1.0"?> 
  <methodResponse> 
    <params> 
      <param> 
        <value><string>South Dakota</string></value> 
      </param> 
    </params> 
  </methodResponse>

Ignoring the POST HTTP headers, this message is 158 bytes. Here is the same response in sexyRPC

  (.(s("South Dakota")))

This response is only 22 bytes (an 87% reduction!).

Here is an XMLRPC fault example

  HTTP/1.1 200 OK
  Connection: close
  Content-Length: 426
  Content-Type: text/xml
  Date: Fri, 17 Jul 1998 19:55:02 GMT
  Server: UserLand Frontier/5.1.2-WinNT
  <?xml version="1.0"?> 
  <methodResponse> 
    <fault> 
      <value> 
        <struct> 
          <member> 
            <name>faultCode</name> 
            <value><int>4</int></value> 
          </member> 
          <member> 
            <name>faultString</name> 
            <value><string>Too many parameters.</string></value> 
          </member> 
        </struct> 
      </value> 
    </fault> 
  </methodResponse>

This fault response is 426 bytes! Here is the same fault in sexyRPC

  (.(!(4 "Too many parameters.")))

We have reduced the fault response from 426 bytes to 32 bytes. A 93% reduction!

The XMLRPC spec was not designed to allow for boxcarring requests together. For network efficiency, it makes sense to be able to send multiple requests and to get multiple responses. sexyRPC allows requested to be easily boxcarred.

Immitation is the sincerest form of flattery. The truth is that sexyRPC is XMLRPC repackaged with an emphasis places on network and computational efficiency. To achieve that efficiency, XML was replaced with S-expressions. sexyRPC would not exist without XMLRPC.


SEE ALSO

Related Web Pages

Matt Sottile's small, fast s-expression library, http://sexpr.sourceforge.net/

UserLand XML-RPC, http://www.xmlrpc.com/

XML-RPC Specification, http://www.xmlrpc.com/spec

Related Man Pages

xdr(3), rpc(3).


TO DO

o Build an XDR compatibility API
o Possibly build a sexyrpcgen program modeled after rpcgen
o Build code to serialize data to xmlrpc for compatibility


AUTHOR

 Matt Massie <massie@CS.Berkeley.EDU>

SourceForge Logo