User:Srobertson

From The Neuros Technology Wiki

Jump to: navigation, search
Srobertson
Other Names: Steven Robertson
Where to Find: mailing list, wiki, IRC
Areas of Expertise: Linux, Sysadmin
Secondary Expertise:
Member Since: 2004
Group Membership: OSD Dev. Sample, Netadmin on OdNT, Wiki & Forums
Accomplishments: Restructured Wiki
Primary Products: N1,N2, OSD
Other Communities:  ???
Location: Central FL, USA
Website: parseit.org


Uh, saw this linked. I hadn't touched it in >2yrs. Per Joe's request (from last time I nuked the page, more than three years ago), I'm leaving the old stuff up, even though it's mostly irrelevant. See my website for what I'm up to now, including my thesis which targets the OSD2 - I'm trying to get in the habit of "less talk, more code." Srobertson 21:05, 10 January 2009 (EST)

Contents

Personal Stuff

I own a Neuros 1. I've dropped it more times than I can count, and even broke it a few times, but opened it right up, gave it a coat of solder (just all over the board -- one time I even used a superheated toothbrush and splatter-painted the brain), and every time it worked again. Pretty impressive for a consumer electronics device.

I'm decent at programming, although I prefer coding tools to fill specific needs than massive applications (owing to an incredibly short attention span -- no, really, you'll notice). I'm also experienced at Linux sysadmin and netadmin chores, and can pick up enough of a language or paradigm to hurt myself in about an hour (as I'm sure most here can). I am extremely good at having opinions.

One of these opinions is the dangerous one that every GUI ever should be written in an interpreted language, leaving the computationally-intensive parts to C code. C is for libraries! This even extends to the OSD. I know, it's somewhat crazy to expect an embedded app to be interpreted, but I honestly believe that the consumer would much rather have a slow, full-featured, community-supported set-top box than a lightning-fast one-trick pony that doesn't work half the time. I think that splitting this work up would be a boon to this organization, because it allows the Xiamen office to work on what they're better suited to - the library heavy lifting in C - and let the community at large work on the more nuanced design tasks that require the understanding of being in the target demo in order to get right.

neuros interprocess communication (nipc, ipcd)

This describes my implementation of a method for interprocess communication. The idea is kind of an amalgamation of D-Bus and XMMS2's IPC, but is tuned for low-overhead operation and ease of programming at the cost of some generalizations. It is designed from the ground-up to support a particular type of remote procedure call and callback mechanism. No programmer (aside from me or whoever the next maintainer) should ever have to interact with it directly. Instead, use the rpc and signals, described below.

neuros rpc and signals/slots

Example protocol definition

This will be referenced throughout the document.

protocol nmsd, version 0;

/* This protocol demonstrates every syntactically valid feature
 * available in the protocol compiler.
 *
 * So far, you've already seen multi-line comments and the protocol
 * header.  The header must come before anything else except comments.
 * We recommend placing it on top of the file like it is above.
 * It must be on its own line, and follow the format:
 * 
 * protocol <protocol name>, version <decimal version>;
 *
 * where protocol name is the ipcd name of whatever implements this protocol,
 * and version is any decimal number (thus 0.0.1 is _invalid_). */

// C++-style single-line comments are also supported.


/* Type directives inform the protocol generator how to deal with complex
 * types.  Replace 'type' with 'typdef struct' and you have a valid C 
 * statement. */
type {
        // These are the only primitive types available.
        uint unsigned_integer;
        int signed_integer;
        char * null_terminated_string;
} example_type;

type {
        // Not here that we've used a complex type within another complex type.
        example_type proof;
        char   *null_str;
} example_type_2;

type {
        // NOW we get FANCY.
        int count;
        char *array_of_strings[count];
        example_type_2 array_of_complex_types[count];
} example_type_3;

type {
        int c;
        example_type_3 array_of_complex_types_which_hold_arrays[c];
} example_type_4;


/* Procedure directives take no or one type.  They return a type, which may
 * be void. The procedure keyword is required. */

procedure example_type  example_rpc_proc_one (uint32_t your_own_argument_name);
procedure void          example_rpc_proc_two ();

/* Signal emitters take no argument.  They emit one type, which may be void.
 * They must be preceded by the signal keyword. */

signal example_signal_emitter (int);

Types

Both the RPC and signal/slot mechanisms make use of a versatile serialization system to sling data around. The system represents complex data types (essentially C structures, but support for Lua tables will happen if necessary) in a portable, endian-correct format capable of rendering almost all of the most common usage patterns of C structures. The below is what you need to know to use RPC and signal/slot mechanisms.

Basic types

There are three basic data types natively understood by this system.

Type C common name C stdint.h name Protocol name
Signed 32-bit integer int int32_t int
Unsigned 32-bit integer unsigned int uint32_t uint
Null-terminated string of 8-bit characters char * char * char *

You use these types within the protocol as organically as you would if you were writing in real C; no declarations or anything required.

A note about whitespace and pointers: the regexes used to parse the protocol are somewhat whitespace-insensitive; "char*", "char *", "char *", and "char\n\n\t\n\t \t\n *" all render the same way. In order to get that flexibility, there's a chance you can seriously mess with the protocol compiler: you might be able to name your type 'type*', because the protocol compiler doesn't truly understand what a pointer is. Of course the C header generated will be worthless. Stick to valid C names and everybody's happy.

Complex types

Specifying more complex types is simple. See the protocol definition above for examples. The syntax for types is almost identical to that of a typedef'd C structure, and indeed if you adjust type names and change 'type' to 'typedef struct' most of the time it will work straight up in C.

You can build complex types out of any type the system understands, meaning not only are the three basic types supported, but complex types as well. When accessing the resulting structures in C, recursive definitions work pretty intuitively - if the instance of type a, "a_instance", has a member of type c called "c_memb", you can go a_instance.c_memb to access it.

Arrays

In one of the great triumphs of modern technology, variable-length arrays are supported. Because the primary user of this serialization system, the RPC system, couldn't have any interaction with it (the whole point is to make RPC transparent to the coder), we take a slightly different approach to serializing arrays than most other systems.

To use an array as part of a complex type, add brackets after the instance name (see above for an example). The brackets must contain the name of a variable that is a member of the complex type, an integer, and preceding the array declaration. See above for a good example. This variable specifies the number of elements in the array, and of course must be zero or greater.

From C, these arrays operate like regular arrays; if you have 'example_type_3 a;' you can call 'a.array_of_strings[0]' for the first element in the array of strings, and so on.

Allocating and destroying arrays

Arrays are allocated with calloc(); you must free() the memory used by the arrays when you destroy an array. This behavior is expected whenever an array is destroyed automatically, as well, so building an array on the stack then using it inside a complex type where manual allocation is necessary would probably segfault or double-free(). Most of the time, the library is designed not to put this obligation on the user, but it is occasionally unavoidable.

When the user has to allocate his or her own type that will be edited by the program - as happens for RPC return values - the program stomps on old array pointers, filling them with pointers to the newly-serialized array. For the same reason, the automatic serializer will not allocate arrays automatically, because it doesn't know how big to make them.

For arrays-within-arrays, the recursive calls to free() can get overwhelming fast. We recommend using the helper function generated (not yet complete, gimme a day or two) as part of protocol compilation for the instances when you're freeing such a data type. This function does everything but the outermost layer, so that you can use stack (or just reuse a heap variable) as the return value expected to a particular function.

RPC

The amount of source-code changes required to use RPC is surprisingly small. Even for providers of RPC calls, the only requirement outside of mainloop integration (below) is simply defining the functions the RPC engine will call. Everything else is handled by the protocol.

For providers

nipc-enabled applications that provide RPC calls must define the functions that implement them. The protocol compiler handles function declarations, and it's best to leave it that way because of some C dumbness. For a protocol line:

protocol int prot_name (example_type some_name);

a function declaration is produced like:

bool prot_name (const example_type *some_name, int *ret, void *priv);

  • Yes, that's right, the function arguments are provided in the proper type, and the procedure argument (some_name here) is named properly . Not much type-safety here, but it affords a bit (plus it's more convenient).
  • Obviously, the function's return value is expected to be placed in *ret.
  • Returning 'false' will result in a no-op on the client end; the return value will be discarded.
  • Both the argument (whatever its' name is) and the return value are pre-allocated; trying to re-allocate them would not go well. (Also, see the note about arrays and pre-allocation.)
  • "void *priv" is a pointer to the private data section you passed while setting up the mainloop (see below). Use it (or global variables) to store the program's internal state and whatever variables the functions need to execute.

For clients

For the above protocol line, a client can make a call like:

prot_name (h, some_name, &int_on_the_stack);

  • h is the ipc_handle_t that connects to the ipcd. the application of interest must also be connected to that ipcd, of course.
  • Both "arg" and "ret", the argument and return value, must be pointers to pre-allocated data of the correct type.
  • here, some_name is the argument. int_on_the_stack is an actual int, and will be dereferenced and filled with the return value. Of course, a pointer to a heap value could have been used here too.
  • This "function" (actually a macro) returns a var of type 'bool'. If it returns false, don't rely on the contents of the return value.

Protocol version, and what the heck fastrpc is

FastRPC is currently deprecated, and will probably be eliminated soon. What better time than now, right?

When an rpc call is made, normally the mechanism encodes the procedure name and argument types, which are then verified on the other end. However, for providers with large numbers of functions, this has the potential to cost hundreds of strcmp()s, and for frequently-used remote procedures this can add up. An alternative is available -- fastrpc -- which simply verifies that versions match and then uses an index to call the appropriate function, which can be considerably faster.

The downside of fastrpc is that it absolutely requires that the version numbers match, and it is the programmer's responsibility to increment the version number for every public change to the protocol - meaning that fastrpc clients would need to have a different version for each version change of every piece of software it uses. Icky, to say the least.

It is important to note that choosing between rpc and fastrpc is done by the client - any provider with a header generated by the protocol compiler allows for both rpc and fastrpc calls to be made with little overhead.

Use fastrpc when:
  • Your software and the software that implements the procedures you want to call are compiled together - like nmsd and main_app.
  • Performance is more important than the potential inconvenience of having to maintain different versions.
Use (normal) rpc when:
  • Your software is not compiled together with the software it calls - like desktop applications and third-party addons.
  • Performance is not absolutely critical.
  • There is any possibility of a version mismatch.
  • You don't know if you should use fastrpc or normal rpc.

Protocol compiler

...takes a protocol file and turns it into headers. Run without arguments for help. Operation is pretty straightforward from there, with one possible exception: prefixing. That's covered below.

Client C header

Contains:

  1. A multiple include check, and then a few includes as necessary.
  2. Type definitions for complex types.
  3. Macro wrappers around rpc_call.
  4. Macro wrappers around sig_register and sig_unregister.
  5. Optionally, type destructors.

The rpc and fastrpc macro wrappers are equivalent to

bool procedure_name(ipc_handle, arg, ret)

both arg and ret must be pre-allocated pointers; so calling a function that takes an int and returns an int can be accomplished by saying:

ipc_handle_t *h,
int a, b;
/* Initialize things */
if (procedure_name(h, &a, &b))
    printf("Success!\n");

The signal macro wrappers are in the form

sig_name_hook(ipc_handle, slot_function)
sig_name_unhook(ipc_handle, slot_function)

You can attach multiple slots to a given signal from a single client. Currently, if a client crashes then reattaches, signals will *not* automatically rehook; you should connect a slot to the ipc client status change signal to monitor for that if you're depending on a callback for a client that might go down unexpectedly.

Provider C header

Contains:

  1. A multiple include check and a few standard includes.
  2. Any includes specified during protocol compilation.
  3. Type definitions.
  4. (OPTIONAL) Function definitions for rpc calls.
  5. A macro (and some associated data structures) that may be called to set up an IPC handle to handle incoming rpc and signal connection requests.
  6. Macros to emit signals.

Function definitions are optional; but if they are not specified, you MUST tell the protocol compiler to include the header that defines them, otherwise you will get compile-time errors. It is, of course, the programmer's responsibility to actually implement the functions no matter how they're defined.

The function definitions take the form

bool procedure_name (const arg_type_name *arg_name, ret_type_name *ret, void *priv);

That's right, they don't use void*, they use the real type and name (except for the private data). arg_name, whatever it is, points to a complete data structure. ret points to a base type, or the first level of a complex type, that has already been allocated. "First level" means that if there is a pointer, as would happen for example with a string or an array, then the pointer will be allocated, but will point to NULL; it's the programmer's responsibility to allocate it. BIG NOTE: At the end of every RPC call, for both arg_name and ret, the entire data structure is freed - including stuff the programmer assigns. This includes pointers! If this is unclear, check the test case in the nipc source; it makes extensive use of complex types which demonstrate these kinds of situations.


Dealing with multiple protocols

It's easy to imagine a client that makes use of several different protocols, each implemented by a different component, and provides a few of its own. This is of course possible, but different between client and server.

For clients

Client RPC calls are simply macro calls, which are largely stateless -- there's no special treatment of these calls based on which ipcd the ipc handle connects to -- so there's no problem in using multiple protocols as a client. Signal slots aren't quite stateless, however. In order to make the callback mechanism work properly, you are highly encouraged to run every protocol you want to be a client of through the protocol compiler all at once. The compiler will sort through any conflicts and duplications and spit out a single header that will pretty much always work. Simple!

For servers

This gets a little more tricky. ipcd was designed from the ground up for providers of services to be named, so that if there was an nmsd connected to a particular ipcd you knew it was the nmsd, and not some fancy-pants imitator. As a result, each protocol is heavily tied to the ipcd name of the intended provider; it would essentially be impossible to simply redirect commands destined from one name to another name. In addition, one ipcd client can register only one name at a time. There are other internal details that makes this process complicated, but you get my point.

The solution seems a little bit brutish at first, but it was the most elegant I could find for this particularly rare problem: simply connect to ipcd multiple times, and register each protocol you want to provide with a different handle.

Note that, unlike for clients, you cannot and should not compile providers together.

Clients and servers

Shouldn't be a problem. The only bit of advice I have is be sure to do all your registering of a slot with a remote signal over one ipc handle, for consistency.

Project Lobster Enhancements

Not gonna lie to you, chief: I don't have much done yet in this category. Deeper infrastructure challenges keep popping up, and I'm chasing those down first so that Lobster can be done right. With a rich butter sauce.








The stuff below is pretty old, and may not be relevant.

Notes on OSD Features

My microwave has a button on it. This button says "Beverage." This button, in the middle of a panel with at least 40 other buttons on it, works this way: You hit the button, enter the beverage type, enter the volume of the beverage in ounces (if you are in a non-ounce country, a grams-to-ounces conversion chart is on the inside of the microwave door), enter how hot you want your beverage to be (from 'Warm' to 'Hot'), and how hot it is now, and it will automatically calculate the correct cook time. Now, if I forget how to do this, I can press the 'Help' button on my microwave, which presents agonizingly slowly scrolled documentation in some terse half-English on my 8-character LED. Once I trawl through that mess, I can conveniently make a note on the half-effective built-in voice recorder and memo system, sitting right next to my answering machine and -- oh yeah -- pen and paper.

Or I could press '2', then 'Add 30 seconds', to get the 2:30 cook on maximum power that I always use to boil water for tea.

Many things can be pulled out from it, but just keep this example in mind as you decide where to spend your time. It is always a personal goal of mine to do everything for everybody, but as someone on the list said, that's the surest way to make a failure of a product, so I write this as much to myself, if not more so, than to anyone else.

Notes on Website Reorganization

I had some stuff here about wiki organization. As most of it is or shortly will not be relevant, I'm going to summarise.

  • We should resist the temptation to abuse hierarchies. Wikis are designed to form natural "idea webs"; with extensive cross-referencing and a search button, little will ever get lost.
  • We should also be mindful about what gets put on a wiki. Though it may be tempting, copying and pasting will *not* benefit us in the long run. We'll work out exactly what gets put where as we go, particularly with global search.
  • The Wiki is not a place to preach (well, not for Neuros anyway -- I seem to be doing a fair job of it). Even a little preachiness will result in users not coming here and contributing information.

Here's my new thoughts:

  • We (the existing core community) should recruit some of the more well-spoken and outspoken community members and give them specific duties in terms of data management. For example, we need a couple of good moderators for the forums. We also really need someone to help us maintain the official site. "Data Movers" could also help by semi-manually shoving information from forums to wiki or wiki to official site as necessary, during the transition.
  • That's it for now, I'm thought out.
Personal tools