[OSC_dev] what is the point of SLIP?

[Kaspar Bumke]

Hi,

What are the advantages of using SLIP OSC over a serial/virtual serial port
when communicating with an embedded device rather than just sending out
straight OSC formatted strings? Will the framing make it more effecient?
Does it not simply add an unnecessary overhead?

Does it make more sense for higher bandwidth communication but maybe not
for low bandwidth?

Regards,

Kaspar

[Adrian Freed]

Hi,
What are the advantages of using SLIP OSC over a serial/virtual serial port when communicating with an embedded device rather than just sending out straight OSC formatted strings? Will the framing make it more effecient? Does it not simply add an unnecessary overhead?

The OSC encoding needs framing of some kind over streaming transports because it is packet oriented and intrinsically doesn't define its boundaries.
If you are using an error free transport (TCP) you can use the count prefix framing (as in OSC 1.0 spec.). We use SLIP for unreliable streaming protocols like USB serial and physical serial.

[Stephen Sinclair]

On Thu, Jan 12, 2012 at 12:06 PM, Adrian Freed

Post by Adrian Freed

Hi,
What are the advantages of using SLIP OSC over a serial/virtual serial port when communicating with an embedded device rather than just sending out straight OSC formatted strings? Will the framing make it more effecient? Does it not simply add an unnecessary overhead?

The OSC encoding needs framing of some kind over streaming transports because it is packet oriented and intrinsically doesn't define its boundaries.
If you are using an error free transport (TCP) you can use the count prefix framing (as in OSC 1.0 spec.). We use SLIP for unreliable streaming protocols like USB serial and physical serial.

But SLIP doesn't add reliability, only framing, no? (SLIP doesn't
include any error checking, does it?) Of course OSC needs framing one
way or another. The advantages / disadvantages of SLIP vs. count
prefix seem to be:

- SLIP, pro: stateless, con: unknown buffer size needed for message
- count prefix, pro: known length ahead of time (allocation), con:
stateful, no standard for count representation

Anything else? On a microcontroller presumably the unknown buffer
size will not be a problem since the decoder will simply support a
maximum message size and reject the rest of the message when the
buffer fills up. The statefulness on the other hand is problematic
since a serial line receiver is more likely to "jump in" in the middle
of a stream, and with count prefix there is no way to know when the
packet begins. With TCP this is not a problem.

On the other hand a "serious" serial line implementation may use its
own lower-level packetization just to include error
checking/correction, making SLIP an unnecessary overhead.

That there is no standard for count representation is also an issue.
We have used a signed 32-bit integer in liblo TCP, (this should really
be unsigned I guess). But since OSC doesn't define a maximum message
size, theoretically the count prefix should be in some "big int"
expandable format. In practice of course 32-bit is more than enough,
and in fact wasteful in most cases.

One nice thing about count-prefix is that the message is represented
as-is, which means a memcpy from the buffer is all that is needed to
"parse" the message, while SLIP encoding requires stepping through
every byte of the message. On the other hand, this is usually needed
regardless to do pattern matching, so maybe it makes no difference.

I don't really know how much the TCP code in liblo is really used, but
I would not be adverse to switching to SLIP encoding if it would
increase interoperability and not break too badly many people's
applications. Unfortunately I'm not sure how to determine this other
than asking on the mailing list, but not everyone who uses liblo is on
the mailing list. The joys of library maintenance ;)

Steve

[Adrian Freed]

Post by Stephen Sinclair
On Thu, Jan 12, 2012 at 12:06 PM, Adrian Freed

Post by Adrian Freed

Hi,
What are the advantages of using SLIP OSC over a serial/virtual serial port when communicating with an embedded device rather than just sending out straight OSC formatted strings? Will the framing make it more effecient? Does it not simply add an unnecessary overhead?

The OSC encoding needs framing of some kind over streaming transports because it is packet oriented and intrinsically doesn't define its boundaries.
If you are using an error free transport (TCP) you can use the count prefix framing (as in OSC 1.0 spec.). We use SLIP for unreliable streaming protocols like USB serial and physical serial.

But SLIP doesn't add reliability, only framing, no? (SLIP doesn't
include any error checking, does it?) Of course OSC needs framing one
way or another. The advantages / disadvantages of SLIP vs. count

It's hard to recover from an error with the count prefix style. Resynch is easy with double SLIP.

[Adrian Freed]

Post by Stephen Sinclair
Anything else? On a microcontroller presumably the unknown buffer
size will not be a problem since the decoder will simply support a
maximum message size and reject the rest of the message when the
buffer fills up. The statefulness on the other hand is problematic
I

It is actually possible to support dispatch as the fragments of the OSC bundle come in
so long as your buffer is larger than the longest data payload expected for each OSC message.
I seem to remember Andy pulled this of in uOSC which runs on a very small memory footprint.

Post by Stephen Sinclair
since a serial line receiver is more likely to "jump in" in the middle
of a stream, and with count prefix there is no way to know when the
packet begins. With TCP this is not a problem.
On the other hand a "serious" serial line implementation may use its
own lower-level packetization just to include error
checking/correction, making SLIP an unnecessary overhead.

I can't find anyone using serious serial line stuff in the DIY community.

Post by Stephen Sinclair
That there is no standard for count representation is also an issue.
We have used a signed 32-bit integer in liblo TCP, (this should really
be unsigned I guess). But since OSC doesn't define a maximum message
size, theoretically the count prefix should be in some "big int"
expandable format. In practice of course 32-bit is more than enough,
and in fact wasteful in most cases.

This is actually an issue in interprocess OSC use in clusters now that
OS's have gone 64-bit.

Post by Stephen Sinclair
One nice thing about count-prefix is that the message is represented
as-is, which means a memcpy from the buffer is all that is needed to

OSC was designed specifically to avoid the need for the memcpy.
You can dispatch with pointers to the data in the inbound packet

Post by Stephen Sinclair
"parse" the message, while SLIP encoding requires stepping through
every byte of the message. On the other hand, this is usually needed
regardless to do pattern matching, so maybe it makes no difference.
I don't really know how much the TCP code in liblo is really used, but
I would not be adverse to switching to SLIP encoding if it would
increase interoperability and not break too badly many people's
applications. Unfortunately I'm not sure how to determine this other
than asking on the mailing list, but not everyone who uses liblo is on
the mailing list. The joys of library maintenance ;)

Andy and I are working on clarifying this. Fortunately it is easy to tell
automatically whether a received stream is framed with a count or a slip wrap.

[Stephen Sinclair]

Post by Adrian Freed

Anything else?  On a microcontroller presumably the unknown buffer
size will not be a problem since the decoder will simply support a
maximum message size and reject the rest of the message when the
buffer fills up.  The statefulness on the other hand is problematic
I

It is actually possible to support dispatch as the fragments of the OSC bundle come in
so long as your buffer is larger than the longest data payload expected for each OSC message.

That's what I meant, yes.

Post by Adrian Freed
I seem to remember Andy pulled this of in uOSC which runs on a very small memory footprint.

since a serial line receiver is more likely to "jump in" in the middle
of a stream, and with count prefix there is no way to know when the
packet begins.  With TCP this is not a problem.
On the other hand a "serious" serial line implementation may use its
own lower-level packetization just to include error
checking/correction, making SLIP an unnecessary overhead.

I can't find anyone using serious serial line stuff in the DIY community.

Fair enough.

Post by Adrian Freed

That there is no standard for count representation is also an issue.
We have used a signed 32-bit integer in liblo TCP, (this should really
be unsigned I guess).  But since OSC doesn't define a maximum message
size, theoretically the count prefix should be in some "big int"
expandable format.  In practice of course 32-bit is more than enough,
and in fact wasteful in most cases.

This is actually an issue in interprocess OSC use in clusters now that
OS's have gone 64-bit.

The code for liblo specifies int32_t, is there another OSC
implementation doing TCP that uses a different type on 64-bit
machines?

Post by Adrian Freed

One nice thing about count-prefix is that the message is represented
as-is, which means a memcpy from the buffer is all that is needed to

OSC was designed specifically to avoid the need for the memcpy.
You can dispatch with pointers to the data in the inbound packet

My point was that SLIP would mess with this since for example an
integer or string could have the escape code embedded in it.

Post by Adrian Freed

"parse" the message, while SLIP encoding requires stepping through
every byte of the message.  On the other hand, this is usually needed
regardless to do pattern matching, so maybe it makes no difference.
I don't really know how much the TCP code in liblo is really used, but
I would not be adverse to switching to SLIP encoding if it would
increase interoperability and not break too badly many people's
applications.  Unfortunately I'm not sure how to determine this other
than asking on the mailing list, but not everyone who uses liblo is on
the mailing list.  The joys of library maintenance ;)

Andy and I are working on clarifying this. Fortunately it is easy to tell
automatically whether a received stream is framed with a count or a slip wrap.

That's pretty interesting, how so? I can see looking for "#bundle",
but for bare messages I can't imagine how to detect it. Maybe finding
the type string by "," and then discovering the end of the message
according to the argument length, and looking for the END code.
Sounds complicated.

Steve

[Adrian Freed]

Post by Stephen Sinclair

Post by Adrian Freed
OSC was designed specifically to avoid the need for the memcpy.
You can dispatch with pointers to the data in the inbound packet

My point was that SLIP would mess with this since for example an
integer or string could have the escape code embedded in it.

ah yes, the state machine that deals with the escape code ( SLIP is a few of lines of code)
has to run somewhere in the stream. For uart based serial it just
lives in the interrupt routine so the packet is never seen in its SLIP wrapped form by the upper leve.

Post by Stephen Sinclair

Post by Adrian Freed

Post by Stephen Sinclair
"parse" the message, while SLIP encoding requires stepping through
every byte of the message. On the other hand, this is usually needed
regardless to do pattern matching, so maybe it makes no difference.
I don't really know how much the TCP code in liblo is really used, but
I would not be adverse to switching to SLIP encoding if it would
increase interoperability and not break too badly many people's
applications. Unfortunately I'm not sure how to determine this other
than asking on the mailing list, but not everyone who uses liblo is on
the mailing list. The joys of library maintenance ;)

Andy and I are working on clarifying this. Fortunately it is easy to tell
automatically whether a received stream is framed with a count or a slip wrap.

That's pretty interesting, how so? I can see looking for "#bundle",
but for bare messages I can't imagine how to detect it. Maybe finding
the type string by "," and then discovering the end of the message
according to the argument length, and looking for the END code.
Sounds complicated.

No worse than some of the other things people do in the old UNIX magic stuff - and you only have
to do it once when the data stream starts.

[Andy W. Schmeder]

Post by Stephen Sinclair

Post by Adrian Freed
OSC was designed specifically to avoid the need for the memcpy.
You can dispatch with pointers to the data in the inbound packet

My point was that SLIP would mess with this since for example an
integer or string could have the escape code embedded in it.

Is this a concern? Its possible to define a variant of SLIP where the SLIP_ESC byte is padded to be 4-bytes long, that would preserve alignment in spite of insertions...


---

Andy W. Schmeder
email: andy [at] cnmat.berkeley.edu
skype: andy.schmeder

Programmer/Analyst II
Research Group
Center for New Music and Audio Technologies
University of California at Berkeley
http://cnmat.berkeley.edu

[Matt Wright]

Post by Adrian Freed
OSC was designed specifically to avoid the need for the memcpy.
You can dispatch with pointers to the data in the inbound packet

...unless the packet contains a bundle whose time tag demands that it be scheduled to take effect in the future...

-Matt

[Kaspar Bumke]

This discussion is very interesting but fails to answer my original
quandry. Let me change the question.

If I have an embedded device sending low-bandwidth OSC messages. Say a
micro-controller sending the pressure value for 16 FSRs. It simply sends
out a message "/pressure,ii" with the first int being FSR number(0-15) and
the second being a pressure value (0-511 let's say). This is a scenario
where MIDI could be used easily instead but in this case OSC allows for a
higher resolution on the pressure value.

Does SLIP make sense in this scenario as opposed to just printing an OSC
formatted string to the serial port or a more custom solution like monome
uses <http://monome.org/data/monome256_protocol.txt>? What would it add in
this case?

The OSC encoding needs framing of some kind over streaming transports

Post by Adrian Freed
because it is packet oriented and intrinsically doesn't define its
boundaries.

What problems would this cause exactly in this case? I figure each message
is bounded by an EOL character.

I guess my example is quite specific and one where data-loss over the
stream is of little significance. This is probably why there is not much
use for SLIP in this case. I guess if the communication start's to get more
complex and data-loss could mean failure of operation a framing protocol
would come into it's own.

What I find interesting about simply using an OSC formatted string is that
it is human readable and decoding would be easy to implement on anything.
Besided the lack of framing and error detection what other potential
problems do you think there are with this?

[Andy W. Schmeder]

What I find interesting about simply using an OSC formatted string is that it is human readable and decoding would be easy to implement on anything.
What problems would this cause exactly in this case? I figure each message is bounded by an EOL character.

OSC is not entirely plain text, its a mix of plain text and binary.

So, at some point you will want to send a number and happens to contain the EOL character in one of its encoded bytes, which will cause the receiver to get confused and something bad will happen.

With the SLIP protocol the end-byte is SLIP_END, decimal 192, but there is also defined a method to send the end byte by escaping it (prepending with SLIP_ESC decimal 219), and a method to send the escape byte, by escaping that also. Thats basically all there is to it.

In the worst case the size of a SLIP encoded message is inflated by 2x, but for a random byte stream the inflation is about 1 percent.


---

Andy W. Schmeder
email: andy [at] cnmat.berkeley.edu
skype: andy.schmeder

Programmer/Analyst II
Research Group
Center for New Music and Audio Technologies
University of California at Berkeley
http://cnmat.berkeley.edu

[Kaspar Bumke]

Post by Andy W. Schmeder
So, at some point you will want to send a number and happens to contain
the EOL character in one of its encoded bytes, which will cause the
receiver to get confused and something bad will happen.

Ah yes, I didn't think about that. I guess the data could be sent as
strings too for the ultimate human readable format. If being human readable
is not a concern then going for a custom protocol like the monome would
seem attractive but I think at that point SLIP seems like the better
option. It doesn't seem like the monome protocol actually accounts for this
EOL issue.

[Adrian Freed]

Post by Andy W. Schmeder
So, at some point you will want to send a number and happens to contain the EOL character in one of its encoded bytes, which will cause the receiver to get confused and something bad will happen.

SLIP solves this.

Post by Andy W. Schmeder
Ah yes, I didn't think about that. I guess the data could be sent as strings too for the ultimate human readable format. If being human readable is not a concern then going for a custom protocol like the monome would seem attractive but I think at that point SLIP seems like the better option. It doesn't seem like the monome protocol actually accounts for this EOL issue.

The problem with a custom encoding is that it is not easily extensible and you are on your own maintaining it.
You can add functionality to OSC streams without having to rebuild code on both sides each time. Also a lot of people are building shim applications between their custom protocols
and OSC and/or MIDI. We have measured unacceptable latencies and jitter in this case for our applications which is why we like to wire OSC in at the lowest level.
Just to clarify:The SLIP OSC option is mature. It has been in use commercially for years on the Make controller board. Our external
slipOSC does the work for Max/MSP users. It is a few lines of Python to unwrap or frame SLIP OSC messages. We have built a couple of dozen
devices at CNMAT over the years that use SLIP OSC. The more recent ones use the $16 Teensy board

You are posting to the developers group where we are happy to speculate about these things, but sometimes you just want to focus on the use of the data.

[Adrian Freed]

Post by Matt Wright

Post by Adrian Freed
OSC was designed specifically to avoid the need for the memcpy.
You can dispatch with pointers to the data in the inbound packet

...unless the packet contains a bundle whose time tag demands that it be scheduled to take effect in the future...

Hi, Matt

Yes, although even in that case if you really need to avoid the copy you can leave the packet where it is, mark at in a side structure as "in use" and then
return it back to the I/O pool when the deadline dispatch completes.

There is something odd about this discussion when people are buying multicore 2GHz cpu's and 8G of memory at Fry's for $400.