[OSC_dev] arrays (again)

[IOhannes m zmoelnig]

i'm trying to understand the semantics of OSC arrays.

while andy claims in

The [ ] typetags are already on the list of optional types from OSC
1.0, several implementations support it, and its existing semantics
are adequately documented.

..., i cannot find any "adequate documentation" (osc.org being not
available doesn't help).

esp. i wonder how the typetag for an array of (say) 1002 floats would
look like.
all references i found seem to hint that it would be:
[ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
ffff]
which i think is just insane.

the references often mention nested arrays and structured data, but if
this is the intended use, why do the specs speak of "arrays" rather than
"lists" (the former makes me think of largish sets of uniform data,
whereas the latter goes more into the "data structs" (small sets of
strutured data) directions)

v,sdr
IOhannes

[Laurent Pointal]

Post by IOhannes m zmoelnig
i'm trying to understand the semantics of OSC arrays.
while andy claims in

The [ ] typetags are already on the list of optional types from OSC
1.0, several implementations support it, and its existing semantics
are adequately documented.

..., i cannot find any "adequate documentation" (osc.org being not
available doesn't help).
esp. i wonder how the typetag for an array of (say) 1002 floats would
look like.
[fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff

<zip>

Post by IOhannes m zmoelnig
fffffffffff ffff]

This is how I understand it too.

Post by IOhannes m zmoelnig
which i think is just insane.

Yes for big arrays.

As OSC seem to have been initially used for many small messages, maybe
protocol authors don't have in mind transport of such collections.

Post by IOhannes m zmoelnig
the references often mention nested arrays and structured data, but if
this is the intended use, why do the specs speak of "arrays" rather than
"lists" (the former makes me think of largish sets of uniform data,
whereas the latter goes more into the "data structs" (small sets of
strutured data) directions)

I agree, current OSC arrays are more structs where you describe each item.
So it is missing a short description allowing to indicate the transport of: a
count of values and the type-code for these values.
As [] is used, maybe a notation like {<typecode>} which would pe packed as an
OSC int32 followed by corresponding values for the typecode.

A+
Laurent.

[Adrian Freed]

yep, you need an "f" for every float in the list.
insane?
It does allow the flexibility of mixing types in an array - a well-trodden path
in lisp, scheme and dynamic programming languages.

Post by Laurent Pointal

Post by IOhannes m zmoelnig
i'm trying to understand the semantics of OSC arrays.
while andy claims in

The [ ] typetags are already on the list of optional types from OSC
1.0, several implementations support it, and its existing semantics
are adequately documented.

..., i cannot find any "adequate documentation" (osc.org being not
available doesn't help).
esp. i wonder how the typetag for an array of (say) 1002 floats would
look like.
[fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff

<zip>

Post by IOhannes m zmoelnig
fffffffffff ffff]

This is how I understand it too.

Post by IOhannes m zmoelnig
which i think is just insane.

Yes for big arrays.
As OSC seem to have been initially used for many small messages, maybe
protocol authors don't have in mind transport of such collections.

Post by IOhannes m zmoelnig
the references often mention nested arrays and structured data, but if
this is the intended use, why do the specs speak of "arrays" rather than
"lists" (the former makes me think of largish sets of uniform data,
whereas the latter goes more into the "data structs" (small sets of
strutured data) directions)

I agree, current OSC arrays are more structs where you describe each item.
So it is missing a short description allowing to indicate the transport of: a
count of values and the type-code for these values.
As [] is used, maybe a notation like {<typecode>} which would pe packed as an
OSC int32 followed by corresponding values for the typecode.
A+
Laurent.
_______________________________________________
OSC_dev mailing list
http://lists.create.ucsb.edu/mailman/listinfo/osc_dev

[IOhannes m zmoelnig]

Post by Adrian Freed
yep, you need an "f" for every float in the list.

well, i need an "f" for every float in the *array*.

Post by Adrian Freed
insane?
It does allow the flexibility of mixing types in an array - a well-trodden path
in lisp, scheme and dynamic programming languages.

i would have thought to call that data type a "list", rather than an
"array".
e.g. python's [] is called "list". python's `array` can only hold values
of a pre-defined type (e.g. you cannot mix floats and ints).

fgam
IOhannes

[Hanns Holger Rutz]

Post by IOhannes m zmoelnig
e.g. python's [] is called "list". python's `array` can only hold values
of a pre-defined type (e.g. you cannot mix floats and ints).

the counter-example is JavaScript (yes, that horrible language), where
an array is as heterogeneous as you want...

[ 123, 'foo', true ]

I agree that if someone says "array" to me, I'd think its elements share
the same type. How large are your arrays/lists? Are the type-tags really
a problem?

[Adrian Freed]

OSC fares pretty well as an encoding in terms of space efficiency compared to XML and JSON.
You can design something more compact and even use OSC blobs to wrap that but
it is rarely worth the trouble. The common case where this is a consideration is power-starved wireless.

We have been sending audio and spectra around networks as OSC packets for many years without
performance issues.


The terminology around arrays, lists, vectors, tables is a mess across languages.

Post by Hanns Holger Rutz

Post by IOhannes m zmoelnig
e.g. python's [] is called "list". python's `array` can only hold values
of a pre-defined type (e.g. you cannot mix floats and ints).

the counter-example is JavaScript (yes, that horrible language), where
an array is as heterogeneous as you want...
[ 123, 'foo', true ]
I agree that if someone says "array" to me, I'd think its elements share
the same type. How large are your arrays/lists? Are the type-tags really
a problem?
_______________________________________________
OSC_dev mailing list
http://lists.create.ucsb.edu/mailman/listinfo/osc_dev

[IOhannes m zmoelnig]

Post by Adrian Freed
The terminology around arrays, lists, vectors, tables is a mess across languages.

sure.
probably a footnote should be added to the 1.0 specs clarifying the term
"array" and it's use.

i wouldn't have written my original email if it had been clear to me.

fgmasdr
IOhannes

[Matt Wright]

Just a few points:

LISP (by which I mean Scheme and Common Lisp) arrays may have heterogeneous types, just like the non-insane lists Adrian mentioned.

The point of OSC's optional type tag support for arrays was not to squeeze a few bytes out of the type tag string, but to let there be a difference on the receiving end between calling a function with 5 arguments versus calling a function with one argument that is an array (or vector, or list...) of 5 elements.

If you have a specific array to transmit then you must know its size and dimension! This points to a possible weakness, though - lack of representation for multi-dimensional arrays. For a 3x3 array of floats I would recommend representing them in OSC as nested arrays:





----- Original Message -----
From: "IOhannes m zmoelnig" <***@iem.at>
To: "osc dev" <***@lists.mat.ucsb.edu>
Sent: Wednesday, October 28, 2015 7:42:55 AM
Subject: Re: [OSC_dev] arrays (again)

Post by Adrian Freed
The terminology around arrays, lists, vectors, tables is a mess across languages.

sure.
probably a footnote should be added to the 1.0 specs clarifying the term
"array" and it's use.

i wouldn't have written my original email if it had been clear to me.

fgmasdr
IOhannes

[Adrian Freed]

Thanks for the suggestion.
The specification does indeed need a clarifyng commentary.

Post by IOhannes m zmoelnig

Post by Adrian Freed
The terminology around arrays, lists, vectors, tables is a mess across languages.

sure.
probably a footnote should be added to the 1.0 specs clarifying the term
"array" and it's use.
i wouldn't have written my original email if it had been clear to me.
fgmasdr
IOhannes
_______________________________________________
OSC_dev mailing list
http://lists.create.ucsb.edu/mailman/listinfo/osc_dev

[Ross Bencina]

Post by Adrian Freed
We have been sending audio and spectra around networks as OSC packets for many years without
performance issues.

How do you encode them? Blobs or arrays?

Post by Adrian Freed
The terminology around arrays, lists, vectors, tables is a mess across languages.

tuples.

Ross.


P.S.

I realise that it may never happen, but single-type vectors using a
length-prefix encoding would be nice. Something like:

Typetag: J
Data: 32 bit item count n, followed by n int32s

Typetag: G
Data: 32 bit item count n, followed by n float32s

Typetag: D
Data: 32 bit item count n, followed by n float64s

Typetag: P
Data: 32 bit item count n, followed by n int16s (with 2 bytes padding
appended for odd n).

[Roger Dannenberg]

I think if you want to base terminology on examples, you would be better
off looking at the historical development of languages and terminology.
I was quite surprised to see Python's non-standard use of "list" (how do
we explain lists and arrays to students of Python when the data
structure names are unconventional?) and surprised again to see that
perhaps Python terminology is supplanting more traditional terms. I'm
referring mainly to Lisp which is quite old, extremely influential, at
the foundation of much early research in programming languages. The main
Lisp data structure is the linked "list" with quite different properties
from Python lists, which I would call "dynamic arrays" or in Lisp terms,
simply "arrays" (as in OSC). Many other languages followed Lisp's
terminology, just as Lisp followed the terminology of earlier languages
(e.g. I think Lisp arrays were introduced later than Fortran arrays.) -Roger

Post by Hanns Holger Rutz

Post by IOhannes m zmoelnig
e.g. python's [] is called "list". python's `array` can only hold values
of a pre-defined type (e.g. you cannot mix floats and ints).

the counter-example is JavaScript (yes, that horrible language), where
an array is as heterogeneous as you want...
[ 123, 'foo', true ]
I agree that if someone says "array" to me, I'd think its elements share
the same type. How large are your arrays/lists? Are the type-tags really
a problem?
_______________________________________________
OSC_dev mailing list
http://lists.create.ucsb.edu/mailman/listinfo/osc_dev

[Hanns Holger Rutz]

I'd also say that for 32-bit values you get an overhead of 25%, for
64-bit values an overhead of 12.5% to store the type tags, so asking
implementations to add runlength decoding of type tags is
disproportionate and makes the otherwise rather simple protocol
unnecessarily complicated. If the performance is an issue in a
particular application, one could still use blobs.

2c, .hh.

Post by Adrian Freed
yep, you need an "f" for every float in the list.
insane?
It does allow the flexibility of mixing types in an array - a well-trodden path
in lisp, scheme and dynamic programming languages.

Post by Laurent Pointal

Post by IOhannes m zmoelnig
i'm trying to understand the semantics of OSC arrays.
while andy claims in

The [ ] typetags are already on the list of optional types from OSC
1.0, several implementations support it, and its existing semantics
are adequately documented.

..., i cannot find any "adequate documentation" (osc.org being not
available doesn't help).
esp. i wonder how the typetag for an array of (say) 1002 floats would
look like.
[fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff

<zip>

Post by IOhannes m zmoelnig
fffffffffff ffff]

This is how I understand it too.

Post by IOhannes m zmoelnig
which i think is just insane.

Yes for big arrays.
As OSC seem to have been initially used for many small messages, maybe
protocol authors don't have in mind transport of such collections.

Post by IOhannes m zmoelnig
the references often mention nested arrays and structured data, but if
this is the intended use, why do the specs speak of "arrays" rather than
"lists" (the former makes me think of largish sets of uniform data,
whereas the latter goes more into the "data structs" (small sets of
strutured data) directions)

I agree, current OSC arrays are more structs where you describe each item.
So it is missing a short description allowing to indicate the transport of: a
count of values and the type-code for these values.
As [] is used, maybe a notation like {<typecode>} which would pe packed as an
OSC int32 followed by corresponding values for the typecode.
A+
Laurent.
_______________________________________________
OSC_dev mailing list
http://lists.create.ucsb.edu/mailman/listinfo/osc_dev

_______________________________________________
OSC_dev mailing list
http://lists.create.ucsb.edu/mailman/listinfo/osc_dev

[IOhannes m zmoelnig]

Post by Hanns Holger Rutz
I'd also say that for 32-bit values you get an overhead of 25%, for

which i think is unacceptable.

Post by Hanns Holger Rutz
64-bit values an overhead of 12.5% to store the type tags, so asking
implementations to add runlength decoding of type tags is
disproportionate and makes the otherwise rather simple protocol
unnecessarily complicated. If the performance is an issue in a
particular application, one could still use blobs.

the issue arose exactly because i was trying to avoid blobs.
blobs are basically proprietary data (in which case i don't see a point
in using OSC with typetags at all).

but i think the blob-representation is a good model for (large) arrays
(of uniform data) in the *data* section:
- ab int32 count followed by that many elements of the specified data.
i don't think that putting the element count into the *type* section is
a particularily good idea (starting with alignment issues).

fgamsdr
IOhannes

[jpff]

followng u on a discussion on liblo-devel list, the arrays I need to
transmit usually have unknown size and dimension, so arrays are not useful
(to me). I will stick with blobs which I a beginning to understand.
==John ff

Post by Laurent Pointal

Post by IOhannes m zmoelnig
i'm trying to understand the semantics of OSC arrays.
while andy claims in

The [ ] typetags are already on the list of optional types from OSC
1.0, several implementations support it, and its existing semantics
are adequately documented.

..., i cannot find any "adequate documentation" (osc.org being not
available doesn't help).
esp. i wonder how the typetag for an array of (say) 1002 floats would
look like.
[fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff

<zip>

Post by IOhannes m zmoelnig
fffffffffff ffff]

This is how I understand it too.

Post by IOhannes m zmoelnig
which i think is just insane.

Yes for big arrays.
As OSC seem to have been initially used for many small messages, maybe
protocol authors don't have in mind transport of such collections.

Post by IOhannes m zmoelnig
the references often mention nested arrays and structured data, but if
this is the intended use, why do the specs speak of "arrays" rather than
"lists" (the former makes me think of largish sets of uniform data,
whereas the latter goes more into the "data structs" (small sets of
strutured data) directions)

I agree, current OSC arrays are more structs where you describe each item.
So it is missing a short description allowing to indicate the transport of: a
count of values and the type-code for these values.
As [] is used, maybe a notation like {<typecode>} which would pe packed as an
OSC int32 followed by corresponding values for the typecode.
A+
Laurent.
_______________________________________________
OSC_dev mailing list
http://lists.create.ucsb.edu/mailman/listinfo/osc_dev

[Stephen Sinclair]

Post by jpff
followng u on a discussion on liblo-devel list, the arrays I need to
transmit usually have unknown size and dimension, so arrays are not useful
(to me). I will stick with blobs which I a beginning to understand.
==John ff

This is perhaps a difficulty with liblo, which strongly encourages you
specify the typetags of the message ahead of time, when adding the
message handler.

However, be aware that you can register a handler with unknown args by
passing NULL as the typespec argument:

lo_server_thread_add_method(st, "/foo/bar", NULL, foo_handler, NULL);

Then, the foo_handler will be called for any arguments, and you just
need to check argc in the handler to see how many arguments it has.
So, you could collect up all the float arguments into an array thusly,

int foo_handler(const char *path, const char *types, lo_arg ** argv,
int argc, void *data, void *user_data)
{
int i, k=0;
float *myarray = calloc(argc, sizeof(float));
for (i=0; i<argc; i++) {
if (types[i] == 'f')
myarray[k++] = argv[i]->f;
}

// .. store myarray somewhere or do something and free it

return 0;
}


hope that helps,
Steve

[Stephen Sinclair]

Post by IOhannes m zmoelnig
i'm trying to understand the semantics of OSC arrays.
while andy claims in

The [ ] typetags are already on the list of optional types from OSC
1.0, several implementations support it, and its existing semantics
are adequately documented.

..., i cannot find any "adequate documentation" (osc.org being not
available doesn't help).
esp. i wonder how the typetag for an array of (say) 1002 floats would
look like.

[.snip..]

Post by IOhannes m zmoelnig
the references often mention nested arrays and structured data, but if
this is the intended use, why do the specs speak of "arrays" rather than
"lists" (the former makes me think of largish sets of uniform data,
whereas the latter goes more into the "data structs" (small sets of
strutured data) directions)

I've also always found this a bit confusing.

I've long thought that although OSC isn't really intended to support
complex data structures, a simple run-length encoding would make a lot
of sense. ie., if a typetag is followed by a number, that typetag
should be considered repeated.

isf4i2, 40 "asdf" 1.2 3.4 5.6 7.8 9 0

'[]' could be used for repeating groups of typetags, for example, but
even that is overly complicating tings for most use cases. Still,
e.g.:

[if]3, 40 1.1 50 2.2 60 3.3

Seeing as numbers are not supported in typetags, and another type of
bracket could be used, such a format could be proposed in the future.
It would not be backward compatible unfortunately.

Note however that sending OSC over any even remotely simple compressed
transport layer would have a similar effect. So, I'm not sure the
backward incompatibility is worth the trouble.

Steve

[Matt Wright]

(Sorry; accidentally sent message without final example; try again:)

Just a few points:

LISP (by which I mean Scheme and Common Lisp) arrays may have heterogeneous types, just like the non-insane lists Adrian mentioned.

The point of OSC's optional type tag support for arrays was not to squeeze a few bytes out of the type tag string, but to let there be a difference on the receiving end between calling a function with 5 arguments versus calling a function with one argument that is an array (or vector, or list...) of 5 elements.

If you have a specific array to transmit then you must know its size and dimension!

This points to a possible weakness, though - lack of representation for multi-dimensional arrays. For a 3x3 array of floats I would recommend representing them in OSC as nested arrays:

,[[fff][fff][fff]]

-Matt



----- Original Message -----
From: "IOhannes m zmoelnig" <***@iem.at>
To: "osc dev" <***@lists.mat.ucsb.edu>
Sent: Wednesday, October 28, 2015 7:42:55 AM
Subject: Re: [OSC_dev] arrays (again)

Post by Adrian Freed
The terminology around arrays, lists, vectors, tables is a mess across languages.

sure.
probably a footnote should be added to the 1.0 specs clarifying the term
"array" and it's use.

i wouldn't have written my original email if it had been clear to me.

fgmasdr
IOhannes

[Matt Wright]

Post by Matt Wright
LISP (by which I mean Scheme and Common Lisp) arrays may have
heterogeneous types, just like the non-insane lists Adrian mentioned.

it's free to send such weak typed lists as a blob or even as a string -
then let the receiver to the ambiguous boxing (if language supports it)
like js eval().

Not sure I understand your point here. I would characterize Lisp's and OSC's typing system as "dynamic" rather than "weak" - your program can tell whether a given data value is an int or float or string or whatever. Converting to a blob means the receiver has to know exactly how to interpret the received bytes, like in the bad old days of OSC before type tags existed at all.

bringing any of such ambiguity or clever "typetag-shortening" to the liblo
datatypes just to break backward compatibility would be insane.

Doing anything "just to break backward compatibility" seems bizarre...

the spec is unclear about it

Sorry there; it seemed clear to me.

and there was no usecase or lib that adopted it.

I wonder if this is true. I personally have never used OSC's array types but I believe others did.

the [] notation should be removed from newer osc specs IMHO.

Don't hold your breath for even one new OSC spec.

Post by Matt Wright
This points to a possible weakness, though - lack of representation for
multi-dimensional arrays. For a 3x3 array of floats I would recommend
,[[fff][fff][fff]]

what's different here to just send fffffffff ? if the dimension is dynamic
and only the sender knows it.. then the receiver must know the dimension,

That is precisely the difference. [[fff][fff][fff]] is how the sender informs the receiver that the dimension is three groups of three rather than 9 individual items or one group of 9...

it could do so by receiving two more ints: iifffffffff

...or 11 individual items

using nested [] or {} statements to describe a custom (i.e. hierarchical
type) structure sounds like json. json is just a string or blob anyway and
liblo shouldn't care.

But the program that receives the OSC messages certainly cares whether it also needs to have a JSON parser.

there are numerous ways to handle any kind of osc messages

Agree there.

and with the
string and blob datatypes it's open for any kind of structures.

Yes, but only structures that have been hard-coded by agreement into both sender and receiver, what I would call "static" typing (aka "strong"), where all types are agreed in advance rather than inspect able at runtime.

not to mind bundles as a further way to "group" things

That's a little tricky because bundles group entire OSC messages, not data within the arguments to a single message.

and the forementioned
generic handler to match any kind of message (may be it by counting how
many f's there are).

I think that's what I'm trying to say about dynamic types.

it has all the tools already and is cooked well :)

Bon apetit!

-Matt

[t***@trellis.ch]

Post by Matt Wright

using nested [] or {} statements to describe a custom (i.e.
hierarchical type) structure sounds like json. json is just a string or
blob anyway and liblo shouldn't care.

But the program that receives the OSC messages certainly cares whether it
also needs to have a JSON parser.

certainly, yes. i don't say this is the solution but closer to having all
that in the typetag(-parser) (it will become a LANGUAGE) :)

[Laurent Pointal (LIMSI)]

Le Wednesday 28 October 2015, 18:02:16 ***@trellis.ch a écrit :
<zip>

the spec is unclear about it and there was no usecase or lib that adopted
it.

I adopted it into my osc4py3.

the [] notation should be removed from newer osc specs IMHO.

I disagree. Document it better, but keep it as it is already, an optional
feature which may not be available in some libs.

A+
L.Pointal

[Hanns Holger Rutz]

Post by Laurent Pointal (LIMSI)
<zip>

the spec is unclear about it and there was no usecase or lib that adopted
it.

I adopted it into my osc4py3.

ScalaOSC supports it, too.

Post by Laurent Pointal (LIMSI)

the [] notation should be removed from newer osc specs IMHO.

I disagree. Document it better, but keep it as it is already, an optional
feature which may not be available in some libs.

+1

[t***@trellis.ch]

Post by Hanns Holger Rutz

Post by Laurent Pointal (LIMSI)
<zip>

the spec is unclear about it and there was no usecase or lib that
adopted it.

I adopted it into my osc4py3.

ScalaOSC supports it, too.

ok sorry for that.

i see it in the docs now
Type tags string: ',ii[iiii]'
Arguments list: [3, 1, [4, 2, 8, 9]]
and wondered if it allows mixes like ii[fsm[T][F]] to create arbitrary new
datatypes

[Matt Wright]

Post by t***@trellis.ch
i see it in the docs now
Type tags string: ',ii[iiii]'
Arguments list: [3, 1, [4, 2, 8, 9]]

Right, like I've been trying to say: three things (the third of which is a list/array/vector of four integers), rather than 6 things.

Post by t***@trellis.ch
and wondered if it allows mixes like ii[fsm[T][F]] to create arbitrary new
datatypes

Not sure "create new datatypes" is appropriate here - this isn't a typedef, but rather an encoding of the arguments to one specific OSC message. This hypothetical message takes three arguments: two ints and an array. It's maybe OK to think of that as a "new datatype", but it's not the case that you've defined something that can be reused in the future: once this one message is sent and received that's the end of the story.

Sorry if I'm being overly pedantic with my vocabulary nitpicking here.

-Matt

[t***@trellis.ch]

Post by Matt Wright

i see it in the docs now Type tags string: ',ii[iiii]'
Arguments list: [3, 1, [4, 2, 8, 9]]

Right, like I've been trying to say: three things (the third of which
is a list/array/vector of four integers), rather than 6 things.

if the tags string contains the grouping/nesting information already it's
redundant to repeat this information again in the arguments. a receiver
could well map 6 things in a flat sequence to a given structure.

Post by Matt Wright

and wondered if it allows mixes like ii[fsm[T][F]] to create arbitrary
new datatypes

Not sure "create new datatypes" is appropriate here - this isn't a
typedef, but rather an encoding of the arguments to one specific OSC
message. This hypothetical message takes three arguments: two ints and
an array. It's maybe OK to think of that as a "new datatype", but it's
not the case that you've defined something that can be reused in the
future: once this one message is sent and received that's the end of the
story.

in my understanding a new type is created as soon as it's a combination of
native types. i'm always thinking at the receiver side, how would it
access a structured such type? i.e. an application could derive a typedef
from the typetags and cast to it from a blob or series of native
arguments. if the type is temporary or to be reused doesn't make a
difference in that need.

in the case of things=[1,2,[a,b]] it's not clear to me how 'a' would be
referenced. mypick=thing[2][0]?

it could be interesting to use a 'message' type that would behave
differently from bundles.
if ii[ii]f would be interpreted as iiMf (where M is type 'message'),
access could be like args[2]->M->args[0]->i to get the first i of first M.

greetings
Thomas

[Matt Wright]

Post by t***@trellis.ch

Post by Matt Wright

i see it in the docs now Type tags string: ',ii[iiii]'
Arguments list: [3, 1, [4, 2, 8, 9]]

Right, like I've been trying to say: three things (the third of which
is a list/array/vector of four integers), rather than 6 things.

if the tags string contains the grouping/nesting information already it's
redundant to repeat this information again in the arguments.

If we're talking about the binary byte format of the OSC-encoded message, then no, the grouping information does not appear in the arguments. The padded type tag string ",ii[iiii]\0\0\0" will be immediately followed by the six binary numbers, flattened into a linear sequence.

If you're talking about the API to a specific OSC implementation, where you're able to pass grouped/nested (and dynamically typed) data structures into a procedure that creates an OSC message to send, then I would agree that it would be redundant (and hence dangerously error-prone) to ask the user to construct the type tag string; much better for this procedure to inspect the data structure and automatically construct the type tag string. (Likewise on receipt the API should handle the type tag automatically.)

Post by t***@trellis.ch

Post by Matt Wright

and wondered if it allows mixes like ii[fsm[T][F]] to create arbitrary
new datatypes

Not sure "create new datatypes" is appropriate here - this isn't a
typedef, but rather an encoding of the arguments to one specific OSC
message. This hypothetical message takes three arguments: two ints and
an array. It's maybe OK to think of that as a "new datatype", but it's
not the case that you've defined something that can be reused in the
future: once this one message is sent and received that's the end of the
story.

in my understanding a new type is created as soon as it's a combination of
native types.

Nothing wrong with this view.

Post by t***@trellis.ch
i'm always thinking at the receiver side, how would it
access a structured such type?

Well this is all 100% dependent on the type semantics of the language on the receiving side. For example, if the receiver is Max then it's dynamically typed and the receiver converts OSC ints, floats, and strings into Max ints, floats, and strings. Max doesn't have nested data structures at all, so the receiver simply doesn't implement []. If the receiver were Lisp then it would work differently.

Post by t***@trellis.ch
i.e. an application could derive a typedef
from the typetags and cast to it from a blob or series of native
arguments.

Your saying "typedef" and "cast" make me think you're talking about C (or ObjC or C++...), in which case I think you're going to get in trouble because these statically (aka "strong") typed languages can only do this kind of stuff at compile time, but receiving a message with type tag string "ii[fsm[T][F]]" is something that happens at runtime. While the program is running and it just received a never-before-seen type tag is too late to do any typedef or cast.

Post by t***@trellis.ch
in the case of things=[1,2,[a,b]] it's not clear to me how 'a' would be
referenced. mypick=thing[2][0]?

(caddr things), obviously. :-)

But seriously, it totally depends on the receiving language. Again, in Max, forget it: there are no nested data structures. Lisp is great for this including this beautiful family of selectors such as CADDR. C can do this kind of thing but only at compile time.

Post by t***@trellis.ch
it could be interesting to use a 'message' type that would behave
differently from bundles.

Years ago James McCartney suggested that it would be useful to be able to have a message (or indeed a full bundle) as the argument to another message, for example so that a query could return the full message to which it is responding (along with the answer).

Post by t***@trellis.ch
if ii[ii]f would be interpreted as iiMf (where M is type 'message'),
access could be like args[2]->M->args[0]->i to get the first i of first M.

Again this looks like C and so I believe you need to think more in terms of statically knowing all possible types before you compile your program.

-Matt

[Matt Wright]

Post by Matt Wright

Post by t***@trellis.ch
in the case of things=[1,2,[a,b]] it's not clear to me how 'a' would be
referenced. mypick=thing[2][0]?

(caddr things), obviously. :-)

Whoops, I mean caaddr - shame on me, my Lisp skills are getting rusty!

-Matt

[t***@trellis.ch]

Post by Matt Wright
Years ago James McCartney suggested that it would be useful to be able to
have a message (or indeed a full bundle) as the argument to another
message, for example so that a query could return the full message to
which it is responding (along with the answer).

Post by t***@trellis.ch
if ii[ii]f would be interpreted as iiMf (where M is type 'message'),
access could be like args[2]->M->args[0]->i to get the first i of first M.

Again this looks like C and so I believe you need to think more in terms
of statically knowing all possible types before you compile your program.

some some thoughts on this, if the osc library is able to represent and
make accessible any information contained in the message (in a
language-specific way), without knowing the exact message format in
advance at compile time, it has that "dynamic" aspect. this is the case
for most libraries (i.e. receive message with arbitrary typetag).
if we look at a message object of any library, it could be seen as an
instance of a special datatype. so the libs have a way to handle the
message "type" containing all the different native types that come as a
flat sequence. now "simply" introducing a recursion by allowing message
objects to contain other message objects (in a given sequence along any
other types) would allow reuse of most what a message object already has
to handle. all the "introspection" and accessing is the same, just in a
nested / recursive way.

if a library would have a policy 'if there is a blob and if it starts with
'/' i'll treat it as a nested osc message', a receiver could *today* get a
"typed tree" of the structure of the message simply by recursively expand
the blobs (treat like osc msg) and then create a "root" object of choice
for the language. i can't think of any tuples that couldn't be structured
this way.
and yes, why not also have quantifiers for types :) /data f256

greetings
Thomas

[t***@trellis.ch]

Post by t***@trellis.ch
if a library would have a policy 'if there is a blob and if it starts
with '/' i'll treat it as a nested osc message', a receiver could *today*
get a "typed tree" of the structure of the message simply by recursively
expand blobs (treat like osc msg) and then create a "root" object of
choice for the language. i can't think of any tuples that couldn't be
structured this way.

here is an example of doing this with liblo.

#get test program
wget -O expand.c
https://raw.githubusercontent.com/7890/liblo/fixmax/nest_test/expand.c

#create OSC message dump
echo -n
"L3Rlc3QAAAAsaWZzYgAAAAAAAIA+TMzNZm9vAAAAAHAvAAAALGJiYmlpZgAAAAAULwAAACxzAABndWd1cyBhYWFhAAAAAAAMzczMPc3MTD6amZk+AAAALC8AAAAsc2IAaW5uZXIAAAAAAAAYLwAAACxmZmYAAAAAPczMzT5MzM0+mZmaAAAAAQAAAAJAWZma"
| base64 -d > /tmp/a

#compile and run test program
gcc -o expand expand.c -llo && ./expand /tmp/a
---output
/test ifsb
i 128
f 0.200000
s foo
b 116 / bbbiif
b 24 / s
s gugus aaaa
b 16 (blob with unknown encoding)
b 48 / sb
s inner
b 28 / fff
f 0.100000
f 0.200000
f 0.300000
i 1
i 2
f 3.400000
---
using type notation with [] this would be:
/test ifs[[s]b[s[fff]]iif]

creating an 'M' datatype would avoid the case of a non-message blob
starting with '/'. 'M' would effectively only give a hint that a blob is
another OSC message and can thus be parsed with already available methods.

avoiding "[]" to be in the typetag by hiding the substructure via 'M'
keeps the simplicity of the typetag and let's receivers introspect the
(fully typed) structure to their needs and create another representation
of the (structured) typetag if needed.

greetings
Thomas

[Roger Dannenberg]

Post by Matt Wright
I would agree that it would be redundant (and hence dangerously error-prone) to ask the user to construct the type tag string; much better for this procedure to inspect the data structure and automatically construct the type tag string.

In my Serpent (a real-time scripting language based on Python) interface
to OSC, I do both: the user provides a type tag string, but the
implementation checks that the arguments match and raises an error if
they do not. This tends to catch errors early at the sending side rather
than trying to diagnose problems caused at the receiver due to
unexpected types (or having to explicitly validate all outgoing and
incoming types in the client code).

You give up the flexibility of dynamic typing in messages, so I can
understand objections, but in practice, having strict, static message
types has made message protocols easier to think about, design,
document, and implement (for me).

-Roger