Topics

Communicating with WSJT-X over UDP #udp


s53c.lado@...
 

I'm working on (yet another) ham logging program written in c#. I do not have big plans for it: It is just a project I am using as a means to learn something new and to help me pass my retirement days. Right now I am trying to add communication between WSJT-X and this logging program. So far I am able to decode most of the messages coming from WSJT-X , Where I am stuck is that I don't understand the schema negotiation between WSJT-X  and UDP server. Datagram from UDP server to WSJT-X seems to be defined in NetworkMessage part of NetworkMessage.hpp. But as I know practically nothing of C++, I am not able to read and interpret what's written there. So I kindly ask if there is someone to show how the message from UDP server to WSJT-X  should be constructed in a way like messages from WSJT-X to UDP servers are described.

Lado, S53C


Bill Somerville
 

On 22/11/2020 07:31, s53c.lado@... wrote:
I'm working on (yet another) ham logging program written in c#. I do not have big plans for it: It is just a project I am using as a means to learn something new and to help me pass my retirement days. Right now I am trying to add communication between WSJT-X and this logging program. So far I am able to decode most of the messages coming from WSJT-X , Where I am stuck is that I don't understand the schema negotiation between WSJT-X  and UDP server. Datagram from UDP server to WSJT-X seems to be defined in NetworkMessage part of NetworkMessage.hpp. But as I know practically nothing of C++, I am not able to read and interpret what's written there. So I kindly ask if there is someone to show how the message from UDP server to WSJT-X  should be constructed in a way like messages from WSJT-X to UDP servers are described.

Lado, S53C
Hi Lado,

firstly you need to understand that there may be many WSJT-X instances *and* interoperating server applications like your logging program working together in what I will refer to as a "mesh". Each may be written and updated at different times and can only be expected to understand the data type schema at the time they were last updated. To allow the "mesh" of applications to interoperate there must be a lowest common denominator schema that they all understand. The schema negotiation is how this is occurs. Each new application joining the "mesh" shall wait for a Heartbeat message and examine the current schema being used, it shall then reply with another Heartbeat message that contains the highest schema number it can work with. The WSJT-X client receiving that Heartbeat message will adjust it's outgoing schema by lowering it to match if necessary. That in turn will be received by all server applications in the "mesh" and they in turn will adjust if necessary. The result, eventually, will be all applications will settle on the adjusted schema number. You application shall always monitor the schema number of incoming traffic and be prepared to move to a lower schema number. It is forbidden to send a message with a higher schema number than any previously received schema number in a given operating session.

Note that the schema number is not the schema of the data messages and fields within them, it is the schema of the underlying data types. This means the schema only changes very rarely when a data type encoding has to change. Note also that neither new fields appended to existing messages, or new message types, even if new data types included,do not require a new schema number. A new schema number is *only* necessary if an data type of a field used in an already published message type has a change in the encoding of that type.

To make this schema scheme work we guarantee that no message field in existing messages will be changed, only new fields appended to existing message types, or new message types will be added. To comply with this your application must be prepared to receive datagrams that are longer than the expected sum of fields, or indeed new unknown message types. Both of these cases should be silently ignored as they represent updates to the messages for a newer version than the one your application was coded to support. Of course you can update you application to recognize those new fields and message types in your own time.

You application shall be prepared to receive messages with any schema up to and equal to the maximum schema it understands. There are currently three possible schema numbers (1, 2, and 3), but as noted in Network/NetworkMessage.hpp schema number 1 is broken and obsolete so will never be used, so currently you must support schema numbers 2 and 3. The difference lies in the encoding of the QDateTime type, in schema 2 QTime components with a value of 0 represent NULL whereas in schema 3 -1 represents NULL. There are also changes in the QDate component but only in the timespec field and we guarantee to only use the UTC timespec where there is no difference be tween schema numbers 2 and 3.

The current list of message types and fields is documented in the WSJT-X project source repository in this file:

https://sourceforge.net/p/wsjt/wsjtx/ci/master/tree/Network/NetworkMessage.hpp

in the commentary at the top. This file may be updated at any WSJT-X public release so authors of interoperating applications should monitor the history of this file for changes at those times if they wish to track any recent changes. No C++ knowledge is required as the commentary has all the information you need, or hyperlinks to further reading for it.

73
Bill
G4WJS.


Tom M0LTE
 

Hi Lado

Rather than starting from scratch, I encourage you to use, and indeed extend via submitting PRs, this:

Thanks
Tom

On Sun, 22 Nov 2020 at 12:04, Bill Somerville <g4wjs@...> wrote:
On 22/11/2020 07:31, s53c.lado@... wrote:
> I'm working on (yet another) ham logging program written in c#. I do
> not have big plans for it: It is just a project I am using as a means
> to learn something new and to help me pass my retirement days. Right
> now I am trying to add communication between WSJT-X and this logging
> program. So far I am able to decode most of the messages coming from
> WSJT-X , Where I am stuck is that I don't understand the schema
> negotiation between WSJT-X  and UDP server. Datagram from UDP server
> to WSJT-X seems to be defined in NetworkMessage part of
> NetworkMessage.hpp. But as I know practically nothing of C++, I am not
> able to read and interpret what's written there. So I kindly ask if
> there is someone to show how the message from UDP server to WSJT-X 
> should be constructed in a way like messages from WSJT-X to UDP
> servers are described.
>
> Lado, S53C

Hi Lado,

firstly you need to understand that there may be many WSJT-X instances
*and* interoperating server applications like your logging program
working together in what I will refer to as a "mesh". Each may be
written and updated at different times and can only be expected to
understand the data type schema at the time they were last updated. To
allow the "mesh" of applications to interoperate there must be a lowest
common denominator schema that they all understand. The schema
negotiation is how this is occurs. Each new application joining the
"mesh" shall wait for a Heartbeat message and examine the current schema
being used, it shall then reply with another Heartbeat message that
contains the highest schema number it can work with. The WSJT-X client
receiving that Heartbeat message will adjust it's outgoing schema by
lowering it to match if necessary. That in turn will be received by all
server applications in the "mesh" and they in turn will adjust if
necessary. The result, eventually, will be all applications will settle
on the adjusted schema number. You application shall always monitor the
schema number of incoming traffic and be prepared to move to a lower
schema number. It is forbidden to send a message with a higher schema
number than any previously received schema number in a given operating
session.

Note that the schema number is not the schema of the data messages and
fields within them, it is the schema of the underlying data types. This
means the schema only changes very rarely when a data type encoding has
to change. Note also that neither new fields appended to existing
messages, or new message types, even if new data types included,do not
require a new schema number. A new schema number is *only* necessary if
an data type of a field used in an already published message type has a
change in the encoding of that type.

To make this schema scheme work we guarantee that no message field in
existing messages will be changed, only new fields appended to existing
message types, or new message types will be added. To comply with this
your application must be prepared to receive datagrams that are longer
than the expected sum of fields, or indeed new unknown message types.
Both of these cases should be silently ignored as they represent updates
to the messages for a newer version than the one your application was
coded to support. Of course you can update you application to recognize
those new fields and message types in your own time.

You application shall be prepared to receive messages with any schema up
to and equal to the maximum schema it understands. There are currently
three possible schema numbers (1, 2, and 3), but as noted in
Network/NetworkMessage.hpp schema number 1 is broken and obsolete so
will never be used, so currently you must support schema numbers 2 and
3. The difference lies in the encoding of the QDateTime type, in schema
2 QTime components with a value of 0 represent NULL whereas in schema 3
-1 represents NULL. There are also changes in the QDate component but
only in the timespec field and we guarantee to only use the UTC timespec
where there is no difference be tween schema numbers 2 and 3.

The current list of message types and fields is documented in the WSJT-X
project source repository in this file:

https://sourceforge.net/p/wsjt/wsjtx/ci/master/tree/Network/NetworkMessage.hpp

in the commentary at the top. This file may be updated at any WSJT-X
public release so authors of interoperating applications should monitor
the history of this file for changes at those times if they wish to
track any recent changes. No C++ knowledge is required as the commentary
has all the information you need, or hyperlinks to further reading for it.

73
Bill
G4WJS.





 

Thanks, Bill.

 

I wasn’t sure I understood that there could be more than one server. You say ‘interoperating’, surely a server would only respond to UDPs addressed to it. Or can an application be server and client? Or several servers having the same address? What other applications are likely to be servers? Do you expect people to have multiple loggers? What other sort of applications would be servers?

 

73 Phil GM3ZZA.

 

Sent from Mail for Windows 10

 

From: Bill Somerville
Sent: 22 November 2020 12:04
To: main@WSJTX.groups.io
Subject: Re: [WSJTX] Communicating with WSJT-X over UDP #udp

 

O

 


--
73 Phil GM3ZZA


 

Thanks Tom,

 

I’ve already implemented a UDP server for WSJT-X in my hb logger, but it doesn’t handle schema negotiation correctly (according to Bill’s reply). I might look at replacing it with your lib.

 

73 Phil GM3ZZA

 

Sent from Mail for Windows 10

 

From: Tom M0LTE
Sent: 22 November 2020 12:08
To: main@wsjtx.groups.io
Subject: Re: [WSJTX] Communicating with WSJT-X over UDP #udp

 

Hi Lado

 

Rather than starting from scratch, I encourage you to use, and indeed extend via submitting PRs, this:

 

Thanks

Tom

 

On Sun, 22 Nov 2020 at 12:04, Bill Somerville <g4wjs@...> wrote:

On 22/11/2020 07:31, s53c.lado@... wrote:
> I'm working on (yet another) ham logging program written in c#. I do
> not have big plans for it: It is just a project I am using as a means
> to learn something new and to help me pass my retirement days. Right
> now I am trying to add communication between WSJT-X and this logging
> program. So far I am able to decode most of the messages coming from
> WSJT-X , Where I am stuck is that I don't understand the schema
> negotiation between WSJT-X  and UDP server. Datagram from UDP server
> to WSJT-X seems to be defined in NetworkMessage part of
> NetworkMessage.hpp. But as I know practically nothing of C++, I am not
> able to read and interpret what's written there. So I kindly ask if
> there is someone to show how the message from UDP server to WSJT-X 
> should be constructed in a way like messages from WSJT-X to UDP
> servers are described.
>
> Lado, S53C

Hi Lado,

firstly you need to understand that there may be many WSJT-X instances
*and* interoperating server applications like your logging program
working together in what I will refer to as a "mesh". Each may be
written and updated at different times and can only be expected to
understand the data type schema at the time they were last updated. To
allow the "mesh" of applications to interoperate there must be a lowest
common denominator schema that they all understand. The schema
negotiation is how this is occurs. Each new application joining the
"mesh" shall wait for a Heartbeat message and examine the current schema
being used, it shall then reply with another Heartbeat message that
contains the highest schema number it can work with. The WSJT-X client
receiving that Heartbeat message will adjust it's outgoing schema by
lowering it to match if necessary. That in turn will be received by all
server applications in the "mesh" and they in turn will adjust if
necessary. The result, eventually, will be all applications will settle
on the adjusted schema number. You application shall always monitor the
schema number of incoming traffic and be prepared to move to a lower
schema number. It is forbidden to send a message with a higher schema
number than any previously received schema number in a given operating
session.

Note that the schema number is not the schema of the data messages and
fields within them, it is the schema of the underlying data types. This
means the schema only changes very rarely when a data type encoding has
to change. Note also that neither new fields appended to existing
messages, or new message types, even if new data types included,do not
require a new schema number. A new schema number is *only* necessary if
an data type of a field used in an already published message type has a
change in the encoding of that type.

To make this schema scheme work we guarantee that no message field in
existing messages will be changed, only new fields appended to existing
message types, or new message types will be added. To comply with this
your application must be prepared to receive datagrams that are longer
than the expected sum of fields, or indeed new unknown message types.
Both of these cases should be silently ignored as they represent updates
to the messages for a newer version than the one your application was
coded to support. Of course you can update you application to recognize
those new fields and message types in your own time.

You application shall be prepared to receive messages with any schema up
to and equal to the maximum schema it understands. There are currently
three possible schema numbers (1, 2, and 3), but as noted in
Network/NetworkMessage.hpp schema number 1 is broken and obsolete so
will never be used, so currently you must support schema numbers 2 and
3. The difference lies in the encoding of the QDateTime type, in schema
2 QTime components with a value of 0 represent NULL whereas in schema 3
-1 represents NULL. There are also changes in the QDate component but
only in the timespec field and we guarantee to only use the UTC timespec
where there is no difference be tween schema numbers 2 and 3.

The current list of message types and fields is documented in the WSJT-X
project source repository in this file:

https://sourceforge.net/p/wsjt/wsjtx/ci/master/tree/Network/NetworkMessage.hpp

in the commentary at the top. This file may be updated at any WSJT-X
public release so authors of interoperating applications should monitor
the history of this file for changes at those times if they wish to
track any recent changes. No C++ knowledge is required as the commentary
has all the information you need, or hyperlinks to further reading for it.

73
Bill
G4WJS.



 


--
73 Phil GM3ZZA


Bill Somerville
 

Hi Phil,

Tom's library doesn't handle schema negotiation either. It appears to force the "mesh" schema number to 2, which is acceptable but not necessarily very friendly. Also it does not decode the QDate field NULL value, which is the difference between schema numbers 2 and 3, so it doesn't grok the different schema either.

73
Bill
G4WJS.

On 22/11/2020 14:10, Philip Rose via groups.io wrote:

Thanks Tom,

 

I’ve already implemented a UDP server for WSJT-X in my hb logger, but it doesn’t handle schema negotiation correctly (according to Bill’s reply). I might look at replacing it with your lib.

 

73 Phil GM3ZZA

 

Sent from Mail for Windows 10

 

From: Tom M0LTE
Sent: 22 November 2020 12:08
To: main@wsjtx.groups.io
Subject: Re: [WSJTX] Communicating with WSJT-X over UDP #udp

 

Hi Lado

 

Rather than starting from scratch, I encourage you to use, and indeed extend via submitting PRs, this:

 

Thanks

Tom

 

On Sun, 22 Nov 2020 at 12:04, Bill Somerville <g4wjs@...> wrote:

On 22/11/2020 07:31, s53c.lado@... wrote:
> I'm working on (yet another) ham logging program written in c#. I do
> not have big plans for it: It is just a project I am using as a means
> to learn something new and to help me pass my retirement days. Right
> now I am trying to add communication between WSJT-X and this logging
> program. So far I am able to decode most of the messages coming from
> WSJT-X , Where I am stuck is that I don't understand the schema
> negotiation between WSJT-X  and UDP server. Datagram from UDP server
> to WSJT-X seems to be defined in NetworkMessage part of
> NetworkMessage.hpp. But as I know practically nothing of C++, I am not
> able to read and interpret what's written there. So I kindly ask if
> there is someone to show how the message from UDP server to WSJT-X 
> should be constructed in a way like messages from WSJT-X to UDP
> servers are described.
>
> Lado, S53C

Hi Lado,

firstly you need to understand that there may be many WSJT-X instances
*and* interoperating server applications like your logging program
working together in what I will refer to as a "mesh". Each may be
written and updated at different times and can only be expected to
understand the data type schema at the time they were last updated. To
allow the "mesh" of applications to interoperate there must be a lowest
common denominator schema that they all understand. The schema
negotiation is how this is occurs. Each new application joining the
"mesh" shall wait for a Heartbeat message and examine the current schema
being used, it shall then reply with another Heartbeat message that
contains the highest schema number it can work with. The WSJT-X client
receiving that Heartbeat message will adjust it's outgoing schema by
lowering it to match if necessary. That in turn will be received by all
server applications in the "mesh" and they in turn will adjust if
necessary. The result, eventually, will be all applications will settle
on the adjusted schema number. You application shall always monitor the
schema number of incoming traffic and be prepared to move to a lower
schema number. It is forbidden to send a message with a higher schema
number than any previously received schema number in a given operating
session.

Note that the schema number is not the schema of the data messages and
fields within them, it is the schema of the underlying data types. This
means the schema only changes very rarely when a data type encoding has
to change. Note also that neither new fields appended to existing
messages, or new message types, even if new data types included,do not
require a new schema number. A new schema number is *only* necessary if
an data type of a field used in an already published message type has a
change in the encoding of that type.

To make this schema scheme work we guarantee that no message field in
existing messages will be changed, only new fields appended to existing
message types, or new message types will be added. To comply with this
your application must be prepared to receive datagrams that are longer
than the expected sum of fields, or indeed new unknown message types.
Both of these cases should be silently ignored as they represent updates
to the messages for a newer version than the one your application was
coded to support. Of course you can update you application to recognize
those new fields and message types in your own time.

You application shall be prepared to receive messages with any schema up
to and equal to the maximum schema it understands. There are currently
three possible schema numbers (1, 2, and 3), but as noted in
Network/NetworkMessage.hpp schema number 1 is broken and obsolete so
will never be used, so currently you must support schema numbers 2 and
3. The difference lies in the encoding of the QDateTime type, in schema
2 QTime components with a value of 0 represent NULL whereas in schema 3
-1 represents NULL. There are also changes in the QDate component but
only in the timespec field and we guarantee to only use the UTC timespec
where there is no difference be tween schema numbers 2 and 3.

The current list of message types and fields is documented in the WSJT-X
project source repository in this file:

https://sourceforge.net/p/wsjt/wsjtx/ci/master/tree/Network/NetworkMessage.hpp

in the commentary at the top. This file may be updated at any WSJT-X
public release so authors of interoperating applications should monitor
the history of this file for changes at those times if they wish to
track any recent changes. No C++ knowledge is required as the commentary
has all the information you need, or hyperlinks to further reading for it.

73
Bill
G4WJS.



 


--
73 Phil GM3ZZA



Bill Somerville
 

On 22/11/2020 14:08, Philip Rose via groups.io wrote:

Thanks, Bill.

 

I wasn’t sure I understood that there could be more than one server. You say ‘interoperating’, surely a server would only respond to UDPs addressed to it. Or can an application be server and client? Or several servers having the same address? What other applications are likely to be servers? Do you expect people to have multiple loggers? What other sort of applications would be servers?

 

73 Phil GM3ZZA.

Hi Phil,

one of the reasons why WSJT-X uses UDP for interoperation with other applications is because that allows a many-to-many topology without having to use different service ports for each server. This is enabled by using multicast IP, a server can bind a service port using SO_REUSEADDR and join a multicast group to express an interest in datagrams sent to that group. The networking infrastructure will deliver such datagrams, by copying them if necessary (multicast is the only networking protocol where packets can be copied) so that every member of a multicast group can receive their own copy of each datagram.

There are already several applications that can run in parallel as servers interoperating with WSJT-X instances. JTAlert, Gridtracker, and WSJT-X Monitor for example. They often do have some overlap in functionality, but that's fine so long as only one of them is responsible for logging QSOs in any main station log.

73
Bill
G4WJS.