Laying the good foundation, with TCP…

In case you are just joining the conversation, we are in the midst of a project to modify the testbed application that we have been slowly assembling over the past year. I would heartily recommend that you take some time and review the past posts.

To this point in our latest expansion project, we have created a remote control interface, embedded it in our testbed and performed some basic testing to verify that the interface works. Our next step is to create the first of several “middleware” processes. I call them middleware because they sort of sit between our application’s basic code and the external applications and users. In future installments we will look at middleware for .NET, ActiveX and WebSockets, but we will start with a more fundamental interface: TCP/IP.

The Roadbed for the Information Highway

Aside from giving me the opportunity to air out some tired metaphors, TCP/IP is a good place to start because it gives us the opportunity to examine the protocol that underlies a host of other connection options.

Just the basics

Although the idea of creating a “server” can have a certain mystique, there really isn’t much to it really – at least when you are working with LabVIEW. The underlying assumption to the process is that there is something monitoring the computer’s network interface waiting for a client application to request a TCP/IP connection. In network parlance, this “something” is called a “listener” because listens to the Ethernet interface for a connection. However a given listener isn’t simply listening for any connection attempt, rather the network standards define the ability to create multiple “ports” on a single interface, and then associate particular ports with particular applications. Thus, when you create the listener you have to tell it what port that it is to monitor. In theory, a port number can be any U32 value, but existing standards specify what sorts of traffic is expected on certain port numbers. For example, by default HTTP connections are expected on ports 80 or 8080, port 21 is the default for FTP and LabVIEW by default listens to port 3363. All you have to do is pick a number that isn’t being used for anything else on your computer. To create a listener in LabVIEW, there is a built-in function called TCP Create Listener. It expects a port number, and returns a reference to the listener that it creates – or an error if you pick a port to which some other application is already listening.

Once you have created the listener, you have to tell it to start listening by calling the built-in function TCP Wait On Listener. As its name implies, it waits until a connection is made on the associated port, though you will typically want to specify a timeout. When this function sees and establishes an incoming connection it outputs a new reference specific to that particular connection. A connection handler VI can then use that reference to manage the interactions with that particular remote device or process.

Finally, when you are done with your work, you kill the server by closing the listener reference (TCP Close Connection), and all connection references that you have open. Put these three phases together, you come up with something like this.

The Simplest TCP Server

This simple code creates a listener, waits for a connection, services that connection (it reads 4 bytes from it), and then quits. While this code works, it isn’t really very useful. For example, what good is a server that only waits for one connection and then quits? Thankfully, it’s not hard to expand this example. All you have to do is turn it into a mini state-machine.

One Step at a Time

As usual, the state-machine is built into the timeout event so the following states include a shift register pointing to the next state to be executed, and a second one carrying the delay that the code will impose before going to that state. But before we get into the specifics, here’s a state diagram showing the process’ basic flow.

State Diagram

Execution starts with the Initialize Listener state. It’s main job at this point is to create the TCP listener. Next is the aptly named state, Wait for Connection. It patiently waits for a connection by looping back to itself with a short timeout. As long as there is no connection established, this state will execute over and over again. This series of short waits gives other events (like the one for shutting down the server) a chance to execute.

When a connection is made, the machine transitions to the Spawn Handler state. Since it is critical that the state machine gets back to waiting for a new connection as soon as possible, this state dynamically launches a reentrant connection handler VI and immediately transitions back to the Wait for Connection state.

The state machine continues ping-ponging between these last two states until the server is requested to stop. At that point, the code transitions to the Close Listener state which disposes of the listener and stops the state machine. So let’s look at some real code to implement these logical states – which, by the way resides in a new process VI named TCP-IP

The Initialize Listener State

This state at present only executes once, and its job is to create the listener that initiates connections with remote clients. The TCP Create Listener node has two inputs, the first of which is the port that the listener will monitor. Although I could have hard-coded this number, I instead chose to derive this value from the application’s Server.Port property. In a standalone executable, the application reads this value from its INI file at start-up, thus making it reconfigurable after the application is deployed. If the server.tcp.port key does not exist in the INI file, the runtime engine defaults to LabVIEW’s official port number, 3363.

Initialize Listener

When running in the development environment, this value is still reconfigurable, but it is set through the My Computer target’s VI Server settings. To change this value, right-click on My Computer in the project explorer window and select Properties. In the resulting dialog box, select the VI Server Category. At this point, the port number field is visible in the Protocols section of the VI Server page, but it is disabled. To edit this value, check the TCP/IP box to enable the setting, make the desired change and then uncheck the TCP/IP box, and click the OK button. It is critical that you remember to uncheck TCP/IP before leaving this setting. If you don’t, the project will be linked to the specified port and the TCP server in the testbed application will throw an error 60 when it tries to start.

The other input to the TCP Create Listener node is a timeout. However, this isn’t the time that the node will wait to finish creating the listener. We will be testing this code on a single computer and so don’t have to worry about such things as the network going down – even momentarily. In the broader world, though, there are a plethora of opportunities for things to go wrong. For example, the network could go south while a client is in the middle of connecting to our server. This timeout addresses this sort of situation by specifying the amount of time that the listener will wait for the connection to complete, once a connection attempt starts.

The Wait for Connection State

This state waits for connection attempts, and when one comes, completes the connection. Unfortunately, LabVIEW doesn’t support events based on a connection attempt so this operation takes the form of a polling operation where the code checks for a connection attempt, and if there is none, waits a short period of time and then checks again. The short wait period is needed to give the process as a whole the chance to respond to other events that might occur.

Wait for Connection

The logic implementing this logic starts with a call to the built-in TCP Wait On Listener node with a very short (5-msec) timeout. If there is no connection attempt pending when the call is made, or an attempt is not received during that 5-msec window, the node terminates with an error code 56. The following subVI (Clear looks for, and traps that error code so its occurrence can be used to decide what to do next. If the subVI finds an error 56, the following logic repeats the current state and sets the timeout to 1000-msec. If there is no error, the next state to be executed is Spawn Handler and the timeout is 0.

If there is a successful connection attempt, the TCP Wait On Listener node also outputs a new reference that is unique to that particular connection. This new reference is passed to a shift-register that makes is available to the next state.

The Spawn Handler State

In this state, the code calls a subVI (Launch Connection that spawns a process to handle the remote connection established in the previous state. This connection handler takes the form of a reentrant VI that accepts two inputs: a reference to a TCP connection and a boolean input that enable debugging operations – which at the current time consists of opening the clone’s front panel when it launches, and closing it when it closes.

Spawn Handler

It is important that the connection handler be a reentrant process for two reasons: First, we want the code to be able to handle more than one connection at a time. Second, the listener need to get back to listening for another new connection as quickly as possible. We’ll discuss exactly what goes into the connection handler in a bit.

The Close Listener State

Finally, when the process is stopping, this event closes open connections, sets the timeout to -1, and stops the event loop.

Close Listener

But why are there two connections to be closed? Doesn’t the connection handler that gets launched to manage the remote connection handle closing that reference? While that point is true, the logic behind it is flawed. There is a small, but finite, delay between when the remote connection is completed and the Spawn Handler starts executing. If the command to stop should occurring during that small window of time, the handler will never be launched, and so can’t close that new connection and its associated reference.

Turning States into a Plugin

Now that we have an understanding of the process’ basic operation, we need to wrap a bit more logic around it to turn it into usable code.

Adding Shutdowns and Error Handling

To begin with, if this new process is going to live happily inside the structure we have already defined for testbed application plugins, it is going to need a mechanism to shut itself down when the rest of the application stops. Since that mechanism is already defined, all we have to do is register for the correct event (Stop Application) and add an event handler to give it something to do.

Loop Shutdown

Nothing too surprising here: When the shutdown event fires, the handler sets the next state to be executed to Close Listener and the timeout to 0. Note that it does not actually stop the loop – if it did the last state (which closes the listener reference) would never get the chance to execute. Finally, we also need to provide for error handling…

Add in Error Handling

…but as with the shutdown logic, this enhancement basically consists of adding in existing code. In this case, the application’s standard error reporting VI.

Defining the Protocol

With the new middleware plugin ensconced happily in the testbed framework, we need to create the reentrant connection handler that will handle the network interactions. However, before we can do that we need to define exactly what the communications protocol will look like. In later posts, I will present implementations of a couple of standardized protocols, but for now let’s explore the overall communications process by “rolling our own”.

As a quick aside, you may have noticed that I have been throwing around the word “protocol” a lot lately. Last time, I talked about creating a safe protocol for remote access. Then this time we discussed the TCP protocol, and now I am using the word again to describe the data we will be sending over out TCP connection. A key concept in networking is the idea of layers. We have discussed the TCP protocol for making connections, but that isn’t whole story. TCP is build on top of a lower-level protocol called IP – which is itself built on even lower level protocols for handling such things as physical interfaces. In addition, this protocol stack can also extend upwards. For example, VI Server is at least partially built on top of TCP, and we are now going to create our own protocol that will define how we want to communicate over TCP.

This layering may seem confusing, but it offers immense value because each layer is a modular entity that can be swapped out without disrupting everything else. For example, say you swap out the NIC (Network Interface Card) in your computer, the only part of the stack that needs to change are the very lowest levels that interface to the hardware.

The first thing we need to do is define the data that will be passed back and forth over the connection, and how that data will be represented while it is in the TCP communications channel. Taking the more basic decision first, let’s look at how we want to represent the data. Ideally, we want a data representation that is flexible in terms of capability, is rigorous in its data representations and easy to generate in even primitive languages like C++. The first standard that was created to fill this niche was a spin-off of HTML called XML. The problem is that while it excels in the first two points, the third is a problem because when used to encode small data structures the same features that make it incredibly flexible and rigorous, conspire to make it is very verbose. Or to put it another way, for small data structures the data density in an XML document is very low.

Fortunately, there is an alternative that is perfect for what we need to do: JSON. The acronym stands for “JavaScript Object Notation”, and as the name implies is the notation originally used to facilitate the passing of data within JavaScript applications. The neat part is that a lot of the JSON concepts map really well to native LabVIEW data structures. For example, in terms of datatypes, you can have strings, numbers and booleans, as well as arrays of those datatypes. When you define a JSON object, you define it as a collection of those basic datatypes – sort of like what we do with clusters in LabVIEW. But (as they say on the infomercials), “Wait there’s more…” JSON also allows you include other JSON objects in the definition of a new object just LabVIEW lets us embed clusters within clusters. Finally, to put icing on the cake, nearly every programming language on the planet (including LabVIEW) incorporates support for this standard.

To see how this will work, let’s consider the case of the temperature controller parameters. When wanting to configure this value, the remote application will need to send the following string: (Note: As with JavaScript itself, the presence of “white space” in JSON representations is not significant. I’m showing this “pretty-printed” to make it easier to understand.)

    "Target":"Dog House TC", 
        "Error High Level":100,
        "Warning High Level":90,
        "Warning Low Level":70,
        "Error Low Level":60,
        "Sample Interval":1

This string defines a JSON object that contains two items. The first is labeled Target and it holds a string identifying the specific plugin that it is wanting to configure – in this case the Dog House TC. In the same way, the name of the second item is labeled Data, but look at its value! If you think that looks like another JSON object definition, you’d be right. This sub-object has 5 values representing the individual parameters needed to configure a temperature controller. In case you’re wondering, this is what the code looks like that parses this string back into a LabVIEW data structure:

Unflattening JSON

That’s right, all it takes is one built-in function and one typedef cluster. The magic lies in the fact that the string and the cluster represent the exact same logical structure so it is very easy for LabVIEW’s built-in functions to map from one to the other.

The Unflattened JSON Data

The other thing to note is that the Sample Interval value in the cluster has a unit associated with it, in this case milliseconds. The way LabVIEW handles this situation is consistent with how it handles units in general: When converting data to a unitless form (like a JSON value) it expresses the value using the base unit for the type of data that it is. In the example shown, Sample Interval is time, and the base unit for time is seconds, so LabVIEW expresses the 1000 msecs as 1 sec in JSON. Likewise when unflattening the string back to a LabVIEW data structure, the function interprets the input value in the value’s base units as defined in the cluster.

We are about done with what our message will look like, but there are still a couple of things we need to add before we can start shooting our data down a wire. To begin with, we need to remember that Ethernet is a serial protocol and as so it’s much easier to uses if a receiver can know ahead of time how much data to be expecting. To meet that need, we will append a 2-byte binary value that is the total message. The other thing we need is someway to tell whether the message arrived intact and without corruption, so we will also append a 2-byte CRC. Moreover, to make the CRC easy for other applications to generate we will use a standard 16-bit CCITT form of the calculation. So this is what one of our command data packets will look like:

Message Format

In the same way, we can use the same basic structure for response messages. All we have to do is redefine the JSON “payload” as a JSON object with two objects: a numeric error code (where 0 = “No Error”), and a string that is a contains any data that the response needs to return. As you would expect, this string would itself be another JSON-encoded data structure.

Creating the Connection Handler

We are finally ready to implement the reentrant command handler that manages these messages, and the important part of that job is to ensure that it is fully reentrant. By that I mean that it does little good to make the VI itself reentrant if its major subVIs are not. So what is a “major” subVI? The two things to consider are:

  • How often does the SubVI execute? If the subVI rarely executes or only runs once during initialization, it might not be advantageous to make it reentrant.
  • How long does it take to execute? In the same way, subVIs that implement simple logic and so execute quickly, might not provide a lot of benefit as reentrant code.

As I am wont to do, I defined the handler’s overall structure as a state machine with three states corresponding to the three phases of the response interaction. So the first thing we need to do (and the first state to be executed) is Read Data Packet. Its job is to read an entire message from the new TCP connection, test it for validity and, if valid, pass the command on to the Process Command state.

Read Data Packet

The protocol we have defined calls for each message to start with a 2-byte count, so the state starts by reading two bytes from the interface, casting the resulting binary value to U16 number and then using that number to read the remainder of the message. Then to validate the message, the code performs a CRC calculation on the entire message, including the CRC at the end. Due to the way the CRC calculation works, if the message and CRC are valid, the result of this calculation will always be 0. Assuming the CRC checks, the code strips the CRC from the end of the string and sends the remaining part of the string to a subVI that converts the JSON object into a LabVIEW object. I chose an object-oriented approach here because it actually simplifies the code (no case structures) and it provides a clear roadmap of what I need to do if I ever decide to add more interface commands in the future. If the CRC does not check, the next state to execute is either Send Response if no error occurred during the network reads, or Stop Handler if there was.

Moving on, the Process Command state calls a dynamic dispatch method (Process that is responsible for interfacing to the rest of the application through the events we defined last time, and formatting a response to be sent to the caller. The object model for this part of the code has 5 subclasses (one for each command) and the parent class is used as the default for when JSON command structure does not contain a valid command object. It should surprise no one that the command processing subclass methods look a lot like the test VIs we created last time to verify the operation of the remote access processor, consequently, I am not going to take the time or space to present them all again here. However I will highlight the part that makes them different:

Parsing Response for Error or Data

This snippet shows the logic that I use to process the response coming back from the remote access engine in response to the event that reads the graph data. Because the variant returned in the response notifier can be either a text error message, or an array of real data, the first thing the code does is attempt to convert the variant into a string. If this attempt fails and generates an error, we know that the response contains data and so can format it for return to the remote caller. If the variant converts successfully to a string, we know the command failed and can pass an error back to the caller.

At this point, we now have a response ready to send back to the caller, so the state machine transitions to the Send Response state. Here we see the logic that formats and transfers the response to the caller:

Send Response

Since the core of the message is a JSON representation of a response cluster, the code first flattens the cluster to a JSON string. Note however, the string that it generates contains no extraneous white space, so it will look different very from the JSON example I showed earlier. The logic next calculates the length of the return message and the CRC of the JSON. Those two values are added to the beginning and end, respectively, of the JSON string and the concatenated result is written back to the TCP connection.

Finally, the Stop Handler state closes the TCP connection and stops the state machine loop, which also stops and removes from memory the reentrant clone that has been running.

Testing the Middleware

Finally, as always we need to again test what we have done, and to do that I have written a small LabVIEW test client program. However, if you know another programming language, feel free to write a short program to implement the transactions that we have defined. The program I created is included as a separate project. The top-level VI opens a window that allows you to select the action you want to perform, the plugin that it should target and (if required) enter the data associated with the action. Because this is a test program, it also incorporates a Boolean control that forces an invalid CRC, so you can test that functionality as well.

So open both projects and run the testbed application, nothing new here – or so it seems. Now run the simple TCP client, its IP address and port number are correct for this test scenario. As soon as the client starts, the waveform graph for displaying the plugin graph data appears, so let’s start with that. You should be able to see the data from each of the 5 testbed plugins by selecting the desired target and clicking the Send Command button. You should also be able to see all 5 graph images.

Now try generating some errors. Turn on the Force CRC Error check-box and retry the tests that you just ran successfully. The client’s error cluster should now show a CRC Error. Next turn the Force CRC Error check-box back off and try doing something illegal, like using the Set Acquisition Rate action on one of the temperature controllers. Now you should see an Update Failed error.

Continue trying things out, verifying that the thing which should work do, and that the things that shouldn’t work, don’t. If you did the testing associated with the last post, you will notice that there is a lag between sending the command and getting the results, but that is to be expected since you are now running over a network interface. Finally, assuming the network is configured correctly, and the desired ports are open, the client application should be able to work from a computer across the room, across the hall or across the world.

Testbed Application – Release 19
Toolbox – Release 16
Simple TCP Client – Release 1

Big Tease

So what is in store for next time? Well, let’s extend things a bit further and look at a way to access this same basic interface, but this time from a web browser! Should be fun.

Until Next Time…

It’s a big interconnected world – and LabVIEW apps can play too

If I were asked to identify one characteristic that set modern test systems apart from their predecessors, my answer would be clear: distributed processing. In the past, test applications were often monolithic programs – but how things have changed! In the past we have talked several times about how the concept of distributed processing has radically altered our approach to system architecture. In fact, the internal design of our Testbed Application is a very concrete expression of that architectural shift. However, the move away from monolithic programs has had a profound impact in another area as well.

In the days of yore, code you wrote rarely had to interact with outside software. The basic rule was that if you wanted your program to do something you had to implement the functionality yourself. You had to assume this burden because there was no alternative. There were no reusable libraries of software components that you could leverage, and almost no way to pass data from one program to another. About all you could hope for were some OS functions that would allow you to do things like read and write disk files, or draw on the screen. In this blog we have looked at ways that our LabVIEW applications can use outside resources through standardized interfaces like .NET or ActiveX. But what if someone else wants to write a program that uses our LabVIEW code as a drop in component? How does that work?

As it turns out, the developers saw this possibility coming and have provides mechanisms that allow LabVIEW application to provide the same sort of standardized interface that we see other applications present. Over the next few posts, we are going to look at how to use incorporate basic remote interfaces ranging from traditional TCP/IP-based networking to building modern .NET assemblies.

What Can We Access?

However, before we can dig into all of that, we need to think about what these interfaces are going to access. In our case, because we have an existing application that we will be retrofitting to incorporate this functionality, we will be looking at the testbed to identify some basic “touchpoints” that a remote application might want to access. By contrast, if you are creating a new application, the process of identifying and defining these touchpoints should be an integral part of you design methodology from day one.

The following sections present the areas where we will be implementing remote access in our testbed application. Each section will describe the remote interface we desire to add and discuss some of the possible implementations for the interface’s “server” side.

Export Data from Plugins

The obvious place to start is by looking at ways of exporting the data. This is, after all, why most remote applications are going to want to access our application: They want the data that we are gathering. So the first thing to consider is, where does the data reside right now? If you go back and look at the original code, you will see that, in all cases, the primary data display for a plugin is a chart that is plotting one new point at a time. Here is what the logic looked like in the Acquire Sine plugin.

Simple Acquisition and Charting

As you can see, the only place the simulated data goes after it is “acquired” is the chart. Likewise, if you looked at the code for saving the data, you would see that it was getting the data by reading the chart’s History property.

Save Chart Data to File

Now, we could expand on that technique to implement the new export functionality, but there is one big consequence to that decision. Approaching the problem in this way would have the side-effect of tying together the number of data points that are saved to the chart’s configuration. Hence, because the amount of data that a chart buffers can’t be changed at runtime, you would have to modify the LabVIEW code to change the amount of data that is returned to the calling application.

A better solution is to leverage what we have learned recently about DVRs and in-place structures to create a storage location the size of which we can control without modifying the application code. A side-effect of this approach is that we will be able to leverage it to improve the efficiency of the local storage of plugin data – yes, sometimes side-effects are good.

To implement this logic we will need three storage buffers: One for each of the two “acquisition” plugins and one for the reentrant “temperature controller” plugin. The interface to each storage buffer will consist of three VIs, the first one of which is responsible for initializing the buffer:

Initialize Buffer

This screenshot represents the version of the initialization routine that serves the Ramp Signal acquisition process. The basic structure of this code is to create a circular buffer that will save the last N samples – where “N” is a reconfigurable number stored in the database. To support this functionality, the DVR incorporates two values: The array of datapoints and a counter that operates as a pointer to track where the current insertion point is in the buffer. These VIs will be installed in the initialization state of the associated plugin screen’s state machine. With the buffer initialized, we next need to be able to insert data. This is typical code for performing that operation:

Insert Data Point

Because the DVR data array is initialized with the proper number of elements at startup, all this logic has to do is replace an existing value in the array with a newly acquired datapoint value, using the counter of course to tell it which element to replace. Although we have a value in the DVR called Counter we can’t use it without a little tweaking. Specifically, the DVR’s counter value increments without limit each time a value is inserted, however, there is only a finite number of elements in the data array. What we need for our circular buffer is a counter that starts at 0, counts to N-1 and then returns to 0 and starts over. The code in the image shows the easiest way to generate this counter. Simply take the limitless count and modulo divide it by the number of points in the buffer. The output of the modulo division operation is a quotient and a remainder. The remainder is the counter we need.

Modulo division is also important to the routine for reading the data from the buffer, but in this case we need both values. The quotient output is used to identify when the buffer is still in the process of being filled with the initial N datapoints:

Read All Data.1

During this initial period, when the quotient is 0, the code uses the remainder to trim off the portion of the buffer contents that are yet to be filled with live data. However, once the buffer is filled, the counter ceases being the a marker identifying the end of the data, and it becomes a demarcation point between the new data and the old data. Therefore, once the quotient increments past 0, a little different processing is required.

Read All Data.2

Once the circular buffer is full, the element that the remainder is pointing at is the oldest data in the array (chronologically speaking), while the datapoint one element up from it is newest. Hence, while the remainder is still used to split the data array, the point now is to swap the two subarrays to put the data in correct chronological order.

Retrieve Graph Images from Plugins

The next opportunity for remote access is to fetch not the data itself, but a graph of the data as it is shown on the application’s GUI. This functionality could form the basic for a remote user interface, or perhaps as an input to a minimalistic web presentation. Simplifying this operation is a control method that allows you to generate an image of the graph and the various objects surrounding it like the plot legend or cursor display. Consequently, the VI that will be servicing the remote connections only needs to be able to access the chart control reference for each plugin. To make those references available, the code now incorporates a buffer that is structurally very similar to the one that we use to store the VI references that allow the GUI to insert the plugins into its subpanel. Due to its similarity to existing code, I won’t cover it in detail, but here are a few key points:

  • Encapsulated in a library to establish a namespace and provided access control
  • The FGV that actually stores the references is scoped as Private
  • Access to the functionality is mediated though publicly-scoped VIs

This FGV is the only new code we will need to add to the existing code.

Adding Remote Control

One thing that both of the remote hooks we just discussed have in common is that they are both pretty passive – or to make this point another way, they both are monitoring what the application is doing without changing what it is doing. Now we want to look at some remote hooks that will allow remote applications control the application’s operation, at least in a limited way.

Since the way the application works is largely dependent upon the contents of the database, it should surprise no one that these control functions will need to provide provisions for the remote application to alter the database contents in a safe and controlled way.

Some Things to Consider

The really important words in that last sentence are “safe” and “controlled”. You see, the thing is that as long as you are simply letting people read the data you are generating, your potential risk is often limited to the value of the data that you are exposing. However, when you give remote users or other applications the ability to control your system, the potential exists that you could lose everything. Please understand that I take no joy in this conversation – I still remember working professionally on a computer that didn’t even have a password. However, in a world where “cyber-crime”, “cyber-terrorism” and “cyber-warfare” have become household terms, this conversation is unavoidable.

To begin with, as a disclaimer you should note that I will not be presenting anything close to a complete security solution, just the part of it that involves test applications directly. The advice I will be providing assumes that you, or someone within your organization, has already done the basic work of securing your network and the computers on that network.

So when it comes to securing applications that you write, the basic principle in play here is that you never give direct access to anything. You always qualify, error-check and validate all inputs coming from remote users or applications. True, you should be doing this input validation anyway, but the fact of the matter is that most developers don’t put a lot of time into validating inputs coming from local users. So here are a couple of recommendations:

Parametrize by Selecting Values – This idea is an expansion on a basic concept I use when creating any sort of interface. I have often said that anything you can do to take the keyboard out of your users’ hands is a good thing. By replacing data that has to be typed with data menus from which they can select you make software more robust and reduce errors. When working with remote interfaces, you do have to support typed strings because unless the remote application was written in LabVIEW, typing is the only option. But what you can do is limit the inputs to a list of specific values. On the LabVIEW-side the code can convert those string values into either a valid enumeration, or a predefined error that cancels the operation and leaves your system unaltered. When dealing with numbers, be sure to validate them also by applying common-sense limits to the inputs.

Create Well-Defined APIs – You want to define a set of interfaces that specify exactly what each operation does, and with as few side-effects as possible. In fancy computer-science terms, this means that operations should be atomic functions that either succeed or fail as a unit. No half-way states allowed! Needless to say, a huge part of being “well-defined” is that the APIs are well-documented. When someone calls a remote function, they should know exactly what is expected of them and exactly what they will get in response.

Keep it Simple – Let’s be honest, the “Swiss Army Knife” approach to interface design can be enticing. You only have to design one interface where everything is parametrized and you’re done, or at least you seem to be for a while. The problem is that as requirements change and expand you have to be constantly adding to that one routine and sooner or later (typically sooner) you will run into something that doesn’t quite fit well into the structure that you created. When that happens, people often try to take the “easy” path and modify their one interface to allow it to handle this new special case – after all, “…it’s just one special case…”. However once you start down that road, special cases tend to multiply like rabbits and the next thing you know, your interface is a complicated, insecure mess. The approach that is truly simple is to create an interface that implements separate calls or functions for each logical piece of information.

With those guidelines in mind, let’s look at the three parameters that we are going to be allowing remote users or applications to access. I picked these parameters because each shows a slightly different use case.

Set the Acquisition Sample Interval

One of the basic ways that you can store a set of parameters is using a DVR, and I demonstrated this technique by using it to store the sample rates that the “acquisition” loops use to pace their operation. In the original code, the parameter was already designed to be changed during operation. You will recall that the basic idea for the parameter’s storage was that of a drop box. It wasn’t important that the logic using the data know exactly when the parameter was changed, as long as it got the correct value the next time it tried to use the data. Consequently, we already have a VI that writes to the DVR (called Sample and, as it turns out, it is all we will need moving forward.

Set Number of Samples to Save

This parameter is interesting because it’s a parameter that didn’t even exist until we started adding the logic for exporting the plugin data. This fact makes it a good illustration of the principle that one change can easily lead to requirements that spawn yet other changes. In this case, creating resizable data buffers leads to the need to be able change the size of those buffers.

To this point, the libraries that we have defined to encapsulate these buffers each incorporate three VIs: one to initialize the buffer, one to insert a new datapoint into it, and one to read all the data stored in the buffer. A logical extension of this pattern would be the addition of a fourth VI, this time one to resize the existing buffer. Called Reset Buffer these routines are responsible for both resizing the buffer, and correctly positioning the existing data in the new buffer space. So the first thing the code does is borrow the logic from the buffer reading code to put the dataset in the proper order with the oldest samples at the top and the newest samples at the bottom.

Put the Data in Order

Next the code compares the new and old buffer sizes in order to determine whether the buffer is growing larger, shrinking smaller or staying the same size. Note that the mechanism for performing this “comparison” is to subtract the two value. While a math function might seem to be a curious comparison operator, this technique makes it easy to identify the three conditions that we need to detect. For example, if the values are the same the difference will be 0, and the code can use that value to bypass further operations. Likewise, if the two numbers are not equal, the sign of the result will indicate which input is larger, and the absolute magnitude of the result tells us how much difference there is between the two.

This is the code that is selected when the result of the subtraction is a positive number representing the number of element that are to be added to the buffer.

Add points to Buffer

The code uses the difference value to create an array of appropriate size and then appends it to the bottom of the existing array. In addition, the logic has to set the Counter value point to the first element of the newly appended values so the next insert will go in the correct place. By contrast, if the buffer is shrinking in size, we need to operate on the top of the array.

Remove points from buffer

Because the buffer is getting smaller, the difference is a negative number representing the number of elements to be removed from the buffer data. Hence, the first thing we need to do is extract the number’s absolute value and use it to split the array, effectively removing the elements above the split point. As before, we also need to set the Counter value, but the logic is a little more involved.

You will remember that the most recent data is on the bottom of the array, so where does the next data point need to go? That’s right, the buffer has to wrap around and insert the next datapoint at element 0 again, but here is where the extra complexity comes in. If we simply set Counter to 0 the data insert logic won’t work correctly. Reviewing the insert logic you will notice that the first pass through the buffer (modulo quotient = 0) is handled differently. What we need is to reinitialize Counter with a number that when subjected to the modulo division will result in a remainder of 0, and a quotient that is not 0. An easily derived value that meets that criteria is the size of the array itself.

Finally we have to decide what to do when the buffer size isn’t changing, and here is that code. Based on our discussions just now, you should be able to understand it.

buffer size not changing

Set Temperature Controller Operating Limits

Finally, there are two reasons I wanted to look at this operation: First, it is an example of where you can have several related parameters that logically form a single value. In this case, we have 5 separate numbers that, together, define the operation of one of the “temperature controller” processes. You need to be on the look-out for this sort of situation because, while treating this information as 5 distinct value would not be essentially wrong, that treatment would result in you needing to generate a lot of redundant code.

However, this parameter illustrates a larger issue, namely that changes in requirements can make design decisions you made earlier – let’s say – problematic. As it was originally designed, the temperature controller values were loaded when the plugins initialized, and they were never intended to be changed during while the plugin was running. However, our new requirement to provide remote control functionality means that this parameter now needs to be dynamic. When confronted with such a situation, you need to look for a solution that will require the least rework of existing code and the fewest side-effects. So you could:

  1. Redesign the plugin so it can restart itself: This might sound inviting at first because the reloading of the operating limits would occur “automatically” when the plugin restarted. Unfortunately, it also means that you would have to add a whole new piece of functionality: the ability for the application to stop and then restart a plugin. Moreover, you would be creating a situation where, from the standpoint of a local operator, some part of the system would be restarting itself at odd intervals for no apparent reason. Not a good idea.
  2. Redesign the plugin to update the limits on the fly: This idea is a bit better, but because the limits are currently being carried through the state machine in a cluster that resides in a shift-register, to implement this idea we will need to interrupt the state machine to make the change. Imposing such an interruption risks disrupting the state machine’s timing.

The best solution (as in all such cases) is to address the fundamental cause: the setups only load when the plugin starts and so are carried in the typedef cluster. The first step is to remove the 5 numbers associated with the temperature controller operating parameters from the cluster. Because the cluster is a typedef, this change conveniently doesn’t modify the plugin itself, though it does alter a couple of subVIs – which even more conveniently show up as being broken. All that is needed to repairs these VIs is to go through them one by one and modify the code to read the now-missing cluster data values with the corresponding values that the buffered configuration VI reads from the database. Said configuration VI (Load Machine also requires one very small tweak:

Reload Enable

Previously, the only time logic would force a reload of the data was when the VI had not been called before. This modification adds an input to allow the calling code to request a reload by setting the new Reload? input to true. To prevent this change from impacting the places where the VI is already being called, the default value for this input is false, the input is tied to a here-to-fore unused terminal on the connector pane, and the terminal is marked as an Optional input.

Building Out the Infrastructure

At this point in the process, all the modifications that need to be done to the plugins themselves have been accomplished, so now we need is a place for the external interface functionality itself to live. One of the basic concepts of good software design is to look at functionality from the standpoint of what you don’t know or what is likely to change in the future, and then put those things into abstracted modules by themselves. In this way, you can isolate the application as a whole, and thus protect it from changes that occur in the modularized functionality.

The way this concepts applies to the current question should be obvious: There is no way that we can in the here and now develop a compete list of the remote access functionality that users will require in the future. The whole question is at its essence, open-ended. Regardless if how much time you spend studying the issue, users have an inherently different view of your application than you do and so they will come up with needs that you can’t imagine. Hence, while today we might be able to shoe-horn the various data access and control functions into different places in the current structure, to do so would be to start down a dead-end road because it is unlikely that those modifications would meet the requirements of tomorrow. What we need here is a separate process that will allow us to expand or alter the suite of data access and control functionality we will offer.

Introducing the Remote Access Engine

The name of our new process is Remote and (like most of the code in the project) it is designed utilizing an event-drive structure that ensures it is quiescent unless it is being actively accessed. The process’ basic theory of operation is that when one of its events fire, it performs the requested operation and then sends a reply in the form of a notification. The name of the notification used for the reply is sent as part of the event data. This basic process is not very different from the concept of “callbacks” used in languages such as JavaScript.

Although this process is primarily intended to run unseen in the background, I have added three indicators to its front panel as aides in troubleshooting. These indicators show the name of the last event that it received, the name of the plugin that the event was targeting, and the name of the response notifier.

The Read Graph Data Event

The description of this event handler will be longer than the others because it pretty much sets the pattern that we will see repeated for each of the other events. It starts by calling a subVI (Validate Plugin that tests to see if the Graph Name coming from the event data is a valid plugin name, and if so, returns the appropriate enumeration.

Validate plugin name

The heart of this routine is the built-in Scan from String function. However, due to the way the scan operation operates, there are edge conditions where it might not always perform as expected when used by itself. Let’s say I have a typedef enumeration named Things I Spend Too Much Time Obsessing Over.ctl with the values My House, My Car, My Cell Phone, and My House Boat, in that order. Now as I attempt to scan these values from strings, I notice a couple of “issues”. First there is the problems of false positives. As you would expect, it correctly converts the string “My House Boat” into the enumerated value My House Boat. However, it would also convert the string “My House Boat on the Grand Canal” to the same enumeration and pass the last part of the string (” on the Grand Canal”) out its remaining string output. Please note that this behavior is not a bug. In fact, in many situations it can be a very useful behavior – it’s just not the behavior that we want right now because we are only interested in exact matches. To trap this situation, the code marks the name as invalid if the remaining string output is not empty.

The other issue you can have is what I call the default output problem. The scan function is designed such that if the input string is not scanned successfully, it outputs the value present at the default value input. Again, this operation can be a good thing, but it is not the behavior that we want. To deal with this difference, the code tests the error cluster output (which generates and error code 85 for a failed scan) and marks the name as invalid if there is an error.

When Validate Plugin finishes executing, we have a converted plugin name and a flag that tells us whether or not we can trust it. So the first thing we do is test that flag to see whether to continue processing the event or return an error message to the caller. Here is the code that executes when the result of the name validation process is Name Not Valid.

Name Not Valid

If the Response Notifier value from the event data is not null, the code uses it to send the error message, “Update Failed”. Note that this same message is sent whenever any problem arises in the remote interface. While this convention certainly results in a non-specific error message, it also ensures that the error message doesn’t give away any hints to “bad guys” trying to break in. If the Response Notifier value is null (as it will be for local requests) the code does nothing – remember we are also wanting to leverage this logic locally.

If the result of the name validation process is Name Valid, the code now considers the Plugin Name enumeration and predicates its further actions based on what it finds there. This example for Sine Source shows the basic logic.

Name Valid - Remote

The code reads the data buffer associated with the signal and passes the result into a case structure that does one of two different things depending on whether the event was fired locally, or resulted from a remote request. For a remote request (Response Notifier is not null), the code turns the data into a variant and uses it as the data for the response notifier. However, if the request is local…

Name Valid - Local

…it sends the same data to the VI that saves a local copy of the data.

The Read Graph Image Event

As I promised above, this event shares much of the basic structure as the one we just considered. In fact, the processing for a Name Not Valid validation result is identical. The Name Valid case, however, is a bit simpler:

Read Graph Image

The reason for this simplification is that regardless of the plugin being accessed, the datatypes involved in the operation are always the same. The code always starts with a graph control reference (which I get from the lookup buffer) and always generates an Image Data cluster. If the event was fired locally, the image data is passed to a VI (Write PNG that prompts the user for a file name and then saves it locally. However, if instead of saving a file, you are wanting to pass the image in a form that is usable in a non-LabVIEW environment, a bit more work is required. To encapsulate that added logic, I created the subVI Send Image

Send Image Data

The idea is to convert the proprietary image data coming from the invoke node into a generic form by rendering it as a standard format image. Once in that form, it is a simple matter to send it as a binary stream. To implement this approach, the code first saves the image to a temporary png file. It then reads back the binary contents of the file and uses it as the data for the response notifier. Finally, it deletes the (now redundant) temporary file.

The Set Acquisition Rate Event

This event is the first one to control the operation of the application. It also has no need to be leveraged locally, so no dual operation depending on the contents of the Response Notifier value.

Set Acquisition Rate

Moreover, because the event action is a command and not a request, the response can only have one of two values: “Update Failed” or “Update Good”. The success message is only be sent if the plugin name is either Sine Source or Ramp Source, and no errors occurs during the update. While on the topic of errors, there are two operations that need to be performed for a complete update: the code must modify both the database and the buffer holding the live copy of the setting that the rest of the application uses. In setting the order of these two operations, I considered which of the two is most likely to generate an error and put it first. When you consider that most of the places storing live data are incapable of generating an error, the database update clearly should go first.

So after verifying the plugin name, the subVI responsible for updating the database (Set Default Sample looks to see if the value is changing. If the “new” value and the “old” value are equal, the code returns a Boolean false to its Changed? output and sets the Result output to Update Good. It might seem strange to return a value to the remote application that the update succeeded when there was no update performed, but think about it from the standpoint of the remote user. If I want a sample period of 1000ms, an output of Update Good tells me I got what I wanted – I don’t care that it didn’t have to change to get there. If the value is changing…

Set Default Sample Period

…the code validates the input by sending it to a subVI that compares it to some set limits (500 < period < 2500). Right now these limits are hardcoded, and in some cases that might be perfectly fine. You will encounter many situations where the limits are fixed by the physics of a process or a required input to some piece of equipment. Still, you might want these limits to be programmable too, but I’ll leave that modification as, “…as exercise for the reader.” In any case, if the data is valid, the code uses it to update the database and sets the subVI’s two outputs to reflect whether the database update generated an error. If the data is not valid, it returns the standard error message stating so.

The Set Data Buffer Depth Event

The basic dataflow for this event is very much like the previous one.

Set Data Buffer Depth

The primary logical difference between the two is that all plugins support this parameter. The logic simply has to select the correct buffer to resize.

The Set TC Parameters Event

With our third and (for now at least) final control event, we return to one that is only valid for some of the plugins – this time the temperature controllers.

Set TC Parameters

The interesting part of this event processing is that, because its data was not originally intended to be reloaded at runtime, the live copy of the data is read and buffered in the object-oriented configuration VIs.

Save Machine Configuration

Consequently, the routine to update the database (Save Machine first creates a Config Data object and then use that object to read the current configuration data. If the data has changed, and is valid, the same object is passed on to the method that writes the data to the database. Note also, that the validation criteria is more complex.

Validate TC Parameters

In addition to simple limits on the sample interval, the Error High Level cannot exceed 100, the Error Low Level cannot go below 30, and all the levels have to be correct relative to each other.


With the last of the basic interface code written and in place, we need to look at how to test it. To aide in that effort, I created five test VIs – one for each event. The point of these VIs is to simply exercise the associated event so we can observe and validate the code’s response. For instance, here’s the one for reading the graph data:

Test Read Graph Data

It incorporates two separate execution paths because it has two things that it has to be doing in parallel: Sending the event (the top path) and waiting for a response (the bottom path). Driving both paths, is the output from a support VI from the notifier library (not_Generic Named Notifier.lvlib:Generate Notifier It’s job is to generate a unique notifier name based on the current time and a 4-digit random number. Once the upper path has fired the event, it’s work is done. The bottom path displays the raw notifier response and graphs of the data that is transferred. Next, the test VI for reading the graph image sports similar logic, but the processing of the response data is more complex.

Test Read Graph Image

Here, the response notifier name is also used to form the name for a temporary png file that the code uses to store the image data from the event response. As soon as the file is written, the code reads it back in as a png image and passes it to a subVI that writes it to a 2D picture control on the VI’s front panel. Finally, the three test VIs for the control operations are so similar, I’ll only show one of them.

Test Resizing Data Buffers

This exemplar is for resizing the data buffers. The only response processing is that it displays the raw variant response data.

To use these VIs, launch the testbed application and run these test VIs manually one at a time. For the VIs that set operating parameters, observe that entering invalid data generates the standard error message. Likewise, when you enter a valid setting look for the correct response in both the program’s behavior and the data stored in the database. For the VI’s testing the read functions, run them and observe that the data they display matches what the selected plugin shows on the application’s GUI.

Testbed Application – Release 18
Toolbox – Release 15

The Big Tease

In this post, we have successfully implemented a remote access/control capability. However, we don’t as of yet have any way of accessing that capability from outside LabVIEW. Next time, we start correcting that matter by creating a TCP/IP interface into the logic we just created. After that introduction, there will be posts covering .NET, ActiveX and maybe even WebSockets – we’ll see how it goes.

Until Next Time…