Plotting Large Datasets Without Waiting Forever

Posted on 2015-07-312015-07-31 by mlportersr

Many years ago in his landmark 1977 book, Exploratory Data Analysis, John Tukey made a comment that in his pre-PC, pre-Excel, pre-most-everything-we-think-we-need-to-do-analysis time was pretty radical, what which is today pretty common-sense:

“…there is never a good reason to not look at a plot of your data.”

Any test engineer worth his or her salt will admit that we often do our best work when we heed that truly sage advice. Looking at a plot allows us to see the data’s basic “shape” and begin to develop something of an intuitive understanding of relationships that might exist in the data. The problem is that in a world where “Big Data” is the catchphrase of the day, that simple advice can be difficult to put into practice. It has become increasingly common to see people on the user forum wondering how to effectively view work with datasets that incorporate hundreds, thousands or even hundreds of thousands of datapoints.

Seeing the Problem

To assist us in visualizing some of the problems inherent in handling large datasets, I have put together a test dataset consisting of 3 traces, each with over 19,000 datapoints. Now when I just read the data and plot it, this is what I get:

Clearly there is an issue here – I mean what is up with the wide vertical bars? But there is an even larger problem. Let’s say I change the size of the plot by making it just 9 pixels wider.

Now what is going on? The white bars have changed and if you look at the peaks in the data carefully, some of them appear to have moved or even disappeared. In order to get your head wrapped around what is happening, consider what LabVIEW is having to do behind the scenes. I mentioned that the dataset had over 19,000 datapoints (19,555, to be exact) but the active plot area of the display is only 350 pixels wide. If you do the math, you discover that to generate this plot, each pixel has to represent about 57 datapoints. The problem of course is that you can’t subdivide a pixel into 57 pieces. So what is LabVIEW to do?

Well it does what any graphing package does when it is confronted with this challenge: it decimates the data. In other words it takes 57-datapoint chunks of the data, performs some sort of statistical operation on each chunk (min, max, mean, etc) and then uses the resulting summary value to represent that chunk of data on the plot. There are several potential problems with this way of handling the situation, but they typically don’t become an issue unless the dataset being plotted is very large relative to the size of the graph. For example, this is why the data on the graph appeared to change as a function of the size of the plot area. As the plot area changed (even slightly) the chunking changes so the data appears to change as well – you can think of it as sort of visual aliasing.

More subtle problems have to do with the way the graphing routines “summarize” the data chunks. Depending upon the shape of your dataset, the operations I mentioned earlier can give dramatically different output and to make matters worse you have no idea what techniques the graphing functions are using. But even if you can live with the visual effects there are good reasons to take action to address the issue.

Finally, in order to plot these huge datasets you have to be carrying them around inside your program. Consequently, rather than having just one copy of these monsters, you can have several – perhaps dozens – it all depends on how your code it written. From this discussion we can then see the two imperatives for our solution:

The approach must minimize the number of copies that LabVIEW has to make of the dataset.
It must reduce the number of datapoints that actually need to be plotted.

Let’s start by looking at the data management aspect of the problem, remembering of course that these two issues are inextricably linked together.

Low-Overhead Storage

Decades ago, people in the nascent computer-science discipline realized that if you had a value, like an array, that consisted of multiple items, the most efficient way of making it available throughout your code was to store it in one location in memory and give the code that needed to access it a “pointer” that served as a reference to that value. Originally this mechanism was pretty primitive with the pointer often consisting of simply the value’s starting address in RAM. In addition, there was no real way of preventing race conditions or security intrusions because there was no way of controlling access to the data. It would be nice to think that we have learned the errors of our ways and fixed all the holes, but such is not always the case. Remember the “Heartbleed” bug panic from last year?

The good news is that LabVIEW does not suffer from the same problems because while we have at our disposal a mechanism that fills the same role as the primitive pointer, it lacks the problems. I am talking about the Data Value Reference, or DVR. It meets the low-overhead storage mandate by accessing the data through a reference that is only 4 bytes long. The DVR is also secure because the buffer that is creates is strongly typed, meaning that you can’t just store anything in it or read whatever you want from it. The data going in and coming out must match the definition of the data structure that was used when the DVR was defined. Finally, the DVR removes problems resulting from simultaneous access to the same resource by defining a new structure that automatically serializes access on a first-come, first-served basis. So the first thing we need to do is get our data into the DVR, and here’s some code to do just that.

The VI starts by reading a binary file containing the data which, to simplify this example, is already formatted correctly for how we are going to use it. The resulting array drives a box called an inplace structure that guarantees there will be no other accesses to the DVR occurring in parallel with this one. However, the structure does something else too: Inplace structures operate something like compiler directives telling the LabVIEW compiler that its OK to attempt additional optimizations that would not otherwise be safe to make. For example, they allow to LabVIEW operates on the inplace data without making the copies that the compiler might otherwise need make.

The other thing to note is that funny-looking function in the middle of the inner inplace structure. It’s called Swap Values and its help description really doesn’t do it justice. If all you did was read the context help you might assume that it is simply some sort of switch for routing signals around, and stifling a yawn, go on to consider matters that you think look more exciting. To see why you should consider this function very exciting, we need to look under LabVIEW’s hood.

To store data internally, LabVIEW uses memory data buffers. In fact much of what we think of as “dataflow” consists of the manipulation of those buffers. Now when LabVIEW stores a complex datatype like a cluster (which is what the DVR in this case is holding) it uses a combination of techniques. For simple fix-sized data like numerics or booleans, LabVIEW simply includes the data values directly in the cluster’s memory space. However, it needs a different approach when storing data values like arrays or strings that can vary in length. When a cluster includes an item that can change in size, the item is stored outside the cluster in its own memory buffer and the cluster only holds a reference to that buffer. That way if the item changes in size it can do so without effecting the memory allocation of the cluster containing it.

However this explanation also reveals why the Swap Values node is so important. Let’s look at this code from the standpoint of buffers. Coming into the inner inplace structure there are two buffers allocated that are holding arrays: One contains the data I read from the file, and one the (empty) array that is contained in the cluster that is the contents of the DVR. Now there are two ways that we could initialize that array. The most obvious one is to leave the unbundle (left) side of the cluster inplace structure unwired and wire the array containing the data directly to the bundle (right) side of the cluster inplace structure. While this would work, coding it in that way would result in LabVIEW needing to copy the data contained in the incoming array’s buffer to the array buffer associated with the cluster – and the larger the dataset is, the longer this copy can take.

Now consider what happens when Swap Values is used. Although the node resides inside an inplace structure, it would seem logical that you can’t replace an empty array with an array containing thousands of datapoints in place. Well actually you can. The key point to remember is that at a very low level, the clusters don’t actually contain the arrays, rather they hold references that point to the arrays that are associated with them. So what Swap Values does is it leaves the two arrays in place and simply swaps the references that the clusters contain. Thanks to this optimization, populating this cluster with data will take the exact same amount of time whether the input data contains 2 datapoints or 200,000 datapoints because the only thing that is really being moved is a pair of 4-byte memory buffer references.

Getting Data Out

So we have gotten our data into the DVR as efficiently as we can, but if this storage is going to be of any use, there clearly needs to be a way to get data out of it as well. However, here we face the issue of plotting data that is too large. At the same time we are pulling it out, we also need to be reducing or decimating it to more closely match the size of the available graphing area. To meet those dual requirements I created this VI.

At first this code might seem intimidating, but if you take it step-by-step and analyse what it’s doing, it isn’t really so very different from the example we looked for initializing the data in the DVR. Starting at the left side, the code unbundles the data array from the DVR and passing it into a loop that will execute three times – once for each plot in the dataset. The first point of optimization is in how this loop operates. Note the node with the “P” in it. The presence of this node means that the for loop is set for parallel operation. There are many situations where, even though you specify a for loop, there is no logical reason that the iterations have to operate sequentially. When LabVIEW encounters a “parallelized” loop the optimizer essentially flattens the loop out, creates the necessary parallel code to execute each iteration simultaneously, and then reassemble the output data in the correct order. To find out if a loop is parallelizable, there is an option under Tools>>Profile called Find Parallelizable Loops…. This operation opens a dialog that allows you to identify the loops that can and cannot be run in parallel mode.

Inside the loop, the array drives an inplace structure that indexes out one element, and the resulting cluster feeds a second inplace structure that unbundles the two items in the cluster. The processing of this data occurs in two distinct steps. First the Start and Length inputs produce a subset of the total dataset representing the portion of the data that is to be displayed. Because this operation causes LabVIEW to copy the selected data into a new memory buffer, the code passes the resulting arrays into another inplace structure to ensure that the subset will also be manipulated inplace.

The code inside this inner-most inplace structure performs the second half of the processing – the decimation to reduce the size of the data being plotted. Note that if the selected portion of the dataset is already smaller than the width of plot area, the following code is bypassed. The first step in the decimation process is to reshape the 1D array into a 2D array where each row contains one chunk of data to be statistically summarized. To obtain the final X values, the code takes the first value of each chunk, while the final Y values are the maximum Y for each chunk. Note that this processing occurs in another parallelized loop that auto-indexes the output arrays, which are swapped into the output dataset as they work their way back out through the inplace structures.

Summarizing Options and Challenges

The real heart of this VI is the function that is being used to summarize the Y values for each chunk of data. Right now, I am using the function that returns the minimum and maximum values contained in the array. One of the advantages that it offers is that it is deals well with datasets containing missing datapoints represented by the value NaN. This consideration is important because it is a common (and valuable) practice to represent missing data points using that value. Without the NaN datapoints, any graph will simply connect the dots on either side of the missing datapoints resulting in a graph that visually misrepresents the data being presented. However, with the NaN values, the missing points are shown as breaks in the line (or gaps between bars), thus highlighting the missing data.

The statistical function I selected to summarize the data in the chunks simply returns the minimum and maximum values of the elements that are not NaN. However, most other analysis routines follow the basic rule that any calculation which has NaN as an operand will return an answer of NaN – which in this situation will not be real helpful. More often, what you will want is, for example, the average value of the datapoints that are present in the dataset chunk. If you are wanting to use the chunk mean or median value to summarize a dataset that you know contains NaN value, you should include something like this before the statistical operation:

Basically it works by first sorting the array to move any NaN values to the end of the array. It then looks for the first NaN and simply trims off it (and anything after it). This works because a mean operation doesn’t care about data order, and the first thing a median function does is sort the data anyway.

Let’s See How it Works

When you run the top-level VI in the linked project, the graph that comes up will look a lot like the first image in this post, but minus the vertical white bars. As you make changes to the display that effect the X axis range, you will notice that the resulting image will zoom in on the data, showing ever greater levels of detail. Try manually typing in new X axis end points, or use the horizontal zoom tool on the graph palette to select a range of data points that you want to zoom in on.

Zoom in far enough and you will see why there were white bars on the original plot: There are a lot of missing datapoints. Using the default decimation resulted in wide white bars because the presence of the NaN values effectively hid dozens of real datapoints.

Plotting Large Datasets – Release 1
Toolbox – Release 11

Hopefully this discussion will give you something to think about, and experiment with.

The Big Tease

One of the things that developers often have to face is adding functionality to an existing VI without disrupting, or even modifying what is already there. One way to accomplish this (seemingly impossible) task is to use what are sometimes called “drop-in” VIs. These routines are simply dropped down on an existing block diagram and they do what they do without interaction with the existing code. To demonstrate how this could work, next time we’ll get back to our test bed application and give it the ability to customize the font and size of the test that are on its various displays.

Until Next Time…
Mike…

More Than One Kind of Modularity

Posted on 2015-07-132015-07-13 by mlportersr

When learning something that you haven’t done before – like .NET – it’s not uncommon to go through a phase where you look at some of the code you wrote early on and cringe (or at least sigh deeply). The problem is that you are often not only learning a new interface or API, but you are learning how to best use that interface or API. The cause of all the cringing and sighing is that as you learn more, you begin to realize that some of the assumptions and design decisions that you made were misguided, if not flat-out wrong. If you look at the code we put together last time to help us learn about .NET in general, and the NotifyIcon assemble in particular, we see a gold-plated example of just such code. Although it is clearly accomplished it’s original goal of demonstrating how to access .NET functionality and illustrating how the various objects can relate to one another, it is certainly not reusable – or maintainable, or extensible, or any of the other “-ables” that good software needs to be.

In fact, I created the code in that way so this time we can take the lesson one step further to fix those shortcomings, and thus demonstrate how you can go about cleaning up code (of your own or inherited) that is making you cringe or sigh. Remember, it is always worth your time to fix bad design. I can’t tell you how many times I have seen people struggling with bad decisions made years before. Rather than taking a bit of time to fix the root cause of their trouble, they continue to waste hours on project after project in order to workaround the problem.

Ok, so where do we start?

Clearly this code would benefit from cleaning-up and refactoring, but where and how should we start? Well, if you are working on an older code base, the question of where to start will not be a problem. You start with where the most pain is. To put it another way, start with the things that cause you the biggest problems on a day-to-day basis.

This point, however, doesn’t mean that you should just sit around and wait for problems to arise. As you are working always be asking yourself if what you are doing has limitations, or embodies assumptions that might cause problems in the future.

The next thing to remember is that this work can, and should, be iterative. In other words you don’t have to fix everything at once. Start with the most egregious errors, and address the others as you have the opportunity. For example, if you see the code doing something stupid like using a string as a state variable, you can fix that quickly by replacing the strings with a typedef enumeration. I have even fixed some long-standing bugs in doing this replacement because it resolved places where states were subtly misspelled or contained extraneous spaces.

Finally, remember that the biggest payoffs, in terms of long-term benefit, come from improved modularity that corrects basic architectural problems. As we shall see in the following discussion, I include under this broad heading modularity in all its forms: modular functionality, modular logic and modular data.

Revisiting Modular Functionality

Modular functionality is the result of taking small reusable bits of code and encapsulating it in routines that simplify access, standardize the interface or ensure proper usage. There are good examples of all these usages in the application modified code, starting with Create NotifyIcon.vi:

Your first thought might be why I bothered turning this functionality into a subVI. After all, it’s just one constructor node. Well, yes that is true but it’s also true that in order to create that one node you have to remember multiple steps and object names. Even though this subVI appears rather simple, if you consider what it would take to recreate it multiple times in the future, you realize that it actually encapsulates quite a bit of knowledge. Moreover, I want to point out that this knowledge is largely stuff for which there is no overwhelming benefit to be gained from you committing it to memory.

Next, let’s consider the question of standardizing interfaces. Our example in this case is a new subVI I created to handle the task of assigning an icon to the interface we are creating. I have named it Set NotifyIcon Icon.vi:

You will remember from out previous discussion that this task involves passing a .NET object encapsulating the icon we wish to use to a property node for the NotifyIcon object. Originally, this property was combined with several others on a single node. A more flexible approach is to breakup that functionality and standardize the interfaces for all the subVIs that will be setting NotifyIcon to simply consist of an object reference and the data to be used to set the property in a standard LabVIEW datatype – in this case a path input. This approach also clearly simplifies access to the desired functionality.

Finally, there is the matter of ensuring proper usage. A good place to highlight that feature is in the last subVI that the application calls before quitting: Drop NotifyIcon.vi.

You have probably been warned many times about the necessity of closing references that you open. However, when working with .NET objects, that action by itself is sometimes not sufficient to completely release all the system resources that the assembly had been using. Most of the time if you don’t completely close out the assembly you many notice memory leaks or errors from attempting to access resources that still think they are busy. However with the NotifyIcon assembly you will see a problem that is far more noticeable, and embarrassing. If you don’t call the Dispose method your program will close and release all the memory it was using, but if you go to the System Tray you’ll still see your icon. In fact, you will be able to open its menu and even make selections – it just doesn’t do anything. Moreover, the only way to make it go away it to restart your computer.

Given the consequences of forgetting to include this method in your shutdown sequence, it is a good idea to make it an integral part of the code that you can’t forget to include.

Getting Down with Modular Logic

But as powerful as this technique is, there can still be situations where the basic concept of modularity needs to be expressed in a slightly different way. To see such a situation, let’s look at the structure that results from simply applying the previous form of modularity to the problem of building the menus that go with the icon.

Comparing this diagram to the original one from last time, you can see that I have encapsulated the repetitive code that generated the MenuItem objects into dedicated subVIs. By any measure this change is a significant improvement: the code is cleaner, better organized, and far more readable. For example, it is pretty easy to visualize what menu items are on submenus. However, in cases such as this one, this improved readability can be a bit of a double-edged sword. To see what I mean, consider that for the structure of your code to allow you to visualize your menu organization, said organization must be hard-coded into the structure of the code. Consequently, changes to the menus will, as a matter of course, require modification to the fundamental structure of the code. If the justifications for modularity is to include concepts like flexibility and reusability, you just missed the boat.

The solution to this situation is to realize that there is more than one flavor of modularity. In addition to modularizing specific functionality, you can also modularize the logic required to perform complex and changeable tasks (like building menus) that you don’t want to hard code. If this seems like a strange idea to you, consider that computers spend most of their time using their generalized hardware to performed specialized tasks defined by lists of instructions called “programs”. The thing that makes this process work is a generalized bit of software called a “compiler” that turns the programs into data structures that the generalized hardware can use to perform specialized actions.

Carrying forward with this line of reasoning, what we need is a simple way of defining a menu structure that is external to our program, and a “menu compiler” that turns that definition into the MenuItem references that our program needs. So let’s build one…

Creating the Data for Our Menu Compiler

So what should this menu definition look like? Well, to answer that question we need to start with the data required to define a single MenuItem. We see that as a minimum, every item in a menu has to have a name for display to the user, a tag to identify it, and a parent tag that says if the item has a parent item (and if so which item is its parent). In addition, we haven’t really talked about it, but the order of references in an array of menu items defines the order in which the items appear in the menu or submenu – so we need a way to specify its menu position as well. Finally, because in the end the menu will consist of a list (array) of menu item references, it makes sense to express the overall menu definition that we will eventually compile into that array of references as a list (and eventually also an array).

But where should we store this list of menu item definitions? At least part of the to this question depends on who you want to be able to modify the menu, and the level of technical expertise that person has. For example, you could store this data in text files as INI keys, or as XML or JSON strings. These files have the advantage of being easy to generate and are readily accessible to anyone who has access to a text editor – of course that is their major disadvantage, as well. Databases on the other hand are more secure, but not as easy to access. For the purposes of this discussion, I’ll store the menu definitions in a JSON file because, when done properly, the whole issue of how to parse the data simply goes away.

To see what I mean, here is a nicely indented JSON file that describes the menu that we have been working using for our example NotifyIcon application:

[
	{
		"Menu Order":0,
		"Item Name":"Larry",
		"Item Tag":"Larry",
		"Parent Tag":"",
		"Enabled":true
	},{
		"Menu Order":1,
		"Item Name":"Moe",
		"Item Tag":"Moe",
		"Parent Tag":"",
		"Enabled":true
	},{
		"Menu Order":2,
		"Item Name":"The Other Stooge",
		"Item Tag":"The Other Stooge",
		"Parent Tag":"",
		"Enabled":true
	},{
		"Menu Order":3,
		"Item Name":"-",
		"Item Tag":"",
		"Parent Tag":"",
		"Enabled":true
	},{
		"Menu Order":4,
		"Item Name":"Quit",
		"Item Tag":"Quit",
		"Parent Tag":"",
		"Enabled":true
	},{
		"Menu Order":0,
		"Item Name":"Curley",
		"Item Tag":"Curley",
		"Parent Tag":"The Other Stooge",
		"Enabled":true
	},{
		"Menu Order":1,
		"Item Name":"Shep",
		"Item Tag":"Shep",
		"Parent Tag":"The Other Stooge",
		"Enabled":true
	},{
		"Menu Order":2,
		"Item Name":"Joe",
		"Item Tag":"Joe",
		"Parent Tag":"The Other Stooge",
		"Enabled":true
	}
]

And here is the LabVIEW code will convert this string into a LabVIEW array (even if it isn’t nicely indented):

JSON has a lot of advantages over techniques like XML: For starters, it’s easier to read, and a lot more efficient, but this is why I really like using JSON: It is so very convenient.

Starting the Compilation

Now that we have our raw menu definition string read into LabVIEW and converted into a datatype that will simplify the next step in the processing, we need to ensure that the data is in the right order. To see why, we need to remember that the final data structure we are building is hierarchical, so the order in which we build it matters. For instance, “The Other Stooge” is a top-level menu item, but it is also a submenu so we can’t build it until we have references to all the menu items that are under it. Likewise, if one of the items under it is a submenu, we can’t build it until all its children are created.

So given the importance of order, we need to be careful how we handle the data because none of the available storage techniques can on their own guarantee proper ordering. The string formats can all be edited manually, and it’s not reasonable to expect people to always type in data in the right order. Even though databases can sort the result of queries, there isn’t enough information in the menu definition to allow it to do so.

The menu definition we created does have a numeric value that specifies the order of items in their respective menus and submenus. We don’t, however, yet have a way of telling the level the items reside at relative to the overall menu structure. Logically we can see that “Larry” is a top-level menu item, and “Shep” is one level down, but we can’t yet determine that information programmatically. Still the information we need is present in the data, it just needs to be massaged a bit. Here is the code for that task:

As you can see, the process is basically pretty simple. I first rewrite the Item Tag value by adding the original Item Tag value to the colon-delimited list that starts with the Parent Tag. I then count the number of colons in the resulting string, and that is my Menu Level value. The exception to this processing are the top-level menu items which are easy to identify due to their null parent tags. I simply force their Menu Level values to zero and replace the null string with a known value that will make the subsequent processing easier. The real magic however, occurs after the loop stops. The code first sorts the array in ascending order and then reverses the array. Due to the way the 1D array sort works when operating on arrays of clusters, the array will be sorted first by Menu Level and then Menu Order – the first two items in the cluster. This sorting, in concert with the array reversal, guarantees that the children of a submenu will always be processed before the submenu item itself.

Some of you may be wondering why we go to all this trouble. After all, couldn’t we just add a value to the menu definition data to hold the Menu Level? Yes, we could, but it’s not a good idea, and here’s why. In some areas of software development (like database development, for instance) the experts put a lot of store in reducing “redundancy” – which they define basically as storing the same piece of information in more than one place. The problem is that if you have redundant information, you have to decide how to respond when the two pieces of information that are supposed to be the same, aren’t. So let’s say we add a field to the menu definition for the menu level. Now we have the same piece of information stored in two different places: It is stored explicitly in the Menu Level value while at the same time it is also stored implicitly in Parent Tag.

Generating the Menu Item “Code”

In order to turn this listing into the MenuItem references we need, we will pass this sorted and ordered array into a loop that will process one element at a time. And here it is:

You can see that the loop carries two shift registers. The top SR holds a 1D array of strings that consists of the submenu tags that the loop has encountered so far. The other SR also carries a 1D array but each element in it is a cluster containing an array of MenuItem references associated with the submenu named in the corresponding element of the top SR.

As the screenshot shows, the first thing that happens in the loop is that the code checks to see if the indexed Item Tag is contained in the top SR. If the tag is missing from the array it means that the item is not a submenu, so the code uses its data to create a non-submenu MenuItem. In parallel with that operation, the code is also determining what to do with the reference that is being created by looking to see if the item’s Parent Tag exists in the top SR. If the item’s parent is also missing from the array, the code creates entries for it in both arrays. If the parent’s tag is found in the top SR, it means that one or more of the item’s sibling items has already been processed so code is executed to add the new MenuItem to the array of existing ones:

Note that the new reference is added to the top of the array. The reason for this departure from the norm is that due to the way the sorting works, the menu order is also reversed and this logic puts the items on each submenu back in their correct order. Note also that during this processing the references associated the menu items are also accumulated in a separate array that will be used to initialize the callbacks. Because the array indexing operation is conditional, only a MenuItem that is not a submenu, will be included in this array.

Generating the Submenu “Code”

If the indexed Item Tag is found in the top SR, the item is a submenu and the MenuItem references needed to create its MenuItem should be in the array of references stored in the bottom SR.

So the first thing the code does is delete the tag and its data from the two array (since they are no longer needed) and uses the data thus obtained to create the submenu’s MenuItem. At the same time, the code is also checking to see if the submenu’s parent exists in the top SR. As before, if the Parent Tag doesn’t exist in the array, the code creates an entry for it, and if it does…

…adds the new MenuItem to the existing array – again at the top of the array. By the time this loop finishes, there should be only one element in each array. The only item left in the top SR should be “[top-menu]” and the bottom SR should be holding the references to the top-level menu items. The array of references is in turn used to create the ContextMenu object which written to the NotifyIcon object’s ContextMenu property.

What Could Possibly Go Wrong?

At this point, you can run the example code and see an iconic system tray interface that behaves pretty much as it did before, but with a few extra selections. However, we need to have a brief conversation about error checking, and frankly in this situation there are two schools of though on this topic. There is ample opportunity for errors to creep into the menu structure. Something as simple as misspelling a parent tag name could result in an “orphan” menu that would never get displayed – or could end up being the only one that is displayed. So the question is how much error checking do we really need to do? There are those that think you should spend a lot of time going through the logic looking for and trapping every possible error.

Given that most menus should be rather minimal, and errors are really obvious, I tend to concentrate on the low-hanging fruit. For example, one simple check that will catch a large number of possible errors, is looking to see if at the end of the processing, there is more than one menu name left in the top SR – and finding an extra one, asserting an error that gives the name of the extra menu. You should probably also use this error as an opportunity to abort the application launch since you could be left in a situation when you can’t shutdown the program because the “Quit” option is missing.

Something else that you might want to consider is what to do if the external file containing the menu definitions comes up missing. The most obvious solution is to, again, abort the application launch with some sort of appropriate error message. However, depending on the application it might be valuable to provide a hard-coded default menu that doesn’t depend on external files and provides a certain minimum level of functionality. In fact, I once worked on an application where this was an explicit requirement because one of the things that the program allowed the user to do was create custom menus, the structure of which was stored in external files.

Stooge Identifier – Release 2
Toolbox – Release 11

The Big Tease

So what are we going to talk about next time? Well something that I have seen coming up a lot lately on the user forum is the need to be able to work with very large datasets. Often, this issue arises when someone tries to display the results of a test that ran for several hours (or days!) only to discover that the complete dataset consists of hundreds of thousands of separate datapoints. While LabVIEW can easily deal with datasets of this magnitude, it should be obvious that you need to really bring you memory management “A” game. Next time will look into how to plot and manage VLDs (Very Large Datasets).

Until Next Time…

Mike…

A Brief Introduction to .NET in LabVIEW

Posted on 2015-07-022015-10-06 by mlportersr

From the earliest days of LabVIEW, National Instrument has recognized that it needed the ability to incorporate code that was developed in other programming environments. Originally this capability was realized through specialized functions called Code Interface nodes, or CINs. However as the underlying operating systems continued to develop, LabVIEW acquired the ability to leverage such things as DLLs, ActiveX controls and .NET assemblies. Unfortunately, while .NET solves many of the problems that earlier efforts to standardize sharable code exhibited, far too many LabVIEW developers feel intimidated by what they see as unmanageable complexity. The truth, however, is that there are many well-written .NET assemblies that are no more difficult to use than VI Server.

As an example of how to use .NET, we’ll look at an assembly that comes with all current versions of Windows. Called NotifyIcon, it is the mechanism that Windows itself uses to give you access to programs through the part of the taskbar called the System Tray. However, beyond that usage, it is also an interesting example of how to utilize .NET to implement an innovative interface for background tasks.

The Basic Points

Given that the whole point of this lesson is to learn about creating a System Tray interface for your application, a good place to start the discussion is with a basic understanding of how the bits will fit together. To begin with, it is not uncommon, though technically untrue, to hear someone say that their program was, “…running in the system tray…”. Actually, your program will continue to run in the same execution space, with or without this modification. All this .NET assembly does is provide a different way for your users to interact with the program.

But that explanation raises another question: If the .NET code allows me to create the same sort of menu-driven interface that I see other applications using, how do the users’ selections get communicated back to the application that is associated with the menu?

The answer to that question is another reason I wanted to discuss this technique. As we have talked about before, as soon as you have more than one process running, you encounter the need to communicate between process – often to tell another process that something just happened. In the LabVIEW world we often do this sort of signalling using UDEs. In the broader Windows environment, there is a similar technique that is used in much the same way. This technique is termed a callback and can seem a bit mysterious at first, so we’ll dig into it, as well.

Creating the Constructor

In the introduction to this post, I likened .NET to VI Server. My point was that while they are in many ways very different, the programming interface for each is exactly the same. You have a reference, and associated with that reference you have properties that describe the referenced object, and methods that tell the object to do something.

To get started, go to the .NET menu under the Connectivity function menu, and select Constructor Node. When you put the resulting node on a block diagram, a second dialog box will open that allows you to browse to the constructor that you want to create. The pop-up at the top of the dialog box has one entry for each .NET assembly installed on your computer – and there will be a bunch. You locate constructors in this list by name, and the name of the constructor we are interested in is System.Windows.Forms. On your computer there may be more than one assembly with this basic name installed. Pick the one with the highest version (the number in parentheses after the name).

In the Objects portion of the dialog you will now see a list of the objects contained in the assembly. Double click on the plus sign next to System.Windows.Forms and scroll down the list until you find the bullet item NotifyIcon, and select it. In the Constructors section of the dialog you will now see a list of constructors that are available for the selected object. In this case, the default selection (NotifyIcon()) is the one we want so just click the OK button. The resulting constructor node will look like this:

But you may be wondering how you are supposed to know what to select. That is actually pretty easy. You see, Microsoft offers an abundance of example code showing how to use the assemblies, and while they don’t show examples in LabVIEW, they do offer examples in 2 or 3 other languages and – this is the important point – the object, property and method names are the same regardless of language so it’s a simple matter to look at the example code and, even without knowing the language, figure out what needs to be called, and in what order. Moreover, LabVIEW property and invoke nodes will list all the properties and methods associated with each type of object. As an example of the properties associated with the NotifyIcon object, here is a standard LabVIEW property node showing four properties that we will need to set for even a minimal instance of this interface. I will explain the first three, hopefully you should be able to figure out what the fourth one does on your own.

Starting at the top is the Text property. It’s function is to provide the tray icon with a label that will appear like a tip-strip when the user’s mouse overs over the icon. To this we can simply wire a string. You’ll understand the meaning of the label in a moment.

Giving the Interface an Icon

Now that we have created our NotifyIcon interface object and given it a label, we need to give it an icon that it can display in the system tray. In our previous screenshot, we see that the NotifyIcon object also has a property called Icon. This property allows you to assign an icon to the interface we are creating. However, if you look at the node’s context help you see that its datatype is not a path name or even a name, but rather an object reference.

But don’t despair, we just created one object and we can create another. Drop down another empty .NET constructor but this time go looking for System.Drawing.Icon and once you find the listing of possible constructors, pick the one named Icon(String fileName). Here is the node we get…

…complete with a terminal to which I have wired a path that I have converted to a string. In case you missed what we just did, consider that one of the major failings of older techniques such as making direct function calls to DLLs was how to handle complex datatypes. The old way of handling it was through the use of a C or C++ struct, but to make this method work you ended up needing to know way too much about how the function worked internally. In addition, for the LabVIEW developer, it was difficult to impossible to build these structures in LabVIEW. By contrast, the .NET methodology utilizes object-oriented techniques to encapsulate complex datatypes into simple-to-manipulate objects that accept standard data inputs and hide all the messy details.

Creating a Context Menu

With a label that will provide the users a reminder of what the interface is for, and an icon to visually identify the interface, we now turn to the real heart of the interface: the menu itself. As with the icon, assigning a menu structure consists of writing a reference to a property that describes the object to be associated with that property. In this case, however, the name of the property is ContextMenu, and the object for which we need to create a constructor is System.Windows.Forms.ContextMenu and the name of the constructor is ContextMenu(MenuItem[] menuItems).

From this syntax we see that in order to initialize our new constructor we will need to create an array of menuItems. You got to admit, this makes sense: our interface needs a menu, and the menu is constructed from an array of menu items. So now we look at how to create the individual menu items that we want on the menu. Here is a complete diagram of the menu I am creating – clearly inspired by a youth spent watching way too many old movies (nyuk, nyuk, nyuk).

Sorry for the small image, but if you click on the image, you can zoom in on it. As you examine this diagram notice that while there is a single type of menuItem object, there are two different constructors used. The most common one has a single Text initialization value. The NotifyIcon uses that value as the string that will be displayed in the menu. This constructor is used to initialize menu items that do not have any children, or submenus. The other menuItem constructor is used to create a menu item that has other items under it. Consequently in addition to a Text initialization value, it also has an input that is – wait for it – an array of other menu items. I don’t know if there is a limit to how deeply a menu can be nested, but if that is a concern you need to be rethinking your interface.

In addition to the initialization values that are defined when the item is created, a menuItem object has a number of other properties that you can set as needed. For instance, they can be enabled and disabled, checked, highlighted and split into multiple columns (to name but a few). A property that I apply, but the utility which might not be readily apparent, is Name. Because it doesn’t appear anywhere in the interface, programmers are pretty much free to use is as they see fit, so I going to use it as the label to identify each selection programmatically. Which, by the way, is the next thing we need to look at.

Closing the Event Loop

If we stopped with the code at this point, we would have an interface with a perfectly functional menu system, but which would serve absolutely no useful purpose. To correct that situation we have to “close the loop” by providing a way for the LabVIEW-based code to react in a useful way to the selections that the user makes via the .NET assembly. The first part of that work we have already completed by establishing a naming convention for the menu items. This convention largely guarantees menu items will have a unique name by defining each menu item name as a colon-delimited list of the menu item names in the menu structure above it. For example, “Larry” and “Moe” are top-level menu items so their names are the same as their text values. “Shep” however is in a submenu to the menu item “The Other Stooge” so its name is “The Other Stooge:Shep”.

The other thing we need in order to handle menu selections is to define the callback operations. To simplify this part of the process, I like to create a single callback process that services all the menu selections by converting them into a single LabVIEW event that I can handle as part of the VI’s normal processing. Here is the code that creates the callback for our test application:

The way a callback works is that the callback node incorporates three terminals. The top terminal accepts an object reference. After you wire it up, the terminal changes into a pop-up menu listing all the callback events that the attached item supports. The one we are interested in is the Click event. The second terminal is a reference for the VI that LabVIEW will have executed when the event you selected is fired. However, you can’t wire just any VI reference here. For it to be callable from within the .NET environment it has to have a particular set of inputs and a particular connector pane. To help you create a VI with the proper connections, you can right-click on the terminal and select Create Callback VI from the menu. The third terminal on the callback registration node is labelled User Parameters and it provides the way to pass static application-specific data into the callback event.

There are two important points here: First, as I stated before, the User Parameters data is static. This means that whatever value is passed to the terminal when the callback is registered is from then on essentially treated as a constant. Second, whatever you wire to this terminal modifies the data inputs to the callback VI so if you are going to use this terminal to pass in data, you need to wire it up before you create the callback VI.

In terms of our specific example, I have an array of the menu items that the main VI will need to handle so I auto-index through this array creating a callback event for each one. In all cases, though, the User Parameter input is populated with a reference to a UDE that I created, so the callbacks can all use the same callback VI. This is what the callback VI looks like on the inside:

The Control Ref input (like User Parameter) is a static input so it contains the reference to the menu item that was passed to the registration node when the callback was created. This reference allows me to read the Name property of the menu item that triggered the callback, and then use that value to fire the SysTray Callback UDE. It’s important to remember when creating a callback VI to not include too much functionality. If fact, this is about as much code as I would ever put in one. The problem is that this code is nearly impossible to debug because it does not actually execute in the LabVIEW environment. The best solution is to get the selection into the LabVIEW environment as quickly as possible and deal with any complexity there. Finally, here is how I handle the UDE in the main VI:

Here you can see another reason why I created the menu item names as I did. Separating the different levels in the menu structure by colons allows to code to easily parse the selection, and simultaneously organizes the logic.

Future Enhancements

With the explanations done, we can now try running the VI – which disappears as soon as you start it. However, if you look in the system tray, you’ll see its icon. As you make selections from its menu you will see factoids appear about the various Stooges. But this program is just the barest of implementations and there is still a lot you can do. For example, you can open a notification balloon to notify the user of something important, or manipulate the menu properties to show checkmarks on selected items or disable selections to which you want block access.

The most important changes you should make, however, are architectural. For demonstration purposes the implementation I have presented here is rather bare-bones. While the resulting code is good at helping you visualize the relationships between the various objects, it’s not the kind of code you would want to ship to a customer. Rather, you want code that simplifies operation, improves reusability and promotes maintainability.

Stooge Identifier — Release 1

The Big Tease

So you have the basics of a neat interface, and a basic technique for exploring .NET functionality in general. But what is in store for next time? Well I’m not going to leave you hanging. Specifically, we are going to take a hard look at menu building to see how to best modularize that functionality. Although this might seem a simple task, it’s not as straight-forward as it first seems. As with many things in life, there are solutions that sound good – and there are those that are good.

Until Next Time…

Mike…

Helping a Window to Remember

Posted on 2015-06-222015-07-13 by mlportersr

One of the most common, most basic, and most mindless, things we do with computers every day is open windows. Launching a program or opening a document is often synonymous (on a practical level at least) with opening a window. As common as this action is, we rarely give any thought to what is going on behind the scenes when we open a window – hence the wisecrack about it being a mindless operation.

However, if we want to make the most of our design efforts, be need to replace this “mindlessness” with “mindfulness” by really thinking about the things that make windows easy and comfortable to use.

Defining a Well Behaved Window

As we begin looking at the behavior of windows, I want to emphasise that I am not talking about user interface design. User interface design deals with the details of what a window does functionally. Rather what I’m talking about is an examination to the behaviors a window should exhibit, regardless of what happens to be on the screen.

If we think about the window as a kind of frame that supports the interface’s core functionality, we see that one of the big things a window can do is remember things. Over the years, people have developed, and posted on the LabVIEW forums, a variety of toolboxes for storing generic window information like screen customizations, positions and settings. One of these toolsets combined with an event-driven structure can make it easy to significantly pump up the convenience factor of just about any application.

To see how these sorts of tools work, we’re going to enhance our undockable windows application with a simple addition that automatically saves a windows last position and restores that position the next time the window undocks. Although the basic logic is simple, it provides us with the opportunity to discuss many of the major issues that impact this sort of functionality.

The Data, and Where to Keep it

When considering the data that this sort of functionality requires and uses, the operative word is: “convenience”. By that I mean that this data may make using the screen more convenient, but nobody is going to be crying if it gets lost. In fact, a valuable behavior is the ability basically “reset” all the stored data back to its default value by simply deleting the data from the file that is storing it and letting the application rebuild it as needed.

Likewise the data should be of low “intelligence” value. In other words, we don’t want to include things in this data that could constitute a security risk. However, having said that, we also want to make sure that a well-meaning user doesn’t mess up the program’s operation by manually editing the data. My approach to blocking such edits has three major points:

Be careful about what you name things: You want to give identifiers that are, of course, meaningful. However, you don’t want to use names that will call attention to themselves in a way that says, “Hi, I’m a setting you might want to play with…”
Use a non-obvious data structure: For example, in our example we don’t save a window’s position as a simple list of four values. The problem is that a user looking at these values might decide to try to edit them manually – a simple act that could have some significant side effects. To see why consider that the way you move a window around the screen is by changing the VI’s WinBounds property. However, this property defines a windows position by essentially specifying the location of the window’s upper-left and lower-right corners. Consequently, while it does set the window’s location, it is also specifying the window’s size.
Provide a simple way to validate the data: Given that there is no way to know ahead of time what sort of data you might be wanting to store, validation might seem like a huge task, but it’s really not. Remember, when validating the data you don’t have to prove the data values are valid, just that they haven’t changed since the program wrote them.

As we get into how the position saving is implemented, you’ll see how I put these ideas into action, but first we need to look at how we are going to implement the capability from a high-level view.

Our Basic Approach

When adding in new or enhanced functionality, you want to do so in a way that requires as few modifications to (and has as little impact on) the existing structure, as possible. This ability to easily incorporate new functionality is a large part of the meaning of the term, “maintainable”. It is also why it is always good to think about your overall application in terms of functional blocks – or specific VIs that do specific things, and handle specific situations.

With that point in mind we know we have two basic operations we need to add: one sets the position when we open a VI and one writes a new position when we close it. Of these two operations, the simplest is the one that reads the last saved location and moves the window to that location. It’s simple because there is only one place in the code where that opening takes place, and that is right here:

This is the VI Float the VI.vi and if you compare it to the version that I presented last week you will note that it has one extra subVI that uses a reference to the VI being opened to look up and set the window position. we’ll look at exactly how it does that in just a moment. The operation that saves a VI’s last open position can also be boiled down to a single subVI, but due to the nature of our application, it will have to be installed in two locations.

Here’s the first of those locations. It occurs in the subVI Unfloat the VI.vi and it handles the case where the user closes any of the floating windows. Again you’ll notice one added subVI. Using the VI reference supplied to it, the subVI determines the target VI’s new window position and saves it. The other place where this VI occurs is in the event that stops the application.

Here the event logic checks to see if a window is docked, and if not calls the same subVI to save the window position of the VI associated with the reference.

Digging for Details

Now that we see where the modification fits into the application, let’s look at how the subVIs work – starting with the routine that saves the window position.

As you can see, I am using the configuration file VIs to store the data in a text file using the INI file structure. However, it’s not the application’s INI file but rather one that I am creating in the user’s “My Documents” directory. This selection has at least a couple of implications. First it means that every user that logs into the computer will have their own set of customizations. Second, if the user wants to reset all their customizations back to default, all they have to do is delete or rename that one file.

Next, notice that this VI was designed to be usable in two different ways. If the VI reference input carries a valid reference the code uses that reference to get the data it wants to save. Alternatively, if the VI reference provided is not valid, the true case of the structure (not shown) open a reference to the VI calling this subVI and save the data for it.

Finally, let’s look at the subVI that the code uses to convert the window bounds data into the string that will be saved to the custom INI file.

In keeping with the concepts I cited earlier, I obfuscate the datatype by flattening the structure to a string, convert the string into an array of U8s, and finally format the array as a string of 2-character hexadecimal values. However, before making that last conversion I provide a mechanism for ensuring data validity: I append a 16 bit CRC. The result is a string that will allow you to detect if it has been manipulated outside the program.

Turning now to the VI that retrieves the data, we see logic that reverses what was done in the position save routine. However, there is one added twist: if this VI is being called for the first time, the position of the target VI might not be in the INI file. Consequently the code needs to be able to recognize that situation and just let the windows open in its default position.

The subVI (below) that converts the string from the INI file back into the LabVIEW data structure for defining a windows bounds, generates a structure containing all 0s when it is passed an empty string – which is what you get when you try to read a string value from an INI file that doesn’t exist.

Also notice how this VI verifies the CRC. It’s commonly believed that in order to verify a CRC you have to split the CRC from the message, calculate a new CRC on the message and compare the result to the CRC received as part of the message. However, such is not the case. Due to the way the CRC calculation works, if you simply perform a CRC on the entire message including the original CRC the results will always be 0 for a valid message and CRC. Hence, the logic only goes to the trouble of splitting the message from the CRC after it has determined that the message is valid. In the case where the second CRC calculation is not zero (indicating a corrupted message) the logic outputs the same invalid data structure that you get from a null input string.

Given these facts, the VI for reading the target VI’s last position only has to look for that known-invalid data structure and if it finds it, bypasses the logic that set the window’s bounds.

Undockable Windows – Release 2
Toolbox – Release 10

The Big Tease

So there we have it, a basic framework that you can use to implement a variety of “window memory” functions. But what about next time? I have talked a lot about processes that run in the background. What happens though, if you want to be able to provide a minimal interface that isn’t always there but can be easily called up when needed. Next time I’m going how you can utilize the Windows system tray to house just such an interface. At the same time we’ll look at one of the more interesting backwaters of LabVIEW development – .net callbacks.

Until Next Time…

Mike…

Creating a Reconfigurable Interface Using Undockable Windows

Posted on 2015-06-152015-06-15 by mlportersr

Something I have always enjoyed doing is creating programs or interfaces that do things you don’t expect LabVIEW to be able to do. Consequently, I thought I would take a couple posts and consider some useful and perhaps surprising interfaces.

The first one I want to look at is actually an idea I got from another LabVIEW Champion – Ben Rayner by name. Some time ago, he posted on the user forum a small proof-of-concept VI for an interface that is so cool I though I’d polish it up, flesh it out and see what came out of it.

What it does

At first glance, the interface appears similar to the test bed application that we have built on this blog. There is a common display area and by making selections from a pop up menu, you can display screens showing different data.

The difference is that if you click a button, the GUI will “undock” the current screen and turn it into an independent floating window that is no longer accessible from the popup menu.

The software allows an operator to undock as many screen as they want at one time. The only limitations are screen space and logic that mandates leaving at least one screen docked. Likewise, if you close one of the floating windows, it will again take up its original position in the selection menu. As is our usual policy, a link to the code is a little further along in this post.

Why This Interface?

I fully plan to get into how the code works, but first we need to consider why you would want to use this interface in the first place. What is the use case that this interface addresses?

Hopefully it should be obvious to anyone who considers the matter for more than a moment or two that user interfaces are always compromises. If you do your due diligence when designing an interface you try to put together in one screen or window the various bits of information that are logically related, or at least will be used together. However, requirements can change, or a particular type of user might need to be able to correlate pieces of data that are on different screens.

You could create a separate screen for just that user, but that solution requires some additional development effort. However, if you create windows that can be undocked to be moved around the screen as desired, users that want to see their data in a particular way may be able to do so without waiting for you to create a custom screen.

So with all that in mind, let’s think about how we might accomplish this sort of GUI.

How it Works in Theory…

The first thing to you need to realize as we begin looking at how to create this interface, is that if you have been following this blog you already know everything required to make this happen. The only thing missing is an understanding of how to make all the pieces fit together in a slightly different way.

For example, we have talked many times about how to create a basic multiscreen interface using subpanels. Likewise, we know how to make windows float on top of one another by defining their behavior as floating. Now if you think about it, these are the two states that our screens can be in: as a subpanel (when docked) and as a floating window (when undocked).

So our problem is really as simple as deciding how to manage the transition from one state to the other. If you are following the recommendations I have been making, the screens are already written as separate processes that don’t know (or care) how they are being displayed – or even if they are being displayed.

So how do we want this to behave? Well for the undocking there are a variety of options. LabVIEW supports drag-and-drop so we might be able to do something with that. Alternatively, we could create a pull down menu with an “Undock Current Window” on it, or even a custom shortcut menu when the user right-clicks on the on the interface’s front panel. But those are really just different ways of triggering the same logic, so for this demo I’m just going to create a button below the subpanel that is labeled something obscure like, “Click to Undock the Current Display”.

For docking one of the floating windows back in the interface, our options are more limited. But when you have a really good technique available, you only need one. How about this: when you click the window’s close button it gets docked back into the interface. After all, it’s what most users would try anyway.

…and in Practice

Now that we basically understand where we are going, lets start looking at some code.

Undockable Windows – Release 1
Toolbox – Release 9

The block diagram of the top-level VI is pretty much what a regular reader of this blog would expect. It starts by running a subVI that loads into memory, and starts executing, all the VIs that are going to be available through the subpanel. References to these VIs are stored in a DVR so they can be used later to populate the subpanel itself. This subVI also outputs an array of strings that are the names of the VIs it loaded. The code uses this array to initialize the strings in a ring control, and then programmatically fires the value change event associated with the ring.

With this work done, the VI next registers to receive a UDE that we will discuss in a moment, and enters the program’s main event loop. This loop includes a value change event for the ring control that changes the VI that is visible in the subpanel, and two additional events that manage the undocking and docking processes.

Getting Undocked

The first of these new events handles the undocking of windows and, understandably, is a value change event on the undocking button. The event logic goes about its work in two steps that are contained in separate subVIs. The first one is called Float the VI.vi.

Its job is to remove the VI that is being undocked from the subpanel using the Remove VI subpanel control method, and then (after looking up its VI reference in the DVR) open its front panel using the FP:Open VI method. The other half of the operation is performed in Process Undock.vi.

This VI’s purpose is to update the user interface to the VI’s change of state – which in this case means simply disabling its selection in the ring control, and updating the DVR. This subVI is also responsible for getting a reference to the next available docked window and inserting it into the subpanel. As a by-product of this operation, the subVI also generates a flag that indicates when there is only one docked window remaining. The event handler uses this flag to disable the undock button when all but one of the displays have been undocked.

Docking a Window

Like the previous one, this event handles its duties as a two-step process. But unlike the previous one, it is driven by a UDE from the VI that is being docked back into the main display. As stated before, this action should be triggered when the user closes the VI’s front panel. The plugins accomplish this task by intercepting the Panel Close? event and instead of closing its front panel, simply fires a UDE (Redock Screen) that tells the GUI to reincorporate it back into the subpanel. The first subVI in the event handler is Unfloat the VI.vi.

After looking up the VI’s reference in the DVR, it closes the VI’s front panel and inserts it back into the subpanel. Note that it is not necessary to remove the VI that is already there. The other subVI that the event handler calls is Process Redock.vi.

Basically reversing the operation that were performed when the window was undocked, this VI removes the window’s name in the ring control from the list of disabled items and updates the status stored in the DVR.

The Big Tease

So there’s a basic implementation of a pretty neat capability, but what now? What sort of enhancements might we reasonably want to make? I have an idea. You notice that when the window’s undock, they always open in the middle of the main screen. This is certainly a reasonable approach, but an enhancement that would be sure to make your users happy is to provide some sort of mechanism whereby each window would remember its last position. With that capability in place a windows would reopen in the same place as it was when it was last closed.

So when we next get together, let’s do that. It will be a good thing to know how to do in general.

Until Next Time…

Mike…

Building a Web Backend in LabVIEW

Posted on 2015-06-082016-04-01 by mlportersr

When building a web-based application, one of the central goals is often to maximize return on investment (ROI) by creating an interface that is truly cross-platform. One of the consequences of this approach is that if you are using a LabVIEW-based program to do so-called “backend” tasks like data acquisition and hardware control, the interface to that program will likely be hidden from the user. You see the thing is, people like consistent user interfaces and while LabVIEW front panels can do a pretty good job as a conventional application, put that same front panel in a web page and it sticks out like the proverbial sore thumb.

Plus simply embedding a LabVIEW front panel in a web pages breaks one of the basic tenets of good web design by blurring the distinction between content and appearance. Moreover, it makes you — the LabVIEW developer — at least partially responsible for the appearance of a website. Trust me, that is a situation nobody likes. You are angered by what you see as a seemingly endless stream of requests for what to you are “pointless” interface tweaks, “So big deal! The control is 3 pixels too far to the left, what difference does that make?” And for their part, the full-time web developers are rankled by what they experience as a lack of control over an application for which they are ultimately responsible.

If you think about it, though, this situation is just another example of why basic computer science concepts like “low coupling” are, well, basic computer science concepts. Distinguishing between data collection and data presentation is just another form of modularity, and we all agree that modularity is A Very Good Thing, right?

It’s All About the (Data)Base

So if the concept that is really in play here is modularity, then our immediate concern needs to be the structure of the interface between that part of the system which is our backend responsibility, and that which is the “web guy’s” customer facing web-based GUI.

Recalling what we have learned about the overall structure of a web application, we know that the basic way most websites work is that PHP (or some other server-side programming tool) dynamically builds the web pages that people see based on data that they extract from a database. Consequently, as LabVIEW developer trying to make data available to the web, the basic question then is really, “How do I get my data into their database?”

The first option is to approach communications with the website’s database as we would with any other database. Although we have already talked at length about how to access databases in general, there is a problem. While most website admins are wonderful people who will share with you many, many things — direct access to their database is usually not one of those things.

The problem is security. Opening access to your LabVIEW program would create a security hole through which someone not as kind and wholesome as you, could wiggle. The solution is to let the web server mediate the data transfer — and provide the security. To explore this technique, we will consider two simple examples: one that inserts data and one that performs a query. In both cases, I will present a LabVIEW VI, and some simple PHP code that it will be accessing.

However, remember that the point of this lesson is the LabVIEW side of the operations. The PHP I present simply exists so you can get a general idea of how the overall process works. I make no claims that the PHP code bears any similarity to code running on your server, or even that it’s very good. In fact, the server you access might not even use PHP, substituting instead some other server-side tool such as Perl. But regardless of what happens on the other end of the wire, the LabVIEW code doesn’t need to change.

As a quick aside, if you don’t have a friendly HTTP server handy and you want to try some of this stuff out, there is a good option available. An organization called Apache Friends produces an all-in-one package that will install a serviceable server on just about any Windows, Linux or Apple computer you may have sitting around not being used. Note that you do not want to actually put a server that you create in this way on the internet. This package is intended for training and experimentation so they give short shrift to things like security.

The following two examples will be working with a simple database table with the following definition:

CREATE TABLE temp_reading (
   sample_dttm  TIMESTAMP PRIMARY KEY,
   temperature  FLOAT
   )
;

Note that although the table has two columns we will only be interacting directly with the temperature column. The sample_dttm column is configured in the DBMS such that if you don’t provide a value, the server will automatically generate one for the current time. I wanted to highlight this feature because the handling of date/time values is one of the most un-standardized parts of the SQL standard. So it will simplify your life considerably if you can get the DBMS to handle the inserting or updating of time data for you.

Writing to the Database

The first (and often the most common) thing a backend application has to do is insert data into the database, but to do that you need to be able to send data to the server along with the URL you are wanting to access. Now HTTP’s GET method which we played with a bit last week can pass parameters, but it has a couple big limitations: First, is the matter of security.

The way the GET method passes data is by appending it to the URL it is getting. For example, say you have a web page that is expecting two numeric parameters called “x” and “y”. The URL sent to the server would be in the form of:

http://www.mysite.html?x=3.14;y=1953

Consequently the data is visible to even the most casual observer. Ironically though, the biggest issue is not the values that are being sent. The biggest security hole is that it exposes the names of the variables that make the page work. If a person with nefarious intent know the names of the variables, they can potentially discover a lot about how the website works by simply passing different values and seeing how the page responds.

A second problem with the GET method’s way of passing data is that it limits the amount of data you can sent. According to the underlying internet standards, URLs are limited to a maximum of 2048 characters.

To deal with both of these issues, the HTTP protocol defines a method called POST that works much like GET, but without exposing the internal operation or limiting the size of the data it can pass. Here is an example of a simple LabVIEW VI for writing a temperature reading to the database.

This code is pretty similar to the other HTTP client functions that we have looked at before, but there are a few differences. To begin with, it is using the POST we just talked about. Next, it is formatting and passing a parameter to the method. Likewise, after the HTTP call it verifies that the text returned to it is the success message. But where do those values (the parameter name and the success message) come from? As it so happens I didn’t just pull them out of thin air.

Check out the URL the code is calling and you’ll note that it is not a web page, but PHP file. In fact, what we are doing is asking the server to run a short PHP program for us. And here is the program:

<?php
	if($dbc = mysqli_connect("localhost","myUserID","myPassword","myDatabase")) {
    	// Check connection
		if (mysqli_connect_errno()) {
			// There was a connection error, so bail with the error data
			echo "Failed to connect to MySQL: " . mysqli_connect_error();
		} else {
			// Create the command string...
			$insert = "INSERT INTO temp_reading (temperature) VALUES (" . $_POST["tempData"] . ")" ;
			
			// Execute the insert
			if(!mysqli_query($dbc,$insert)){
				// The insert generated an error so bail with the error data
				die('SQL Error: ' . mysqli_error($dbc));
			}
			
			// Close the connection and return the success message
			mysqli_close($dbc);
			echo "1 Record Added";
		}
	}
?>

Now the first thing that you need to know about PHP code is that whenever you see the keyword echo being used, some text is being sent back to the client that is calling the PHP. The first thing this little program does it try to connect to the database. If the connection fails the code echoes back to the clients a message indicating that fact, and telling it why the connection failed.

If the connection is successful, the code next assembles a generic SQL statement to insert a new temperature into the database. Note in particular the phrase $_POST["tempData"]. This statement tells the PHP engine to look through the data that the HTTP method provided and insert into the string it’s building the value associated with the parameter named tempData. This is where we get the name of the parameter we have to use to communicate the new temperature value. Finally, the code executes the insert that it assembled and echoes back to the client either an error, or a success message.

Although this code would work “as is” in many situations, it does have one very large problem. There is an extra communications layer in the process that can hide or create errors. Consider that a good response tells you that the process definitely worked. Likewise receiving one of the PHP generated errors tells you that the process definitely did not work. A network error, however, is ambiguous. You don’t know if the process failed because the transmission to the server was interrupted; or if the process worked but you don’t know about it because the response from the server to you got trashed.

PHP Mediated Database Reads

The other thing that backend applications have to do is fetch information (often configuration data) from the database. To demonstrate the process go back to the GET method:

As with the first example, the target URL is for a small PHP program, but this time the goal is to read a time ordered list of all the saved temperatures and return them to the client. However, the data needs to be returned in such a way that it will be readable from any client — not just one written in LabVIEW. In the web development world the first and most common standard, is to return datasets like this in an XML string. This standard is very complete and rigorous. Unfortunately, it tends to produce strings that are very large compared to the data payload they are carrying.

Due to the size of XML output, and the complexity of parsing it, a new standard is gaining traction. Called JSON, for JavaScript Object Notation, this standard is really just the way that the JavaScript programming language defines complex data structures. It is also easy for human beings to read and understand, and unlike XML, features a terseness that makes it very easy to generate and parse.

Although, there are standard libraries for generating JSON, I have chosen in this PHP program to generate the string myself:

<?php
    if($dbc = mysqli_connect("localhost","myUserID","myPassword","myDatabase")) {

    	// Check connection
		if (mysqli_connect_errno()) {
			echo "Failed to connect to MySQL: " . mysqli_connect_error();
		} else {
			// Clear the output string and build the query
			$json_string = "";
			$query = "SELECT temperature FROM temp_reading ORDER BY sample_dttm ASC";
			
			// Execute the query and store the resulting data set in a variable (called $result)
			$result = mysqli_query($dbc,$query);
			
			// Parse the rows in $result and insert the values into a JSON string
			while($row = mysqli_fetch_array($result)) {
				$json_string = $json_string . $row['temperature'] . ",";
			}
			
			// Replace the last comma with the closing square bracket and send the
			// to the waiting client
			$json_string = "[" . substr_replace($json_string, "]", -1);
			echo $json_string;
		}
	}
	mysqli_close($dbc);
?>

As in the first PHP program, the code starts by opening a database connection and defining a small chunk of SQL — this time a query. The resulting recordset is inserted into a data structure that the code can access on a row by row basis. A while loop processes this data structure, appending the data into a comma-delimited list enclosed in square brackets, which the JSON notation for an array. This string is echoed to the client where a built-in LabVIEW function turns it into a 1D array of floats.

This technique works equally well for most complex datatypes so you should keep it in mind even if you are not working on web project. I once had an application where the client wanted to write the GUI in (God help them) C# but do all the DAQ and control in LabVIEW-generated DLLs. In this situation the C# developer had a variety of structs containing configuration data that he wanted to send me. He, likewise, wanted to receive data from me in another struct. The easy solution was to use JSON. He had a standard library that would turn his structs into JSON strings, and LabVIEW would turn those strings directly into LabVIEW clusters. Then on the return trip my LabVIEW data structure (a cluster containing a couple of arrays) could be turned, via JSON, directly into a C# struct. Very, very cool.

The Big Tease

So what is in store for next time? Well something I have been noticing is that people seem to be looking for new ways to let users interact with their applications. With that in mind I thought I might take some time to look at some nonstandard user interfaces.

For my first foray into this area I was thinking about a user interface where there are several screens showing different data, but by clicking a button you can pop the current screen out into a floating window. Of course, when you close the floating window, it will go back into being a screen the you can display in the main GUI. Should be fun.

Until Next Time…

Mike…

Understanding Web I/O

Posted on 2015-06-012015-06-01 by mlportersr

A question that seems to be popping up with increased urgency is how to incorporate LabVIEW into an environment that is largely web-based. As you would expect, in a technology that is developing as fast as the web, it can be hard to keep track of where things are right now. In fact at least one major tech publisher (O’Reilly Media) is diving full throttle into electronic publishing because with the lead time inherent in conventional publishing, web development books tend to be out of date before they can even be printed.

This is important to us because we don’t have the time or money to waste on technological dead ends. For example, I wouldn’t recommend you buy stock in companies manufacturing compact fluorescent lightbulbs. I mean who really wants to have to call in a hazmat team to clean up the mercury contamination when one breaks? The “curly light bulb” may be all the rage right now, but it is a technological dead end — to be replaced by LEDs. In the same way, I believe that many of the technologies that are in use today to access online resources are also technological dead ends.

If your company is online right now or wants to offer some service online, the standard way of doing that is with an “app”. Or rather, I should say, several apps. You will need an app for Android, one iOS, one for Windows Mobile, and perhaps others as well. But to get them you will need separate at least three separate development team because none of those operating systems are compatible with one another — or for that matter even very similar. This approach is simply not sustainable, and is another technological dead end. The solution is to build a system based on the platform independent standards that drive the web. So we need to learn where LabVIEW fits into this picture.

The Web Landscape

The first thing to understand is that the web as a collection of static HTML pages is gone. The web of today is a dynamic environment where the web pages you view (including this one) are built on the fly and in some cases may only exist for the few milliseconds it takes for the server to transmit it to you. While the underlying technology that accomplishes this magic is fascinating, the good news is that we don’t really need to know very much about it to accomplish our goals. Instead it is adequate for our purpose to know who the “main players” are from our (rather limited) point of view and how they interact.

One of the key developments in the past few years has been the understanding that each piece of the overall web environment needs to have a clearly defined role and that the various components don’t step outside their designated roles. To see what I mean, let’s consider a few of the basic pieces that you will encounter working with LabVIEW:

HTML — The Content

Tn the beginning the need was simple, Researchers at CERN needed a way to use this new “internet” to share research results with colleagues. But they wanted more than just a way to ship text file around. They wanted documents that could include pictures and graph, and could be linked to one another. In short, they wanted (to borrow a buzzword from an even earlier time) hyper-text.

The approach that they used to reach for this goal was simple: take bare text documents and embed in them tags (which were also text) that could tell a custom designed program called a “browser” how to treat the text contained in the tags. The thing to remember is that this idea was hardly a new one — so-called “markup languages” had been used for decades in the mainframe environment to implement word processors and report generators.

Even today, HTML (hyper text markup language) documents are just text files with embedded tags. This point is an important one to remember because it means that while browsers have one way of interpreting tags, other programs — even LabVIEW, or programs you write in LabVIEW — can read the same files and parse them to extract the data, the same as they would any other text file.

CSS — How Things Should Look

As it was originally created, HTML mushed together a lot of different things. Some tags defined the structure of the document, some were links and some simply defined what the text should look like. But this approach soon began to present some limitations. For example, if the logic defining what a particular bit of text looks like is embedded in the document, everybody has to see the same thing no matter how big their browser window might be or what the physical characteristics of their screen might be.

The solution to this problem with as easy. All you had to do is define a separate syntax for defining the content’s appearance, and then take all that logic and move it to a separate file with a .css file extension called a style-sheet. Now two people could view the same HTML document and would see it formatted to best fit their system because their browsers were using style sheets that were optimized for their system hardware.

As a practical example of this flexibility in action, consider the very webpage you are reading right now. Regardless of whether you are looking at it on a monitor on your desk, or a smartphone in the palm of your hand, everybody gets to see a nicely styled layout that fits their device — and I only had to generate the content once.

PHP — Server-Side Programmability

But even with that styling flexibility, there is still a very large problem. If all you have at your disposal is HTML, then each web page becomes a separate program that somebody has to write. Imagine if you will how much effort it would take to maintain a site such as Amazon if every on of its thousands of products needed its own handcrafted and programmed web page. The solution is to automate the creation of web pages using a programming language that can automatically assemble pages from predefined common components as they are needed.

One of the most popular language for this kind of work is PHP — though there are many others. It works by inserting itself into the HTML process in one of two ways. First, there are special HTML tags that cause the server to momentarily stop sending the data from the HTML file and run a PHP function. The purpose of this function would be to programmatically build and echo to the waiting browser some portion of the page, like a standardized menu structure or footer.

The second way PHP may appear is that some web servers (like Apache, for instance) will automatically call a predefined file called index.php if you access the site without specifying a specific page. Consequently, while it might look to you like you are accessing a webpage, you are really running a PHP program that builds the initial screen that you see on the fly. In addition, if you watch your URL bar while browsing you will occasionally notice that you are accessing pages with the same php file extension.

Database — Data Storage

If you have PHP building web pages dynamically, a logical question to ask is where it gets the data that it uses to build the dynamic pages that we see in our browsers? Not surprisingly, PHP uses a database, and it does so for the same reason that we use databases when building a test application: because it’s better to reconfigure than it is to recode. In fact, one of the major functions of PHP programs is to query a local database and pass the result to the browser for display. You will also find PHP programs storing to a database the information that you provide when filling out forms to do things like ordering pizza online or registering for a blog.

The type of databases that are used can vary widely though the most common is MySQL. The important thing to remember is that the specific database that is used to fill this particular niche in the overall web ecosystem is largely irrelevant.

Getting back to LabVIEW

Now that we understand how some of the major pieces fit together, we can begin to explore how our LabVIEW programs fit into this picture. Since the point of all this structure is to manage test files being sent to browsers, it shouldn’t be too much of a surprise to learn that one of the major ways that LabVIEW programs fit in is by pretending to be browsers. In support of that “deception” LabVIEW comes with a set of HTTP client VIs that support all the major HTTP methods as well as routines for dealing with access control, and setting session parameters that you are wanting to use. As an example of how the interface works, I have created a VI which illustrates about the simplest transaction possible:

The code starts by opening an HTTP session, It then uses the VI that implements the HTTP GET method to fetch a URL and display the contents in a string indicator. Finally, it closes the connection. As you will see, the tricky part of this operation isn’t so much how you do it, but rather what do you do with the data you get back. For example, if you run this VI with the default value in the url control you get something that looks like this:

Now you have to admit that the data in the Response indicator would seem to fit the definition that most people have for “unintelligible garbage”. But wait a minute, look at the first 6 bytes of the file. Many proprietary file formats will put a short identifier in the first few bytes of the file. For example, if you look at a Zip archive, the first three characters are “ZIP”. Applying this understanding to the data we just read from the internet, “GIF89a” is how files are identified that meet the 1989 Graphics Interchange Format standard. In other words, it’s a gif format graphic. To see the image all you have to do it provide a way to save the data to a file by adding a bit of logic to the block diagram.

You see the url is the link that the LabVIEW forum uses to fetch one of the predefined avatars, in this case one called “fatbot”. But there are more serious things that you can do also. For example, by parsing the HTML that you get back from reading an actual webpage, you can obtain data that you might not otherwise be able to access. However, if you plan to do such data harvesting, be sure that you have the permission to do so. Otherwise the practice could lead to serious ethical/karmic/legal issues.

So why would someone need to do this sort of data collection as part of a professional test system? One possible scenario would be that the application you are needing to test is inaccessible, and all you can see is the data that it publishes online. Or perhaps, the web interface is the thing that you are wanting to test.

The big risk, of course, is that if you don’t have control over the page that you’re accessing, you can have your code suddenly stop working because the owner changed their webpage. Ideally, you should be able to negotiate changes with the site owner, but if not you need to remember that your number 1 mitigation strategy is good design. While you might not be able to keep changes from breaking your code, you can at least limit the damage to perhaps 1 or 2 VIs.

Assuming that you can negotiate with the folks that are working on the webpage, one of the things that you can request is that they surround the data that you need with special tags like so: (Imagine this is in the middle of a block of text on the web page)

… the test was performed by <lvData id="testOp">John Doe</lvData> using test fixture <lvData id="testLoc">5</lvData> …

This snippet defines a unique HTML tag lvData that allows the LabVIEW code that is parsing the content to quickly find the values it want, while the id="…" clauses allow the LabVIEW application identify the label to associate with each value. Moreover this creation of an ad hoc tag won’t impact the viewing of the page with a standard browser because fundamental to the operation of all browsers is the simple rule: If you see a tag you don’t recognize, ignore it. Here is code that you could use to locate and parse these tags.

This idea could also be expanded by embedding the custom tags in HTML comments. Like this:

<!-- <lvData id="testOp">John Doe</lvData> <lvData id="testLoc">5</lvData> -->

In this implementation, the comment tag would hide the custom tags from view when the page is being displayed in a browser, but they would still be available to a LabVIEW application parsing the page as text. Using this technique you could hide data inside any page — perhaps even this one.

The Big Tease

This should be enough for now. Next time we’ll look at how to use LabVIEW as a data source for a web-based application. We’ll check out the HTTP POST method, and even look at a little PHP code and some JSON data.

Until Next Time…

Mike…

Using VI Server to Interact with Executables

Posted on 2015-05-252015-05-25 by mlportersr

We all want to be able to reuse code and a good way of accomplishing that goal is by repurposing executables that you wrote for other projects. The problem is how you control them. Last week we started addressing this challenge by looking at some of the general tools that are at our disposal for manipulating executables — regardless of where you got them. This time out we will complete the discussion by looking at some of the things you can do that are specific to LabVIEW-created executables.

First, we need an executable

As the title says, if we are going to talk about making VI Server calls to an executable, the first thing we need is an executable — and an executable, we have. Although the functionality it implements is, to be honest, rather sparse, it is sufficient to demonstrate what we need. Here’s what its front panel looks like:

Starting in the top left, it sports an indicator where you can see the command line that was used to launch it. Immediately below that string is an indicator showing the current time, a button for stopping the program manually, and a pair of LEDs that indicate when two different events are triggered: Application Instance Close? and Panel Close?.

To the right is a path indicator that displays one of two different paths depending on the state of the checkbox that is next to it. Below the path indicator are two numerics. One is the PID of the instance that is running. The other is the TCP/IP port number that was assigned to the executable when it was launched. Note that if you don’t provide a port number in the command line parameters, the executable will terminate almost immediately — though you may see it briefly appear in the Windows Task Manager.

Handling the front panel

Most of the code that makes this interface work is pretty straight forward, so I won’t take the time to describe it. The one exception is this bit in the initialization logic:

The reason that I’m pulling it aside for special attention is that it illustrates (at least part of) the solution for a problem that you will encounter the first time you create a VI that is designed to run entirely in the background. The heart of this problem is mismatched expectation: When LabVIEW runs an executable it expects to open the window associates with the top-level VI. You, on the other hand, wanting the executable is run unseen in the background, expect the window to stay closed. Consequently, what happens is that LabVIEW opens the window and starts the VI running in that order and your code immediately hides the front panel. What the user sees is a windows that open and then immediately closes without explanation, and if there is one things that worries users more than things happening too slowly, its things happening too fast — like windows flashing open and then closing.

The solution lies in a VI property called Transparency. The setting to control it can be found in the custom appearance dialog.

When the box is checked and the percentage is set to 100, the window will be open but totally transparent. Hence, when the runtime engine launches the application, the window will still open but it will be invisible. A moment later, the code above will hide the window and set the transparency to 0 so that when we do decide to open it, we will be able to see it.

VI Server Operations

Last time, I presented this block of settings from the application’s INI file. Before we continue, we need to take a moment all consider what these settings mean — at least to the extent that anyone knows what they mean…

server.tcp.enabled=True
; server.tcp.port=3363
; server.tcp.serviceName=""
server.tcp.acl="290000000A000000010000001D00000003000000010000002A10000000030000000000010000000000"
server.vi.access="+*"
server.vi.callsEnabled=True
server.app.propertiesEnabled=True
server.vi.propertiesEnabled=True
server.control.propertiesEnabled=True

The first four parameters in this list control overall access to the application via TCP/IP. Consequently, they are the four that you are most likely need to muck with:

server.tcp.enabled=True: This setting enables the TCP/IP interface that VI Server uses. If this setting is False, nothing is happening.
; server.tcp.port=3363: This setting specifies the port that the associated TCP/IP listener will be monitoring for a connection. Note that I have this line commented out because we will be assigning this value via command line parameter.
; server.tcp.serviceName="": Also commented out, this optional parameter allows you to define a name that you can then use to reference the application, instead of a Port number.
server.tcp.acl=???: This setting defines the TCP/IP access control list (ACL) — or who is allowed to connect to the application. Already I can hear you wondering, what is the deal with that long string? Well, if you ever find out be sure and let me know. The bottom line is that the original interface included an ACL that simply listed the IP addresses that were and were not allowed to access the application. For reasons unknown, NI decided to change this common sense approach to something more enigmatic. So how do you generate this string? Glad you asked. According to LabVIEW’s documentation, you need to set up your development environment to have the same access list as you want your application to have, and then copy and paste the resulting string from LabVIEW’s INI file to your application’s INI file, seriously…

The remaining parameters specify in one way or another the specific resources that the remote program can access in the target application.

server.vi.access="+*": This parameter contains a list of VIs that are accessible through the VI Server interface. The default value shown allows access to all VIs.
server.vi.callsEnabled=True: This parameter specifies whether the remote program is allowed to run VIs contained in the executable. We leave this True so I can demonstrate that ability.
server.app.propertiesEnabled=True: This parameter gives the remote program access to the executable’s application-class properties — like the names of all the VIs currently in memory.
server.vi.propertiesEnabled=True: This parameter gives the remote program access to the VI-class properties of individual VIs. This category include things like a reference to the VI’s front panel.
server.control.propertiesEnabled=True: This parameter gives the remote program access to the properties associated with individual controls on VI front panels. For example, you need to have this parameter enabled to do things like programatically set the value of a control. This value is True as I will be demonstrating this ability as well.

Finally, I want to state one thing that I hope is obvious. All these “security” settings are contained in a plain text file that can be edited by anyone who knows how to use a simple text editor. The point here is that while recent versions of Windows are making it harder and harder to modify files in the “Programs” directories, it is not by any stretch of the imagination bullet-proof. Hence, if there are truly sensitive things that need restricted access, don’t depend on these settings.

What we can do with these controls

So let’s put some of what we have been learning about into action. If you download the code from the SVN repository you will find, in addition to the source code, a compiled executable. For the following tests, you can either run the executable I have included, or compile it on your own — it’s up to you. You will also want to be sure to update your copy of the toolbox as I have added a couple useful VIs. One of the executable management VIs is where we will start:

Launching the Executable

Start by opening small test executable.lvproj and then open the routine Launch 3x.vi. It’s job is to launch three copies of the test application (small test executable.exe) so on the front panel click the path browser button next to the path control and navigate to and then select the test application. Now, run the VI.

When it finishes, launch the Windows Task Manager. Nothing new under Apps, or Programs (depending on your version of Windows), so look in the Background Processes. Ah, there’s the executable, but why is it listed here? And why is there only one instance? The VI clearly looped 3 times, and there were no errors. Go to the directory where the test application is located and open its INI file. There are your answers. There is only one instance running because the INI file has multiple instances turned off, and the one executable that did launch shows up as a background process because the INI file also says to hide the root window. Leave the root window setting the way it is, but change the AllowMutipleInstances to True, and save and close the file.

Now back in the Task Manager, abort the one instance of the test application that is running now, and rerun Launch 3x.vi. You should see when it finishes that there are now 3 instances of the test application running. The instances were given sequential TCP/IP port numbers from 3365 to 3367.

Firing Remote Events

The next thing I want to do is open the front panels of the 3 instances so we can observe their operation. Now if you look at the test application’s source code, there is a UDE that will make the front panel visible, so all I need to do is fire that event. But wait, those are three instances of a compiled executable — you can’t fire events in other executables! Well actually, you can. To see how, open the test VI, Open the Executable’s Window.vi.

No magic here. All the code is doing is dynamically running a VI. But check out the function before Open VI Reference, it’s called Open Application Reference. Its job is to open a reference to a copy of LabVIEW or the LabVIEW runtime engine that is running somewhere else. That “somewhere else” is defined in terms of a machine name and a port number or service name. The machine name can be a DNS name, an IP address or (as in our case here) localhost to point to the local computer.

By the way, if you think it sounds like I just said that you could make this same code access an executable residing on a remote computer by simply changing localhost to an IP address, your right. I did just say that.

But as cool as that feature might be, how does it allow me to fire an event in a compiled executable? Look at the name of the file being run: Open Window.lvlib:Generate Event.vi. It’s the VI that fires the event, and since VIs called in this way actually execute in the remote LabVIEW environment, the event gets fired in the targeted executable.

To see this code in action, run it three times with the port number 3365, 3366 and 3367. Three windows will open.

Setting Control Values

Another way of interacting with an executable is to directly manipulate controls on its front panel. However, if the target VI is event-driven like our test application, we need to remember that there is a difference between setting a value and firing any value change events associated with that control. If all you need to do is set a value, there is a VI method called Control Value:Set that will do the job nicely. However, if you want to fire the value change event you have to set the control’s Value(Signaling) property — which frankly is a bit more work.

This picture is the block diagram of the test VI Toggle the selector checkbox.vi, but the good news is that for this little bit of extra effort, you can set (or read) any control property that can be changed while a VI is running.

Shutting Down the Executable

Finally, we need to be able to stop an application that is running. But the problem here is figuring out how to test it such that we can see that it really did what it was supposed to do. The solution is to turn to the trace technique we discussed a while back when we were learning about command line arguments. I have written the code such that if the executable is run with the argument “d1” in the command line, the code will write a line to the trace file saying how the instance was stopped. And to help demonstrate how this works, I have created a test VI (Stop the Executable.vi) that can execute some of these termination paths.

To start off, leave both controls in their default state, and run the VI. This example stops the targeted executable by clicking the Stop button on its front panel. The instance with the port number 3365 will immediately close.

Now increment the Port Number to 3366 and set the Method control to Windows shutdown - Forced. This example stops the targeted executable by telling to Windows to abort it. The instance with the port number 3366 will immediately close.

Finally, we want to test the remaining instance’s response to the Application Instance Close event. To do that, restart your computer now. (That’s right, restart your computer. Don’t stop anything, don’t shut anything down — just restart.) When your computer is restarted and you are logged back in, go to the directory where the test application is installed and open the trace file. You will see two lines that look something like this:

07:29:39 05/24/2015 -- Shutdown 3365 -- Just Stop
07:42:52 05/24/2015 -- Shutdown 3367 -- Appl Inst Close

The second line shows that when you restarted your computer Windows did in fact generate the Application Instance Close? event and the application caught the event. You’ll note that there is no entry for the instance with the port number 3366. Remember, we stopped it by forcing an abort and a Windows abort is very much like clicking the red abort button when LabVIEW is running a VI: It just stops. No orderly shutdown. No deinitialization.

A Small Test Executable — Release 1
Toolbox — Release 9

The Big Tease

So that was, I hope, interesting. Starting next time I’m going to start delving into how to use LabVIEW code as the data collection and control backend for an application that has as its only customer-facing interface a web site. While there are many companies offering options that claim to be developer-friendly I have found that many of the marketing claims are largely based on FUD (Fear, Uncertainty and Doubt). Simply put, they build up a “strong man” of supposed complexity and complication, and then tell you that the best (and perhaps, only) way to get past this obstacle is to buy their product. The truth, however, is that their “strong man” is really made of straw, and if you understand how it all fits together, doing it yourself isn’t really very hard.

Until Next Time…

Mike…

Managing Standalone Executables

Posted on 2015-05-182015-05-22 by mlportersr

A while back when we were discussing opportunities for code modularization, I made the comment that it is even possible to incorporate processes into a new application that have already been compiled into standalone executables. At the time we didn’t have the time to go into detail on the point, but now we will take the time to consider the various tasks required to manage an existing standalone executable that you wish to leverage for a new application.

So What’s to Manage?

When I talk about managing an executable, I’m including all those tasks needed to control the executable from an operating system level. Therefore, to complete these tasks successfully we need a certain basic understanding of Windows and how it interacts with the programs that it is running. As you might imagine, most of these tasks will involve simply starting and stopping programs, but there are nuances that are important too. For example, when I double-click on a program the first time, Windows will launch it, but what if I double-click on it again? In this situation Windows has two possible options: launch another instance of the program or activate (bring to the front) the instance that is already running. One of the things that we will look at is how to specify that action for the executables we create,

Likewise, sometimes it isn’t enough to simply launch a program, sometimes we need to take high level look at its operation to determine such things as: How many instances of it are currently running or how much memory it is using. This information can help us decide how to best husband the computer resources that our application as a whole uses. Consequently, we will spend a little time considering the alternatives.

Finally, there can even be subtleties when it comes to stopping an executable. Can we simply signal the executable that it needs to stop, or do we need to force it to stop Now.

Getting Things Started

The first thing we want to cover is how to programmatically launch standalone applications. So to state the obvious, this topic is first and foremost about how to tell the operating system to launch the application for us. Luckily, this task is easily handled by the built-in System Exec.vi. This function allows you to do from within LabVIEW anything you would do from a command prompt — though at times the operations aren’t done in the exact the same way. In fact, for some older versions of Windows the differences can be significant. But even in Windows 8 there can be small inconsistencies. For example, say you have a program called Small Test Executable.exe that can accept a couple of command-line arguments like p3365 fpd2. You could launch this program from the command line like so:

C:\"Program Files\Test Executable\Small Test Executable.exe" -- p3365 fpd2

Unfortunately, this doesn’t work with System Exec.vi, which simply returns an (undocumented) error. To get the command to work you have to either write it to a temporary batch file and then execute the batch file, or modify the command like this:

"C:\Program Files\Test Executable\Small Test Executable.exe" -- p3365 fpd2

Don’t see a difference? Check out the location of the first double quote… To simplify dealing with these variations, I have created a subVI to encapsulate the functionality called Launch Executable w-Command Line Parameters.vi and included it in the new release of the toolbox.

The other thing about getting an executable running is that there are a few important launch parameters that we can only set in the application INI file, so we’ll deal with those next.

A Windows Convention

One of the conventions that Windows imposes is that the INI for an executable will have the name as the executable (but with the file extension ini) and the standard configuration parameters will reside is a section with the same name as the file (but minus the file extension). It is in this section that the runtime engine expects to find the parameters that it needs to successfully launch the executable.

The first such parameter we need to discuss controls the behavior when the executable is called more than once. By default, when you create an executable with LabVIEW only one copy of it will launch. If you call it again, Windows will simple bring to the front the one instance of the program that is already running. However, you can change this behavior by including this line in the INI file:

AllowMultipleInstances = TRUE

With this parameter set as shown, Windows will launch another instance of the program each time it is called. An important point to remember, however, is that no matter how many instances you launch, they all share the same INI file. Consequently, if you want to pass unique parameters to each instance, you will have to do so through the command line when the instances are launched.

The second INI file parameter I want to mention controls the program’s visibility to the computer user. Regardless of whether its GUI is visible, when a program normally launches it has a tab that appears on the taskbar. But what if you are creating a program that is intended to run unseen in the background? This setting handles that case.

HideRootWindow = True

When this setting is true, the program can still have a user interface but there is no tab associated with its window on the taskbar. As a side effect, the program also doesn’t appear in the task manager as an “App”, but rather lists it as a “Background Process”. Please note that this terminology is what Windows 8.x uses, older versions make the same distinction but use the word “Programs” and “Processes” to describe it.

The third INI file setting to discuss, is really not so much of a setting, as it is a family of settings that work together to control the program’s ability to respond to external connections via TCP/IP and VI Server.

server.tcp.enabled=True
server.tcp.port=3363
server.tcp.serviceName=""
server.tcp.acl="290000000A000000010000001D00000003000000010000002A10000000030000000000010000000000"
server.vi.access="+*"
server.vi.callsEnabled=True
server.app.propertiesEnabled=True
server.vi.propertiesEnabled=True
server.control.propertiesEnabled=True

Some of these settings are easy to understand, while others are rather obtuse. For right now, don’t worry about what these settings all mean. Next week we will.look at these parameters in detail.

Passing Parameters

To reiterate a point I made earlier, creating a standalone executable that will support multiple instances pretty much necessitates the use of command line arguments. In some ways, this situation is analogous to the situation concerning reentrant clones. The instance needs to be able to identify itself in the midst of a cloud of identical instances.

Given that point, one of the things you should consider when creating such an application is how the program should respond if the required command line parameters are missing. The answer to that question depends, to a large extent, on the nature of the parameters. If the input is essentially a customization for which there is a valid default value, it might be acceptable to simply accept that default and go on. However, in some situations there is no default value possible, so you should consider shutting down the program if the command line parameter is invalid or missing.

Getting Status

After we have gotten the instance (or instances) of a program up and running, one of the things that we might want to do is check on how, and perhaps what, it is doing. For now we will consider this matter in terms of the performance aspects that are visible to Windows. It is certainly possible to access the executable programmatically to obtain internal status information, and even control its operation, but that discussion will have to wait until next week.

General Information

One of the most basic indicators of the health of a program is its memory usage. You should expect the memory that a program has allocated will rise and fall over time. It should not treat trend steadily upwards without regard for what the program is actually doing. If a program’s memory consumption does consistently increase over time, that condition is referred to as a “memory leak” and it is a condition that needs to be addressed. Typically this remediation takes the form of figuring out why the program wants ever-increasing amounts of memory (like programming errors resulting in file or I/O being left open) and fixing the problem. Occasionally however, you will find that the leak is occurring in a place where you have no visibility — like inside LabVIEW itself. In those situations all you can do is try forcing Windows to deallocate the memory or change the way you are doing something so the problem doesn’t arise.

A simple way to read memory usage is to call the simple command line function tasklist and parse the result. Here is the code I wrote to do the job. Its name is Read Task List.vi.

If you leave the Image Name control empty, the call returns information on all the tasks currently running. However, if you populate that control with the name of a program, it returns a list of all the programs with that name. For example, on my computer right now LabVIEW.exe returns 1 item, while chrome.exe returns 13. (Why does Chrome need 13 instances of itself running in the background?)

The two most valuable pieces of information that you can get from this command is the application’s memory usage (expressed in kilobytes) and its PID. The PID is a numeric tag that uniquely identifies each program that is currently running. It is important because other commands or system calls will often require the PID of the program that you are wanting to check.

Digging for the Details

Beyond this basic information there is a variety of other details that you can find, For example, here you can find a community-developed VI that makes direct calls to a .NET assembly in order to return a plethora of information about your computer in general, and the programs that are running in particular. From the standpoint of accessing this information programmatically, this VI has two big problems:

The code is at least 7 years old: This is a problem because there are places where the same operations could be done much more efficiently using techniques that LabVIEW has introduced in the intervening time.
It’s essentially undocumented: To figure out how to really use it you will have to spend time researching the calls and the specific parameters — which can vary from one version of Windows to the next.
It was designed for manual operation: In other words, it was designed to be used interactively, and not programmatically.

Because we need something that is usable from within a program, I created two subVIs that essentially repackage the functionality in a more program-friendly form. The first subVI is called Get Available Instances.vi. To see why this VI is needed, think back to the output of the tasklist function. In that output multiple instances of the same executable (like the 13 copies of chrome) all had the same name. They were distinguished from one another by a numeric id number called a PID. The designers of the .NET interface must have thought this approach confusing because they took a different approach that did away with the PID in favor of a name that was modified to make it unique — as in chrome through chrome#12. Likewise, any file extension was also dropped from the name.

The other subVI (Get Task Performance.vi) accepts as inputs an instance name and an enumeration that lists the available performance parameters that the VI can fetch under Windows 8.

But what is the deal with the loop and all the “extra” logic? This loop is needed because, while some of the parameters are instantaneous values that can be read at any time, others are not. As a case in point, consider the processor usage measurements. Everyone knows (or at least everyone in this business should know) that a CPU can really only do one thing at a time. In fact, the illusion that is central to much of modern technology is the programmatic slight of hand that makes it look like computers are doing multiple things at once.

Now since a CPU can only do one thing at a time, logically there are only two possible immediate answers to the question of how much processor are you’re using. Either you are not executing (in which case you have 0%) or you are executing (resulting in 100% usage). In short, for this type of measurement to have any sort of meaning it has to be approached as a statistic, not an absolute measurement. In the VI, this task is implemented by making two calls to the same measurement counter. The first call — which always returns a 0 — starts the measurement process, with the second and subsequent reads returning the statistical results for the previous measurement period.

Oh, and the logic for stopping the loop after just one iteration? That’s to optimize operation for parameters that are absolute.

Stopping: Fast and Half-Fast

It has been said that, “All good things must come to an end” and this is true for programs as well. However Windows actually provides two different types of shutdown events. The first causes Windows to simply abort the targeted program. This approach, while very fast, is logically equivalent to clicking the abort button on a LabVIEW program — and carries with it all the same dangers. The other type of shutdown event takes longer, but is much safer. It basically sends a message to the targeted program that requests it to stop. However, to reiterate something we learned before, for this messaging to work properly the executable must be written such that the Application Instance Close event will always initiate an orderly shut down.

In addition to using these messages for its own purposes, Windows also provides a command line function that allows users and individual programs to generate these messages as well. I implement both types of shut down requests in the subVI Task Killer.vi.

As a polymorphic VI, the routine incorporates instances that can shut down processes by name and by PID. You simply pick the version that gives you the level of control that you need. For instance, if I wanted to shut down all 13 instances of chrome, using the name version of the code would get the job done all at once. However, if I wanted to stop just a few select instances, I would use the version that uses the PID input. Here is the name version of the code.

But why start and stop at all?

Before ending this time, there’s one more idea that we should consider. To this point we have been thinking about the management of executables in a rather conventional sort of way: Start something just before we use it and close it when we are done. But might not there be situations where the “conventional” doesn’t make sense? I would assert that there are more than you might at first imagine.

The whole point of interacting with a process that is deployed as a separate standalone executable is that it obviously implements some bit of functionality that more than one application would need. Given that fact, starting and stopping it — especially stopping it — can get complicated. Just because one application is stopping and no longer needs it, that doesn’t mean there might not still be another application that does. One obvious solution to this problem is to simply let Windows handle it all. Install the background task such that it starts when either the computer boots, or the user logs in; and then stops when Windows shuts down. This solution might seem wasteful, but what is the real impact of such a solution? Oh sure, the program will be in memory, but if it isn’t being used it will probably be shuffled off to the page file pretty quickly. Likewise, if the executable is written correctly, it will be event driven which means that unless it is accessed by a program that is using it, the process won’t be burning up any CPU cycles either — so what is there to lose?

Just something to be thinking about.

Toolbox Release 8

The Big Tease

Ok, this is all for now. Next time I’ll introduce a little executable that will let us practice some of the theory we have examined today — and expand the conversation to include how to use VI Server to interact with executables created in LabVIEW.

Until Next Time…

Mike…

Expanding Data Processing Bandwidth — Automatically

Posted on 2015-05-112015-05-11 by mlportersr

Well-written software can typically deal with any performance requirement pretty easily as long as the requirement is constant. It’s when requirements change over time that things can get dicey. For example, if your test system generates a new data packet to process every second and it consistently takes 5 seconds for the data to be processed, a little simple math will tell you how much processing bandwidth you need to create to keep up with the flow of data. But how are you to properly size things when variability is inserted into the process? What if the time between data packets can vary between 100 msec and several minutes? Or what if the data processing time can change dramatically due to things like network traffic?

These are the kind of situations where the processing needs to be more than simply “flexible”, it has to be able to automatically maintain its own operation and reconfigure itself on the fly. To demonstrate one possible implementation of this “advanced” technique, we will build on the simple pieces that we have learned in the past. In some ways, good software design techniques are like Lego blocks. Each one by itself is not very impressive, but when you stick them together, magic happens. But before we can stick anything together, we need to understand…

…what we’re going to do.

You’ll notice that any of the bandwidth management challenges that I mentioned earlier can be addressed by either adding more data processing clones, or removing existing ones that are being underutilized. Consequently, the question of how to implement this self-maintenance functionality really gets down to a matter of how to dynamically manage the number of data processing clones that are currently available — which in turn boils down to answering two very simple questions:

1. How do we know we need more?

Given that the whole point of the exercise is to manage a queue, the current state of that queue will give us all the information that we need to answer this question. Specifically, we can know when more processing bandwidth is needed by monitoring how many items are currently in the queue waiting processing. When the code starts to see the depth going steadily up, it can launch additional processes to handle the data backlog. Of course, this functionality assumes that there is a process that is constantly monitoring the queue and managing that aspect of its operation — which we actually have already in the test code from last week (Data Processing Queue Handler.vi). All we have to do is repurpose this VI to be a permanent part of the final application.

2. How do we know a clone is no longer needed?

The one part of the system that knows whether a clone is being under-utilized is, in fact, the clone itself. As a part of its normal operation, it knows and can keep track of how often is it being used. Having said that, there are (at least) a couple of ways to quantify how much a clone is being utilized. We could, for instance, consider how much time the clone is spending processing data versus how much time it spends waiting to receive data to process. If the utilization percentage drops below a given limit, the clone could then shut itself down. However, for this demonstration, I’m going to use a much simpler criteria that, quite frankly, works pretty well. The code will simply keep track of how many times in a row it goes to the queue and doesn’t find any data.

Code Modifications

Before I start describing the changes that will we will need to make in order to fashion this new ability, I want to consider for a moment the thing that won’t have to change: Queue Test.vi. You might be tempted to say, “Well big deal. All it does is stuff some data into the queue every so often. Who cares if it doesn’t have to change? It’s not even deliverable code”

While that is undoubtedly true, the fact of the matter is that this test routine is important, but not because of what it is or what it does. Rather we care about Queue Test.vi because in our little test environment, it represents the rest of our application — or at least that part of it that is generating data. Consequently, the fact that it doesn’t need modification means that your main application, likewise, won’t need modification if you decide to upgrade from a data processing environment that uses a fixed number of data processors to one that dynamically manages itself.

Data Processing Queue Handler.vi

First, note that previously this routine’s primary job was to simply report how deep the data queue was — a bit of functionality that would likely have not been needed in a real application. Now however, this routine is going to be taking an active part in the process, so I started the modifications by adding the error reporting VI that will transfer errors it generates to the exception handler.

In addition, because the software will initially only start a single data processing clone, I also modified the timeout event handler that performs the VI’s initialization, by removing the loop around the clone launching subVI.

The remainder of the modifications to this routine occurs in the event handler for the Check Queue Size UDE. Previously this event only reported how deep the queue was getting. While it still performs that function, that queue depth information in addition now drives the logic that determines whether we have enough processing bandwidth online.

Note that the queue depth is compared to a new configuration value called Max Queue Size that defines how large the queue can grow before a new data processing clone is launched. Regardless of whether it launches a new clone, event handler calls another new subVI that returns the number clones that are currently running. As you will see in a moment, one of the modifications to the data processing VI is the addition of logic that keeps track of the names of the clones that are running. The subVI that we are calling here returns a count of the number of names that have been recorded so far.

Data Processor.vi

Turning now the data processing code itself, the first stop is in the state-machine’s Initialize state. Here we have all the same logic that existed before, but with a couple minor additions

First, there is a new subVI that registers a clone is starting up. This subVI writes the clone’s name to the FGV that is maintaining the clone count. Second, there is also a new shift register carrying a cluster of internal data that clone will need to do its work. All that is needed during initialization is to set a timestamp value. The Check for Data state is next and it has likewise seen some tweaks — the most significant of which is moving the logic for responding to the dequeue operation into a subVI.

The justification for this move lies in the fact that this logic is now also responsible for determining whether or not the clone is being adequately utilized. As I stated before, each time the clone goes to the queue and comes up empty, the logic will increment a counter that is being carried in the new shift register’s data. If this count exceeds a new configuration value Clone Idle Count, the code will branch to a new state that will shutdown the clone. Likewise, anytime the clone does get data to process, it will reset the count to 0. The changes to the Process Data state, which comes next, are pretty trivial.

All that happens here is that the timestamp extracted from the data to be “analyzed” updates the new cluster data — as well as the indicator on the front panel. Finally, there is the new state: Self Shutdown…

…which simply calls a subVI that removes the clone’s name from FGV maintaining a list of all running clones, and stops the event loop.

Let’s talk about “Race Conditions”

All we have left to do now is test this work and see the differences that it makes, but before we can do that, we need to have a short conversation about race conditions. Very often developers and instructors (myself included) will talk about the necessity of avoiding race conditions. The dirty little secret is that as long as you have multiple things happening in parallel, race conditions will always be present. The real point that these admonitions attempt to make is that you should avoid the race conditions that are unrecognized and potentially problematic.

I bring this point up because as you do the following testing, you may get the chance to see this concept in action. The way it will appear is that the system will launch a new clone immediately after one kills itself off for being underutilized. The reason for this apparent logical lapse is that a race condition exists between the part of the code that is checking to see if another clone is needed and the several places where the clones are deciding whether or not they are being used. There are two causes for this race condition, one we can ameliorate a bit and one over which we have no possible control.

Starting with the cause we can’t control, a simple immutable law of nature is that no matter how sophisticated our logic or algorithms might be, they can not see so much as a nanosecond into the future. Consequently, the first source of a race condition is that when the queue checking logic sees that there are three elements enqueued, it has no way of knowing that a currently active clone will be available in a few milliseconds. While it is true that under certain circumstances it might be possible to provide this logic with a bit of “foresight”, there is no generalized solution to this aspect of the problem. Consequently, this is an issue that we may just have to live with.

The news, however, is better for the second cause. Here the problem is that with all the clones having the same timeout between data checks, it is probable that sooner or later one of the clones is going to become “synchronized” with the others such that it is always checking the queue just after it was emptied by one of the others. However the solution to this problem lies in its very definition. The cure is to see to it that the clones do not have constant timeouts from one check to the next. To implement this concept in our test code I modified the routine that returns the delay to add a small random difference that changes each time it’s called.

The bottom line is that while completely removing all race conditions is not possible, they can be managed such that their impacts are minimized.

New Tests for New Code

The testing of the modified code starts the same as it did before: open and launch Data Processing Queue Handler.vi and Queue Test.vi. The first difference that you will notice is that only 1 clone is initially launched, but at the default data rate, 1 clone is more than adequate.

Now decrease the delay between data packets to 2 seconds. Here the queue depth will bounce around a bit but the clone count will stabilize at 3 or 4.

Finally, take the delay all the way down to 1 second. Initially the clone count may shoot up to 8 or 9, but on my system the clone count eventually settled down to 6 or 7.

At this point, you can begin increasing the delay again and the slowly the clones will start dropping out from disuse. Before you shutdown the test, however, you might want to set the delay back to 2 seconds and leave the code running while you go about whatever else you have to do today. It could be instructive to notice how other things you are doing on the same computer effect the queue operation. You might also want to rerun the test but start Queue Test.vi first and let it run for a minute or so before you startData Processing Queue Handler.vi — just to see what happens.

Further Enhancements?

So we have our basic scalable system completed, but are there things we could still do to improve its operation? Of course. For example, we know that due to timing issues which we can only partially control, the number of clones that is running at one time can vary a bit, even if the data rate is constant. One thing that could be done to improve efficiency would be to change the way clones are handled. For example, right now a data processor is either in memory and running or it is closed. One thing you could do is create a new state that a clone could be in — like loaded into memory, but inactive. You could implement this zombie state by setting the timeout to 0, thus effectively turning off the state machine.

It might also be helpful to change to queue depth limit at which a new clone is created by making it softer. Instead of launching a new clone anytime the queue depth exceeds 3, it might be useful in some situations to maintain a running average and only create a new clone if the average queue depth over the last N checks is greater than 3.

Who knows? Some of you might think of still other modifications and enhancements. The point is to experiment and see what works best for your specific application.

Parallel Data Processing — Release 2

The Big Tease

So what is in store for next time? Well in the past we have discussed how to dynamically launch and use VIs that run as separate processes. But what if the code you want to access dynamically like this happens to exist in a process that you have already compiled into a standalone application? If this application is working you don’t want to risk breaking something by modifying it. As it turns out there are ways to manage and reuse that code as well, even if it was created in an older version of LabVIEW. Next time we’ll start exploring how to do it.

Until Next Time…

Mike…

Seeing the Problem

Low-Overhead Storage

Getting Data Out

Summarizing Options and Challenges

Let’s See How it Works

The Big Tease

Share this:

Ok, so where do we start?

Revisiting Modular Functionality

Getting Down with Modular Logic

Creating the Data for Our Menu Compiler

Starting the Compilation

Generating the Menu Item “Code”

Generating the Submenu “Code”

What Could Possibly Go Wrong?

The Big Tease

Share this:

The Basic Points

Creating the Constructor

Giving the Interface an Icon

Creating a Context Menu

Closing the Event Loop

Future Enhancements

The Big Tease

Share this:

Defining a Well Behaved Window

The Data, and Where to Keep it

Our Basic Approach

Digging for Details

The Big Tease

Share this:

What it does

Why This Interface?

How it Works in Theory…

…and in Practice

Getting Undocked

Docking a Window

The Big Tease

Share this:

It’s All About the (Data)Base

Writing to the Database

PHP Mediated Database Reads

The Big Tease

Share this:

The Web Landscape

HTML — The Content

CSS — How Things Should Look

PHP — Server-Side Programmability

Database — Data Storage

Getting back to LabVIEW

The Big Tease

Share this:

First, we need an executable

Handling the front panel

VI Server Operations

What we can do with these controls

Launching the Executable

Firing Remote Events

Setting Control Values

Shutting Down the Executable

The Big Tease

Share this:

So What’s to Manage?

Getting Things Started

A Windows Convention

Passing Parameters

Getting Status

General Information

Digging for the Details

Stopping: Fast and Half-Fast

But why start and stop at all?

The Big Tease

Share this:

…what we’re going to do.

1. How do we know we need more?

2. How do we know a clone is no longer needed?

Code Modifications

Data Processing Queue Handler.vi

Data Processor.vi

Let’s talk about “Race Conditions”