A New Data File Format for Archival Storage of Engineering Data

For decades, engineers have relied on traditional text file formats like .csv and .tab and native Excel formats like .xls and .xlsx for the archival storage and analysis of engineering data. While these formats have served users well, they have not kept pace with the exponential changes in data acquisition technologies and requirements over the same period.

Engineers now collect more data and more types of data, at faster rates, than ever before. The traditional methods of storing and analysing data can’t keep up with the rate at which we acquire it. There have been some estimates that less than 10% of currently acquired engineering data is ever likely to be analysed.

The traditional file formats have significant limitations when it comes to storing these new data mixes:

• They are not designed to efficiently read and write large data sets. Very large files can be grindingly slow to open or close.
• They do not cope well with data acquired at different rates. Slower data is typically padded with additional values to align it to the acquisition rate of the fastest data.
• They do not easily deal with complex data mixes that may include: Standard analog and digital IO points; Block data from non-standard transducers such as scanning lasers or imaging sensors; Data packets from communications-based sources; and Ad-Hoc events.
• They are not structured to allow detailed metadata to be readily associated with the base IO data. Critical context and configuration information needed to make sense of the data is often stored externally. If this external support data becomes detached from the parent file for any reason, the parent data may become unusable.
• Storing test data as a collection of individual flat files makes it difficult to compare and analyse data acquired from multiple tests performed at different times.

Recognising these challenges, some engineering disciplines have developed proprietary file formats tailored to specific needs. While these formats address some issues, their adoption remains restricted to niche applications supported by a limited set of specialised software tools. There is a clear gap in the market for a data file format that can address all these issues in a consistent and meaningful way for all engineering fields.

One standout candidate is the .tdms file format (TDMS), initially developed by National Instruments, but now released as an open standard. TDMS is optimised for storing time-dependent data channels, with each channel associated with its time base. So channels acquired at different rates can be stored in the same file without wasteful data padding.

TDMS also supports time-synchronised channels of text, communications & and event-based data, and non-time-based parametric (x-y) data channels, enabling a unified storage structure for diverse mixed-signal data. It is a binary format that can efficiently manage vast datasets, up to a terabyte or more, without degradation of performance.

It provides for three levels of predefined and user-configurable metadata within the file:

1) Descriptive of the overall file;
2) Descriptive of each channel in the file; and
3) Descriptive of one or more logical channel groups, such as “all temperature channels”.

So a file can fully self-describe its contents, making it ideal for long-term archival data storage.

Transitioning to TDMS is a strategic decision that requires careful implementation. Legacy formats such as .csv, .tab and Excel will persist for the foreseeable future due to familiarity and entrenched workflows. However, their limitations make them increasingly unsuited to the needs of modern engineering data storage.

TDMS offers a robust, efficient, and scalable alternative, addressing the challenges associated with high-volume, multi-source and asynchronous data acquisition. TDMS is integral to the entire National Instruments software line, making it the simplest and quickest way to access the format’s benefits.

However, NI has released full details of the standard if you want to add support for TDMS to your custom software developments. And there is a growing number of third-party products that offer high-level TDMS support for specific environments such as Python, Excel, and others.

If your only reason for continuing to use the traditional formats is “…because that’s the way we have always done it”. In that case, it might be a good time to consider if a modern format like TDMS would deliver better outcomes for your engineering team.

TDMS offers long-term advantages for engineering teams seeking to optimise data storage, enhance analysis capabilities, and future-proof their workflows. By embracing modern data storage solutions, your engineers will unlock the full potential of their datasets, enabling higher-level insights, improved decision-making, and enhanced long-term data integrity.

This is Part 1 of a 4-Part series of articles on engineering data management. In Part 2, we will take you on a deeper dive into the TDMS format, and give you the details you need to bring the benefits of TDMS into your engineering workspace.