5 Tips for Flood Experts Managing Large Datasets

If you work with flood models or manage flood risk, chances are you’ve felt the pain of losing a day’s work while trying to transfer large datasets across networks or onto encrypted hard drives...

The truth is, nearly everyone has a story of losing a model or having their client lose one, sometimes spending weeks trying to track down missing model files because the main modeller is on holiday.

We've put together 5 simple tips to make model management a little bit more enjoyable...

1. Use a Third-Party Application for File Compression

Compressing large flood datasets before transferring or archiving them is standard practice across the industry. Smaller files mean faster uploads, quicker sharing with partners and clients, and reduced long‑term storage requirements ‑ especially important when working with large model, ensemble outputs, or detailed LiDAR‑derived rasters.

Windows’ built‑in compression tool (the “Send to > Compressed (zipped) folder” option) is simple and convenient for small files. However, it is not designed with large technical datasets in mind. For larger folders, third‑party tools such as 7‑Zip, WinRAR, or PeaZip consistently deliver far better performance and efficiency.

One of the key advantages is that these applications allow you to configure compression strength and speed to suit your workflow. Just as importantly for flood modellers, they are multi‑threaded, meaning they can use multiple processor cores in parallel. Windows’ native compressor is single‑threaded, so it makes use of only a fraction of modern CPU capability ‑ significantly extending compression time.

If you already rely on multi‑threading to accelerate model runs, mesh generation, or hydraulic post‑processing, the same principle applies to your file compression. Most of these tools are free, and they let you unlock the full power of your hardware!

Example:

Compressing a 3 GB flood model dataset using Windows’ native zip compression took 15 minutes. Using 7‑Zip with default settings took 48 seconds. Both produced a .7z file with roughly 50% size reduction, but with vastly different efficiency.

Beyond speed, third‑party compression tools also provide:

• Higher compression ratios, enabling smaller archives for long‑term storage

• Support for modern archive formats (.7z, .xz, .tar.zst, .zipx, etc.)

• Error‑checking options for added data integrity

• Split archives, which are useful when sending files via email or upload portals with size limits

If you regularly exchange large flood models, calibration data, or simulation outputs, adopting a dedicated compression tool is a small change that can save significant time ‑ and streamline collaboration.

2. Include Metadata With Your Files

Metadata is information about your data, and it’s essential for helping others (and your future self) understand, manage, reuse, and contextualise model datasets quickly and accurately. Good metadata reduces ambiguity, prevents duplicated effort, and ensures that your modelling work remains transparent, traceable, and easy to build upon.

Most modellers recognise that maintaining a model log is good practice: documenting decisions made, assumptions applied, scenarios tested, and issues encountered. However, very few dedicate the necessary time to do this consistently. Investing a small amount of effort upfront can save days or even weeks for whoever reuses or reviews your work later ‑ especially in large organisations where models frequently change hands.

Metadata is particularly valuable when:

• A model is revisited after a long time gap

• A colleague inherits your work mid‑project

• Models are adapted for new scenarios or catchments

• Audits or verification exercises require clear, defensible documentation

• Regulators or clients need transparency on methods and assumptions

A good set of metadata should include:

• Purpose and scope of the model

• Software versions (e.g., TUFLOW build, GIS versions, plugins)

• Dates when the model was developed and delivered

• Key inputs and where they came from (e.g., LiDAR source, hydrology files, boundaries)

• Scenario definitions (events, durations, climate allowances, blockages)

• Model changes made over time, and the reasons for them

• Known limitations or unresolved issues

• Contacts or points of responsibility

If you’re new to TUFLOW modelling, there is a helpful sample model log available on the TUFLOW website: TUFLOW Modelling Log – TUFLOW. Download it, complete it as you develop your model, review it periodically, and save it alongside your model files.

Building good metadata habits now will make your modelling outputs vastly more usable, maintainable, and defensible ‑ and will make life far easier for the next modeller who picks up your work (very possibly your future self).

Continue Reading

Uncover the remaining 3 practical tips, including:

• How to safely move massive flood model folders without failed transfers or silent data loss

• Why cloud‑based storage removes the pain of VPNs, hard drives, and duplicated datasets

• How purpose‑built platforms can let you store, share, and review flood models without downloading files