Let’s face it – time is money; waiting more than a few
seconds for a web page to load makes most people impatient to the point of
switching tasks. So why would anyone put
up with design software causing excessive delays of minutes or longer? Over time, this adds up dramatically.
In this post, we measure speed in design software, initially focusing on the time it takes to load files.To compare apples-to-apples, we used the same machine with each application running side-by-side. The machine is a Sony VGN-FW, a relatively high-end machine in 2010 and average by today’s standards. The operating system was Windows Vista 64-bit, with a Windows Experience Index of 5.3.
To rule out any randomness from operating system loading and paging, all tests were based on warm-starts: that is, tests were run multiple times and the initial results were thrown out such that any upfront costs of library loading or initialization were disregarded. Even so, such initialization turned out to be negligible with all applications. The machine had 6 GB RAM, enough for each application to load the largest files without paging.
All testing was recorded using in-band video capture to provide backup for the reported results – such video capture consumed 25% (average mean) of available CPU, however impacted other applications evenly. The video capture had a frame rate of 8 fps at 1920x1080 pixels; the observed jumpiness is a based on this frame rate; application responsiveness was under 50 ms (better than the shown by the video frame rate), with exception of periods when some applications had the main thread blocked while loading.
Four applications were tested:
- Graphisoft ArchiCAD 15, 64-bit (www.graphisoft.com)
- Autodesk Revit 2012 Architectural Suite, 64-bit (www.autodesk.com)
- Constructivity Model Editor 0.9.5 (www.constructivity.com)
- Tekla BIMsight 1.4.1 (www.tekla.com)
- Frame (1335 KB) - http://cic.nist.gov/vrm/cis2.html
- Institute (2448 KB) - http://www.iai.fzk.de
- House (5271 KB) - http://www.iai.fzk.de
- Office-A (4004 KB) - http://buildingsmartalliance.org/index.php/projects/commonbimfiles/
- Office-S (10809 KB) - http://buildingsmartalliance.org/index.php/projects/commonbimfiles/
- Office-M (64288 KB) - http://buildingsmartalliance.org/index.php/projects/commonbimfiles/
- Clinic (17717 KB) - http://buildingsmartalliance.org/index.php/projects/commonbimfiles/
- Complex (62594 KB) - http://buildingsmart.no/pub/ifcfiles/HITOS/
The file open test measured file load time, specifically
the time span between picking a file to open, and being able to navigate the building
in 3D. Each application tended to have
several phases within this span – reading the file, transforming, and
rendering.
Each file was compared four times, rotating the order of applications, where all results were within 5% of each other.
Figure 1 shows elapsed time for loading a file and rendering on a logarithmic scale. In this chart, lower values are better, where each range reflects an order of magnitude. |
| Figure 1 - Elapsed time loading and rendering |
Constructivity was the leader in every scenario, and was ten times faster than Autodesk Revit on average. Tekla BimSight came in second, averaging three times faster than Revit. ArchiCAD came in third, averaging two times faster than Revit. Finally, Revit took last place, taking over 15 minutes to load the larger files.
Figure 2 shows load time relative to file size, to observe trends in scalability. In this chart, higher values are better, and particularly more so as the file size increases (going upwards).
 |
| Figure 2 - Throughput loading and rendering |
As
files got larger, the differences became more pronounced. Revit peaked around 200 KB/s and became slower as files got larger. ArchiCAD maintained speed around 200 KB/s as file size increased. BIMsight and Constructivity became faster as files got larger, with Constructivity exceeding 1 MB/s.
Videos were recorded for each scenario, and are provided here to illustrate the methodology and verification. Videos that took longer than 15 minutes are not included, due to YouTube length restriction.
