Sunday, March 18, 2012

BIM design software performance review

In evaluating software for building information modeling, there are various alternatives, some have more useful features than others, some are easier to use than others, and some have more compelling value propositions.  However, when dealing with realistically sized buildings, such comparisons can become meaningless for software that can’t deliver on performance and/or has stability issues. 

Let’s face it – time is money; waiting more than a few seconds for a web page to load makes most people impatient to the point of switching tasks.  So why would anyone put up with design software causing excessive delays of minutes or longer?  Over time, this adds up dramatically.
In this post, we measure speed in design software, initially focusing on the time it takes to load files.

To compare apples-to-apples, we used the same machine with each application running side-by-side.  The machine is a Sony VGN-FW, a relatively high-end machine in 2010 and average by today’s standards.  The operating system was Windows Vista 64-bit, with a Windows Experience Index of 5.3.

To rule out any randomness from operating system loading and paging, all tests were based on warm-starts: that is, tests were run multiple times and the initial results were thrown out such that any upfront costs of library loading or initialization were disregarded.  Even so, such initialization turned out to be negligible with all applications.  The machine had 6 GB RAM, enough for each application to load the largest files without paging.

All testing was recorded using in-band video capture to provide backup for the reported results – such video capture consumed 25% (average mean) of available CPU, however impacted other applications evenly.  The video capture had a frame rate of 8 fps at 1920x1080 pixels; the observed jumpiness is a based on this frame rate; application responsiveness was under 50 ms (better than the shown by the video frame rate), with exception of periods when some applications had the main thread blocked while loading.

Four applications were tested:
To observe trends in scalability and variation of buildings, eight different IFC BIM files were used of various sizes and disciplines.  The building models include the following:
The file open test measured file load time, specifically the time span between picking a file to open, and being able to navigate the building in 3D.  Each application tended to have several phases within this span – reading the file, transforming, and rendering.
Each file was compared four times, rotating the order of applications, where all results were within 5% of each other.
Figure 1 shows elapsed time for loading a file and rendering on a logarithmic scale.  In this chart, lower values are better, where each range reflects an order of magnitude.
Figure 1 - Elapsed time loading and rendering

Constructivity was the leader in every scenario, and was ten times faster than Autodesk Revit on average.  Tekla BimSight came in second, averaging three times faster than Revit.  ArchiCAD came in third, averaging two times faster than Revit.  Finally, Revit took last place, taking over 15 minutes to load the larger files. 

Figure 2 shows load time relative to file size, to observe trends in scalability.  In this chart, higher values are better, and particularly more so as the file size increases (going upwards).
Figure 2 - Throughput loading and rendering
As files got larger, the differences became more pronounced.  Revit peaked around 200 KB/s and became slower as files got larger.   ArchiCAD maintained speed around 200 KB/s as file size increased.  BIMsight and Constructivity became faster as files got larger, with Constructivity exceeding 1 MB/s.
Videos were recorded for each scenario, and are provided here to illustrate the methodology and verification.  Videos that took longer than 15 minutes are not included, due to YouTube length restriction.

  


Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)