AECBytes Architecture Engineering Construction Newsletters

AECbytes Special Report (October 10, 2007)

Top Criteria for BIM Solutions: AECbytes Survey Results

Executive Summary

It has been close to four years since the Building Information Modeling (BIM) phenomenon started to gain momentum, and we are at the point where it has reached a certain level of maturity. Several BIM solutions are now available for individual disciplines such as architecture, structural engineering, MEP engineering, and construction. Leading vendors such as Autodesk and Bentley also offer integrated suites of multi-disciplinary BIM solutions that can be used to improve collaboration between the different disciplines. An increasing number of third-party developers are building supporting technologies for BIM solutions to extend their capabilities to cover varied aspects of building planning, design, and construction. BIM implementation is well underway in several AEC firms, and many other firms are in the process of evaluating different solutions and drawing up a strategy to transition to BIM.

At this juncture, it seems timely for a “status check” of sorts on BIM—to better understand what everyday users from AEC firms of various sizes and demographics, with different levels of experience and expertise in BIM, are looking for from their BIM solutions. BIM is a technology with manifold aspects to it: coordination, collaboration, automated drawing production, intelligent objects, detailed performance simulations, interoperability, and so on. While all of these are important, there are some aspects that will be more critical than others at different points of time. To determine which are the most critical requirements for BIM solutions at the present time, AECbytes was recently commissioned by a leading BIM vendor to conduct a detailed survey among close to 5,500 of its registered subscribers. The excellent response rate of 12% to the survey indicated that users were eager to provide their feedback as well, so that BIM solutions could be further developed to better serve their needs.

This report captures the detailed results of the survey, which was entitled “Top Criteria for BIM Solutions.” It also provides some more details about the survey and the questions that were included, along with the demographic information of those who responded to it. While architecture was the predominant discipline that was represented, there were also a sizable proportion of respondents practicing engineering, construction, and facilities management and operations. The number of respondents from large, multi-firm offices was almost the same as those from smaller, single-office firms. In terms of BIM solutions being used or evaluated, Revit formed the majority followed by ArchiCAD and the Bentley BIM solutions. Overall, the diverse background of the respondents and their solutions provided a rich context for their inputs. The survey results in this report capture not just the consolidated feedback of all the respondents but also an analysis of the data across various segments to determine if there were any significant differences in how respondents in different categories rated the various criteria. In addition to responding to the survey questions, many respondents took the time and the initiative to provide additional feedback on their requirements for BIM, an overview of which is also presented towards the end of this report.

The results of this survey indicate that at the present time, the need for drawing production is still paramount, making this the top ranking criterion for BIM solutions across all categories of firms and respondents. Other highly rated criteria include enhanced modeling capabilities with smart objects, the ready availability of object libraries so that effort is not wasted in re-creating standard building components, support for distributed work processes, and the ability to work efficiently on large projects. The quality of the Help and supporting documentation earned a surprisingly high rating, which should come as a wake-up call to those vendors who spend far too little time on providing good learning resources to accompany their solutions. Aspects such as analysis, performance simulations, and interoperability rated a little lower on the importance scale, indicating that while these aspects are also significant benefits of BIM, they are not “burning issues” right now. Also, the importance of generating renderings and animations seems to have diminished considerably, indicating that the AEC industry is moving beyond the need to create jazzy presentations for clients and is focusing more on higher-value aspects of BIM such as efficiency and quality.

Even though this survey was commissioned by one BIM vendor, its results should be valuable to all the BIM vendors as they plan on the future direction of their solutions, as well as to AEC firms as they move ahead with their BIM evaluations and implementations. It will also be interesting to compare the results of this survey with future surveys of a similar nature to get a sense of the evolution of AEC technology and our progress as an industry.

About the Survey

The main objective of this survey was to identify the most important requirements that AEC professionals would like BIM (building information modeling) solutions to fulfill. It was conducted by AECbytes on behalf of Bentley Systems, who were interested in identifying which of the top criteria were well served by their BIM solutions and which weren’t, so that they could use this information to guide the further development of their solutions. They were also interested in a comparative analysis of their solutions versus Autodesk’s Revit BIM solutions vis-à-vis the top criteria that were identified through this survey. Given that Autodesk is currently the undisputed market leader in the AEC technology space and its Revit platform is getting the most buzz when it comes to BIM, Bentley wanted to find out—by circumventing the hype and the marketing—how its solutions compare to Revit when it comes to what AEC professionals really want from BIM.

AECbytes was commissioned to design and conduct this survey as a neutral and independent resource on BIM. The publication has been focused on covering, analyzing, and reviewing BIM technologies, implementations, and conferences since its inception, and has published indepth reviews of all the leading BIM applications, including Revit, Bentley’s BIM solutions, and ArchiCAD. The survey questions were designed by me based on this knowledge, and the analysis of the survey results captured in this report also reflect my understanding of the core strengths, capabilities, and limitations of the Revit and Bentley platforms. Those who don’t necessarily concur with its assessments should still find the survey results insightful, as they directly capture the inputs of over 650 professionals who took the time to provide their feedback on what they want from BIM.

The survey was launched on June 20 and was closed on July 9. In order to get the maximum number of respondents to participate in the survey and receive unbiased feedback, the vendor who had commissioned the survey was not identified in the questionnaire.

The Survey Questions

The survey was divided into three sections. The first section listed a number of criteria that can be used to evaluate BIM solutions. For each of these criteria, respondents were asked to indicate the degree to which it was important to them and their firms by rating it on a 5 point scale, where 1 indicates “extremely important” and 5 indicates “relatively less important.” Respondents were requested to be as judicious as possible in using the top ratings, reserving it for only those aspects that were really critical to them. The criteria listed for this “stand-alone rating” were:

  1. Automated setup, management, and coordination, reducing traditional CAD management tasks
  2. Ability to work on large projects
  3. Ability to support distributed work processes, with multiple team members working on the same project
  4. The market share leadership position of the vendor offering the BIM solution
  5. Multi-disciplinary capability that serves architecture, structural engineering, and MEP
  6. IFC compatibility
  7. Ability to support preliminary conceptual design modeling
  8. Full support for producing construction documents so that another drafting application need not be used
  9. Built-in ability to generate highly-photorealistic renderings and animations
  10. Smart objects, which maintain associativity, connectivity, and relationships with other objects
  11. Availability of object libraries
  12. Quality of Help and supporting documentation, tutorials, and other learning resources
  13. Extensibility and customization of the solution
  14. Number of third-party developers developing add-on applications for the tool
  15. Direct integration with structural analysis applications
  16. Direct integration with energy analysis applications
  17. Direct integration with cost estimating applications
  18. Direct integration with project management applications
  19. Support for construction-related tasks such as quantity take-off, estimating, and 4D scheduling

The second section of the survey listed several pairs of relative criteria, which represent contrasting choices or approaches to BIM. For these criteria pairs, respondents were asked to indicate their preference on a sliding scale where the left position indicates a solid preference for the first criterion, the right position indicates a solid preference for the second criterion, and the middle position indicates neutrality. The criteria listed for this “relative rating” were:

  1. Scalable solution supporting collaboration and distributed work processes, VERSUS, (b) Single database solution featuring easier setup, organization, and management.
  2. BIM application leveraging the powerful documentation and visualization capabilities of a CAD platform, VERSUS, (b) A BIM-only application with more efficient modeling but less well-developed documentation and visualization tools.
  3. Modeling governed by mandatory constraints, better guaranteeing model integrity, VERSUS, (b) More flexible modeling, where the user takes charge of model integrity.
  4. Fully automated change management and coordination, VERSUS, (b) Semi-automated change management with faster performance.
  5. Intuitiveness and ease of use promising a short learning curve, VERSUS, (b) Enhanced scope and better capability to model complex forms.
  6. Support for 3D PDF for electronic publishing and distribution, VERSUS, (b) Support for 3D DWF.

In the third section, respondents were requested to identify any additional criteria that were not mentioned in the first two sections, but which were important to them. They were also given the opportunity to add any additional comments regarding the subject of this survey.

About the Respondents

The survey invitation was sent out to all the registered subscribers of AECbytes, the count of which was 5,486 at the time the survey was launched. The survey invitation was personalized by email, so that every respondent who received the invitation could only take it once. It could not be forwarded for others to take. This was done so as to guarantee the integrity of the results—it was not possible to take the survey multiple times and skew the data.

The total number of respondents who took the survey was 651, making the respondent rate close to 12% and providing a sufficient amount of data from which useful conclusions about the top criteria for BIM solutions could be drawn.

Towards the end of the survey, the respondents were requested to provide some basic information about themselves and their firms, which provided a useful context for evaluating the results. This information is captured in the following set of graphs.

Professional Role of Respondents (single choice only)
Disciplines Practiced by Respondents’ Firms (multiple choices allowed)
Firm Sizes of the Respondents
Number of Offices of the Firms
BIM Solutions Currently Being Used or Evaluated (multiple choices allowed)

Consolidated Results for all Respondents

Table 1 below shows the weighted average of the stand-alone criteria on a scale of 0 to 10 based on the input by all the respondents. Recall that respondents had rated these on a five point scale, ranging from “Extremely important” to “Relatively less important.” Thus, if all the respondents had selected the “Extremely important” rating for a criterion, its weighted average would be 10, whereas if all the respondents had selected the “Relatively less important” option, its weighted average would be 0. As shown below, the weighted average for each criterion—as calculated from its original five point scale—lies somewhere between these two extremes. The criteria have subsequently been ranked in Table 1 based on this weighted average, clearly indicating the most important requirements that AEC professionals would like BIM solutions to fulfill.

Table 1. The stand-alone criteria, ranked according to their order of importance for all the respondents.


While this overall ranking will serve as very useful feedback to all the BIM vendors, it provides some particularly useful insights on what is most important and what isn’t. For instance, despite the ability of a BIM model to capture information about the building in an intelligent format that can be used for indepth analysis and simulation, the need to produce construction documents is still paramount, making this the top-ranking criterion. At the same time, professionals are looking for smarter and more efficient ways to model (and generate drawings from those models), which is why they also want BIM applications to have smart objects that maintain associativity, connectivity, and relationships with other objects. They would also prefer to have object libraries available, so that they spend less of their time remodeling standard objects to use in their projects. Even though 63% of the respondents came from firms with less than 100 people and 51% from single-office firms, the ability to support distributed work processes and large projects ranked high. The importance of the quality of the Help and supporting documentation should come as a wake-up call to those vendors who spend far too little time on providing good learning resources to accompany their solutions. Firms are also looking to reduce traditional labor-intensive CAD management tasks as much as possible, so any support on that front from the applications is well appreciated.

While multi-disciplinary capability does not rank among the top criteria, it is still relatively important, followed by support for analysis and other downstream tasks. Direct integration seems to be preferable for interfacing with analysis tools and other supporting technologies, as opposed to interoperability through open standards such as the IFC. It is interesting to note that the ability to support preliminary conceptual design modeling ranks much higher than the ability to generate highly-photorealistic renderings and animations. This indicates that despite the availability of popular tools like SketchUp, professionals would still prefer to have conceptual modeling capabilities built within the BIM solution so that there is a seamless connection between the conceptual design and the subsequent design development process. The low ranking of the ability to generate renderings and animations did come as a surprise, but it was a good one, as it indicates that the industry is moving beyond the need to create jazzy presentations for clients and is focusing more on higher-value aspects of BIM such as analysis, quality, and efficiency. The biggest surprise of all was to find that for most of the respondents, the market share leadership position of the vendor offering the BIM solution was not very critical. This was despite the fact that 67% of the respondents were using or evaluating Revit, as opposed to 15% for Bentley and 32% for ArchiCAD. In other words, not all Revit users were using it simply because it was an Autodesk product.  

Looking at the same results from a Bentley-Revit comparative standpoint, the top criteria seem to be well balanced out against their respective strengths, which again is surprising given the significantly larger proportion of respondents using Revit. For example, the #1 criterion—full support for construction documentation—is definitely a key strength of the Bentley platform as it is built on top of the powerful CAD capabilities of MicroStation. (See the AECbytes reviews of Bentley’s Building Suite and Bentley Architecture and Structural.) On the other hand, the #2 criterion—smart objects that maintain associativity, connectivity, and relationships with other objects—is definitely a key strength of the Revit platform, having being built into the application from the start. (See the AECbytes reviews of Revit Architecture and Revit Structure.) With regard to the development of object libraries, there is more activity happening on this front for Revit (see the recent AECbytes article, Supporting Technologies for BIM Exhibited at AIA 2007), while Bentley’s federated database approach lends itself better to distributed work processes, varied workflows, and large projects.

Other key criteria on which Revit ranks higher include automated setup, management, and coordination, reducing traditional CAD management tasks, and direct integration with energy analysis and cost estimating applications. In contrast, Bentley does better with extensibility and customization by having a fully open API (application programming interface) and also features better integration with project management through its ProjectWise solution. Both solutions have multi-disciplinary capability that serves architecture, structural engineering, and MEP, both feature direct integration with structural analysis applications, and both are certified as IFC-compatible. In both of them, however, the quality of the Help and supporting documentation—rated as the #5 criterion—is less than stellar. Also, neither of them offers fluid and intuitive conceptual design tools or full support for construction-related tasks such as quantity take-off, estimating, and 4D scheduling.

Let’s move on to look at the pairs of relative criteria representing contrasting choices or approaches to BIM, which were presented in the second section of the survey. These were also rated by the respondents on a five point sliding scale, where the extreme left position indicated a solid preference for the first criterion, the extreme right position indicated a solid preference for the second criterion, and the middle position indicated neutrality. Table 2 shown below captures the weighted average of the stand-alone criteria on a scale of 0 to 10. Thus, if all the respondents had selected the extreme left position for a criterion-pair, the weighted average of Criterion #a of that pair would be 10 while that of Criterion #b of that pair would be 0. On the other hand, if all the five options for a criterion-pair had an even selection of 20% each, the corresponding weights would be 3 for Criterion #a, 3 for Criterion #b, and 2 for Neutral.

Table 2. The pairs of relative criteria, showed the weighted preference between the two contrasting choices.

In looking at these results, some of the preferences resonate with the stand-alone ranking of criteria captured in Table 1. There is a definite preference for a scalable solution that can support collaboration and distributed work processes over a single database solution, even if the single database involves easier setup, organization, and management. However, this preference is not as dramatic as the one for a BIM application that leverages the powerful documentation and visualization capabilities of a CAD platform, in contrast to a BIM-only application with more efficient modeling but less well-developed documentation and visualization tools. This reiterates the paramount importance of the ability to generate construction documents to AEC professionals and to a lesser extent of the ability to collaborate on large projects efficiently, even if this comes at the expense of less efficient modeling capability and more complexity in setup, organization, and management. As mentioned earlier, both these aspects are the strong points of Bentley’s BIM solutions.

But then, there is again a strong preference for fully automated change management and coordination, which is a key strength of Revit. Intuitiveness and ease of use—another area where Revit does better in comparison to Bentley—is also preferable, even if this comes at the expense of reduced scope and diminished capability to model complex forms. Of course, it would be ideal to have a solution that can model complex forms and is easy and intuitive, judging from the actual weights of these two contrasting choices—which makes it a good goal for BIM solutions to aspire to.

The most even distribution of preference came for the aspect related to model integrity. Almost an equal number of respondents preferred the modeling to be constrained to better guarantee model integrity, as the number of respondents who preferred a more flexible modeling environment where the onus of model integrity was on the user. This aspect of model integrity has often been discussed in AECbytes as being very important for downstream analysis of the model, and no solution currently guarantees complete model integrity, although Revit’s constraint-based modeling does slightly better at this than Bentley’s.

On the issue of electronic publishing and distribution, support for 3D PDF rated significantly higher than support for 3D DWF, although a large percentage of the respondents also expressed neutrality on this option. Bentley shares a strong partnership with Adobe and is actively promoting 3D PDF, while Autodesk has been aggressively promoting its 3D DWF format and building support for it across its applications, including Revit.  

Comparing Results for Different Segments (Segmentation Analysis)

In addition to analyzing the overall results of the survey, it is interesting to see if there are any significant differences in how respondents in different categories rated the various criteria. Let’s start by looking at how the respondents using or evaluating Revit differed from those who are evaluating or using Bentley’s solutions. The following graph shows the weighted average of each of the stand-alone criteria for three different categories—all the respondents, those using Revit, and those using Bentley—on the same 10-point scale used in Table 1. The weighted averages for the Revit and Bentley segments were calculated using the same method that was used to derive the weighted average of the stand-alone criteria for all the respondents.

Comparison of stand-alone criteria for all the respondents, those using Revit, and those using Bentley:

As you will see, the results for the Revit and Bentley segments are not dramatically different from the overall ranking of criteria. There are, however, some differences worth noting. For example, the importance of the availability of object libraries is a notch lower for Bentley users, and is preceded by the ability to support distributed work processes, work on large projects, and multi-disciplinary capability. Interestingly, the support for drawing production rates highest for the Revit users—its weighted average was 9.00 for this segment compared to 8.83 for all the respondents and 8.71 for the Bentley users. Another difference that stands out is with regard to the criterion of IFC compatibility: it ranks 12th for Bentley compared to a rank of 16th for all the respondents as well as for the Revit segment, indicating that Bentley users find it considerably more important than Revit users. It was not surprising to find that the overall low rating of the vendor market leadership criterion was even lower for the Bentley segment: 3.95 compared to 4.85 for all the respondents and 5.15 for the Revit segment.

Let’s move on to look at how these three categories rated the pairs of relative criteria, the weights for which were derived by using the same method described in Table 2. [If all the respondents had selected the “Definitely prefer (a)” option for a criterion-pair, the weighted average of Criterion #a of that pair would be 10 while that of Criterion #b of that pair would be 0. On the other hand, if all the five options for a criterion-pair had an even selection of 20% each, the corresponding weights would be 3 for Criterion #a, 3 for Criterion #b, and 2 for Neutral.] The differences between the Revit and Bentley segments compared to all the respondents are captured in the following set of graphs. One graph has been created for each criterion-pair.

Comparison of relative criteria for all the respondents, those using Revit, and those using Bentley:

As expected, the preference of the respondents is in tune with the strengths of their respective solutions for most of the criteria. Thus, for Bentley users, the preference for a scalable solution versus a single database solution is significantly higher compared to Revit users, as is the preference for 3D PDF support compared to 3D DWF support. Also, enhanced scope and better capability to model complex forms is rated as a little more important to Bentley users versus intuitiveness and ease of use, whereas Revit users have a definite preference the other way around.  What does come as a surprise is to find that the preference for a BIM application that leverages the power of a CAD platform versus a BIM-only application is considerably higher for Revit users (weighted average is 5.21) than for Bentley users (weighted average is 4.23), indicating that Revit users would like to see the application improve upon its documentation and visualization capabilities so that they match up to those provided by CAD-based BIM applications. 

A similar segmentation analysis was performed by creating two separate categories, one for small (firm size of 1-99), single-office firms and the other for the larger, multi-office firms. The following graph shows the weighted average of each of the stand-alone criteria for these two categories along with that calculated earlier for all the respondents. 

As before, the overall pattern of criteria ranking is quite similar across all three categories, with the exception of a few criteria. For the larger, multi-office firms, aspects such as the support for distributed work processes, ability to work on large projects, and multi-disciplinary capability rank high. In contrast, for the smaller, single-office firms, aspects such as smart objects, availability of object libraries, the quality of Help and supporting documentation, and the support for preliminary conceptual design modeling rank higher. What is common across the board is the need for BIM applications to provide full support for producing construction documents so that another drafting application need not be used—this was ranked first for all categories of firms.

The corresponding graphs for the relative criteria are shown next.

As you can see, one of the most significant differences between the smaller, one-office firms and the larger, multi-office firms that is evident from these graphs is the much higher preference of the latter for a scalable solution supporting collaboration and distributed work processes. In contrast, the smaller, one-office firms are almost divided on their preference between a scalable solution and a single database solution featuring easier setup, organization, and management. Both kinds of firms have a definite preference for fully automated change management and coordination, as well as for a BIM application that leverages the capabilities of a CAD platform, although it was a little surprising to find that the preference level for the latter was higher for the smaller, single-office firms than for the larger, multi-office firms. Likewise, the preference for intuitiveness and ease of use versus enhanced scope and better capability to model complex forms was more marked for smaller, single-office firms. The larger firms had a higher preference than the smaller firms for more flexible modeling as opposed to more constrained modeling that better guarantees model integrity. While both types of firms choose 3D PDF support over 3D DWF support, it was interesting to find that the PDF requirement was higher for the smaller firms than for the larger firms, who seem to be a little more open to the idea of collaborating using DWF. 

Additional Input Offered by Respondents        

Over 200 of the respondents took the time and the initiative to provide additional feedback on their requirements for BIM. While a full, or even an extensive, discussion of their inputs is beyond the scope of this report, some of the most frequently mentioned criteria are listed below.

  1. Simplicity and ease of use
  2. Design freedom
  3. Interference checking
  4. Ability to be localized—to capture local building codes and standards
  5. Integration with facilities management
  6. Integration with space programming and planning tools
  7. Integration with related disciplines such as urban design, landscape design, civil engineering, and GIS
  8. Compatibility with other BIM applications
  9. Improved link from design to fabrication
  10. Support for rapid prototyping/3D printing
  11. Ability for specification data development and management within the application
  12. Support for LEED
  13. The ability to leverage advances in computer technology such as multi-processing, multi-threading, 64 bit OS, etc.
  14. Faster performance
  15. Efficient use of computer resources
  16. Ability to work efficiently across a WAN
  17. Affordability
  18. Quality of technical support
  19. Security and access controls
  20. Ability to make multiple changes in real time by multiple operators
  21. Efficiency and speed of operations (how much time a user needs to create a good model)
  22. Rule systems for automating trivial tasks
  23. Ease of use for making custom components
  24. Ability to easily share content and standards from one project to another
  25. Manageable file size
  26. Quality assurance for information in the model

Conclusions

The results of this survey clearly indicate that the AEC industry is still very much reliant on drawings for conducting its business of designing and constructing buildings, which is why the most important requirement for BIM applications that has emerged—for all categories of firms and respondents—is the ability to provide full support for producing construction documents so that another drafting application need not be used. At the same time, AEC professionals also realize the power of BIM for more efficient and intelligent modeling by placing a high premium on smart objects that maintain associativity, connectivity, and relationships with other objects and the availability of object libraries. Comparing these requirements to the BIM applications that are available, the first is currently better served by a BIM application based on a CAD platform such as Bentley while the second is the strength of a BIM-from-the-ground-up application like Revit. This dichotomy is also evident in many of the other top requirements. Users want scalable solutions that support distributed work processes and large projects; at the same time, they also want automated setup, management, and coordination, reducing traditional CAD management tasks. They want a BIM application that leverages the powerful documentation and visualization capabilities of a CAD platform, but they also want fully automated change management and coordination. In short, they want the best of both worlds—all the capabilities of CAD that they have maximized the use of over the last few decades, as well as all the new capabilities that BIM applications have the potential to provide.

BIM as a technology is still in its formative stage and solutions in the market are continuing to evolve as they respond to users’ specific needs. The main requirements of users as identified in this survey are satisfied to varying extents by both Revit and Bentley, two of the leading BIM solutions available today. From the segmentation analysis results, it is clear that the extent to which a specific criterion is important to an individual user depends upon the type of organization the user is involved in, the type and size of projects, and the level of multi-disciplinary interaction required between teams and stakeholders. The criteria identified by this study, along with their relative importance to the diverse group of users that participated in the survey, can be a useful aid for firms in evaluating different BIM solutions to determine which one to implement.

Even though this survey was commissioned by Bentley Systems, its results should be valuable to all the BIM vendors as they plan on the future direction of their solutions. It also serves as a very useful benchmark for assessing the current state of the art of AEC technology to which future survey results can be compared to chart the course of change. I would like to take this opportunity to thank all those who took the time to respond to this survey, providing a minefield of information from which many useful insights will continue to be harvested.

About the Author

Lachmi Khemlani is founder and editor of AECbytes. She has a Ph.D. in Architecture from UC Berkeley, specializing in intelligent building modeling, and consults and writes on AEC technology. She can be reached at lachmi@aecbytes.com.

Have comments or feedback on this article? Visit its AECbytes blog posting to share them with other readers or see what others have to say.

 

 
© 2003-2007 Lachmi Khemlani, AECbytes. All rights reserved.
Site design by Vitalect, Inc