Skip to main content
Solved

Decimal precision in results grid


Forum|alt.badge.img

All, we have fields for a graphic’s width and height. In the results grid I have a formula to calculate the Square Inches; however, it’s rounding to 0 precision (no decimal places). How do I extend this Square inches calculation to 3 places?
 

=CInt([InventoryItem.UsrCFCHeight] * [InventoryItem.UsrCFCWidth])

 

Best answer by Naveen Boga

@swartzfeger   In the above formula, you're using CINT, which converts to an Integer type and won't include any decimal places. This conversion isn't necessary if you're using default values of 0.00 for those fields.

If you still need to perform a conversion for some reason, please use the CDEC/CDBL function instead and check the results.

Below is the screenshot for reference.

 

 

View original
Did this topic help you find an answer to your question?

3 replies

Naveen Boga
Captain II
Forum|alt.badge.img+19
  • Captain II
  • 3382 replies
  • Answer
  • June 4, 2024

@swartzfeger   In the above formula, you're using CINT, which converts to an Integer type and won't include any decimal places. This conversion isn't necessary if you're using default values of 0.00 for those fields.

If you still need to perform a conversion for some reason, please use the CDEC/CDBL function instead and check the results.

Below is the screenshot for reference.

 

 


martinxfe
Jr Varsity II
Forum|alt.badge.img
  • Jr Varsity II
  • 22 replies
  • June 4, 2024

Try CDbl instead of CInt. Integers don’t like dots.

You could also try the Round function under Math. Round( [insert calculation] , 2 ) should give you at least a precision of two.


Forum|alt.badge.img
  • Author
  • Freshman II
  • 129 replies
  • June 4, 2024
martinxfe wrote:

Try CDbl instead of CInt. Integers don’t like dots.

You could also try the Round function under Math. Round( [insert calculation] , 2 ) should give you at least a precision of two.

Thank you Naveen and Martin, CDbl worked! (I should’ve known this, Data Types 101!)


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings