Skip to main content
Solved

Decimal precision in results grid

  • 4 June 2024
  • 3 replies
  • 42 views

All, we have fields for a graphic’s width and height. In the results grid I have a formula to calculate the Square Inches; however, it’s rounding to 0 precision (no decimal places). How do I extend this Square inches calculation to 3 places?
 

=CInt(CInventoryItem.UsrCFCHeight] * tInventoryItem.UsrCFCWidth])

 

3 replies

Naveen Boga
Rank
Userlevel 7
Badge +19

@swartzfeger   In the above formula, you're using CINT, which converts to an Integer type and won't include any decimal places. This conversion isn't necessary if you're using default values of 0.00 for those fields.

If you still need to perform a conversion for some reason, please use the CDEC/CDBL function instead and check the results.

Below is the screenshot for reference.

 

 

Naveen B
martinxfe
Rank
Userlevel 3
Badge

Try CDbl instead of CInt. Integers don’t like dots.

You could also try the Round function under Math. Round( [insert calculation] , 2 ) should give you at least a precision of two.

S
Rank
Userlevel 5
Badge

Try CDbl instead of CInt. Integers don’t like dots.

You could also try the Round function under Math. Round( [insert calculation] , 2 ) should give you at least a precision of two.

Thank you Naveen and Martin, CDbl worked! (I should’ve known this, Data Types 101!)

Reply


Terms & ConditionsCookie settings