Subject: Re: double line bug & nautical miles<br/>
Date: Nov 25, 2001 @ 16:12<br/>
Author: Grant Hutchison (&quot;Grant Hutchison&quot; &lt;granthutchison@...&gt;)<br/>
<a href="4669.htm">Prev</a>&nbsp;&nbsp;&nbsp;&nbsp;Post in Topic&nbsp;&nbsp;&nbsp;&nbsp;<a href="4682.htm">Next</a>&emsp;[<a href="topic4627.htm">All</a> Posts]<br/>
<a href="4675.htm">Prev</a>&nbsp;&nbsp;&nbsp;&nbsp;Post in Time&nbsp;&nbsp;&nbsp;&nbsp;<a href="4677.htm">Next</a><br/>
<br/><br/>
<div id="ygrps-yiv-2133699887">Michael:<br/>
<blockquote><span title="ireply"> &gt; & do you agree i can run the computation out to the 5th decimal<br/>
 </span></blockquote>place without any loss of authenticity by simply using the full 5<br/>
decimal places of the submillimetric nautical mile standard<br/>
<br/>
Your difficulty here is getting the other figures to the same<br/>
accuracy. To calculate the radius of a parallel of latitude, correct<br/>
to the nearest couple of microns to match the definition of the US<br/>
nautical mile, requires the ellipsoidal trig to faithfully carry *13<br/>
digits* - because you&#39;re measuring a very large thing to a very small<br/>
scale.<br/>
That&#39;s beyond the reach of my little calculator, and I&#39;m not sure I<br/>
completely trust my computer to faithfully sum the trigonometrical<br/>
series to enough significant digits. Sure, they sprawl reassuringly<br/>
across the screen, but just square the square root of two and you&#39;ll<br/>
sometimes get an answer like 1.999999999798.<br/>
<br/>
Grant</div>