Author

Topic: Numerical precision of computed avalues

Paul

posted May 22, 2007 06:46 PM
Throughout the cel nav course avalues in example problems are computed to a precision of a tenth of a minute (0.1'). But this involves an arithmetic combination of an Ho possibly known to 0.1' and a tabulated Hc value from HO 249 known only to 1' of precision. I believe that the computed avalue can be no more precise than 1', the less precise of the two numbers used in the arithmetic.
From: Muncie


David Burch

posted May 22, 2007 10:24 PM
Yes in a sense this is true using the Pub 249 tables, but it is not quite that bad even with these tables. As we discuss in the materials, you have the corection of dec minutes which you can interpolate to the tenth, and then with the solutions given to nearest 1' meaning ±0.5' I believe the final results are somewhat better than just ± 1.0'.
However, we need to keep all the decimals along the way so we do not end up losing precision we do not have to. For example, it would be a big error to just record Hs to the nearest full minute.
We also relay on taking many sights and averaging them to help pull out a bit more precision, and then finally, there is no need at all to rely on Pub 249. There are other tables such as Pub 229 that give solutions precise to the 0.1' or we can compute the Hc and zn directly with trig functions and not have any uncertainly there at all.
So another answer to the question is we are at this stage developing our techniues and after we become comforatble with the full process  of which the use of Pub 249 is the easiest  we are then prepared to move on for more precision with other methods.
In the end if we stick with the standard almanac data (± 0.1' for the most part) and we are practiced at good sight taking, and we do other things right, then we might hope for a final result accuracy of about ± 0.5 nmi. Much better than that would be fortuitous but much worse than say 2 nmi means we are not doing our best.
From: Starpath, Seattle, WA


