This article criticising the Nautical Almanac's calculator instructions was published in the Royal Institute of Navigation's quarterly Journal in 1994 (Vol 47, p 89) Though it was in the "Forum" section, it was so critical of the Royal Greenwich Observatory, that the editor had it peer-reviewed. (No changes were suggested.)
In 1989 the Nautical Almanac appeared with some calculator mathematics and a set of sight reduction tables. As best I can figure out, the tables were included because the good ol' boys of the US Power Squadron wanted a freebie. The tables were opposed by the RGO but the Americans insisted. Perhaps thinking that two wrongs make a right, the RGO contributed some ratty ideas of how to use a calculator.
The article is polite, as becomes a learned journal, but in fact the instructions are largely rubbish. If the RGO was going to exploit its Almanac - an annual publication required on ships by law - to propagate instructions, they could at least be correct.
Navigation is a field where experience matters and which has a low tolerance for error. There are myriad errors, the worst being the instructions for computing position. They are breathtakingly silly. In all the letters I have received from enthusiastic amateurs offering celestial navigation solutions there is nothing to rival them.
Despite this article, nothing was changed. It is the sort of arrogance the RGO displayed in the eighteenth century regarding the longitude and Harrison's clocks. It is an arrogance which stems from a legally enforced monopoly.
* * *
THE NAUTICAL ALMANAC'S FAULTY CALCULATOR INSTRUCTIONS
Mike Pepperday
The calculator instructions published in the Nautical Almanac since 1989 are impractical. The sextant corrections should include a passage correction and exclude semi-diameter; the celestial triangle solution is clumsy and not sufficiently comprehensive; the specified computation of position line is a radical departure from standard method and will not work; the computation of the fix omits error assessment and the directions for "iterating" the fix are eccentric and superfluous. The instructions should be re-written to conform with the practice of celestial navigation or deleted from the Almanac.
In 1989 the Nautical Almanac, published jointly by the UK and US Governments, appeared with two new inclusions: a discussion of calculator sight reduction mathematics and a set of sight reduction tables. The inclusions have reappeared every year since. The calculator instructions, presented on Nautical Almanac pages 277-283 and written by the Royal Greenwich Observatory, cover altitude corrections, position line (azimuth and intercept), and least squares fix.
Twenty years ago, the electronic calculator swept the trigonometric tables from the shelves of all the professions except one: navigation. Even today, standard navigation text books barely recognise the existence of calculators and whereas in other professions the educational institutions are in the vanguard of innovation, celestial navigation is being taught much as it was fifty years ago - with tables, not calculators.
Apparently the popular demand for calculator mathematics to be presented in the Almanac was nil.
This is not only a contrast with other professions but is something of a paradox as the past decade has seen a boom in sales of "navigation computers". These are ready-made pocket computers for navigation - essentially celestial navigation - at sea. Total sales by now must be several hundred thousand. At one stage in the mid eighties there were fourteen brands vying for the English-speaking navigator's dollar but the market has settled down and there are now five. They are the Celesticomp V, the CN2000, the Merlin II, the NC99, and the Petrel. They are available in the chandleries of the USA, Australia and New Zealand but generally not offered in Britain. All are pocket computers programmed in BASIC. All five contain sun, moon and Aries almanacs; four contain stars, and three contain planet almanacs. All compute the "least squares" fix.
Each computer is the outcome of a lineage of previous models reflecting years of customer feedback and competitive pressure. Though they vary in many details, they long since converged to a common operating procedure: you enter clock time and sextant altitude and the computer displays azimuth and intercept. After entering two or more sights, if you press the fix button you'll see the fix so formed. You can delete sights, enter more sights, and look at the fix at any stage. One brand limits a fix to forty sights; the others set no limit to the number of sights per fix. Position lines may, of course, be plotted in the usual manner.
Where did the computer makers find the mathematics to program their products? Since the late seventies (at least), the RGO privately published a series of papers, called "Technical Notes", which set out some of the astronomical mathematics. These "Notes" are available from the Observatory and some of them have been used as source material by some of the commercial computer makers. Effectively the navigation and marketing skills of the computer makers enabled the astronomical and mathematical skills of the RGO (among others) to reach and assist thousands of sailors. It was, and is, an effective process.
The American and British Almanac offices have never said why they decided to publish "customer direct" in the Nautical Almanac. Nor have they said why they decided to publish navigation mathematics (which is available in books off the chandlery shelf) rather than ephemeris mathematics (not so readily available) which would reflect their expertise. Whatever the intent, the publication has not been a success. They seem to have prompted no review, no public comment, no apparent acknowledgement at all. In short, despite the prestige of the Nautical Almanac, the instructions have achieved no acceptance.
There is a simple reason: they are impractical. In every sphere of activity, the introduction of electronic computation compels changes to traditional procedures. However, the departures from both traditional and modern practices which are set out in the Almanac are idiosyncratic and inappropriate. As far as computers in navigation are concerned, it is no longer early days. Although the navigation classes tend to ignore the commercially available computers, for years virtually every recreational sailor who navigates with a sextant has been using one. They are a standard against which anyone presuming to tell navigators how to do it, may be judged.
The major faults occur in Section 8 to Section 11 but there are problems of one sort or another throughout.
Section 1, the Introduction, states that the calculator, or "microcomputer" (the usual term is pocket computer or hand-held computer), should preferably be programmable. Quite right. In practice the computing of position line with an ordinary, non-programmable, scientific calculator is unworkable. Any number of people have promoted the idea which has never got beyond the lounge room. Even if such calculation is restricted to solving the bare astronomical triangle, it is just too lengthy and too strange for most sailors.
The potential user of these Almanac instructions would thus be someone who owns a programmable calculator or pocket computer. Perhaps a civil engineer planning to go cruising; or a ship's officer with a taste for trigonometry. Who the intended reader is, is not stated. Eight pages covering sight reduction could never be for novices and the academic style - passive voice, mathematical terminology - also presumes a certain background. Though the prospective readership is limited, knowledge is a good thing and for those interested, it would be handy to have a reference. To function as a reference such instructions would need to be reasonably comprehensive and to reflect the practice of (calculator) navigation.
The introduction says the astronomical data are assumed to be taken from the Nautical Almanac. This is probably unrealistic. While it would depend on individual enthusiasm, chances are that those who went as far as programming the fix would also program almanacs, at least for the sun and Aries. But in that case, the textbook which provides the almanac algorithms would also provide the other mathematics of sight reduction. That in turn raises the question: What is the point of including these mathematics in the Nautical Almanac?
Section 2 sets out the notation with sign conventions and numerical limits. It is clear and to the point. Abbreviations rather than symbols are used which is a help to readability. There are two quibbles. The concept of "apparent altitude" is superfluous when a calculator is being applied since only a sextant altitude and a final corrected altitude are needed. Also, the quantity defined as "index error" would be better termed "index correction," a correction being the amount to be added to an erroneous value to correct it. It is always clearer to speak of corrections rather than errors.
Section 3 is a discussion of how to convert minutes to degrees by dividing by 60. It is a funny thing to be explaining to navigators.
Sections 4 and 5 comprise a lengthy explanation of how to interpolate for GHA or Dec from two values taken from the Almanac. To interpolate means to find, by proportion, a value in between the ones listed. Rather than explain it, however, it should be omitted altogether. One of the computers in the eighties took this approach and quickly vanished. It is unnecessary bother. It is much easier to enter the v and d corrections. It might have been better to set out the arithmetic of these v and d corrections.
It would also have been insightful to show how Aries can be computed from its value at the beginning of the month, e.g.:
GHA Aries = 0.98565 (D-1) + 15.0411 x T + A
where D is the day of the month, T is Universal Time in hours, and A is GHA Aries at O hours on the first of the month. It is also possible to write a complete, Almanac-independent, Aries as an even simpler expression providing it is accompanied by a short list of month and year codes.
Corresponding formulae to compute the position of the sun are lengthier but would have greatly enhanced the usefulness of these mathematical instructions.
Sections 6 and 7 present a celestial triangle solution. That is, the computation of azimuth and altitude given lat, long, Dec and GHA.
It says "LHA = GHA + long" and then "add or subtract multiples of 360° to set LHA in the range 0° to 360°." This is poor advice. LHA is not of interest to anyone and may be left as it is.
Only where a result is for human consumption must it be brought within 0° and 360°. In such a case it is not necessary to go testing for whether it is within range and then adding or subtracting 360s till it is. Instead test nothing but simply subtract 360 x INT(N/360) where N is the value to be brought into range and INT has the BASIC language meaning of "integer value less than". This situation does not arise during sight reduction.
Directions are given to find azimuth and altitude by the spherical cosine rule and the five parts formula. The formulae are divided in an unconventional way with some intermediate parameters introduced. The reason for this is not stated. Then it says:
If X > +1 set X = +1
If X < -1 set X = -1
A = cos-1 X
where cos-1 is the inverse function of cosine.
X has been computed from spherical formulae so how could X possibly exceed 1? Normally it cannot. Perhaps it has something to do with the way the formulae have been divided. The Royal Greenwich Observatory has been in the business of manipulating spherical triangles for centuries so there must be some explanation. It ought to be given, if only because those "IFs" can be a nuisance to program.
If you didn't know what cos-1 was, would the statement about it make you wiser? What, actually, is an "inverse function"?
The choice of formulae to use to solve the celestial triangle seems to be to some extent a matter of taste but "IF" is always bad taste. IF-testing is intellectually sloppy, introduces the risk of overlooking some contingency, eats computer space, makes a program hard to read and is practically always unnecessary. For example the instructions
A = cos-1X.
If LHA > 180° then Z = A
Otherwise Z = 360 - A
could have been avoided with the single expression
Z = (cos-1X - 180°) x SGN sin LHA + 180°
where SGN means the sign of sin LHA.
That is still clumsy and the first 180° would vanish if the previous five parts formula had had its signs reversed so X was its reciprocal.
There is often a better way. Cosine rule and five parts formula are suited to some BASIC computers but calculators should apply the tangent formula. Where a result lies in the range 0° - 360°, a formula using tangent will give the unique result since arctan, unlike arcsin or arccos, is defined through 360°. It is not necessarily briefer but some calculators don't provide IF, SGN, or INT whereas such machines always provide the two-argument arctan, namely the rectangular-polar key. Final azimuth, Z, is given in a single expression (without the X or A above) by
Z = tan-1 [ sin LHA / (cos LHA sin lat - tan Dec cos lat) ] + 180°
assuming the ratio is not divided but directly evaluated with the two-argument arctan.
The computed altitude may be found by cosine rule or, if the calculator space is tight, via the sine rule by multiplying the other value which the rec-pol function yielded by cos Dec to give the cosine of the altitude. (This altitude will be unsigned.)
On computers with a BASIC which lacks the rec-pol function, the tangent quotient could be multiplied by the sign of the numerator, ie SGN sin LHA. There would be no IFs and certainly no impossible values exceeding 1 but there would be a disadvantage since a division requires the program (the programmer) to ensure that the denominator is never zero. It cannot be zero when computing azimuth but on a computer the formulae will usually be written as a general purpose subroutine for spherical triangle solution which means that the other applications (such as great circle, star identification, ephemeris computation) must take a possible zero denominator into account. The five parts formula also involves division but since the denominator is a value output from the previous cosine rule it would be practically impossible for it to be zero.
In sum, a reference to solving the celestial triangle might set out three possibilities. For most computers or calculators: tan formula followed by cos rule; for small calculators: tan followed by sine rule by-product; for BASIC computers without rec-pol: cosine rule and five parts formula. The azimuth formula should always be structured to come out in reverse so that with 180° added it falls automatically within the 0°-360° range.
Solving the celestial triangle does not finish finding the computed altitude. When Dec and GHA were found the semi-diameter correction was also determined. This should now be applied to computed altitude and not, as the Almanac directs, to the sextant reading (see below).
Sections 8 and 9 set out the sextant altitude corrections. There are two bouquets: the refraction correction given is the sensible formula - it works on altitudes from zero to 90° and few knew of it before this publication - and the relationship between moon's semi-diameter and horizontal parallax is presented which saves an entry if looking up the moon in the Almanac. There are three brickbats.
The first is the minor one that the concept "apparent altitude" is superfluous: the sextant reading has to be turned into the corrected altitude and this intermediate quantity is not needed. Mention made of "the height of eye above the horizon" should read "above sea level."
The second is more important: the correction for semi-diameter should not be applied to observed altitude but to computed altitude. The reason for this is so that a lower limb becomes effectively (administratively) a different body from the centre or the upper limb. It saves having the computer pester the navigator about which limb was observed; instead the user simply nominates the body - which includes the limb. All the commercial machines do this. It is actually more logical than the traditional way. Computers are logical sometimes.
The third deficiency is vital. The Almanac instructions have omitted what might be termed "passage correction". This is a correction to the observed altitude for the passage of the vessel during the observations - i.e., to advance or retire the position line over the time between the sight and the required time of fix. Passage correction in degrees is:
cos (course - azimuth) x (fix time - sight time) x Speed / 60
where time is in hours and speed is in knots. Advancing a position line is not traditionally thought of as an altitude correction but in fact this formula simply "runs up" or "runs back" the position line in the same way as the navigator has always done - by moving it in the direction of the course. It can be a large correction and all the commercial computers apply it. Other RGO publications set it out but these instructions in the Almanac omit it and, instead, account for the passage of the vessel by moving the DR position - a method which is quite inappropriate (see below).
In sum, the corrections to the sextant reading should be: index, dip, refraction, passage and parallax. The corrected altitude boils down to one long but fairly simple expression:
Ho = Hs + I/60 - .03 √h - .0167/tan (Hs + 7.31/(Hs + 4.4))
+ passage correction as above + HP x cos Hs
The Almanac also gives the pressure, temperature and oblateness corrections. Though usually ignored, it is reasonable that they be set out since a reference should be complete. Presenting them lets the reader decide and shows, if nothing else, why they may be neglected.
Section 10 , headed "Position from intercept and azimuth using a chart", is mistitled - in two senses. The section applies to the computed fix as well as a chart fix - it is so applied on the very next page - and the section does not actually discuss position. It discusses position line. The only mention of finding position is the uninformative last sentence: "Two or more position lines are required to determine a fix."
The first thing the Section says to do is estimate a fix position and then compute a DR lat and long at the time of each sight. This is incorrect.
It is the great virtue of the intercept method that there is no need to be fussy about DR position. DR uncertainty is of no consequence. Not even a hundred mile error would matter. Yet in the case of the example given in the Almanac the three different DRs for the three stars are only two to five miles apart!
Such precision not only involves much unnecessary computing but to find the fix on the chart - which is what this section is allegedly about - you'd have to plot the three DRs in order to plot the three intercepts. If you took seven shots you'd have to plot seven different DRs each with its own associated intercept. If you took twenty...
On the commercially available computers all intercepts are, naturally, from a single DR latitude/DR longitude. Apart from ease of plotting, the single DR position also means that where several sights of a body are taken, the intercepts are directly comparable and any erroneous reading is obvious. These machines accept any number of observations except for one model which limits the number of sights per fix to forty.
The Almanac instructions are an astonishing departure from standard procedure. No explanation or justification is offered. On the contrary it says: "The position is calculated..." as though it were the normal thing to do. It isn't and it won't work. The whole of section 10 should be deleted.
Before leaving it, however, we should note that the formulae it gives to compute the DR position are not suitable for that purpose. To compute dead reckoning find the latitude first then the longitude whereby the longitude formula uses, not the old latitude (as advised in the Almanac), and not the newly-computed latitude, but the average of the two. This is not for an exact result but for a consistent result. The navigator who updates DR will occasionally make a mis-entry and will require to undo it so the formulae should work the same for a negative time or distance as for a positive.
Section 11 sets out the mathematics for computing the fix by the "least squares" method. This yields the "best fit" fix when there are more than two position lines. As presented it is fairly concise but (a) it can bear some simplifying, (b) an extra calculation for accuracy should be added, (c) the concoction about "iteration" should be excised.
Instead of finding A, B, C, find n, B', C' where n is the count of the number of sights (so far - i.e., at any stage of entering sights) B' is the sum of sin 2Z, and C' is the sum of cos 2Z. Find D and E as shown. Also find F where F is the sum of p2. F is for error estimation and is set out in other RGO publications but has been omitted from these Almanac instructions. Every time a sight - i.e., an azimuth-intercept pair - is computed, this set of summations n, B', C', D, E, F is added to. For deletion of a sight from the fix, subtract the values.
To view the fix formed from the lines entered at any stage, compute B = B'/2, C = (n - C')/2, A = n - C, and compute G as shown in the Almanac then
dlat = (DC - EB) / G
dL = (AE - BD) / G
S = 60 x √[(F - D x dlat - E x dL) / (n - 2)]
where S is the "standard error", in miles, of the fix which is given by:
lat = DR lat + dlat
long = DR long + dL / cos lat
And that is the end of the calculation. Essentially, it is a mathematical way of drawing the lines and marking the fix in the middle of them, much as navigators have always done. Sights can be deleted or more added. The definition of best fit fix - i.e., the point where the sum of the squares of the observational discrepancies is the minimum - should be stated and illuminated with some discussion of the problem of systematic error: the example given has three bodies well distributed around the horizon but there is no explanation of why. Error S, without which the navigator has no measure of the quality of the fix, would also bear some discussing. Essentially it may be viewed as a sort of average error: if it is larger than a mile or two look for an explanation.
But that is not the end of the calculation for the Almanac. According to the Almanac offices of the United States and Great Britain the above computation does not give the fix but only a new estimate of the fix. After a century of use at sea one would have thought that if the intercept method were deficient someone might have noticed. Yet the Almanac instructions simply presume it is deficient. The whole calculation is, allegedly, now to be repeated with this new "estimate" instead of the previous one. As well as repeating the above calculations for the best fit fix, this includes computing yet another set of DR positions for each and every sight!
This eccentric "iterative" procedure is also presented in other RGO publications - there also as though it were normal. Even land surveyors, who are at home with iterative procedures and who look for an accuracy of metres rather than miles, do not iterate a celestial fix. It is a extraordinary complication and if the Royal Greenwich Observatory and the United States Naval Observatory weren't promoting it, year in, year out, in their official Nautical Almanac, it would not warrant serious discussion.
The Almanac's fix example is defective in two other respects. Firstly, real navigators who compute least squares fixes enter sextant altitudes in degrees and minutes. The Almanac example (revised every year) gives for entry corrected altitudes in decimal degrees, perhaps with the idea that this would help the programmer. It probably won't because the programmer using this example as a test of a least squares routine will already have the altitude corrections programmed and will find it necessary to insert some corrections backwards to reconstruct some sextant readings.
Secondly, the example consists of three bodies with one sight each whereas several sights of each body should be taken. Back in Section 1, the Introduction, the possibility of removing doubtful sights from the fix is mentioned. How can a doubtful sight be noticed if there is only one? Single sights are often presented in examples and it tends to leave the wrong impression. If the example is only to illustrate a principle then that should be stated.
* * *
The Nautical Almanac is printed annually and distributed in tens of thousands of copies. It has been around for a couple of centuries and though it changed a few times to reflect evolving technology and new ideas, it has probably always confined itself to astronomical positions. The inclusion of sight reduction mathematics and of sight reduction tables would therefore appear to be significant innovations.
As far as this author is aware there was no prior discussion of, or public notification of, either innovation and in the years since there has appeared no explanation for them and no review of them. Except for a couple of passing mentions of the sight reduction tables in the American press they seem to have been ignored.
The three-century history of the Royal Greenwich Observatory has been a colourful one with seasons of flowerings and witherings. Through it all the problem of - the necessity of - navigation by the heavens remained. Perhaps the RGO is now in terminal decline. It was created to help navigators: the Nautical Almanac was its triumph and its ongoing raison d'être. But sextant navigation has been irrelevant for ships for a decade and for aircraft for much longer. These users carry the printed Almanac only to satisfy obsolescent regulations. As long as recreational sailors needed celestial, the RGO's publication of their astronomical "Technical Notes" provided a service (albeit effectively a public subsidy to private yachtsmen) which kept up with the times and was in line with the RGO's purpose.
The inclusion of calculator instructions in the Almanac could be criticised for competing directly with private enterprise texts on calculator navigation. As it stands private publishers have nothing to fear and owing to its brevity and limited scope this would probably never be a serious objection. Nevertheless, now that the GPS has brought the History of Navigation to an end, sextant navigation is becoming irrelevant to yachts and soon there will remain only the hobby (lounge room) market. Perhaps consideration should be given to privatising the RGO's navigation publishing functions.
Private or public, as long as the Nautical Almanac is being printed and being bought, a reference setting out appropriate, practical mathematics could be a convenience and could influence celestial navigation classes (where there is probably an enormous potential readership). Such instructions would take into account how navigation is thought of and how it is performed. Any deviation should be regarded as extraordinary and would need to be explicitly justified. In its present form the Nautical Almanac's calculator segment is unsatisfactory and should either be set to rights or deleted.