Helwig F. Van Der Grinten

Master Mariner

The Accuracy of Celestial Fixes as Compared to NAVSAT Positions
Navigation Journal of the Institute of Navigation, Vol 22, No2, Summer 1975


FORTY-SIX CELESTIAL LINES of position obtained aboard a ship at sea are compared with the position as interpolated between NAVSAT fixes. Least squares adjustment of the combined celestial lines of position yielded a fix with an essentially circular error ellipse. The directional distribution of stars was well balanced. The NAVSAT positions and the combined celestial fix were found to be substantially in agreement. Using the NAVSAT positions as a reference the mean error of well balanced celestial fixes as derived from the data and the method of least squares can be expected to be less than 0.8 nm for a 3-star fix and 0.1 nm for a 46-star fix. These results permit the construction of probability contours around well balanced celestial fixes consisting of various numbers of observations.


In the march of progress towards better navigation a great deal of attention is given to proving new methods; but little is done to establish the accuracy of the old, once the new system is adopted. This paper is an attempt to establish how good the best is, and was. Discussions of this sort often involve difficulties with the words accuracy, precision, and reliability. Accuracy is generally considered to mean how close a given measurement comes to the truth. Precision means the increment by which repeated measurement will differ. Precision, thus, implies repeatability without regard to the truth of the measurement. Reliability extends these concepts to include the human decision making process. Reliability implies probability or the risk involved in accepting or rejecting a given measurement. Because the celestial fix and the NAVSAT position are obtained by radically different methods it can be said that if they agree, then both have measured the true position accurately. This is the premise of this study.

The Data  

During September of 1972 aboard R/V THOMAS G. THOMPSON (AGOR-9), forty-six star sights were obtained in the North Pacific over a 10 day period for comparison with the position provided by a Magnavox 706 (AN/WRN-4) Transit Satellite receiver and computing system. This data was obtained as part of Ref. 1. Fourteen NAVSAT fixes were used in the comparison. The maximum elevation of these fourteen passes ranged from 9° to 68° with a mean of 33°. The fixes were recorded by hand to the
nearest tenth of a minute in both latitude and longitude after each pass and the best estimate of the ship's true course and speed as determined by the mate on watch was generally entered into the system before the pass. A check of the NAVSAT positions
obtained while THOMPSON was berthed in Seattle resulted in a standard error of 0.03 nm for 29 passes which ranged between 10° and 75° in maximum elevation.

The celestial observations were taken by the author using a standard light metal Plath sextant with a 4-power scope. All adjustable errors in the sextant were eliminated as nearly as possible using the standard shipboard techniques (see Ref. 2). Altitudes were recorded to the nearest tenth of a minute. Time was determined to the nearest second by starting a stopwatch at the time of the observation and stopping it when reading the digital time presentation of the NAVSAT set. The stars observed were
those recommended H.O. 249, Sight Reduction Tables for Air Navigation (Ref. 3).  Altitudes ranged from 23° to 59°. The astronomical coordinates for the stars and the sextant altitude corrections were taken from the Nautical Almanac (Ref. 4). The sights were taken during seven different twilights over the ten day period. No sights were deleted from the data or arbitrarily corrected for gross errors or blunders. The general track of the vessel during this time was south from 42° N, 155° W to 26° N, 155° W.  During this time the seas were generally rough, the sky cover ranged from clear to overcast, and the horizon visibility was generally good.

Hand plotting was used to determine the ship's position at the times of the celestial observations. This was accomplished by establishing a dead reckoning track from one satellite fix to the next using course and speed by log. The current which acted on
the ship during that time was obtained from the difference between the DR position and the second fix. This current was applied to all DR points giving a corrected track which passed through each satellite fix. The times of the star sights were marked off
along this track and the latitude and longitude of these points was used in calculating the computed altitude and azimuth of the star in the Marq St.-Hilaire method of altitude intercept sight reduction (see Ref. 2).

In this study the adjusted NAVSAT position takes the place of the assumed position.  Thus the celestial line of position can be transformed on to an x,y-coordinate system where the origin, defined here as the NAVSAT position, always moves with the ship
along the corrected tract as determined by the satellite fixes (see Figs. 1 and 4).  Computers were used to reduce the celestial data to intercepts and azimuths, and to perform the least squares adjustment of the combined data (see Ref. 1).

Least Squares Adjustment  

By using the altitude intercept method of sight reduction the celestial lines of position can be utilized for least squares adjustment. This is the process of determining the most probable position so that the sum of the squares of the deviations ( vi2) is a minimum. Least squares adjustment is only one of the possible methods for resolving multi-observation fixes. Reference 7 describes the method of bisectors which is most useful for resolving a well balanced three star fix by hand plotting. Least squares has the following advantages over the method of bisectors:

1. The solution lies closer to all of the lines of position collectively.

2. A forty-six star fix is resolved with nearly the same ease as a three star fix by using a digital computer.3. A standard error ellipse can easily be constructed from the results of the computation.


Fig. 1 – Geometry of the celestial line of position (LOP) showing its relationship to the NAVSAT position at 0 and the most probably position at T. (after Ref. 5)

The observation equations are derived from the following relationships, see Fig. 1 and subsequent List of Terms:

and thus the general relation is 

where the trigonometric functions are assumed to be calculated without error. The subscript i has been attached to the terms A,  h, and v to indicate the number of the observation, i.e. where i = 1 the terms have the values determined by the first star sighting. In the form of observation equations, this relation becomes:

Placing the observation equations in matrix form gives:

In the method of least squares the coordinates x and y are calculated by minimizing the matrix V'V.
This is accomplished by solution of the normal equation (after Ref. 6):

   X = (A'A)-1 A'L

where the prime indicates the transpose and the exponent indicates the matrix inverse.

Residual Dip and Azimuth Factor  

Navigators have long recognized that the quality of a celestial fix depends heavily upon controlling errors in sextant altitude and in providing a well balanced distribution of star azimuths. For the purpose of this discussion all errors in altitude will be called residual dip and the imbalance of azimuths will be called the azimuth factor. Residual dip in this sense includes all of the unaccounted for errors inherent in sextant observations.

The azimuth factor can be evaluated by assigning a magnitude of unity to each azimuth, adding them vectorially and dividing by the number of observations. Thus where the observations are perfectly balanced when f = 0. and totally imbalanced when f = 1.

An azimuth histogram of the data appears as Fig. 2. This gives an f = 0.24. The red area indicates where observations should be deleted from the data to improve the azimuth balance.

Fig. 2 - Azimuth histogram for all data. Green area indicates where sights must be deleted from the data to achieve symmetry.

The average residual dip can be determined from Eq. (2) by matrix subtraction:

V = L - AX

and then dividing the sum of the elements by n gives

Figure 3 shows the residual dip histograms and the fitted normal distribution curve for all of the observations before and after correction for the average. The distribution is somewhat improved by the correction. Since the author does not consider himself to be unusually proficient with the sextant, Figure 3A may be taken to be typical of the average navigator.

Error ellipses  

A three dimensional picture of the normal distribution of two dimensional errors, looks like a bell-shaped mound. The focus of the points on the surface of this mound where the slope is the steepest describes a curve which is commonly called the standard error ellipse. The following equations for constructing the standard error ellipse are from Refs. 8 and 9.

Fig. 3- Residual dip: (A) before and (B) after correction for v

The weight coefficient matrix is:

The standard error of unit weight is:

The length of the semi-major and semi-minor axis respectively are:

The azimuth  of the semi-major axis measured clockwise from north is found from

where the proper quadrant of the angle 2 and the octant of the angle 20 is found in Table 1. An additional value of interest is the mean error:

The results of the computations carried out on the data of this paper are summarized in Table 2 and Fig. 4. The so called uncorrected results are for the observations where the altitudes have been corrected only for height of eye and refraction from the tables in the Nautical Almanac. The remaining columns indicate second order corrections which were applied to the data after the uncorrected computation. The correction for azimuth factor was carried out randomly deleting observations from the overweighted sectors shown in Fig. 2. These results indicate that correcting the data for residual dip and azimuth factor improves the quality of the data only slightly. It should also be noted that the origin lies inside all of the ellipses except the one for the uncorrected data. If it can be shown that the uncorrected ellipse does not deviate significantly from being centered at the origin then the same conclusion would apply as well to the other ellipses.

Fig. 4 - Error ellipses for the celestial positions as listed in Table 3. The NAVSAT position is at the origin.

The error factors from Table 3, compiled from Refs. 9 and 10, can be used as multipliers of the standard error ellipse forming probability contours about the most probable position. If one were to construct these contours about the most probable position for the uncorrected data (Position # 1) in Fig. 4, the origin of the coordinate system would be found to lie on the 40% contour. This means that the NAVSAT position can be expected to lie inside the 40% contour in four out of ten similar trials (or outside in six out of ten) given that the NAVSAT and celestial positions are exactly the same and that any differences are purely normal random errors.

Reference 1 provides the same results by use of the more rigorous analysis of variance. The finding there bears out the conclusion that the data does not reveal any statistically significant differences between the positions provided by the two navigation methods.

Reliability of Celestial Fixes  

Since the foregoing suggests that the NAVSAT and celestial data are in reasonable agreement; by using the NAVSAT position as a reference, an estimate on the reliability of celestial fixes can be made.

It is generally accepted that the mean error (σm) is a satisfactory approximation of the two dimensional standard error of observations particularly when the standard error ellipse is nearly circular (see Ref. 8). Substituting Eq. (8) into Eq. (12) gives:

Substitution from the computations of the forty six

Fig. 5-Mean error of well balanced celestial fixes as a function of the number of observations (n).

observation uncorrected for average residual dip gives:

This function represents the mean error which may be expected in celestial fixes obtained by the average navigator and having nearly the same geometry (azimuth factor and ellipse axes ratio) as the combined forty-six star fix obtained here but having differing numbers of observations. Figure 5 is a graph of this function. The data points are values of m calculated from observations randomly selected from the data of this paper. These observations were selected so as to limit the azimuth factors of the fixes to less than 0.24. These points can be seen to roughly conform to the curve.

Table 3 can be used, together with Fig. 5, to determine the probability that the true position will lie within a given distance of the celestial position. For example, the probability that the true position lies within 1.1 miles of the position determined by seven well balanced star sightings is 99%. In other words, if the navigator accepts such a fix with this radius as being true; he has a 1% chance of being wrong.


It has been shown that it is reasonable to assume that the NAVSAT and celestial positions are not systematically different in the Pacific Ocean north of Hawaii. It can thus be concluded that both navigation methods measure the true position in this region.

The NAVSAT radial error for deep water operations reported in Ref. 11 is 0.06 nm. This yields a standard error which will account for only a small part of the error shown in Fig. 5; thus, the reliability of a fix as determined by Fig. 5 and Table 3 should be regarded a slightly conservative estimate.

It is hoped that this study has served to reassure the navigator that NAVSAT and celestial fixes are comparable, and moreover, that the reliability of celestial fixes can be quantitatively ascertained.