Various studies on different grade-level documents have identified technical weaknesses of readability formulas; thus, experts have referred to the "inherent unreliability" of readability formulas.

Readability formulas produce fluctuations in scores for the same text and estimate the difficulty of text differently. Formulas vary in which attributes of words and sentences that they take into account and how they measure them. When you apply different readability formulas to your text, you can expect slightly different results on the same text. Sometimes the differences are large enough to confuse you. Depending on which method you use and how you use it, readability scores for the same text can differ by two, three, or more grade levels.

Different formulas produce different taxonomical results. For example, literacy specialists warn that Flesch-Kincaid scores underestimate the actual reading grade level because the scores are several grade levels below what other readability formulas output.

NOTE: The Flesch-Kincaid is included in many versions of well-known word-processing packages such as Microsoft Word and WordPerfect, and it produces a grade level score. Be careful not to confuse the Flesch-Kincaid with the other Flesch test, which is called the Flesch Reading Ease formula. The Flesch Reading Ease Scale (also known as the Flesch Index or Readability Score) is a score from 0 to 100 that you interpret by referring to a chart. For this formula, higher scores mean easier to read. For example, a score of 70 to 80 means "fairly easy" and approximately seventh grade reading level.

Computerized scoring also outputs unreliable measurements of text. Some computerized assessments using the same formula produce different scores for the same text. Additionally, if you use a computerized formula, you need to prepare your text first to avoid misleading results.

Grade level scores from readability formulas are not precise.

Most readability formulas produce a grade level score, such as "5th grade." Some formulas add a decimal, such as "5.3 grade." Scores with a decimal point are not as precise as they sound, for several reasons:

1) Scores can vary greatly for the same text. As we saw in the previous section, grade level scores for the same piece can differ by formula, and computer programs that use the same formula can yield different results.

2) Scores have a margin of error. Grade level scores are approximations.

3) Scores can differ depending on which part of a document is scored. Sometimes readability scores are based on scoring the entire document. Other times, especially for long documents, readability formulas base their scores on samples drawn from the document. Usually, the formulas base the average score across three, 300-word samples. Readability formulas may output different results for the same document if they select different samples throughout the document.

Readability formulas are not perfect in recommending reading levels, plus it is easy to misinterpret the results. For example, you might think from the name, "readability formulas," that the formulas measure reading ease or comprehension – but they do not.

What does a readability score actually mean?

Does it mean that people need to have at least six years of schooling to read and understand the book? Does it mean that people with a 6th grade education can read and understand the book? Does it mean that words and sentences in the book are roughly the same average length as words and sentences in textbooks for sixth graders?

Why not?

Because readability formulas do not measure how well people understand material. Readability formulas measure the difficulty of individual words and sentences (based on length), which is only one of many factors that affect comprehension and usability of written material.

1) Years of education is only a blunt indicator of reading skills. Some people who have completed 6th grade struggle to read at all, while others may can read at the high school or college level. In general, adults tend to read several grade levels below the highest grade in school that they had completed.

2) Reading skills change over time. For example, certain literacy skills can decline substantially in later life.