Last week I wrote a blog post titled Logic & Fallacies in Genealogy. That post was an offshoot of a presentation that I have been working on for several months highlighting the potential errors in logic that genealogists may encounter during their research. At the end of that article I promised a follow-up article on how AI introduces additional errors into your research and how to deal with them. As I have been using AI more frequently in my research I have noticed that the AI platforms frequently come to conclusions without the supporting date. I asked the AI platform to identify the errors in logic that have occurred as it produces conclusions. Sometimes the AI platforms have caught their errors but the majority of the errors were caught by me. I am working with the AI to catalog the errors and develop ways to resolve or reduce their frequency. The results have been interesting.
Let's start at the beginning. I have found it helpful to upload various background documents prior to starting a research project in AI. These documents set the stage for what is to come. The background documents may include a locality guide to provide background on the resources available for the area, the role that AI will take in the process (research assistant) and its expertise, and a list of logical fallacies to be aware of (see the previous post for the list). I then ask AI to produce a research plan, research log, and a tracking sheet of the errors that occur during the research project. Each of these documents is updated regularly throughout the process.
I will be focusing on the error tracking sheet in this blog post. The following are the result of three individual projects spanning several weeks of research in multiple sessions.
Foundations: What the Error Tracking Sheet Established
1) Reasoning from Unverified Readings
This was the most common and costly error across the projects. This error consisted of the AI building structural claims on readings that were marked uncertain and never tested. Examples include misread digits (reading a year, age, or date incorrectly), misinterpreted initials or letters, and outlier readings treated as significant (a last name misspelled or incorrectly listed in one record).
The fix: Ask AI to list all information and compare it in a table. Ask AI to place uncertain readings in brackets [...]. This gives you the opportunity to decide on the correct interpretation.
2) Jurisdiction Assumed Rather Than Established
This error happened frequently during these research projects. The AI repeatedly inferred jurisdiction from the nearest named place rather than the subject's documented location. For example, parish records indicated that the individuals attended St. Augustine Catholic Church in Minster, Ohio. The AI inferred that this meant the individuals lived in Minster, Auglaize County and the AI created research plan focused on Auglaize County records. However, the individuals lived across the county boarder in Shelby County. Additionally, Auglaize County was established in 1848 from Mercer County. The AI did not consider that early parishioners would have records in Mercer County.
The fix: Township, county, parish, and birthplace are distinct fields that must be independently verified. Clearly identify the timeline of locality formation, records locations, and potential for living in different jurisdictions.
3) Predicting Document Content from General Practice
AI repeatedly assumed that records would contain specific information such as age, heirs, parents, and place of birth, based on previous records. For example, the AI assumed that marriage records would have the parents' names or that naturalization records would have the person's place of birth. These predictions failed multiple times based on local record keeping practices. This demonstrates the need to confirm record structure before applying methods.
The fix: Do not allow the AI to assume record content without reviewing the specific records. Provide a baseline rule that records need to be reviewed prior to assuming content.
4) Argument from Ignorance
Nulls or negative results were repeatedly treated as evidence about the past rather than evidence about the finding aid or record. Every negative finding has three possible explanations: the event did not happen; it happened but was not recorded; it was recorded but the finding aid cannot reach it. There were several cases where this occurred in these projects. One was during a cholera epidemic. The parish death register did not list the several hundred people who died of cholera during 1849-1850 with the burials so AI assumed that these deaths were not recorded. On the contrary, there was a separate list which just listed the names and month of death for these victims.
The fix: Make sure the AI is aware of the types of records available for an area. Are the church records more or less complete than the civil records? Are there overlapping volumes where the same dates are recorded in more than one place? Are there actual gaps in the record keeping?
5) Unexamined Premises About Sources
Assumptions about what a record series must contain (or does not contain) were repeatedly overturned by sampling adjacent pages. A property observed on one page is a property of that page only and is not the rule for all other pages. This error may occur when different people are providing individual pages, i.e., a new clerk or priest versus the previous clerk or priest or a different enumerator in the census.
The fix: Let AI know that the handwriting has changed or the format of the entries have changed as you review pages in the records. Different formats may include more or less information than previous record formats.
6) Pseudo-replication
Multiple observations depending on a single contested reading were counted as independent witnesses. One example that occurred frequently was establishing the death certificate, obituary, and headstone as individual pieces of evidence when there is a high likelihood that the informant on the death certificate also provided the information for the obituary and the headstone.
The fix: Establish a rule to count witnesses, not documents, and state certainty against the weakest link.
7) Failure of Execution
This is not a fallacy per se but it does cause research problems. This consists of naming a check but not running it and then using the ledger to outrank the evidence. The AI may suggest that a specific record be reviewed to check the accuracy of information from another document based on the assumption the record will have the information. It then takes the assumption that hasn't been proven and treats it as evidence.
The fix: Either ensure that all suggested records are reviewed or indicate that the specific record has not been reviewed and that no conclusions can be inferred from it until that check has been performed.
Additional Errors that May Occur
Conclusion by preponderance of trees
I will provide information from online family trees as a baseline document for a project. These may include family group sheets with sources or screen shots from FamilySearch, Ancestry, etc. However, I always provide a caveat with this information highlighting that they are online trees and subject to error. I also ask AI to review them and point out all conflicts, problems, and unsupported information prior to moving forward.
Name similarity as identity
AI may interpret people with the same or similar names as a single individual. As the researcher, you need to point out when there are several people with the same name in a locality and that they may appear in the same records. Make sure the AI is informed when information about an individual is being added as information versus as a specific detail for the research subject.
Age arithmetic as a fact
AI will do the math if you provide a record such as a death record that states the person was 76y 5m 13d to determine the birth date. I have had records where the birth record was a couple days, months, or even a year off from the math and AI inferred they were different individuals. Remember, you are the researcher and you make the final decision on what is the correct information.
Naming convention overreach
AI is aware of typical naming conventions and will use it to invent grandparents or other relatives. Be aware of this and tell AI that names will be based on the records researched and that a naming convention is a potential research clue, not a fact.
Confident sounding plausibility
AI will convince itself that something is a fact because it is plausible. I always provide AI with a rule that conclusions are based on the quality of the record. The AI is instructed to determine if the source is original, derivative, or authored; the information is primary, secondary, or undetermined; and the evidence is direct, indirect, or negative. The AI is also told to assess the quality of any conclusions it makes by classifying it as possible, plausible, probable, highly probable, proven, or disproven.
Invented citations
AI can and will invent citations that fit the conclusion. Make sure that you look up every citation the AI presents and tell the AI when something does not exist. It is your responsibility to push back and not accept everything the AI gives you.
Plausible transcription of illegible text
AI does a good job of transcribing text but it does depend on the quality of the record to begin with. It will make up words to fill in the gaps if it is allowed. I once had an AI produce an entire paragraph in a Will that was not actually there. When having AI do transcriptions you should tell it to produce the transcription exactly as the document is written, maintain all punctuation, line breaks, and spelling errors, and indicate any uncertain text with brackets [...]. You can then help the AI by providing your reading of any bracketted text or indicating where it made errors.
Safeguards that Actually Work
- Label speculation immediately
- State refutation conditions in advance
- Record negative findings with reasons they may be false
- Survey surrounding pages before accepting nulls
- Control for informant reliability
Pre-Statement Checklist
- Is any part built on an uncertain reading? Is the uncertainty marked on the claim?
- Does the evidence require a list or comparison table rather than a selection?
- Is jurisdiction established from the subject’s location?
- Is the AI predicting record content? Do the fields exist in the record?
- Is the AI weighting a null? Has the finding-aid's/record's coverage been established?
- Is the AI generalizing the record content from one page or enumerator?
- How many independent witnesses are there really?
- Is the AI applying a group-level pattern to an individual?
- Is the AI stating a prediction without also applying a refutation condition?
- Is this fact, inference, or speculation? Has it been labelled as such?
Closing
Across projects, most errors were caught externally by the researcher, not the AI. The discipline’s strength lies not in preventing errors but in bounding their cost. The combined framework above provides a unified, method-focused reference for preventing propagation, designing tests, and maintaining evidential integrity.
I hope that some of this helps you as you explore the use of AI in your genealogy research and gives you the comfort to push back on or accept the input AI gives.
2 comments:
Outstanding work - thanks, Miles!
Outstanding work - thanks, Miles!
Post a Comment