Guest Lecture on "AI Patent Quality: Avoiding Flaws Inherent In Human-Performed Work" at Stanford Law
I would like to thank Jason Du Mont and his students for inviting me to speak today at Stanford Law on "AI Patent Quality: Avoiding Flaws Inherent In Human-Performed Work," as part of the Law, Science, and Technology LLM Colloquium. I was impressed by their thoughtful and challenging questions.
To summarize the presentation, most of what patent attorneys do today can be done better by machines using existing technology as applied to patent law. Because AI can optimize strategies based on tens of thousands of data points simultaneously, I believe AI will make better decisions, on average, than the human patent attorney counterparts. The underlying algorithms and technology might be ready to take us there, but the patent-specific tools are not yet fully developed. In the meantime, we can make use of as much data and stats-based analytics as possible. Based on wisdom gained from studies outside of patent law, I think we should defer decisions to AI or stats-based analytics when the algorithms or analytics themselves are shown to be predictive, withstand scrutiny, and are otherwise well-founded. That said, they don't need to be perfect to be better than us.
Most law schools don't teach their students to trust well-founded AI to make decisions. Instead, law schools focus on exposing the students to enough data points so the students themselves can get better gut feelings. To achieve measurably better outcomes that take advantage of the information available from half a million U.S. patent filings each year and thousands of recent patent cases, we need to unlearn some of that more traditional training so the AI can be trusted despite our gut feelings.
The full slide deck (with fancy animations) is available for download here or in video form below. In case you're interested, the suggested readings for the course follow the video. In addition to the suggested readings, I suggest "Thinking, Fast and Slow," by Daniel Kahneman, which is cited in the McAfee articles.
As usual, I would love to hear your thoughts. Although none of the students said this out loud, "Eric, you're crazy and this will never happen" is a valid thought (even though AI is already solving problems of greater complexity than patent work).
Non-Patent Readings
To summarize the presentation, most of what patent attorneys do today can be done better by machines using existing technology as applied to patent law. Because AI can optimize strategies based on tens of thousands of data points simultaneously, I believe AI will make better decisions, on average, than the human patent attorney counterparts. The underlying algorithms and technology might be ready to take us there, but the patent-specific tools are not yet fully developed. In the meantime, we can make use of as much data and stats-based analytics as possible. Based on wisdom gained from studies outside of patent law, I think we should defer decisions to AI or stats-based analytics when the algorithms or analytics themselves are shown to be predictive, withstand scrutiny, and are otherwise well-founded. That said, they don't need to be perfect to be better than us.
Most law schools don't teach their students to trust well-founded AI to make decisions. Instead, law schools focus on exposing the students to enough data points so the students themselves can get better gut feelings. To achieve measurably better outcomes that take advantage of the information available from half a million U.S. patent filings each year and thousands of recent patent cases, we need to unlearn some of that more traditional training so the AI can be trusted despite our gut feelings.
The full slide deck (with fancy animations) is available for download here or in video form below. In case you're interested, the suggested readings for the course follow the video. In addition to the suggested readings, I suggest "Thinking, Fast and Slow," by Daniel Kahneman, which is cited in the McAfee articles.
As usual, I would love to hear your thoughts. Although none of the students said this out loud, "Eric, you're crazy and this will never happen" is a valid thought (even though AI is already solving problems of greater complexity than patent work).
Non-Patent Readings
(required) McAfee,
Andrew. “Big
Data’s Biggest Challenge? Convincing People NOT to Trust Their Judgment”
& “When
Human Judgment Works Well, and When it Doesn’t” Harvard Bus. Rev. 2013-14.
(optional) Logg, Jennifer; Minson, Julia; Moore, Don. “Algorithm
Appreciation: People Prefer Algorithmic To Human Judgment.” Harvard
Business School NOM Unit Working Paper No. 17-086. Mar. 28, 2017
(optional) Kleinberg, Jon; Lakkaraju, Himabindu; Leskovec,
Jure; Ludwig, Jens; Mullainathan, Sendhil. “Human Decisions And
Machine Predictions.” Quarterly Journal of Economics, 2017.
Patent Readings
(required) Sutton,
Eric. “Pursuit
of Extremely Short Patent Claims.” IPWatchdog. May 17, 2016.
(required) Sutton,
Eric. “Do
You Know What Your Provisional Application Did Last Summer?”
Patently-O. Nov. 9, 2017.
(required) Sutton,
Eric. “Navigating
Art Unit 3689, If You Must” & (optional) “Are
Applicants ALWAYS Able To Survive Art Unit 2138?” (optional) Patnotechnic.
May 21, 2018.
(optional) Patent Information Users Group, Inc. “Patent
Analysis, Mapping, and Visualization Tools.” Last Updated Feb. 9,
2017.
(optional, if background needed) Quinn, Gene; Benson,
Michael. “Understanding
U.S. Patent Prosecution.” Posted June 30, 2018.
(optional marketing video) Innography. “Innography’s True Semantic
Search.” Posted Jan. 31, 2017.
(optional marketing video) LexisNexis
IP. “What
Color Is Your Examiner Red, Yellow or Green Customizing Your Prosecution
Strategies.” Posted Apr. 19, 2018.
On LinkedIn, I received this question (more or less): Isn't it traditionally difficult for AI to explain itself? How is that different in the patent space?
ReplyDeleteIn law, and particularly in patent law, I think it is slightly easier to provide automated rationale in large sets of helpful scenarios. The law is so complicated and the drafts so rich that even experienced drafters cannot keep all of the rules in their heads and apply them consistently.
For example, in drafting, there are hundreds or thousands of words that can seriously hurt you but not help you (a.k.a. "patent profanity"), and the average human drafter does not know most of them (or perhaps doesn't have the time to stew over it). Each of these words has one or more reasons inherently tied to it, and more experienced attorneys spend their days explaining these problems to others, much like robots would/could/should.
As another example, in patent claims, even the most experienced drafters have difficulty using proper antecedent basis ("an" item..."the" item) throughout the claims. In this case, they know the rules but cannot consistently follow them because the environment is too complex. Even patent examiners miss these problems, and they persist until litigation. In litigation, they can cause serious problems.
A third example makes you start to realize that the exceptions to human-driven quality may outweigh the potential benefits. Often, inventors or in-house patent attorneys provide shorthand information with the expectation that the drafter will expand out the shorthand to cover all practical scenarios. When this task involves flipping the world upside-down and completely rethinking and rewording what was provided from a different perspective, humans are still significantly better and will continue to be better for a while. If the client already has experienced patent counsel, though, then the initially provided version typically just needs expansion without reformulation, especially if the drafter was there to ask questions when it is provided. In fact, later reformulation may (but not always) violate some of the careful considerations that went into the first version.
As a real-world example of expanding on a discussion, if I'm asking the drafter to list all wireless technologies or to find a few paragraphs of text that describe machine learning, a machine will outperform a human who is on the same task with a confined budget.
In the cases of patent profanity and antecedent basis, machines could easily provide rationale. In the case of expanding on a topic, the machine could say, "I used this paragraph because it is the most-used paragraph on point in yours or competitors' patents," or "this list of items came from these different sources."
The three pieces above account for most of the drafting work and most of the correction needed on human drafting work.
In prosecution with the patent office, humans are routinely making the wrong decisions because they don't have enough information in front of them. Human attorneys will continually pursue 0.X% chances with client funds, thinking that the odds are closer to 50/50 and that their skill will tip the scales. Telling the attorney the odds for their scenario sometimes answers the strategy question. (Many attorneys would still suggest using client funds in those scenarios, which is one reason why AI should already be calling the shots.)
Excellent points, Eric. I'd like to add that the difficulty in explaining AI decision making rests primarily on how the AI is designed.
DeleteOne form of "AI" involves training deep learning models to extract features from data and classify the data based on those the features. The common example used in image processing is the convolutional neural network (CNN), which is highly effective at object recognition. The CNN is a "deep" structure, in that it has many layers which abstract individual pixels (at the input layer) into increasingly higher order features (edges => corners => shapes => collections of shapes => etc.).
Let's say you decide to train a CNN to distinguish between cats and dogs. You use a bunch of training data and traditional training techniques (backpropagation) to tune up the CNN so that it accurately classifies images as containing a cat, a dog, both, or neither.
If you simply look at the input and output, it's a black box of magic. But in this layered design, we can examine the layers (say, near the output) to see what kinds of abstractions lead to a classification of a cat vs. a classification of a dog. The decision may rest on, for example, the shape of the nose (cats having a distinctive upside-down triangular nose vs dogs having a rounder nose)--which can be extracted by looking not at the output of the CNN, but at the internal layers of the CNN.
To the extent that explaining the decision making is important, patent law-related AI designers could employ particular tools or algorithms that are easier to interpret. Referring to your article regarding success and failure rates of particular word stems in art unit 3689, at least two approaches could be used: (1) train a model that simply estimates the likelihood of allowance for an entire patent claim; or (2) train a model that estimates the extent to which particular word stems affect allowance rates, and aggregates each of the word stem probabilities to determine a likelihood of allowance. Approach (2) allows us to examine which words affect patentability the most, even though approach (1) might still accomplish the goal of predicting patentability to some degree of accuracy.
My point is that we can design the machine learning/AI systems in a way that allows us to extract its rationale, to some extent.