Guest Lecture on "AI Patent Quality: Avoiding Flaws Inherent In Human-Performed Work" at Stanford Law

I would like to thank Jason Du Mont and his students for inviting me to speak today at Stanford Law on "AI Patent Quality: Avoiding Flaws Inherent In Human-Performed Work," as part of the Law, Science, and Technology LLM Colloquium. I was impressed by their thoughtful and challenging questions.

To summarize the presentation, most of what patent attorneys do today can be done better by machines using existing technology as applied to patent law. Because AI can optimize strategies based on tens of thousands of data points simultaneously, I believe AI will make better decisions, on average, than the human patent attorney counterparts. The underlying algorithms and technology might be ready to take us there, but the patent-specific tools are not yet fully developed. In the meantime, we can make use of as much data and stats-based analytics as possible. Based on wisdom gained from studies outside of patent law, I think we should defer decisions to AI or stats-based analytics when the algorithms or analytics themselves are shown to be predictive, withstand scrutiny, and are otherwise well-founded. That said, they don't need to be perfect to be better than us.

Most law schools don't teach their students to trust well-founded AI to make decisions. Instead, law schools focus on exposing the students to enough data points so the students themselves can get better gut feelings. To achieve measurably better outcomes that take advantage of the information available from half a million U.S. patent filings each year and thousands of recent patent cases, we need to unlearn some of that more traditional training so the AI can be trusted despite our gut feelings.

The full slide deck (with fancy animations) is available for download here or in video form below. In case you're interested, the suggested readings for the course follow the video. In addition to the suggested readings, I suggest "Thinking, Fast and Slow," by Daniel Kahneman, which is cited in the McAfee articles.

As usual, I would love to hear your thoughts. Although none of the students said this out loud, "Eric, you're crazy and this will never happen" is a valid thought (even though AI is already solving problems of greater complexity than patent work).

Non-Patent Readings

(optional) Logg, Jennifer; Minson, Julia; Moore, Don. “Algorithm Appreciation: People Prefer Algorithmic To Human Judgment.” Harvard Business School NOM Unit Working Paper No. 17-086. Mar. 28, 2017

(optional) Kleinberg, Jon; Lakkaraju, Himabindu; Leskovec, Jure; Ludwig, Jens; Mullainathan, Sendhil. “Human Decisions And Machine Predictions.” Quarterly Journal of Economics, 2017.

Patent Readings

(required) Sutton, Eric. “Pursuit of Extremely Short Patent Claims.” IPWatchdog. May 17, 2016.

(required) Sutton, Eric. “Do You Know What Your Provisional Application Did Last Summer?” Patently-O. Nov. 9, 2017.

(required) Sutton, Eric. “Navigating Art Unit 3689, If You Must& (optional) “Are Applicants ALWAYS Able To Survive Art Unit 2138?” (optional) Patnotechnic. May 21, 2018.

(optional) Patent Information Users Group, Inc. “Patent Analysis, Mapping, and Visualization Tools.” Last Updated Feb. 9, 2017.

(optional, if background needed) Quinn, Gene; Benson, Michael. “Understanding U.S. Patent Prosecution.” Posted June 30, 2018.

(optional marketing video) Innography. “Innography’s True Semantic Search.” Posted Jan. 31, 2017.

(optional marketing video) LexisNexis IP. “What Color Is Your Examiner Red, Yellow or Green Customizing Your Prosecution Strategies.” Posted Apr. 19, 2018.


  1. On LinkedIn, I received this question (more or less): Isn't it traditionally difficult for AI to explain itself? How is that different in the patent space?

    In law, and particularly in patent law, I think it is slightly easier to provide automated rationale in large sets of helpful scenarios. The law is so complicated and the drafts so rich that even experienced drafters cannot keep all of the rules in their heads and apply them consistently.

    For example, in drafting, there are hundreds or thousands of words that can seriously hurt you but not help you (a.k.a. "patent profanity"), and the average human drafter does not know most of them (or perhaps doesn't have the time to stew over it). Each of these words has one or more reasons inherently tied to it, and more experienced attorneys spend their days explaining these problems to others, much like robots would/could/should.

    As another example, in patent claims, even the most experienced drafters have difficulty using proper antecedent basis ("an" item..."the" item) throughout the claims. In this case, they know the rules but cannot consistently follow them because the environment is too complex. Even patent examiners miss these problems, and they persist until litigation. In litigation, they can cause serious problems.

    A third example makes you start to realize that the exceptions to human-driven quality may outweigh the potential benefits. Often, inventors or in-house patent attorneys provide shorthand information with the expectation that the drafter will expand out the shorthand to cover all practical scenarios. When this task involves flipping the world upside-down and completely rethinking and rewording what was provided from a different perspective, humans are still significantly better and will continue to be better for a while. If the client already has experienced patent counsel, though, then the initially provided version typically just needs expansion without reformulation, especially if the drafter was there to ask questions when it is provided. In fact, later reformulation may (but not always) violate some of the careful considerations that went into the first version.

    As a real-world example of expanding on a discussion, if I'm asking the drafter to list all wireless technologies or to find a few paragraphs of text that describe machine learning, a machine will outperform a human who is on the same task with a confined budget.

    In the cases of patent profanity and antecedent basis, machines could easily provide rationale. In the case of expanding on a topic, the machine could say, "I used this paragraph because it is the most-used paragraph on point in yours or competitors' patents," or "this list of items came from these different sources."

    The three pieces above account for most of the drafting work and most of the correction needed on human drafting work.

    In prosecution with the patent office, humans are routinely making the wrong decisions because they don't have enough information in front of them. Human attorneys will continually pursue 0.X% chances with client funds, thinking that the odds are closer to 50/50 and that their skill will tip the scales. Telling the attorney the odds for their scenario sometimes answers the strategy question. (Many attorneys would still suggest using client funds in those scenarios, which is one reason why AI should already be calling the shots.)

    1. Excellent points, Eric. I'd like to add that the difficulty in explaining AI decision making rests primarily on how the AI is designed.

      One form of "AI" involves training deep learning models to extract features from data and classify the data based on those the features. The common example used in image processing is the convolutional neural network (CNN), which is highly effective at object recognition. The CNN is a "deep" structure, in that it has many layers which abstract individual pixels (at the input layer) into increasingly higher order features (edges => corners => shapes => collections of shapes => etc.).

      Let's say you decide to train a CNN to distinguish between cats and dogs. You use a bunch of training data and traditional training techniques (backpropagation) to tune up the CNN so that it accurately classifies images as containing a cat, a dog, both, or neither.

      If you simply look at the input and output, it's a black box of magic. But in this layered design, we can examine the layers (say, near the output) to see what kinds of abstractions lead to a classification of a cat vs. a classification of a dog. The decision may rest on, for example, the shape of the nose (cats having a distinctive upside-down triangular nose vs dogs having a rounder nose)--which can be extracted by looking not at the output of the CNN, but at the internal layers of the CNN.

      To the extent that explaining the decision making is important, patent law-related AI designers could employ particular tools or algorithms that are easier to interpret. Referring to your article regarding success and failure rates of particular word stems in art unit 3689, at least two approaches could be used: (1) train a model that simply estimates the likelihood of allowance for an entire patent claim; or (2) train a model that estimates the extent to which particular word stems affect allowance rates, and aggregates each of the word stem probabilities to determine a likelihood of allowance. Approach (2) allows us to examine which words affect patentability the most, even though approach (1) might still accomplish the goal of predicting patentability to some degree of accuracy.

      My point is that we can design the machine learning/AI systems in a way that allows us to extract its rationale, to some extent.

  2. Such a great blog. Got many useful inforamations about this thechnology.
    Very useful for the freshers. Good keep it up.
    machine learning course in chennai
    machine learning training in chennai
    machine learning institute in chennai

  3. This is an awesome blog for this technology. Keep updating the information in this technology and
    it will be useful for the freshers who seeks for the information.
    oracle training in chennai
    orcale course in chennai
    Advanced excel training in chennai
    android training in chennai
    dot net training in chennai
    oracle training in porur
    oracle training in adyar

  4. Awesome post. Thanks for sharing this post with us.In general, however, 2 things appear clear: 1st, the term computer science (AI) is older than the term machine learning (ML), and second,Machine learning course most of the people think about machine learning to be a set of computer science.

  5. Information is very informative also you can click Nexter ,and get such type of info, this is the great resource to get such type of information.

  6. Thank you sharing this kind of noteworthy information. Nice Post.


  7. I was very pleased to find this site.I wanted to thank you for this great read!! I definitely enjoying every little bit of it and i am sharing Nexter this blog with you if you have any trouble you can visit here for more news.

  8. Valentine's day, the day of love, which falls on 14th February every year is celebrated by the couple all across the globe. Here is the world famous valentine day website visit here: Happy Valentine’s Day Images

  9. nice post, thank you for giving valuable information.

  10. nice blog, thank you.

    pmp certification
    project management certification
    project management courses
    project management training
    project management professional certification
    pmp course
    pmp training
    pmp certification cost

  11. nice info, thank you.

    data science course malaysia
    data science course

  12. great post , thank you for sharing such an good info....


Post a Comment