Guest Lecture on "AI Patent Quality: Avoiding Flaws Inherent In Human-Performed Work" at Stanford Law

I would like to thank Jason Du Mont and his students for inviting me to speak today at Stanford Law on "AI Patent Quality: Avoiding Flaws Inherent In Human-Performed Work," as part of the Law, Science, and Technology LLM Colloquium. I was impressed by their thoughtful and challenging questions.

To summarize the presentation, most of what patent attorneys do today can be done better by machines using existing technology as applied to patent law. Because AI can optimize strategies based on tens of thousands of data points simultaneously, I believe AI will make better decisions, on average, than the human patent attorney counterparts. The underlying algorithms and technology might be ready to take us there, but the patent-specific tools are not yet fully developed. In the meantime, we can make use of as much data and stats-based analytics as possible. Based on wisdom gained from studies outside of patent law, I think we should defer decisions to AI or stats-based analytics when the algorithms or analytics themselves are shown to be predictive, withstand scrutiny, and are otherwise well-founded. That said, they don't need to be perfect to be better than us.

Most law schools don't teach their students to trust well-founded AI to make decisions. Instead, law schools focus on exposing the students to enough data points so the students themselves can get better gut feelings. To achieve measurably better outcomes that take advantage of the information available from half a million U.S. patent filings each year and thousands of recent patent cases, we need to unlearn some of that more traditional training so the AI can be trusted despite our gut feelings.

The full slide deck (with fancy animations) is available for download here or in video form below. In case you're interested, the suggested readings for the course follow the video. In addition to the suggested readings, I suggest "Thinking, Fast and Slow," by Daniel Kahneman, which is cited in the McAfee articles.

As usual, I would love to hear your thoughts. Although none of the students said this out loud, "Eric, you're crazy and this will never happen" is a valid thought (even though AI is already solving problems of greater complexity than patent work).

Non-Patent Readings

(optional) Logg, Jennifer; Minson, Julia; Moore, Don. “Algorithm Appreciation: People Prefer Algorithmic To Human Judgment.” Harvard Business School NOM Unit Working Paper No. 17-086. Mar. 28, 2017

(optional) Kleinberg, Jon; Lakkaraju, Himabindu; Leskovec, Jure; Ludwig, Jens; Mullainathan, Sendhil. “Human Decisions And Machine Predictions.” Quarterly Journal of Economics, 2017.

Patent Readings

(required) Sutton, Eric. “Pursuit of Extremely Short Patent Claims.” IPWatchdog. May 17, 2016.

(required) Sutton, Eric. “Do You Know What Your Provisional Application Did Last Summer?” Patently-O. Nov. 9, 2017.

(required) Sutton, Eric. “Navigating Art Unit 3689, If You Must& (optional) “Are Applicants ALWAYS Able To Survive Art Unit 2138?” (optional) Patnotechnic. May 21, 2018.

(optional) Patent Information Users Group, Inc. “Patent Analysis, Mapping, and Visualization Tools.” Last Updated Feb. 9, 2017.

(optional, if background needed) Quinn, Gene; Benson, Michael. “Understanding U.S. Patent Prosecution.” Posted June 30, 2018.

(optional marketing video) Innography. “Innography’s True Semantic Search.” Posted Jan. 31, 2017.

(optional marketing video) LexisNexis IP. “What Color Is Your Examiner Red, Yellow or Green Customizing Your Prosecution Strategies.” Posted Apr. 19, 2018.


  1. On LinkedIn, I received this question (more or less): Isn't it traditionally difficult for AI to explain itself? How is that different in the patent space?

    In law, and particularly in patent law, I think it is slightly easier to provide automated rationale in large sets of helpful scenarios. The law is so complicated and the drafts so rich that even experienced drafters cannot keep all of the rules in their heads and apply them consistently.

    For example, in drafting, there are hundreds or thousands of words that can seriously hurt you but not help you (a.k.a. "patent profanity"), and the average human drafter does not know most of them (or perhaps doesn't have the time to stew over it). Each of these words has one or more reasons inherently tied to it, and more experienced attorneys spend their days explaining these problems to others, much like robots would/could/should.

    As another example, in patent claims, even the most experienced drafters have difficulty using proper antecedent basis ("an" item..."the" item) throughout the claims. In this case, they know the rules but cannot consistently follow them because the environment is too complex. Even patent examiners miss these problems, and they persist until litigation. In litigation, they can cause serious problems.

    A third example makes you start to realize that the exceptions to human-driven quality may outweigh the potential benefits. Often, inventors or in-house patent attorneys provide shorthand information with the expectation that the drafter will expand out the shorthand to cover all practical scenarios. When this task involves flipping the world upside-down and completely rethinking and rewording what was provided from a different perspective, humans are still significantly better and will continue to be better for a while. If the client already has experienced patent counsel, though, then the initially provided version typically just needs expansion without reformulation, especially if the drafter was there to ask questions when it is provided. In fact, later reformulation may (but not always) violate some of the careful considerations that went into the first version.

    As a real-world example of expanding on a discussion, if I'm asking the drafter to list all wireless technologies or to find a few paragraphs of text that describe machine learning, a machine will outperform a human who is on the same task with a confined budget.

    In the cases of patent profanity and antecedent basis, machines could easily provide rationale. In the case of expanding on a topic, the machine could say, "I used this paragraph because it is the most-used paragraph on point in yours or competitors' patents," or "this list of items came from these different sources."

    The three pieces above account for most of the drafting work and most of the correction needed on human drafting work.

    In prosecution with the patent office, humans are routinely making the wrong decisions because they don't have enough information in front of them. Human attorneys will continually pursue 0.X% chances with client funds, thinking that the odds are closer to 50/50 and that their skill will tip the scales. Telling the attorney the odds for their scenario sometimes answers the strategy question. (Many attorneys would still suggest using client funds in those scenarios, which is one reason why AI should already be calling the shots.)

    1. Excellent points, Eric. I'd like to add that the difficulty in explaining AI decision making rests primarily on how the AI is designed.

      One form of "AI" involves training deep learning models to extract features from data and classify the data based on those the features. The common example used in image processing is the convolutional neural network (CNN), which is highly effective at object recognition. The CNN is a "deep" structure, in that it has many layers which abstract individual pixels (at the input layer) into increasingly higher order features (edges => corners => shapes => collections of shapes => etc.).

      Let's say you decide to train a CNN to distinguish between cats and dogs. You use a bunch of training data and traditional training techniques (backpropagation) to tune up the CNN so that it accurately classifies images as containing a cat, a dog, both, or neither.

      If you simply look at the input and output, it's a black box of magic. But in this layered design, we can examine the layers (say, near the output) to see what kinds of abstractions lead to a classification of a cat vs. a classification of a dog. The decision may rest on, for example, the shape of the nose (cats having a distinctive upside-down triangular nose vs dogs having a rounder nose)--which can be extracted by looking not at the output of the CNN, but at the internal layers of the CNN.

      To the extent that explaining the decision making is important, patent law-related AI designers could employ particular tools or algorithms that are easier to interpret. Referring to your article regarding success and failure rates of particular word stems in art unit 3689, at least two approaches could be used: (1) train a model that simply estimates the likelihood of allowance for an entire patent claim; or (2) train a model that estimates the extent to which particular word stems affect allowance rates, and aggregates each of the word stem probabilities to determine a likelihood of allowance. Approach (2) allows us to examine which words affect patentability the most, even though approach (1) might still accomplish the goal of predicting patentability to some degree of accuracy.

      My point is that we can design the machine learning/AI systems in a way that allows us to extract its rationale, to some extent.

  2. Awesome post. Thanks for sharing this post with us.In general, however, 2 things appear clear: 1st, the term computer science (AI) is older than the term machine learning (ML), and second,Machine learning course most of the people think about machine learning to be a set of computer science.

  3. Thank you sharing this kind of noteworthy information. Nice Post.


  4. It is an amazing post and it is very informative. Keep sharing this kind of useful information.

    Guest posting sites

  5. Education is something nearly all kids in the developed nations take for granted including myself. So many adults and children write and talk about how the education so let me know and read this click here

  6. Thanks, for sharing this post with us. Many students needs this type of post. My team share this post with QandA team.

  7. I read awesome post after a long time.I am very satisfied with your way of writing.Expect the next post in the upcoming days.
    German Classes in Anna nagar
    Spoken English Class in Anna Nagar
    French Classes in Anna nagar
    IELTS Coaching in Anna nagar

  8. This comment has been removed by the author.

  9. The information you have shared is more useful to us. Thanks for your blog.

    Python Course | Digital Marketing Course | Java Course

  10. very good post, I have not seen a good article since many days, but when I read this article I felt it much interesting and want to read such article regular. I would request you people to provide more of such articles.

    data science training in aurangabad
    data science course in aurangabad

  11. This comment has been removed by the author.


  12. This was not just great in fact this was really perfect your talent in writing was great. ExcelR Machine Learning Course Pune

  13. I know, it is not an easy task to write such a big article in one day, I've tried that and I've failed. But, here you are, trying the big task and finishing it off and getting good comments and ratings. That is one hell of a job done.

    IELTS Coaching in chennai

    German Classes in Chennai

    GRE Coaching Classes in Chennai

    TOEFL Coaching in Chennai

    spoken english classes in chennai | Communication training

  14. You are providing good knowledge. It is really helpful and factual information for us and everyone to increase knowledge. Continue sharing your data. Thank you. patent attorney adelaide

  15. I am very thankful to you that you have shared this information with us. Read more info about Trademark Registration Applications Online. I got some different kind of knowledge from your web page, and it is really helpful for everyone. Thanks for share it.

  16. i'm unequipped for perusing articles online especially every now and again, but Im cheerful I did these days. it is chosen dexterously composed, and your focuses are skillfully communicated. I request you agreeably, engage, dont at any point lower composing.! Sai2 Crack

  17. I wish you love everlasting, peace for the rest of your days, may your life be as beautiful as you are, and joy fill your heart this Christmas day. Merry .
    Christmas Birthday Wishes

  18. I have read your blog. It is really helpful. If you want to excel in your class 11 studies, enroll in our online tuition for 11th class today and experience the benefits of personalized learning from the comfort of your home.
    For more info Contact us: +91-9654271931, +971-505593798 or visit Online Tuition for 11th Class

  19. Thanks for taking the time to discuss this, I feel strongly that love and read more on this topic. Unlock the door to GCSE success with Ziyyara Edutech's premier online GCSE tuition.
    For more info visit GCSE online home tuition


Post a Comment