Guest Lecture on "AI Patent Quality: Avoiding Flaws Inherent In Human-Performed Work" at Stanford Law

I would like to thank Jason Du Mont and his students for inviting me to speak today at Stanford Law on "AI Patent Quality: Avoiding Flaws Inherent In Human-Performed Work," as part of the Law, Science, and Technology LLM Colloquium. I was impressed by their thoughtful and challenging questions.

To summarize the presentation, most of what patent attorneys do today can be done better by machines using existing technology as applied to patent law. Because AI can optimize strategies based on tens of thousands of data points simultaneously, I believe AI will make better decisions, on average, than the human patent attorney counterparts. The underlying algorithms and technology might be ready to take us there, but the patent-specific tools are not yet fully developed. In the meantime, we can make use of as much data and stats-based analytics as possible. Based on wisdom gained from studies outside of patent law, I think we should defer decisions to AI or stats-based analytics when the algorithms or analytics themselves are shown to be predictive, withstand scrutiny, and are otherwise well-founded. That said, they don't need to be perfect to be better than us.

Most law schools don't teach their students to trust well-founded AI to make decisions. Instead, law schools focus on exposing the students to enough data points so the students themselves can get better gut feelings. To achieve measurably better outcomes that take advantage of the information available from half a million U.S. patent filings each year and thousands of recent patent cases, we need to unlearn some of that more traditional training so the AI can be trusted despite our gut feelings.

The full slide deck (with fancy animations) is available for download here or in video form below. In case you're interested, the suggested readings for the course follow the video. In addition to the suggested readings, I suggest "Thinking, Fast and Slow," by Daniel Kahneman, which is cited in the McAfee articles.

As usual, I would love to hear your thoughts. Although none of the students said this out loud, "Eric, you're crazy and this will never happen" is a valid thought (even though AI is already solving problems of greater complexity than patent work).


Non-Patent Readings


(optional) Logg, Jennifer; Minson, Julia; Moore, Don. “Algorithm Appreciation: People Prefer Algorithmic To Human Judgment.” Harvard Business School NOM Unit Working Paper No. 17-086. Mar. 28, 2017

(optional) Kleinberg, Jon; Lakkaraju, Himabindu; Leskovec, Jure; Ludwig, Jens; Mullainathan, Sendhil. “Human Decisions And Machine Predictions.” Quarterly Journal of Economics, 2017.

Patent Readings

(required) Sutton, Eric. “Pursuit of Extremely Short Patent Claims.” IPWatchdog. May 17, 2016.

(required) Sutton, Eric. “Do You Know What Your Provisional Application Did Last Summer?” Patently-O. Nov. 9, 2017.

(required) Sutton, Eric. “Navigating Art Unit 3689, If You Must& (optional) “Are Applicants ALWAYS Able To Survive Art Unit 2138?” (optional) Patnotechnic. May 21, 2018.

(optional) Patent Information Users Group, Inc. “Patent Analysis, Mapping, and Visualization Tools.” Last Updated Feb. 9, 2017.

(optional, if background needed) Quinn, Gene; Benson, Michael. “Understanding U.S. Patent Prosecution.” Posted June 30, 2018.

(optional marketing video) Innography. “Innography’s True Semantic Search.” Posted Jan. 31, 2017.

(optional marketing video) LexisNexis IP. “What Color Is Your Examiner Red, Yellow or Green Customizing Your Prosecution Strategies.” Posted Apr. 19, 2018.



Comments

  1. On LinkedIn, I received this question (more or less): Isn't it traditionally difficult for AI to explain itself? How is that different in the patent space?

    In law, and particularly in patent law, I think it is slightly easier to provide automated rationale in large sets of helpful scenarios. The law is so complicated and the drafts so rich that even experienced drafters cannot keep all of the rules in their heads and apply them consistently.

    For example, in drafting, there are hundreds or thousands of words that can seriously hurt you but not help you (a.k.a. "patent profanity"), and the average human drafter does not know most of them (or perhaps doesn't have the time to stew over it). Each of these words has one or more reasons inherently tied to it, and more experienced attorneys spend their days explaining these problems to others, much like robots would/could/should.

    As another example, in patent claims, even the most experienced drafters have difficulty using proper antecedent basis ("an" item..."the" item) throughout the claims. In this case, they know the rules but cannot consistently follow them because the environment is too complex. Even patent examiners miss these problems, and they persist until litigation. In litigation, they can cause serious problems.

    A third example makes you start to realize that the exceptions to human-driven quality may outweigh the potential benefits. Often, inventors or in-house patent attorneys provide shorthand information with the expectation that the drafter will expand out the shorthand to cover all practical scenarios. When this task involves flipping the world upside-down and completely rethinking and rewording what was provided from a different perspective, humans are still significantly better and will continue to be better for a while. If the client already has experienced patent counsel, though, then the initially provided version typically just needs expansion without reformulation, especially if the drafter was there to ask questions when it is provided. In fact, later reformulation may (but not always) violate some of the careful considerations that went into the first version.

    As a real-world example of expanding on a discussion, if I'm asking the drafter to list all wireless technologies or to find a few paragraphs of text that describe machine learning, a machine will outperform a human who is on the same task with a confined budget.

    In the cases of patent profanity and antecedent basis, machines could easily provide rationale. In the case of expanding on a topic, the machine could say, "I used this paragraph because it is the most-used paragraph on point in yours or competitors' patents," or "this list of items came from these different sources."

    The three pieces above account for most of the drafting work and most of the correction needed on human drafting work.

    In prosecution with the patent office, humans are routinely making the wrong decisions because they don't have enough information in front of them. Human attorneys will continually pursue 0.X% chances with client funds, thinking that the odds are closer to 50/50 and that their skill will tip the scales. Telling the attorney the odds for their scenario sometimes answers the strategy question. (Many attorneys would still suggest using client funds in those scenarios, which is one reason why AI should already be calling the shots.)

    ReplyDelete
    Replies
    1. Excellent points, Eric. I'd like to add that the difficulty in explaining AI decision making rests primarily on how the AI is designed.

      One form of "AI" involves training deep learning models to extract features from data and classify the data based on those the features. The common example used in image processing is the convolutional neural network (CNN), which is highly effective at object recognition. The CNN is a "deep" structure, in that it has many layers which abstract individual pixels (at the input layer) into increasingly higher order features (edges => corners => shapes => collections of shapes => etc.).

      Let's say you decide to train a CNN to distinguish between cats and dogs. You use a bunch of training data and traditional training techniques (backpropagation) to tune up the CNN so that it accurately classifies images as containing a cat, a dog, both, or neither.

      If you simply look at the input and output, it's a black box of magic. But in this layered design, we can examine the layers (say, near the output) to see what kinds of abstractions lead to a classification of a cat vs. a classification of a dog. The decision may rest on, for example, the shape of the nose (cats having a distinctive upside-down triangular nose vs dogs having a rounder nose)--which can be extracted by looking not at the output of the CNN, but at the internal layers of the CNN.

      To the extent that explaining the decision making is important, patent law-related AI designers could employ particular tools or algorithms that are easier to interpret. Referring to your article regarding success and failure rates of particular word stems in art unit 3689, at least two approaches could be used: (1) train a model that simply estimates the likelihood of allowance for an entire patent claim; or (2) train a model that estimates the extent to which particular word stems affect allowance rates, and aggregates each of the word stem probabilities to determine a likelihood of allowance. Approach (2) allows us to examine which words affect patentability the most, even though approach (1) might still accomplish the goal of predicting patentability to some degree of accuracy.

      My point is that we can design the machine learning/AI systems in a way that allows us to extract its rationale, to some extent.

      Delete
  2. Awesome post. Thanks for sharing this post with us.In general, however, 2 things appear clear: 1st, the term computer science (AI) is older than the term machine learning (ML), and second,Machine learning course most of the people think about machine learning to be a set of computer science.

    ReplyDelete
  3. Thank you sharing this kind of noteworthy information. Nice Post.

    Education
    Technology

    ReplyDelete
  4. It is an amazing post and it is very informative. Keep sharing this kind of useful information.

    nationalreviewcouncil
    Guest posting sites

    ReplyDelete
  5. Education is something nearly all kids in the developed nations take for granted including myself. So many adults and children write and talk about how the education so let me know and read this click here https://queryok.com/2019/03/13/5-things-to-add-to-your-cycling-habits/

    ReplyDelete
  6. Thanks, for sharing this post with us. Many students needs this type of post. My team share this post with QandA team.

    ReplyDelete
  7. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...

    Education
    Technology

    ReplyDelete
  8. I am really enjoying reading your well written articles.
    It looks like you spend a lot of effort and time on your blog.
    I have bookmarked it and I am looking forward to reading new articles. Keep up the good work..
    ccna Training in Chennai
    ccna Training institute in Chennai
    Angularjs Course in Chennai
    Ethical Hacking Course in Chennai
    Tally Course in Chennai
    ccna course in chennai
    ccna training in chennai

    ReplyDelete
  9. I read awesome post after a long time.I am very satisfied with your way of writing.Expect the next post in the upcoming days.
    German Classes in Anna nagar
    Spoken English Class in Anna Nagar
    French Classes in Anna nagar
    IELTS Coaching in Anna nagar

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. Such a wonderful article! I learned more info to your post and It's attractive to me. Surely I will share with my friends and Keep it up a great job!
    IELTS Coaching in Jaipur

    ReplyDelete
  12. The information you have shared is more useful to us. Thanks for your blog.

    Python Course | Digital Marketing Course | Java Course

    ReplyDelete
  13. Great blog ! I am impressed with suggestions of author. Driving school in Alexandria

    ReplyDelete
  14. I found one successful example of this truth through this blog. I am going to use such information now. ISO Risk Consulting

    ReplyDelete

Post a Comment