What's new in the world of AI
June 2, 2023
A rare consensus has been building between the US high-tech companies creating AI models and the US government on the need to regulate AI. The impetus behind such need for consensus is the extraordinary success of GLLMM (Generative Large Language Multi Modal) models. The consensus has its critics though, who argue that regulating AI will have the effect of allowing China to catch up in the AI race. There are many facets to this argument, and the reader should consult independent credible sources who promote both sides of this argument. A well researched article arguing for AI regulation was posted today on the Foreign Affairs website: The Illusion of China’s AI Prowess, with the subtitle "Regulating AI Will Not Set America Back in the Technology Race".Excellent as the article may be, in my opinion this issue needs a more careful look than the one the article supports. First, although most AI researchers indeed agree that sensible AI regulation is needed, over-regulation is not needed, so the key word here is the word sensible. The problem is that the US government does not have at this moment the necessary human resources to drive this regulatory action on its own. Therefore, a more cooperative stance between the government and high-tech is essential if we are to get this regulatory action right and not hinder the development of an ever more useful and aligned AI. A mixed working group needs to be created whereby regulatory experts work side by side with AI experts to draft sensible regulation.
April 20, 2023
Large Language Models (LLMs) have been dominating the news, showing astonishing capabilities. At the same time, they can easily make quantitative blunders or spew nonsense or simply make stuff up. What is to make of all this? I have been asked this question many times. Here is one way to look at it, we'll use ChatGPT as an example because this is the model most talked about.Imagine two people sitting on a bench in a park. They ask each other questions, and they may go into deep details about various questions. They can hold the context of their conversation alive for a very long time. One may ask the other to draw a picture of something that was mentioned. Or ask the other to sing a tune that was mentioned. Or open up YouTube and play a song that was mentioned. Or get directions to a restaurant. Or solve an algebra problem. And so on. The user interface between these two people is very rich and when they do not know the answers, they appeal to additional tools, like a Google search on their phones, or a stick to draw a diagram in the sand.
One way to look at ChatGPT is as a glimpse into the richest user interface of the future, doing all the things we mentioned about the conversation between people and more. Just like in the conversation between the two people, ChatGPT does not have to do all the work; it can answer various questions by connecting to rich sets of additional AI models: tools to draw, tools to produce music, tools to recommend products. I found that this way of thinking of ChatGPT as a user interface into the entire brave new world of AI provides a template for many of the new developments that will come up. Here is a presentation of this idea by one of the co-founders of OpenAI, the company that built ChatGPT:
Let's push this a bit further. ChatGPT is not a formal system. It cannot make inferences based on sets of axioms. Therefore it cannot derive knowledge or compute things with certainty (relative to a set of axioms). It is a language model, it is only able to learn patterns, in a statistical, NOT logical (=computational) way. This is very important to understand. Among the rich set of tools that ChatGPT can connect to are tools that will in fact mitigate its mathematical/computational/logical/knowledge shortcomings. If you ask ChatGPT a quantitative question, it may give an incorrect answer without deferring to such tools. In my somewhat biased view (biased towards mathematical/logical/computational) here is one of the best examples of such integration, described in Stephen Wolfram's Wolfram|Alpha as the Way to Bring Computational Knowledge Superpowers to ChatGPT. It is more popularly described in the following video:
January 24, 2023
David Chalmers' invited talk at NeurIPS 2022 ties in a few threads between our article on Artificial Consciousness, the note on AI Transformers and the Self-Attention Mechanism, and the Main AI Concepts article. The talk is here: Could a Large Language Model be Conscious?):November 4, 2022
The interaction between AI and mathematics is growing rapidly. That advanced math is being used more and more in AI projects has been visible for some time and to a large degree predictable. But it is the other direction that is raising eyebrows. For example, we have seen neural networks-based solvers of partial differential equations achieve remarkable successes. It is not easy to explain these successes. I watched yesterday this clip with two most accomplished researchers, one a Fields medalist (highest recognition in Mathematics) and the other a Turing Award recipient (highest recognition in Computer Science). Although I took a break from updating the site for a couple of months, I am eager to include that clip right away (you may want to suplement the clip with this article, published yesterday too, Teaching AI advanced mathematical reasoning):October 5, 2022
There is hardly a day without some news about the popularity of diffusion models. Moreover, there are quite a number of websites that allow even non-technical users to experiment with diffusion models. They are a lot of fun, but how do they work? Here is a presentation of one particular class of diffusion models, for more technical users: How diffusion models work: the math from scratchSeptember 30, 2022
Graph Neural Networks are becoming mainstream. They are a bit harder to understand and explain, in a similar way to how non-euclidean geometries are with respect to the euclidean one. The GNN workshop at Stanford, which ended yesterday, gives a good explanation of their importance. 8 hours seem grueling, so pick and choose whatever talk interests you the most. But do not skip the first one because it gives a good overview and shows how GNNs generalize many of the other neural network architectures, including the very successful transformers:May 12, 2022
One of the most consistent and important trends in AI has been the quest for increased generalization power in transformers, very large neural network architectures initially built for NLP. Without being tweaked for various other applications, in other words using exactly the same weights obtained during training, some transformers can now chat, play games, caption images or control a robot arm. The linked-in PDF describes such a system built by DeepMind: A Generalist AgentMay 11, 2022
I am very reluctant to refer to video clips of Elon Musk because he is being used too many times to simply draw attention. But there are also times when his interviews are simply too relevant to the future of AI, for anyone, including myself, to pretend otherwise. The WELT interview below, published three weeks ago, mixes in many themes important to our website and brings in current events, I hesitate to even add comments about it. So, without further ado, please watch it:April 22, 2022
Some interesting statistics about the conferences, the organizations, the individuals, and the countries with most published papers in ML and NLP in 2021: ML and NLP Publications in 2021.April 19, 2022
New architectures for neural networks allow them to learn entire operators, not just individual functions. Operators are mapping (infinite-dimensional) spaces of functions to other such spaces. This trend is one of the most exciting developments, as the cross fertilization between AI and mathematics continues. Many important natural processes studied in science and engineering (turbulence around airplanes for example) are described by Partial Differential Equations, notoriously difficult to solve. These new neural architectures allow for much faster solutions, practically impossible to find with older solvers. Latest Neural Nets Solve World’s Hardest Equations Faster Than Ever Before. One such architecture, the Fourier Neural Operator, is described in the video below. It's quite technical and long. I'm working on a larger note, "AI and Mathematics", in which I will attempt to explain these new developments in a more popular tone.March 10, 2022
The neural network architectures that make up transformers continue to be tweaked so that they can be used for data other than their original application, NLP.
This Quanta article reviews the progress: Will Transformers Take Over Artificial Intelligence?.From the article: "The success of transformers prompted the AI crowd to ask what else they could do. The answer is unfolding now, as researchers report that transformers are proving surprisingly versatile. In some vision tasks, like image classification, neural nets that use transformers have become faster and more accurate than those that don’t. Emerging work in other AI areas — like processing multiple kinds of input at once, or planning tasks — suggests transformers can handle even more."
February 1, 2022
3D convolutional networks are good at aggregating local context while the self-attention mechanisms in transformers are good at capturing global dependencies. To better learn from video data, a combined architecture is proposed. This is one trend that is sure to strengthen in the coming months and years: UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning.January 25, 2022
One of the most intriguing approaches to AI is taken by Numenta, a company started by Jeff Hawkins. Numenta's approach is described in very readable terms in the excellent blog The Path to Machine Intelligence: Classic AI vs. Deep Learning vs. Biological Approach. The blog makes references to Hawkins' Thousand Brain Theory, described in more detail in the book with that same title. One of the main ideas in all this work is that intelligence is still an exclusive and distinguished property of the human brain and it is not yet present in machine learning in general or in deep learning in particular. Numenta follows that dual track, neuroscience and machine learning, in their research. The following interview with Hawkins and Subutai Ahmad, who leads the machine learning track, covers some of these ideas. It is being conducted by Peter Norvig, formed Director of Research at Google, now a Distinguished Education Fellow at the Stanford Institute for Human-Centered AI, and the author with Berkeley's Stuart Russell of the most authoritative textbook on modern AI: Artificial Intelligence: A Modern Approach. Here it is:November 18, 2021
This is one of the most fascinating areas for me, I have been interested in what is now called Geometric Deep Learning for some time. And more generally, the synergy between AI and various areas of mathematics. The synergy with more applied kind of mathematics was well understood, but it wasn't obvious at all that AI can do geometry (or algebra or analysis or topology or number theory) or that these mathematical disciplines can be used in AI.
This is changing rapidly. While the mathematics used in standard deep learning is not complicated, Graph Neural Networks (GNNs) have been more difficult to tackle with elementary methods. Differential geometry and algebraic topology bring a new perspective. This Quanta article reviews the progress: Graph Neural Networks through the lens of Differential Geometry and Algebraic Topology.
November 1, 2021
Below is yesterday's 60 MINUTES interview with Yuval Noah Harari about the future of AI. AI is being increasingly given data not just about where we go, what we do, what we buy, and how we vote, but also biometric data about our biological makeup. The notion of hacking another human being (ethically or not), which Harari says it means knowing that person better than they know themselves, is identical with the concept of a digital twin in the merged national graphs, which is a central concept on our website.June 3, 2021
This is an interview with Dr. Henry Kissinger in which he explains how he got interested in AI. It brings a unique perspective to our topic from an icon of US diplomacy, and it's especially relevant to the article AI in a Bipolar World. You might also want to listen to his interview at the Bush Center about the US, Russia, China, and AI, for additional insights. In more recent interviews with Dr. Kissinger you will also find support for one of our main points, namely the need to ramp up our AI efforts in the US considerably, and not just in the private sector.May 20, 2021
Transformers have been the most interesting new development in neural networks, especially in language applications, where in the beginning neural networks did not enjoy the same success as in computer vision applications. I've started to write a short technical note about them here: Transformers.February 15, 2021
Will AI replace or augment humans in the workforce? This is a difficult question and many very different answers have been given. This is a thoughtful presentation of this dilemma: Artificial Intelligence And The End Of Work. Most technologists lean towards augmentation. But the author of the article is leaning towards replacement, not augmentation, and reflects on what this might mean for humanity. From the article: "The myth of augmentation has spread far and wide in real-world contexts, too. One powerful reason why: job loss from automation is a frightening prospect and a political hot potato. Let’s unpack that. Entrepreneurs, technologists, politicians and others have much to gain by believing—and by persuading others to believe—that AI will not replace but rather will supplement humans in the workforce. Employment is one of the most basic social and political necessities in every society in the world today. To be openly job-destroying is therefore a losing proposition for any technology or business."February 10, 2021
AI can pose new mathematical conjectures, a first result of this kind. Posing good conjectures in mathematics seemed to be a very creative human endeavor, so the fact that AI can sift through enough math and create interesting conjectures is pretty significant. Machines Are Inventing New Math We've Never Seen, published by researchers from the Technion in Haifa and from Google's operation in Tel Aviv.December 17, 2020
AI has been used in many ways to fight the COVID-19 pandemic, especially for the development of new vaccines. This time it is being used after the vaccines are administered, to track the after effects. From the article: "The technology will also track issues or trends related to ethnicity, age, gender, or other demographic factors that can come into play with the vaccine" : How AI is Helping with COVID-19 Vaccine Rollout and Tracking.November 11, 2020
Mathematics can be looked at as a game, played according to rules specified within what we call formal systems. In theory, a computer can be programmed to produce all the theorems of such a formal system. So it could produce all the algebra, the geometry, and the calculus you have learned in school. In practice though, mathematicians still have to rely on a great deal of inventiveness to find such proofs. Nevertheless, the strive to develop proof assistants has never ceased and there have been many notable but disparate successes. Now this work is being pushed even further, and in a more organized way. From the article The Effort to Build the Mathematical Library of the Future: "Digitizing mathematics is a longtime dream. The expected benefits range from the mundane—computers grading students’ homework—to the transcendent: using artificial intelligence to discover new mathematics and find new solutions to old problems. Mathematicians expect that proof assistants could also review journal submissions, finding errors that human reviewers occasionally miss, and handle the tedious technical work that goes into filling in all the details of a proof."
If this interests you in a more serious way, I have written a program of study for formal software development, in which these ideas are treated in more depth: Formal Software Development Program. It's in a slide presentation format, which allows a smooth gradual viewing of each page; view it in Adobe Acrobat Reader (free download from Adobe), not in your browser.
November 3, 2020
An interesting view on the future of Artificial Intelligence, in an interview with Geoffrey Hinton, a pioneer of deep learning techniques. For his contributions, Hinton was awarded the Turing Award last year, together with Yann LeCun and Yoshua Bengio. Interviewer: "You think deep learning will be enough to replicate all of human intelligence. What makes you so sure?"
Hinton: "I do believe deep learning is going to be able to do everything, but I do think there’s going to have to be quite a few conceptual breakthroughs. For example, in 2017 Ashish Vaswani et al. introduced transformers, which derive really good vectors representing word meanings. It was a conceptual breakthrough. It’s now used in almost all the very best natural-language processing. We’re going to need a bunch more breakthroughs like that."
October 12, 2020
A timely Forbes article on some future directions of AI, with an informative look at the types of networks called transformers, the recent success they have had in NLP, and their potential uses in other areas, like computer vision. The Next Generation Of Artificial Intelligence. I'll probably add it to the permanent "Further Reading" list as a complement to the Main AI Concepts article.October 6, 2020
NVIDIA Uses AI to Slash Bandwidth on Video Calls. Obviously, this is even more meaningful given the current reliance on video calls by businesses and individuals, given the covid health crisis. From the article:"What the researchers have achieved has remarkable results: by replacing the traditional h.264 video codec with a neural network, they have managed to reduce the required bandwidth for a video call by an order of magnitude. In one example, the required data rate fell from 97.28 KB/frame to a measly 0.1165 KB/frame – a reduction to 0.1% of required bandwidth. The mechanism behind AI-assisted video conferencing is breathtakingly simple. The technology works by replacing traditional full video frames with neural data. Typically, video calls work by sending h.264 encoded frames to the recipient, and those frames are extremely data-heavy. With AI-assisted video calls, first, the sender sends a reference image of the caller. Then, instead of sending a stream of pixel-packed images, it sends specific reference points on the image around the eyes, nose, and mouth."