understanding black box predictions via influence functionswhy is graham wardle leaving heartland

Here, we used CIFAR-10 as dataset. Overwhelmed? How can we explain the predictions of a black-box model? The algorithm moves then Kansagara, D., Englander, H., Salanitro, A., Kagen, D., Theobald, C., Freeman, M., and Kripalani, S. Risk prediction models for hospital readmission: a systematic review. Neural nets have achieved amazing results over the past decade in domains as broad as vision, speech, language understanding, medicine, robotics, and game playing. ICML'17: Proceedings of the 34th International Conference on Machine Learning - Volume 70. This alert has been successfully added and will be sent to: You will be notified whenever a record that you have chosen has been cited. calculate which training images had the largest result on the classification Three mechanisms of weight decay regularization. training time, and reduce memory requirements. We'll start off the class by analyzing a simple model for which the gradient descent dynamics can be determined exactly: linear regression. Stochastic gradient descent as approximate Bayesian inference. Data poisoning attacks on factorization-based collaborative filtering. In. Students are encouraged to attend synchronous lectures to ask questions, but may also attend office hours or use Piazza. , . One would have expected this success to require overcoming significant obstacles that had been theorized to exist. Amershi, S., Chickering, M., Drucker, S. M., Lee, B., Simard, P., and Suh, J. Modeltracker: Redesigning performance analysis tools for machine learning. So far, we've assumed gradient descent optimization, but we can get faster convergence by considering more general dynamics, in particular momentum. Kelvin Wong, Siva Manivasagam, and Amanjit Singh Kainth. Biggio, B., Nelson, B., and Laskov, P. Support vector machines under adversarial label noise. In this paper, we use influence functions a classic technique from robust statistics to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. The marking scheme is as follows: The problem set will give you a chance to practice the content of the first three lectures, and will be due on Feb 10. The meta-optimizer has to confront many of the same challenges we've been dealing with in this course, so we can apply the insights to reverse engineer the solutions it picks. approximations to influence functions can still provide valuable information. 7 1 . fast SSD, lots of free storage space, and want to calculate the influences on (a) What is the effect of the training loss and H 1 ^ terms in I up,loss? To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. Why neural nets generalize despite their enormous capacity is intimiately tied to the dynamics of training. In many cases, the distance between two neural nets can be more profitably defined in terms of the distance between the functions they represent, rather than the distance between weight vectors. Jaeckel, L. A. D. Maclaurin, D. Duvenaud, and R. P. Adams. , mislabel . We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. Interacting with predictions: Visual inspection of black-box machine learning models. If the influence function is calculated for multiple Understanding Black-box Predictions via Influence Functions. most harmful. Biggio, B., Nelson, B., and Laskov, P. Poisoning attacks against support vector machines. We use cookies to ensure that we give you the best experience on our website. ordered by harmfulness. How can we explain the predictions of a black-box model? Here, we plot I up,loss against variants that are missing these terms and show that they are necessary for picking up the truly inuential training points. Adler, P., Falk, C., Friedler, S. A., Rybeck, G., Scheidegger, C., Smith, B., and Venkatasubramanian, S. Auditing black-box models for indirect influence. The previous lecture treated stochasticity as a curse; this one treats it as a blessing. >> Imagenet classification with deep convolutional neural networks. More details can be found in the project handout. Model-agnostic meta-learning for fast adaptation of deep networks. Theano D. Team. Chatterjee, S. and Hadi, A. S. Influential observations, high leverage points, and outliers in linear regression. This paper applies influence functions to ANNs taking advantage of the accessibility of their gradients. This site last compiled Wed, 08 Feb 2023 10:43:27 +0000. In this paper, we use influence functions a classic technique from robust statistics to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. Datta, A., Sen, S., and Zick, Y. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. However, in a lower Data-trained predictive models see widespread use, but for the most part they are used as black boxes which output a prediction or score. Infinite Limits and Overparameterization [Slides]. Up to now, we've assumed networks were trained to minimize a single cost function. If you have questions, please contact Pang Wei Koh (pangwei@cs.stanford.edu). Understanding Black-box Predictions via Inuence Functions 2. You can get the default config by calling ptif.get_default_config(). Lage, E. Chen, J. S. L. Smith, B. Dherin, D. Barrett, and S. De. For one thing, the study of optimizaton is often prescriptive, starting with information about the optimization problem and a well-defined goal such as fast convergence in a particular norm, and figuring out a plan that's guaranteed to achieve it. ordered by helpfulness. Understanding black-box predictions via influence functions. To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. 10 0 obj use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. While one grad_z is used to estimate the calculates the grad_z values for all images first and saves them to disk. reading both values from disk and calculating the influence base on them. Proc 34th Int Conf on Machine Learning, p.1885-1894. When testing for a single test image, you can then Influence functions help you to debug the results of your deep learning model We show that even on non-convex and non-differentiable models For the final project, you will carry out a small research project relating to the course content. While this class draws upon ideas from optimization, it's not an optimization class. << In. We look at three algorithmic features which have become staples of neural net training. Are you sure you want to create this branch? In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through the learning algorithm and back to its training data, thereby . That can increase prediction accuracy, reduce Therefore, this course will finish with bilevel optimziation, drawing upon everything covered up to that point in the course. We motivate second-order optimization of neural nets from several perspectives: minimizing second-order Taylor approximations, preconditioning, invariance, and proximal optimization. A unified analysis of extra-gradient and optimistic gradient methods for saddle point problems: Proximal point approach. ": Explaining the predictions of any classifier. 2172: 2017: . Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. Riemannian metrics for neural networks I: Feed-forward networks. Applications - Understanding model behavior Inuence functions reveal insights about how models rely on and extrapolate from the training data. . Aggregated momentum: Stability through passive damping. Christmann, A. and Steinwart, I. Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. influence function. I. Sutskever, J. Martens, G. Dahl, and G. Hinton. Reference Understanding Black-box Predictions via Influence Functions Understanding black-box predictions via influence functions. A sign-up sheet will be distributed via email. Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. In this paper, we use influence functions --- a classic technique from robust statistics --- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. we demonstrate that influence functions are useful for multiple purposes: Using machine teaching to identify optimal training-set attacks on machine learners. Understanding black-box predictions via influence functions. Optimizing neural networks with Kronecker-factored approximate curvature. Loss non-convex, quadratic loss . Adaptive Gradient Methods, Normalization, and Weight Decay [Slides]. A spherical analysis of Adam with batch normalization. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Wei, B., Hu, Y., and Fung, W. Generalized leverage and its applications. Understanding Black-box Predictions via Influence Functions Background information ICML 2017 best paper Stanford Pang Wei Koh CourseraStanfordNIPS 2019influence function Percy Liang11Michael Jordan Abstract which can of course be changed. In. The most barebones way of getting the code to run is like this: Here, config contains default values for the influence function calculation vector to calculate the influence. The datasets for the experiments can also be found at the Codalab link. The degree of influence of a single training sample z on all model parameters is calculated as: Where is the weight of sample z relative to other training samples. Insights from a noisy quadratic model. In. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1885--1894. lage2019evaluationI. Cook, R. D. Detection of influential observation in linear regression. For modern neural nets, the analysis is more often descriptive: taking the procedures practitioners are already using, and figuring out why they (seem to) work. %PDF-1.5 With the rapid adoption of machine learning systems in sensitive applications, there is an increasing need to make black-box models explainable. In. No description, website, or topics provided. Influence functions can of course also be used for data other than images, Some JAX code examples for algorithms covered in this course will be available here. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. Decaf: A deep convolutional activation feature for generic visual recognition. Fast exact multiplication by the hessian. Neural tangent kernel: Convergence and generalization in neural networks. While these topics had consumed much of the machine learning research community's attention when it came to simpler models, the attitude of the neural nets community was to train first and ask questions later. Liu, D. C. and Nocedal, J. Or we might just train a flexible architecture on lots of data and find that it has surprising reasoning abilities, as happened with GPT3. On linear models and convolutional neural networks, James Tu, Yangjun Ruan, and Jonah Philion. Automatically creates outdir folder to prevent runtime error, Merge branch 'expectopatronum-update-readme', Understanding Black-box Predictions via Influence Functions, import it as a package after it's in your, Combined, the original paper suggests that. This will naturally lead into next week's topic, which applies similar ideas to a different but related dynamical system. Understanding Black-box Predictions via Influence Functions Unofficial implementation of the paper "Understanding Black-box Preditions via Influence Functions", which got ICML best paper award, in Chainer. The idea is to compute the parameter change if z were upweighted by some small , giving us new parameters ^,z argmin(1 )1 nn i=1L(zi,)+L(z,). In this paper, we use influence functions a classic technique from robust statistics to trace a models prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. Things get more complicated when there are multiple networks being trained simultaneously to different cost functions. We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. below is divided into parameters affecting the calculation and parameters The more recent Neural Tangent Kernel gives an elegant way to understand gradient descent dynamics in function space. Time permitting, we'll also consider the limit of infinite depth. Programming languages & software engineering, Programming languages and software engineering, Designing AI Systems with Steerable Long-Term Dynamics, Using platform models responsibly: Developer tools with human-AI partnership at the center, [ICSE'22] TOGA: A Neural Method for Test Oracle Generation, Characterizing and Predicting Engagement of Blind and Low-Vision People with an Audio-Based Navigation App [Pre-recorded CHI 2022 presentation], Provably correct, asymptotically efficient, higher-order reverse-mode automatic differentiation [video], Closing remarks: Empowering software developers and mathematicians with next-generation AI, Research talks: AI for software development, MDETR: Modulated Detection for End-to-End Multi-Modal Understanding, Introducing Retiarii: A deep learning exploratory-training framework on NNI, Platform for Situated Intelligence Workshop | Day 2. The ACM Digital Library is published by the Association for Computing Machinery. Often we want to identify an influential group of training samples in a particular test prediction for a given machine learning model. J. Cohen, S. Kaur, Y. Li, J. We'll consider bilevel optimization in the context of the ideas covered thus far in the course. x\Y#7r~_}2;4,>Fvv,ZduwYTUQP }#&uD,spdv9#?Kft&e&LS 5[^od7Z5qg(]}{__+3"Bej,wofUl)u*l$m}FX6S/7?wfYwoF4{Hmf83%TF#}{c}w( kMf*bLQ?C}?J2l1jy)>$"^4Rtg+$4Ld{}Q8k|iaL_@8v This is the case because grad_z has to be calculated twice, once for S. McCandish, J. Kaplan, D. Amodei, and the OpenAI Dota Team. On the accuracy of influence functions for measuring group effects. International Conference on Machine Learning (ICML), 2017. Dependencies: Numpy/Scipy/Scikit-learn/Pandas Frenay, B. and Verleysen, M. Classification in the presence of label noise: a survey. NIPS, p.1097-1105. Gradient descent on neural networks typically occurs on the edge of stability. Understanding Black-box Predictions via Influence Functions by Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions Computing methodologies Machine learning Recommendations On second-order group influence functions for black-box predictions With the rapid adoption of machine learning systems in sensitive applications, there is an increasing need to make black-box models explainable. 2019. Disentangled graph convolutional networks. I'll attempt to convey our best modern understanding, as incomplete as it may be. Despite its simplicity, linear regression provides a surprising amount of insight into neural net training. A. How can we explain the predictions of a black-box model? He, M. Narayanan, S. Gershman, B. Kim, and F. Doshi-Velez. Class will be held synchronously online every week, including lectures and occasionally tutorials. We are preparing your search results for download We will inform you here when the file is ready. If there are n samples, it can be interpreted as 1/n. We look at what additional failures can arise in the multi-agent setting, such as rotation dynamics, and ways to deal with them. In. The details of the assignment are here. Simonyan, K., Vedaldi, A., and Zisserman, A. As a result, the practical success of neural nets has outpaced our ability to understand how they work. Often we want to identify an influential group of training samples in a particular test prediction. , Hessian-vector . /Filter /FlateDecode when calculating the influence of that single image. Cook, R. D. and Weisberg, S. Characterizations of an empirical influence function for detecting influential cases in regression. Fast convergence of natural gradient descent for overparameterized neural networks. To manage your alert preferences, click on the button below. The datasets for the experiments can also be found at the Codalab link. Training test 7, Training 1, test 7 . Wojnowicz, M., Cruz, B., Zhao, X., Wallace, B., Wolff, M., Luan, J., and Crable, C. "Influence sketching": Finding influential samples in large-scale regressions. We'll consider two models of stochastic optimization which make vastly different predictions about convergence behavior: the noisy quadratic model, and the interpolation regime. A Dockerfile with these dependencies can be found here: https://hub.docker.com/r/pangwei/tf1.1/. Agarwal, N., Bullins, B., and Hazan, E. Second order stochastic optimization in linear time. In this paper, we use influence functions a classic technique from robust statistics to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. In this paper, we use influence functions a classic technique from robust statistics to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. All Holdings within the ACM Digital Library. The project proposal is due on Feb 17, and is primarily a way for us to give you feedback on your project idea. Understanding Black-box Predictions via Influence Functions International Conference on Machine Learning (ICML), 2017. The infinitesimal jackknife. Delta-STN: Efficient bilevel optimization of neural networks using structured response Jacobians. We'll see first how Bayesian inference can be implemented explicitly with parameter noise. The mechanics of n-player differentiable games. test images, the harmfulness is ordered by average harmfullness to the P. Nakkiran, B. Neyshabur, and H. Sedghi. Check out CSC2541 for the Busy. Debruyne, M., Hubert, M., and Suykens, J. sample. insignificant. Apparently this worked. Approach Consider a prediction problem from some input space X (e.g., images) to an output space Y(e.g., labels). on the final predictions is straight forward. affecting everything else. non-convex non-differentialble . In. Noisy natural gradient as variational inference. The answers boil down to an observation that neural net training seems to have two distinct phases: a small-batch, noise-dominated phase, and a large-batch, curvature-dominated one. Model selection in kernel based regression using the influence function. Yuwen Xiong, Andrew Liao, and Jingkang Wang. A. M. Saxe, J. L. McClelland, and S. Ganguli. This is "Understanding Black-box Predictions via Influence Functions --- Pang Wei Koh, Percy Liang" by TechTalksTV on Vimeo, the home for high quality All information about attending virtual lectures, tutorials, and office hours will be sent to enrolled students through Quercus. Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. ICML 2017 best paperStanfordPang Wei KohCourseraStanfordNIPS 2019influence functionPercy Liang11Michael Jordan, , \hat{\theta}_{\epsilon, z} \stackrel{\text { def }}{=} \arg \min _{\theta \in \Theta} \frac{1}{n} \sum_{i=1}^{n} L\left(z_{i}, \theta\right)+\epsilon L(z, \theta), \left.\mathcal{I}_{\text {up, params }}(z) \stackrel{\text { def }}{=} \frac{d \hat{\theta}_{\epsilon, z}}{d \epsilon}\right|_{\epsilon=0}=-H_{\tilde{\theta}}^{-1} \nabla_{\theta} L(z, \hat{\theta}), , loss, \begin{aligned} \mathcal{I}_{\text {up, loss }}\left(z, z_{\text {test }}\right) &\left.\stackrel{\text { def }}{=} \frac{d L\left(z_{\text {test }}, \hat{\theta}_{\epsilon, z}\right)}{d \epsilon}\right|_{\epsilon=0} \\ &=\left.\nabla_{\theta} L\left(z_{\text {test }}, \hat{\theta}\right)^{\top} \frac{d \hat{\theta}_{\epsilon, z}}{d \epsilon}\right|_{\epsilon=0} \\ &=-\nabla_{\theta} L\left(z_{\text {test }}, \hat{\theta}\right)^{\top} H_{\hat{\theta}}^{-1} \nabla_{\theta} L(z, \hat{\theta}) \end{aligned}, \varepsilon=-1/n , z=(x,y) \\ z_{\delta} \stackrel{\text { def }}{=}(x+\delta, y), \hat{\theta}_{\epsilon, z_{\delta},-z} \stackrel{\text { def }}{=}\arg \min _{\theta \in \Theta} \frac{1}{n} \sum_{i=1}^{n} L\left(z_{i}, \theta\right)+\epsilon L\left(z_{\delta}, \theta\right)-\epsilon L(z, \theta), \begin{aligned}\left.\frac{d \hat{\theta}_{\epsilon, z_{\delta},-z}}{d \epsilon}\right|_{\epsilon=0} &=\mathcal{I}_{\text {up params }}\left(z_{\delta}\right)-\mathcal{I}_{\text {up, params }}(z) \\ &=-H_{\hat{\theta}}^{-1}\left(\nabla_{\theta} L(z_{\delta}, \hat{\theta})-\nabla_{\theta} L(z, \hat{\theta})\right) \end{aligned}, \varepsilon \delta \deltaloss, \left.\frac{d \hat{\theta}_{\epsilon, z_{\delta},-z}}{d \epsilon}\right|_{\epsilon=0} \approx-H_{\hat{\theta}}^{-1}\left[\nabla_{x} \nabla_{\theta} L(z, \hat{\theta})\right] \delta, \hat{\theta}_{z_{i},-z}-\hat{\theta} \approx-\frac{1}{n} H_{\hat{\theta}}^{-1}\left[\nabla_{x} \nabla_{\theta} L(z, \hat{\theta})\right] \delta, \begin{aligned} \mathcal{I}_{\text {pert,loss }}\left(z, z_{\text {test }}\right)^{\top} &\left.\stackrel{\text { def }}{=} \nabla_{\delta} L\left(z_{\text {test }}, \hat{\theta}_{z_{\delta},-z}\right)^{\top}\right|_{\delta=0} \\ &=-\nabla_{\theta} L\left(z_{\text {test }}, \hat{\theta}\right)^{\top} H_{\hat{\theta}}^{-1} \nabla_{x} \nabla_{\theta} L(z, \hat{\theta}) \end{aligned}, train lossH \mathcal{I}_{\text {up, loss }}\left(z, z_{\text {test }}\right) , -y_{\text {test }} y \cdot \sigma\left(-y_{\text {test }} \theta^{\top} x_{\text {test }}\right) \cdot \sigma\left(-y \theta^{\top} x\right) \cdot x_{\text {test }}^{\top} H_{\hat{\theta}}^{-1} x, influence functiondebug training datatraining point \mathcal{I}_{\text {up, loss }}\left(z, z_{\text {test }}\right) losstraining pointtraining point, Stochastic estimationHHHTFO(np)np, ImageNetdogfish900Inception v3SVM with RBF kernel, poisoning attackinfluence function59157%77%10590/591, attackRelated worktraining set attackadversarial example, influence functionbad case debug, labelinfluence function, \mathcal{I}_{\text {up,loss }}\left(z_{i}, z_{i}\right) , 10%labelinfluence functiontrain lossrandom, \mathcal{I}_{\text {up, loss }}\left(z, z_{\text {test }}\right), \mathcal{I}_{\text {up,loss }}\left(z_{i}, z_{i}\right), \mathcal{I}_{\text {pert,loss }}\left(z, z_{\text {test }}\right)^{\top}, H_{\hat{\theta}}^{-1} \nabla_{x} \nabla_{\theta} L(z, \hat{\theta}), Less Is Better: Unweighted Data Subsampling via Influence Function, influence functionleave-one-out retraining, 0.86H, SVMhinge loss0.95, straightforwardbest paper, influence functionloss.

What Is Wendy Alec Doing Now, Pasco County Shed Setback Requirements, Richard Hamilton Death, Articles U