1、Python security insightsSeptember 2021 tl;drIntroductionThe security footprint of a typical Python projectThe most common security issues in Python projectsDealing with vulnerabilities in Python containersVulnerability spotlight:ipaddressThe most commonly downloaded Python packagesThe most common vu

2、lnerable packages in Python projectsPackage spotlight:urllib3Fixing vulnerabilitiesKeep your code secureTackling known vulnerabilities Picking the right container imageConclusion345796171819Table of contents*tl;drPyPI stat?316,360 package?2,739,363 version release?4,595,611 package file?8

3、.3TB total size of packages on PyPIPython project?An average project has 35 dependencies with an almost 50/50 split between direct and indirect dependencie?An average vulnerable project consists of 33 known vulnerabilitiesSecurity takeawaysBad new?47%of Python projects contain known vulnerabilitie?3

4、3%of all known Python vulnerabilities are high and critical severity issue?In over 60%of Python projects,code related elements of OWASP Top 10 2021 list of issues can be found(e.g.Command Injection,XSS).Good new?You can eliminate 87%of known vulnerabilities by upgrading the vulnerable packag?Most co

5、ntainer vulnerabilities can be fixed using slimmer image?The most popular Python packages are healthy,receiving an average Snyk Advisor health score of 81%Python container?Over 1500 different Python image tags were pulled down from Docker in the last month alone?python:latest(AKA python:3.9),has 431

6、 pre-installed packages with 353 vulnerabilities?Most vulnerabilities can quickly be“fixed”by using slimmer images?python:slim(Debian-based)has only 94 packages and 78 vulnerabilitie?python:alpine has just 37 packages and 0 vulnerabilitiesData taken from the Snyk Intel Vulnerability Database,hundred

7、s of thousands of Python projects monitored by Snyk,the same number of projects used as a training set for Snyk Code,and Snyk Advisor.Python known vulnerabilitie?On average,60 new Python vulnerabilities are added to Snyks vulnerability database on a monthly basi?5%of Python vulnerabilities are criti

8、cal severity issues,27%are high severity,56%are medium severity and 12%are low severit?87%of vulnerabilities in Python packages have available remediationTypical types of Python issues:(1)XSS(2)TLS Certificate verification disabled(3)Path traversal(4)Hard coded secret(non crypto)(5)SQL Injection9%of

9、 XSS,Path Traversal,and SQL Injections were interfile.Especially,Django and Flask XSS and Path Traversal fall into this category.With a huge and ever-growing ecosystem of libraries,frameworks,and tools used by a community of over 10 million developers worldwide,the stage is set for Python to overtak

10、e contenders to become the most popular programming language.Being a high-level,versatile,object-oriented language,Python is easy to learn while also being useful and powerful.This makes it an ideal choice by programmers of all backgrounds for a variety of projects:data analysis,web development,inte

11、rnet of things development(IoT),machine learning,DevOps,scripting,and plenty of other computing uses.Python application development has changed over the years.Today,a typical Python repository will consist of much more than just the Python code written by the developer,but will also pull in more ope

12、n source packages.It will likely include container images as well as configuration files used for provisioning the infrastructure required to run them.A lot of what used to be the responsibility of IT is now code built and managed by developers.From a security perspective,this means one thing-the at

13、tack surface is much wider than before.Scanning a Python project will shed light on all the different nuts and bolts of your application-the vulnerabilities they introduce and(in the best case)the remediation path.But its likely you will still have questions.How do your Python projects compare to ot

14、hers?How are other Python developers using open source packages?Are there any security trends in the Python ecosystem as a whole?This report aims to cover these questions,and includes the following:?The security footprint of a typical Python projec?The most common vulnerabilities seen in Python appl

15、ication?The most commonly used Python packages as well as the common vulnerable packagesIntroductionSnyk performs millions of monthly scans of hundreds of thousands of Python projects.We now have the ability to describe what an average Python project looks like and can give you an idea of what you m

16、ight find when you scan one of your Python projects.Known vulnerabilitiesPython projects monitored by Snyk contain 1.3M distinct direct dependencies and 1.4M distinct transitive dependencies.An average Python project has around 35 dependencies.Out of these,17 are direct dependencies and 18 are indir

17、ect dependencies.In 47%of these projects,dependencies are introducing vulnerabilities.An average vulnerable project consists of 33 known vulnerabilities,out of which 10%are critical severity vulnerabilities,26%are high severity,26%medium severity,and 38%low severity Container issuesRunning Python ap

18、plications in containers is a popular option as it eases the management of Python versions and dependencies.Docker provides official Python images going back as far as Python 2.7.There are also images available in popular Python data science packages like pandas,NumPy,and Jupyter.The security footpr

19、int of a typical Python projectVulnerabilities in projects by severityLow 30%High26%Critical10%Medium30%Indirectdependancies51%49%DirectdependanciesAverage makeup of a dependency treeSnyks customers scan 10s of thousands of new containers each month and Python is the 3rd most popular language-based

20、container in use,which aligns with the overall popularity of Python.Images built directly from Dockers official Python image family are the most popular:93%of the readily identifiable images we scan are built on Dockers Official images.Unfortunately,while the use of slimmer images is a well-known be

21、st practice,and Docker provides many slim options,relatively heavy images like python:3.7 are still the most common Python base images detected by Snyk.Python:3.7 has over 400 packages in it,resulting in over 300 vulnerabilities detected.Multistage builds are a great way to allow for the bigger imag

22、es in development,while ensuring the final production image is as light and secure as possible.Security weaknessesSnyk Code support for Python is relatively new but Snyk is already performing static code analysis for thousands of Python projects.Roughly speaking,20%of the security weaknesses identif

23、ied by Snyk Code are related to Python projects.Out of these,25%are high severity issues,60%are medium severity issues and 15%are low severity issues.Overall,issues show up in over 60%of the Python projects scanned by Snyk Code.Based on the use of Dockers official images,as detected by Snyk Containe

24、r.Python is the fourth most popular language-based container and Python 3.7 is the most popular version amongst Snyks users.Relative popularity of language-specific container imagesPython versions in use based on Snyk Container scans nodejavagolangpythonrubyaspnetphpThe most common security issues i

25、n Python projectsTop 5 Python Issues Reported by Snyk Code12%14%23%23%28%Lets start our deeper dive into the state of security for Python projects with a look at the most common security issues found in Python projects.The findings you see here are based on the training set used for Snyk Code-Snyks

26、Static Application Security Testing(SAST)solution,which numbered over 120k public Python projects on GitHub.A comparison to a recent study comparing vulnerabilities types(CWE)found in PyPI,confirms the pattern.Unsurprisingly,you can see OWASPs Top 10 reflected in the results.There are two issues spe

27、cific to Python,though?Unicode issues:Like most programming languages older than 10 years,Python has a history of issues related to handling encoding in strings.?Closing API calls:When interacting with external resources such as file or network streams,calling the close function signals the system t

28、o flush the content as well as to free any handles.Developers in other languages-such as C+or Java-are a bit more disciplined in managing these resources.Pro Tip:Snyks Python cheatsheet provides a list of typical issue types with concrete advice.SQL InjectionHardcoded secretPath TraversalTLS Cert Di

29、sabledXSSWithin the training set for Snyk Code of 131,910 Python Open Source packages,we see the following distribution of issue types:Type:Rel.within all issuesExplanationCWEXSS(Cross Site Scripting)TLS Certificate Verification DisabledPath TraversalHard Coded Secret(non-crypto)SQL InjectionHandled

30、 UnicodeMissing closing call to APIBinary WriteWait can DeadlockCommand Injection20%16%16%10%9%7%7%7%4%3%CWE-79CWE-295CWE-23CWE-798CWE-259CWE-89CWE-176CWE-400CWE-116CWE-1322CWE-78Attacker injects client-side scripts into website.Certificate verification is disabled by disabling verification calling

31、certain functions.Possible man-in-the-middle attacks.User supplied file names or paths are used and attackers could traverse the filesystem.Hard coded values like account names,passwords,paths or file names should be kept a secret.User supplied string is used to construct SQL queries and could be us

32、ed for SQL injection attacksCalling functions that need encoded parameters or provides encoded results without proper formatting itAccess to external or buffered resource without proper closing may lead to data loss(e.g.when storing a resource as global)When opening a file to write,not using binary

33、mode may result in issues if you are using Python 3(or on Windows).When working with child processes,wait can lead to deadlocks on long running processes.Unsanitized external data is used to build a command line call.Attackers might inject commands to be executed.Running Python apps in containers is

34、 quite common,because its an elegant way of dealing with Pythons dependencies and virtual environments.In fact,the open source project that would go on to become Docker was first announced and demonstrated on stage by Solomon Hykes at PyCon 2013.While its pretty easy to get a Python app running in a

35、 container,as with so much in life the“easy way”isnt always the best way.Containers come with pre-installed Linux packages,which may or may not be important to making your app run but will impact your vulnerability reports.Lets take a quick look at some Python base image options to see how they vary

36、 in size and vulnerabilities:All of these images are very popular,ranking in the top tags pulled from Dockers Official Images for Python.But“fat”images like:3.9,:3.8,and:3.7 are the most common,despite all the best practices stating you should use a slimmer base image to reduce attack surface.You ca

37、n get both worlds in containers through the use of multi-stage builds,starting with the bigger images to simplify building and testing your code and then moving the required production packages to a slim image in the final stage.The key takeaway here is that smaller usually is better,when it comes t

38、o security.But diligence is still required.In addition,as fixes for vulnerabilities are published by the Linux maintainers,Docker updates their images.For this reason,even if your code doesnt change,it is worthwhile to have a process that rebuilds,tests,and redeploys your container images.For compar

39、ison purposes,we included Googles distroless image as well.Distroless images remove the Linux shell,package manager,and other components resulting in a very small image.But Dockers python:3.9-alpine image,while slightly larger,actually has fewer vulnerabilities.Dealing with vulnerabilities in Python

40、 containersIn March this year,security researchers Sick Codes,Victor Viale,Kelly Kaoudis,John Jackson,and Nick Sahler uncovered a critical IP validation vulnerability in the netmask package.The vulnerability existed in both npm and Perl versions of netmask but was also identified later as impacting

41、the ipaddress stdlib package in Python.ipaddress enables developers to easily create IP addresses,networks,and interfaces,and to parse/normalize IP addresses inputted in different formats.The uncovered vulnerability in ipaddress is related to the way the package parses IP addresses,improperly valida

42、ting octal strings,and rendering IPv4 addresses that contain certain octal strings as integers.This opens up those Python applications using ipaddress to Server-Side Request Forgery(SSRF),Remote File Inclusion(RFI),and Local File Inclusion(LFI)attacks.The vulnerability affects Python version 3.8.0 t

43、hrough 3.10.As reported in Snyk Advisor,ipaddress continues to be downloaded millions of times a week.Its worth noting that the vulnerability was actually first introduced in 2019,two full years before it was discovered.During this time,attackers could have exploited this vulnerability as a 0-day wi

44、thout this becoming public knowledge!2019-03-30-vulnerability introduced2021-03-29 vulnerability discovered2021-03-29 CVE requested2021-04-30 CVE Assigned CVE-2021-29921 2021-05-01 PoC exploit published2021-05-02 commit 60ce8f0(branch 3.10)2021-05-02 commit 5374fbc(branch 3.9)2021-05-03 Python 3.9.5

45、 released2021-05-06 CVE added to NVDVulnerability spotlight:ipaddress TimelinePackage data?Name:ipaddres?Last release:Oct 18,201?Popularity:1.4M weekly download?Security:severity vulnerabilities affecting all versions-1 high severity,2 medium severit?Community:6 contributor?Direct inclusion rate*:19

46、%*Calculated based on the number of projects which include this package as a direct dependency.Vulnerability data?CVE:CVE-2021-2992?Severity level:Hig?CVSS score:7.?Disclosure date:April 30,202?Fix version:no fixed version availab767 daysThe most commonly downloaded Python packages What open source

47、packages are Python developers using?How healthy are these projects?Are these packages secure?How are they included in projects?To answer these questions we looked at two key datasets-the 1000 most downloaded packages from PyPI and the 1000 most common packages used in the Python projects monitored

48、by Snyk.Python packages in general are healthy!To examine PyPI package downloads,we used Snyk Advisor-a free,online,research tool that helps you decide which open source packages or container base images to use to build your Python project.Containing all projects from PyPI,Snyk Advisor calculates a

49、health score based on packages popularity,security,maintenance,and community strength.For the 1000 most downloaded Python packages from PyPI,Snyk Advisor suggests an average high health score of 81%!Usage patterns:Direct vs.Indirect Taking a closer look at the 1000 most common packages used in proje

50、cts monitored by Snyk,we found some interesting findings related to usage patterns.The average direct inclusion rate for these packages is 63%implying that Python packages are mostly included in projects as a direct dependency.Drilling a bit deeper into inclusion rates,popular utility packages,such

51、as boto3(91%)or markupsafe(96%),are more likely to be pulled into a project as a direct dependency.Other popular packages,such as urllib3(31%),six(32%)and certifi(29%)have a much lower direct inclusion rate and are mostly introduced as an indirect dependency.Packageurllib3boto3sixbotocorerequestscer

52、tifisetuptoolsidnachardetpython-dateutilHealth score98.50%95.50%86.93%97.00%97.00%85.70%95.64%83.60%88.43%94.14%#of downloads*158,893,228134,136,171128,572,888128,304,032116,360,262113,424,297113,270,733112,275,170107,130,519100,233,788*Average monthly downloads for the period of March 21-June 21.*A

53、t the time of writing.Last release date*Jun 25,2021Jul 28,2021May 5,2021Jul 27,2021Jul 13,2021May 30,2021Jul 19,2021May 29,2021Dec 10,2020Jul 14,2021Age12 years7 years11 years9 years10 years10 years15 years8 years15 years13 yearsContributors250303802040110LicenseMITApache-2.0MITApache-2.0

54、Apache-2.0MPL-2.0MITBSD-3-ClauseLGPL-2.1Apache-2.0 OR BSD-2-ClauseDependencies0604700001The most common vulnerable packages in Python projectsWeve taken a look at the packages that are downloaded more frequently.Of course,not all packages are vulnerable.Lets now look at the top ten vulnerable Python

55、 packages that are currently impacting Snyks users the most,because they most frequently appear in Snyk project scans.Included in the table are the minimum version upgrades you need to make to move to a vulnerability-free version.As seen in the data,some particularly popular packages are vulnerable.

56、urllib3 stands out here(we take a closer look at this package in the next section)but so do others.pillow-the popular(8.3M weekly downloads)imaging library-is riddled with vulnerabilities.While the latest version is safe to use,previous and widely-used versions include 1 critical severity vulnerabil

57、ity and 16 high severity vulnerabilities.The same goes for Django-a popular web framework for Python.Versions 3.1.13 and above are safe to use but older versions are not as secure.ipaddress and cryptography-downloaded together over 12.7M times a week,currently have no remediation path.Packageurllib3

58、pillowPyYAMLipaddresscryptographydjangojinja2pygmentsrequestsrsaVulnerabilitiesCritical-0 High-3 Medium-6 Low-1Critical-1 High-18 Medium-7 Low-Critical-4Critical-0 High-1 Medium-2 Low-0 Critical-0 High-4 Medium-3 Low-0 Critical-0 High-8 Medium-15 Low-5Critical-0 High-1 Medium-4 Low-5Critical-1 High-

59、2 Medium-0 Low-0Critical-1 High-0 Medium-5 Low-0Critical-0 High-2 Medium-3 Low-0UseHTTP client Imaging libraryYAML parser and emitter IPv4/IPv6 manipulation libraryCryptographic recipes and primitivesHigh-level Python Web frameworkA fast and expressive template engineA syntax highlighting packageHTT

60、P client RSA implementationMinimum knownvuln free version1.26.6(latest)8.3.1(latest)5.4-3.1.132.11.32.7.42.20.04.7Weekly downloads38M8.3M21M1.5M11.2M1.6M17M6M37M16MData as of Aug/2021Package spotlight:urllib3urllib3 is an extremely popular HTTP client for Python,supporting a lot of greatly-needed fu

61、nctionality missing in Python libraries such as thread safety,connection pooling,client-side SSL/TLS verification,file uploads with multipart encoding,and plenty more.With millions of downloads a week,urllib3 is the most downloaded Python package on PyPI.It is also the 3rd most used package in the p

62、rojects monitored by Snyk.From a health perspective,urllib receives top scores with an impressive 100/100 Snyk Advisor health score?Popularity-urllib3 is downloaded over 42 million times a week,marking it as a key project in the Python ecosyste?Maintenance-with 146 open issues,13 open PRs,a recent r

63、elease under a month ago and a commit from just a few days ago,urllib3 shows strong vitality?Security-the latest version of urllib3,version 1.26.6,is free of known security vulnerabilities and safe to us?Community-with 250 developers collaborating on the project,urllib is clearly receiving strong ex

64、ternal contributionsThe devil,as always,is in the details.When looking at usage patterns for urllib3,two worrying data points immediately stand out:Older vulnerable versions-while the latest version of the package is safe to use,previous versions include security vulnerabilities,including high and m

65、edium severity issues.Version 1.24.3,downloaded over 2 million times a week,includes a high severity CRLF injection vulnerability.Version 1.26.3,downloaded over 1.5 million times a week contains an Improper Certificate Validation vulnerability.Indirect inclusion-in the projects monitored by Snyk,url

66、lib3 is pulled in as a direct dependency in only 31%of the cases.This means that vulnerabilities in older versions of the package are more difficult to identify and fix.#1Most downloaded package on PyPI100%Snyk Advisor Health Score2ndMost used package in projects monitored by SnykFixing vulnerabilit

67、iesChoose your dependencies wiselyAdding dependencies using PyPI(or other Python package managers for that matter)is extremely easy.But PyPI does not scan packages before listing them nor does it guarantee the authenticity of a package.Before you add a dependency into your project,some due diligence

68、 is a good best practice.Using Snyk Advisor for example,you will be able to check for known vulnerabilities or license issues as well as gauge how strong the community behind the package is.snyk.io/advisor/Pro tip:Snyk Advisor provides you with the command line to install the package.By copy and pas

69、ting it,you prevent typosquatting.Keep your code secureA modern application consists of only 10-20%proprietary code-the rest of the code base is actually made up of open source code brought in by open source dependencies.But it is this 10-20%that makes the application unique.Containing the intellect

70、ual property of the organization,this portion of code is not maintained by a community of open source developers-it is all yours to take care of!Python provides some build-in mechanisms that can help you develop stable apps-support for virtual environments is one example.Reviewing the top 10 securit

71、y issues found in application code listed above,leads to six general points of advice:?Use modern static code analysis:Linters like Pylint and scanners like Bandit are a good start.But nasty problems are interfile(aka the issue happens as the application execution flows between various source files)

72、.Finding these kinds of issues manually is near impossible?Sanitization of data:Try to sanitize inflowing data from any external sources(including databases)at the entry point in the application.?ORM:Use modern Object Relational Mapping(ORM)tools to abstract the database interactions and prevent SQL

73、 injection opportunities.If you are using packages like Django or Flask,use libraries like Django ORM or SQLAlchemy which are well-vetted?Unicode:If possible,standardize all strings to a certain unicode encoding-we recommend UTF-8.Be careful when converting unicode strings into ASCII?Close APIs:Make

74、 sure to close your network connections(e.g.external read and writes).This ensures that data written in their buffers are actually stored,the state is stored correctly and it frees up handles in your system.?Guard your secrets:This is not Python specific but we see secrets like usernames,passwords,A

75、PI access tokens,but also file paths or file names leak into the source code.It is a good practice to keep them in separate files,or better yet,secret stores like HashiCorp Vault,AWS Key Management Service,etc.For more information on best practices for keeping your Python code secure,check out Snyks

76、 Python cheatsheetTackling known vulnerabilities Once a vulnerability is found,project maintainers will typically include a fix(if possible)in a future version,so keeping your dependencies up to date is generally a good way to stay on top of known security vulnerabilities.In some cases,though,upgrad

77、ing a dependency is challenging because of the way dependencies interact with each other and your code.Fixing vulnerabilities in direct dependencies is usually straightforward.Upgrade the dependency to the minimum version that includes the fix.Fixing vulnerabilities in indirect dependencies requires

78、 two things:a fixed version of the indirect dependency and a version of the direct dependency that utilizes that fixed version.If these two conditions are met,upgrading the associated direct dependency to a version that utilizes the fixed version of the indirect dependency will fix the issue.If no f

79、ix is available at the level of the direct dependency,you can upgrade the indirect dependency to resolve the issue.Note,however,that this has the potential to break your code due to compatibility issues between the dependencies.Pro tip:The Snyk CLI can be used to scan your Python dependencies locall

80、y or as part of a CI/CD pipeline.A new fix command(currently in beta)will also fix the vulnerability by updating your requirement.txt file.Picking the right container imageAt first glance,one might see a container vulnerability report and think that dealing with 375 vulnerabilities is an overwhelmin

81、g task or that containers are far too risky.Or one might wonder why Docker doesnt fix these vulnerabilities themselves.Fundamentally,this thinking usually stems from the belief that every vulnerability needs a patch and diligent sysadmins work to stay on top of these 375 issues.Instead,the theory be

82、hind containers is that they should only contain what your app needs to run,and nothing else.That means“fixing”is not strictly limited to installing a patch;rather,removing unneeded packages is also a fix.Which brings us back to the best practice of using slimmer images.But there are literally thous

83、ands of image tags within the Docker Official Image Python repository alone-if youre using python:latest,how do you know which alternatives might be better,short of testing them one-by-one?To pick a good container base image?Use Snyk Advisor to look up good starting points.Tags with the word slim an

84、d Alpine images will be the smallest,reducing the likelihood of vulnerabilities?Use Snyk Container for scanning your Dockerfile from your git repo to get immediate recommendations and fix PRs even before you build and scan any images?Rebuild,scan and redeploy often,even if there are no code changes,

85、to get the latest updates from Docker and the latest versions of your tools.Snyk Container will monitor for new vulnerabilities,out-of-date images,and incorporate the running configuration to prioritize issues to fix?Multistage builds not only help produce small,secure images for production,but are

86、a great way to provide a single source of image instructions for every stage of development and release.ConclusionAs applications get more complex,so does the task of securing them.Malicious actors have a wide variety of attack vectors to use when attacking a Python app-whether via known vulnerabili

87、ties introduced via direct and indirect dependencies,security issues in the apps proprietary code,or container vulnerabilities.Just under half of all Python projects monitored by Snyk are indeed vulnerable.But not all is bad.In fact,the data here suggests the Python ecosystem is well equipped to suc

88、cessfully tackle security.An impressive 87%of known vulnerabilities have a fix.Not only are the most commonly used packages generally healthy,but 63%of these packages are also included via a direct dependency and so applying a fix is relatively simple.Using the python:slim image will quickly remove

89、hundreds of container vulnerabilities.At Snyk,our goal is to help development and security teams develop fast while staying secure.A big part of that is providing developer-first tools that help these teams find,prioritize and fix issues quickly and efficiently.Snyk Advisor helps developers research

90、 packages on PyPi.And we facilitate the reporting of new vulnerabilities.Currently the Python ecosystem does not have a centralized place to report vulnerabilities in open source libraries.Snyk is a CVE Numbering Authority(CNA),which means we are able to assign a new vulnerability a CVE number and add the vulnerability to relevant databases.As a CNA,Snyk can help you responsibly report vulnerabilities.Secure your Python projects with SnykScan your Python code for quality and security issues,and get fix advice right in your IDE.Get started with Snyk for free.Get started for free