We\u2019ll start with a quick, non-comprehensive overview of the fundamental data structures in pandas to get you started. The fundamental behavior about data types, indexing, and axis labeling \/ alignment apply across all of the objects. To get started, import numpy and load pandas into your namespace:<\/p>\n
In [477]: import numpy as np\n\n# will use a lot in examples\nIn [478]: randn = np.random.randn\n\nIn [479]: from pandas import *<\/pre>\n<\/div>\n<\/div>\nHere is a basic tenet to keep in mind:\u00a0data alignment is intrinsic<\/strong>. The link between labels and data will not be broken unless done so explicitly by you.<\/p>\n
We\u2019ll give a brief intro to the data structures, then consider all of the broad categories of functionality and methods in separate sections.<\/p>\n
When using pandas, we recommend the following import convention:<\/p>\n
\n\nimport pandas as pd<\/pre>\n<\/div>\n<\/div>\n\nSeries<\/h2>\n
Series<\/tt>\u00a0is a one-dimensional labeled array (technically a subclass of ndarray) capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.). The axis labels are collectively referred to as the\u00a0index<\/strong>. The basic method to create a Series is to call:<\/p>\n
\n\n>>> s = Series(data, index=index)<\/pre>\n<\/div>\n<\/div>\nHere,\u00a0data<\/tt>\u00a0can be many different things:<\/p>\n
\n\n
- a Python dict<\/li>\n
- an ndarray<\/li>\n
- a scalar value (like 5)<\/li>\n<\/ul>\n<\/blockquote>\n
The passed\u00a0index<\/strong>\u00a0is a list of axis labels. Thus, this separates into a few cases depending on what\u00a0data is<\/strong>:<\/p>\n
From ndarray<\/strong><\/p>\n
If\u00a0data<\/tt>\u00a0is an ndarray,\u00a0index<\/strong>\u00a0must be the same length as\u00a0data<\/strong>. If no index is passed, one will be created having values\u00a0[0,\u00a0...,\u00a0len(data)\u00a0-\u00a01]<\/tt>.<\/p>\n
\n\nIn [480]: s = Series(randn(5), index=['a', 'b', 'c', 'd', 'e'])\n\nIn [481]: s\nOut[481]: \na 0.405\nb 0.577\nc -1.715\nd -1.039\ne -0.371\ndtype: float64\n\nIn [482]: s.index\nOut[482]: Index([a, b, c, d, e], dtype=object)\n\nIn [483]: Series(randn(5))\nOut[483]: \n0 -1.158\n1 -1.344\n2 0.845\n3 1.076\n4 -0.109\ndtype: float64<\/pre>\n<\/div>\n<\/div>\n\nNote<\/p>\n
Starting in v0.8.0, pandas supports non-unique index values. In previous version, if the index values are not unique an exception will\u00a0not<\/strong>\u00a0be raised immediately, but attempting any operation involving the index will later result in an exception. In other words, the Index object containing the labels \u201clazily\u201d checks whether the values are unique. The reason for being lazy is nearly all performance-based (there are many instances in computations, like parts of GroupBy, where the index is not used).<\/p>\n<\/div>\n
From dict<\/strong><\/p>\n
If\u00a0data<\/tt>\u00a0is a dict, if\u00a0index<\/strong>\u00a0is passed the values in data corresponding to the labels in the index will be pulled out. Otherwise, an index will be constructed from the sorted keys of the dict, if possible.<\/p>\n
\n\nIn [484]: d = {'a' : 0., 'b' : 1., 'c' : 2.}\n\nIn [485]: Series(d)\nOut[485]: \na 0\nb 1\nc 2\ndtype: float64\n\nIn [486]: Series(d, index=['b', 'c', 'd', 'a'])\nOut[486]: \nb 1\nc 2\nd NaN\na 0\ndtype: float64<\/pre>\n<\/div>\n<\/div>\n\nNote<\/p>\n
NaN (not a number) is the standard missing data marker used in pandas<\/p>\n<\/div>\n
From scalar value<\/strong>\u00a0If\u00a0data<\/tt>\u00a0is a scalar value, an index must be provided. The value will be repeated to match the length of\u00a0index<\/strong><\/p>\n
\n\nIn [487]: Series(5., index=['a', 'b', 'c', 'd', 'e'])\nOut[487]: \na 5\nb 5\nc 5\nd 5\ne 5\ndtype: float64<\/pre>\n<\/div>\n<\/div>\n\nSeries is ndarray-like<\/h3>\n
As a subclass of ndarray, Series is a valid argument to most NumPy functions and behaves similarly to a NumPy array. However, things like slicing also slice the index.<\/p>\n
\n\nIn [488]: s[0]\nOut[488]: 0.40470521868023651\n\nIn [489]: s[:3]\nOut[489]: \na 0.405\nb 0.577\nc -1.715\ndtype: float64\n\nIn [490]: s[s > s.median()]\nOut[490]: \na 0.405\nb 0.577\ndtype: float64\n\nIn [491]: s[[4, 3, 1]]\nOut[491]: \ne -0.371\nd -1.039\nb 0.577\ndtype: float64\n\nIn [492]: np.exp(s)\nOut[492]: \na 1.499\nb 1.781\nc 0.180\nd 0.354\ne 0.690\ndtype: float64<\/pre>\n<\/div>\n<\/div>\nWe will address array-based indexing in a separate\u00a0section<\/em><\/a>.<\/p>\n<\/div>\n
\nSeries is dict-like<\/h3>\n
A Series is like a fixed-size dict in that you can get and set values by index label:<\/p>\n
\n\nIn [493]: s['a']\nOut[493]: 0.40470521868023651\n\nIn [494]: s['e'] = 12.\n\nIn [495]: s\nOut[495]: \na 0.405\nb 0.577\nc -1.715\nd -1.039\ne 12.000\ndtype: float64\n\nIn [496]: 'e' in s\nOut[496]: True\n\nIn [497]: 'f' in s\nOut[497]: False<\/pre>\n<\/div>\n<\/div>\nIf a label is not contained, an exception is raised:<\/p>\n
\n\n>>> s['f']\nKeyError: 'f'<\/pre>\n<\/div>\n<\/div>\nUsing the\u00a0get<\/tt>\u00a0method, a missing label will return None or specified default:<\/p>\n
\n\nIn [498]: s.get('f')\n\nIn [499]: s.get('f', np.nan)\nOut[499]: nan<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\nVectorized operations and label alignment with Series<\/h3>\n
When doing data analysis, as with raw NumPy arrays looping through Series value-by-value is usually not necessary. Series can be also be passed into most NumPy methods expecting an ndarray.<\/p>\n
\n\nIn [500]: s + s\nOut[500]: \na 0.809\nb 1.154\nc -3.430\nd -2.079\ne 24.000\ndtype: float64\n\nIn [501]: s * 2\nOut[501]: \na 0.809\nb 1.154\nc -3.430\nd -2.079\ne 24.000\ndtype: float64\n\nIn [502]: np.exp(s)\nOut[502]: \na 1.499\nb 1.781\nc 0.180\nd 0.354\ne 162754.791\ndtype: float64<\/pre>\n<\/div>\n<\/div>\nA key difference between Series and ndarray is that operations between Series automatically align the data based on label. Thus, you can write computations without giving consideration to whether the Series involved have the same labels.<\/p>\n
\n\nIn [503]: s[1:] + s[:-1]\nOut[503]: \na NaN\nb 1.154\nc -3.430\nd -2.079\ne NaN\ndtype: float64<\/pre>\n<\/div>\n<\/div>\nThe result of an operation between unaligned Series will have the\u00a0union<\/strong>\u00a0of the indexes involved. If a label is not found in one Series or the other, the result will be marked as missing (NaN). Being able to write code without doing any explicit data alignment grants immense freedom and flexibility in interactive data analysis and research. The integrated data alignment features of the pandas data structures set pandas apart from the majority of related tools for working with labeled data.<\/p>\n
\nNote<\/p>\n
In general, we chose to make the default result of operations between differently indexed objects yield the\u00a0union<\/strong>\u00a0of the indexes in order to avoid loss of information. Having an index label, though the data is missing, is typically important information as part of a computation. You of course have the option of dropping labels with missing data via thedropna<\/strong>\u00a0function.<\/p>\n<\/div>\n<\/div>\n
\nName attribute<\/h3>\n
Series can also have a\u00a0name<\/tt>\u00a0attribute:<\/p>\n
\n\nIn [504]: s = Series(np.random.randn(5), name='something')\n\nIn [505]: s\nOut[505]: \n0 1.644\n1 -1.469\n2 0.357\n3 -0.675\n4 -1.777\nName: something, dtype: float64\n\nIn [506]: s.name\nOut[506]: 'something'<\/pre>\n<\/div>\n<\/div>\nThe Series\u00a0name<\/tt>\u00a0will be assigned automatically in many cases, in particular when taking 1D slices of DataFrame as you will see below.<\/p>\n<\/div>\n<\/div>\n
\nDataFrame<\/h2>\n
DataFrame<\/strong>\u00a0is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table, or a dict of Series objects. It is generally the most commonly used pandas object. Like Series, DataFrame accepts many different kinds of input:<\/p>\n
\n\n
- Dict of 1D ndarrays, lists, dicts, or Series<\/li>\n
- 2-D numpy.ndarray<\/li>\n
- Structured or record<\/a>\u00a0ndarray<\/li>\n
- A\u00a0Series<\/tt><\/li>\n
- Another\u00a0DataFrame<\/tt><\/li>\n<\/ul>\n<\/blockquote>\n
Along with the data, you can optionally pass\u00a0index<\/strong>\u00a0(row labels) and\u00a0columns<\/strong>\u00a0(column labels) arguments. If you pass an index and \/ or columns, you are guaranteeing the index and \/ or columns of the resulting DataFrame. Thus, a dict of Series plus a specific index will discard all data not matching up to the passed index.<\/p>\n
If axis labels are not passed, they will be constructed from the input data based on common sense rules.<\/p>\n
\nFrom dict of Series or dicts<\/h3>\n
The result\u00a0index<\/strong>\u00a0will be the\u00a0union<\/strong>\u00a0of the indexes of the various Series. If there are any nested dicts, these will be first converted to Series. If no columns are passed, the columns will be the sorted list of dict keys.<\/p>\n
\n\nIn [507]: d = {'one' : Series([1., 2., 3.], index=['a', 'b', 'c']),\n .....: 'two' : Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}\n .....:\n\nIn [508]: df = DataFrame(d)\n\nIn [509]: df\nOut[509]: \n one two\na 1 1\nb 2 2\nc 3 3\nd NaN 4\n\nIn [510]: DataFrame(d, index=['d', 'b', 'a'])\nOut[510]: \n one two\nd NaN 4\nb 2 2\na 1 1\n\nIn [511]: DataFrame(d, index=['d', 'b', 'a'], columns=['two', 'three'])\nOut[511]: \n two three\nd 4 NaN\nb 2 NaN\na 1 NaN<\/pre>\n<\/div>\n<\/div>\nThe row and column labels can be accessed respectively by accessing the\u00a0index<\/strong>\u00a0andcolumns<\/strong>\u00a0attributes:<\/p>\n
\nNote<\/p>\n
When a particular set of columns is passed along with a dict of data, the passed columns override the keys in the dict.<\/p>\n<\/div>\n
\n\nIn [512]: df.index\nOut[512]: Index([a, b, c, d], dtype=object)\n\nIn [513]: df.columns\nOut[513]: Index([one, two], dtype=object)<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\nFrom dict of ndarrays \/ lists<\/h3>\n
The ndarrays must all be the same length. If an index is passed, it must clearly also be the same length as the arrays. If no index is passed, the result will be\u00a0range(n)<\/tt>, where\u00a0n<\/tt>\u00a0is the array length.<\/p>\n
\n\nIn [514]: d = {'one' : [1., 2., 3., 4.],\n .....: 'two' : [4., 3., 2., 1.]}\n .....:\n\nIn [515]: DataFrame(d)\nOut[515]: \n one two\n0 1 4\n1 2 3\n2 3 2\n3 4 1\n\nIn [516]: DataFrame(d, index=['a', 'b', 'c', 'd'])\nOut[516]: \n one two\na 1 4\nb 2 3\nc 3 2\nd 4 1<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\nFrom structured or record array<\/h3>\n
This case is handled identically to a dict of arrays.<\/p>\n
\n\nIn [517]: data = np.zeros((2,),dtype=[('A', 'i4'),('B', 'f4'),('C', 'a10')])\n\nIn [518]: data[:] = [(1,2.,'Hello'),(2,3.,\"World\")]\n\nIn [519]: DataFrame(data)\nOut[519]: \n A B C\n0 1 2 Hello\n1 2 3 World\n\nIn [520]: DataFrame(data, index=['first', 'second'])\nOut[520]: \n A B C\nfirst 1 2 Hello\nsecond 2 3 World\n\nIn [521]: DataFrame(data, columns=['C', 'A', 'B'])\nOut[521]: \n C A B\n0 Hello 1 2\n1 World 2 3<\/pre>\n<\/div>\n<\/div>\n\nNote<\/p>\n
DataFrame is not intended to work exactly like a 2-dimensional NumPy ndarray.<\/p>\n<\/div>\n<\/div>\n
\nFrom a list of dicts<\/h3>\n
\n\nIn [522]: data2 = [{'a': 1, 'b': 2}, {'a': 5, 'b': 10, 'c': 20}]\n\nIn [523]: DataFrame(data2)\nOut[523]: \n a b c\n0 1 2 NaN\n1 5 10 20\n\nIn [524]: DataFrame(data2, index=['first', 'second'])\nOut[524]: \n a b c\nfirst 1 2 NaN\nsecond 5 10 20\n\nIn [525]: DataFrame(data2, columns=['a', 'b'])\nOut[525]: \n a b\n0 1 2\n1 5 10<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\nFrom a Series<\/h3>\n
The result will be a DataFrame with the same index as the input Series, and with one column whose name is the original name of the Series (only if no other column name provided).<\/p>\n
Missing Data<\/strong><\/p>\n
Much more will be said on this topic in the\u00a0Missing data<\/em><\/a>\u00a0section. To construct a DataFrame with missing data, use\u00a0np.nan<\/tt>\u00a0for those values which are missing. Alternatively, you may pass a\u00a0numpy.MaskedArray<\/tt>\u00a0as the data argument to the DataFrame constructor, and its masked entries will be considered missing.<\/p>\n<\/div>\n
\nAlternate Constructors<\/h3>\n
DataFrame.from_dict<\/strong><\/p>\n
DataFrame.from_dict<\/tt>\u00a0takes a dict of dicts or a dict of array-like sequences and returns a DataFrame. It operates like the\u00a0DataFrame<\/tt>\u00a0constructor except for the\u00a0orient<\/tt>\u00a0parameter which is\u00a0'columns'<\/tt>\u00a0by default, but which can be set to\u00a0'index'<\/tt>\u00a0in order to use the dict keys as row labels.<\/p>\n
DataFrame.from_records<\/strong><\/p>\n
DataFrame.from_records<\/tt>\u00a0takes a list of tuples or an ndarray with structured dtype. Works analogously to the normal\u00a0DataFrame<\/tt>\u00a0constructor, except that index maybe be a specific field of the structured dtype to use as the index. For example:<\/p>\n
\n\nIn [526]: data\nOut[526]: \narray([(1, 2.0, 'Hello'), (2, 3.0, 'World')], \n dtype=[('A', '<i4'), ('B', '<f4'), ('C', '|S10')])\n\nIn [527]: DataFrame.from_records(data, index='C')\nOut[527]: \n A B\nC \nHello 1 2\nWorld 2 3<\/pre>\n<\/div>\n<\/div>\nDataFrame.from_items<\/strong><\/p>\n
DataFrame.from_items<\/tt>\u00a0works analogously to the form of the\u00a0dict<\/tt>\u00a0constructor that takes a sequence of\u00a0(key,\u00a0value)<\/tt>\u00a0pairs, where the keys are column (or row, in the case oforient='index'<\/tt>) names, and the value are the column values (or row values). This can be useful for constructing a DataFrame with the columns in a particular order without having to pass an explicit list of columns:<\/p>\n
\n\nIn [528]: DataFrame.from_items([('A', [1, 2, 3]), ('B', [4, 5, 6])])\nOut[528]: \n A B\n0 1 4\n1 2 5\n2 3 6<\/pre>\n<\/div>\n<\/div>\nIf you pass\u00a0orient='index'<\/tt>, the keys will be the row labels. But in this case you must also pass the desired column names:<\/p>\n
\n\nIn [529]: DataFrame.from_items([('A', [1, 2, 3]), ('B', [4, 5, 6])],\n .....: orient='index', columns=['one', 'two', 'three'])\n .....:\nOut[529]: \n one two three\nA 1 2 3\nB 4 5 6<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\nColumn selection, addition, deletion<\/h3>\n
You can treat a DataFrame semantically like a dict of like-indexed Series objects. Getting, setting, and deleting columns works with the same syntax as the analogous dict operations:<\/p>\n
\n\nIn [530]: df['one']\nOut[530]: \na 1\nb 2\nc 3\nd NaN\nName: one, dtype: float64\n\nIn [531]: df['three'] = df['one'] * df['two']\n\nIn [532]: df['flag'] = df['one'] > 2\n\nIn [533]: df\nOut[533]: \n one two three flag\na 1 1 1 False\nb 2 2 4 False\nc 3 3 9 True\nd NaN 4 NaN False<\/pre>\n<\/div>\n<\/div>\nColumns can be deleted or popped like with a dict:<\/p>\n
\n\nIn [534]: del df['two']\n\nIn [535]: three = df.pop('three')\n\nIn [536]: df\nOut[536]: \n one flag\na 1 False\nb 2 False\nc 3 True\nd NaN False<\/pre>\n<\/div>\n<\/div>\nWhen inserting a scalar value, it will naturally be propagated to fill the column:<\/p>\n
\n\nIn [537]: df['foo'] = 'bar'\n\nIn [538]: df\nOut[538]: \n one flag foo\na 1 False bar\nb 2 False bar\nc 3 True bar\nd NaN False bar<\/pre>\n<\/div>\n<\/div>\nWhen inserting a Series that does not have the same index as the DataFrame, it will be conformed to the DataFrame\u2019s index:<\/p>\n
\n\nIn [539]: df['one_trunc'] = df['one'][:2]\n\nIn [540]: df\nOut[540]: \n one flag foo one_trunc\na 1 False bar 1\nb 2 False bar 2\nc 3 True bar NaN\nd NaN False bar NaN<\/pre>\n<\/div>\n<\/div>\nYou can insert raw ndarrays but their length must match the length of the DataFrame\u2019s index.<\/p>\n
By default, columns get inserted at the end. The\u00a0insert<\/tt>\u00a0function is available to insert at a particular location in the columns:<\/p>\n
\n\nIn [541]: df.insert(1, 'bar', df['one'])\n\nIn [542]: df\nOut[542]: \n one bar flag foo one_trunc\na 1 1 False bar 1\nb 2 2 False bar 2\nc 3 3 True bar NaN\nd NaN NaN False bar NaN<\/pre>\n<\/div>\n<\/div>\n<\/div>\n\nIndexing \/ Selection<\/h3>\n
The basics of indexing are as follows:<\/p>\n