{"id":131,"date":"2013-03-31T23:41:34","date_gmt":"2013-04-01T04:41:34","guid":{"rendered":"http:\/\/homepages.uc.edu\/~yaozo\/wordpress\/?p=131"},"modified":"2013-03-31T23:41:34","modified_gmt":"2013-04-01T04:41:34","slug":"10-minutes-to-pandas","status":"publish","type":"post","link":"https:\/\/zhuoyao.net\/index.php\/2013\/03\/31\/10-minutes-to-pandas\/","title":{"rendered":"10 Minutes to Pandas"},"content":{"rendered":"

This is a short introduction to pandas, geared mainly for new users.<\/p>\n

Customarily, we import as follows<\/p>\n

\n
\n
In [1]: import pandas as pd\n\nIn [2]: import numpy as np<\/pre>\n<\/div>\n<\/div>\n
\n
Object Creation<\/h2>\n
See the\u00a0Data Structure Intro section<\/em><\/a><\/p>\n
Creating a\u00a0Series<\/tt>\u00a0by passing a list of values, letting pandas create a default integer index<\/p>\n
\n
\n
In [3]: s = pd.Series([1,3,5,np.nan,6,8])\n\nIn [4]: s\nOut[4]: \n0     1\n1     3\n2     5\n3   NaN\n4     6\n5     8\ndtype: float64<\/pre>\n<\/div>\n<\/div>\n
Creating a\u00a0DataFrame<\/tt>\u00a0by passing a numpy array, with a datetime index and labeled columns.<\/p>\n
\n
\n
In [5]: dates = pd.date_range('20130101',periods=6)\n\nIn [6]: dates\nOut[6]: \n<class 'pandas.tseries.index.DatetimeIndex'>\n[2013-01-01 00:00:00, ..., 2013-01-06 00:00:00]\nLength: 6, Freq: D, Timezone: None\n\nIn [7]: df = pd.DataFrame(np.random.randn(6,4),index=dates,columns=list('ABCD'))\n\nIn [8]: df\nOut[8]: \n                   A         B         C         D\n2013-01-01  0.469112 -0.282863 -1.509059 -1.135632\n2013-01-02  1.212112 -0.173215  0.119209 -1.044236\n2013-01-03 -0.861849 -2.104569 -0.494929  1.071804\n2013-01-04  0.721555 -0.706771 -1.039575  0.271860\n2013-01-05 -0.424972  0.567020  0.276232 -1.087401\n2013-01-06 -0.673690  0.113648 -1.478427  0.524988<\/pre>\n<\/div>\n<\/div>\n
Creating a\u00a0DataFrame<\/tt>\u00a0by passing a dict of objects that can be converted to series-like.<\/p>\n
\n
\n
In [9]: df2 = pd.DataFrame({ 'A' : 1.,\n   ...:                      'B' : pd.Timestamp('20130102'),\n   ...:                      'C' : pd.Series(1,index=range(4),dtype='float32'),\n   ...:                      'D' : np.array([3] * 4,dtype='int32'),\n   ...:                      'E' : 'foo' })\n   ...:\n\nIn [10]: df2\nOut[10]: \n   A                   B  C  D    E\n0  1 2013-01-02 00:00:00  1  3  foo\n1  1 2013-01-02 00:00:00  1  3  foo\n2  1 2013-01-02 00:00:00  1  3  foo\n3  1 2013-01-02 00:00:00  1  3  foo<\/pre>\n<\/div>\n<\/div>\n
Having specific\u00a0dtypes<\/em><\/a><\/p>\n
\n
\n
In [11]: df2.dtypes\nOut[11]: \nA           float64\nB    datetime64[ns]\nC           float32\nD             int32\nE            object\ndtype: object<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
Viewing Data<\/h2>\n
See the\u00a0Basics section<\/em><\/a><\/p>\n
See the top & bottom rows of the frame<\/p>\n
\n
\n
In [12]: df.head()\nOut[12]: \n                   A         B         C         D\n2013-01-01  0.469112 -0.282863 -1.509059 -1.135632\n2013-01-02  1.212112 -0.173215  0.119209 -1.044236\n2013-01-03 -0.861849 -2.104569 -0.494929  1.071804\n2013-01-04  0.721555 -0.706771 -1.039575  0.271860\n2013-01-05 -0.424972  0.567020  0.276232 -1.087401\n\nIn [13]: df.tail(3)\nOut[13]: \n                   A         B         C         D\n2013-01-04  0.721555 -0.706771 -1.039575  0.271860\n2013-01-05 -0.424972  0.567020  0.276232 -1.087401\n2013-01-06 -0.673690  0.113648 -1.478427  0.524988<\/pre>\n<\/div>\n<\/div>\n
Display the index,columns, and the underlying numpy data<\/p>\n
\n
\n
In [14]: df.index\nOut[14]: \n<class 'pandas.tseries.index.DatetimeIndex'>\n[2013-01-01 00:00:00, ..., 2013-01-06 00:00:00]\nLength: 6, Freq: D, Timezone: None\n\nIn [15]: df.columns\nOut[15]: Index([A, B, C, D], dtype=object)\n\nIn [16]: df.values\nOut[16]: \narray([[ 0.4691, -0.2829, -1.5091, -1.1356],\n       [ 1.2121, -0.1732,  0.1192, -1.0442],\n       [-0.8618, -2.1046, -0.4949,  1.0718],\n       [ 0.7216, -0.7068, -1.0396,  0.2719],\n       [-0.425 ,  0.567 ,  0.2762, -1.0874],\n       [-0.6737,  0.1136, -1.4784,  0.525 ]])<\/pre>\n<\/div>\n<\/div>\n
Describe shows a quick statistic summary of your data<\/p>\n
\n
\n
In [17]: df.describe()\nOut[17]: \n              A         B         C         D\ncount  6.000000  6.000000  6.000000  6.000000\nmean   0.073711 -0.431125 -0.687758 -0.233103\nstd    0.843157  0.922818  0.779887  0.973118\nmin   -0.861849 -2.104569 -1.509059 -1.135632\n25%   -0.611510 -0.600794 -1.368714 -1.076610\n50%    0.022070 -0.228039 -0.767252 -0.386188\n75%    0.658444  0.041933 -0.034326  0.461706\nmax    1.212112  0.567020  0.276232  1.071804<\/pre>\n<\/div>\n<\/div>\n
Transposing your data<\/p>\n
\n
\n
In [18]: df.T\nOut[18]: \n   2013-01-01  2013-01-02  2013-01-03  2013-01-04  2013-01-05  2013-01-06\nA    0.469112    1.212112   -0.861849    0.721555   -0.424972   -0.673690\nB   -0.282863   -0.173215   -2.104569   -0.706771    0.567020    0.113648\nC   -1.509059    0.119209   -0.494929   -1.039575    0.276232   -1.478427\nD   -1.135632   -1.044236    1.071804    0.271860   -1.087401    0.524988<\/pre>\n<\/div>\n<\/div>\n
Sorting by an axis<\/p>\n
\n
\n
In [19]: df.sort_index(axis=1, ascending=False)\nOut[19]: \n                   D         C         B         A\n2013-01-01 -1.135632 -1.509059 -0.282863  0.469112\n2013-01-02 -1.044236  0.119209 -0.173215  1.212112\n2013-01-03  1.071804 -0.494929 -2.104569 -0.861849\n2013-01-04  0.271860 -1.039575 -0.706771  0.721555\n2013-01-05 -1.087401  0.276232  0.567020 -0.424972\n2013-01-06  0.524988 -1.478427  0.113648 -0.673690<\/pre>\n<\/div>\n<\/div>\n
Sorting by values<\/p>\n
\n
\n
In [20]: df.sort(columns='B')\nOut[20]: \n                   A         B         C         D\n2013-01-03 -0.861849 -2.104569 -0.494929  1.071804\n2013-01-04  0.721555 -0.706771 -1.039575  0.271860\n2013-01-01  0.469112 -0.282863 -1.509059 -1.135632\n2013-01-02  1.212112 -0.173215  0.119209 -1.044236\n2013-01-06 -0.673690  0.113648 -1.478427  0.524988\n2013-01-05 -0.424972  0.567020  0.276232 -1.087401<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
Selection<\/h2>\n
See the\u00a0Indexing section<\/em><\/a><\/p>\n
\n
Getting<\/h3>\n
Selecting a single column, which yields a\u00a0Series<\/tt>, equivalent to\u00a0df.A<\/tt><\/p>\n
\n
\n
In [21]: df['A']\nOut[21]: \n2013-01-01    0.469112\n2013-01-02    1.212112\n2013-01-03   -0.861849\n2013-01-04    0.721555\n2013-01-05   -0.424972\n2013-01-06   -0.673690\nFreq: D, Name: A, dtype: float64<\/pre>\n<\/div>\n<\/div>\n
Selecting via\u00a0[]<\/tt>, which slices the rows.<\/p>\n
\n
\n
In [22]: df[0:3]\nOut[22]: \n                   A         B         C         D\n2013-01-01  0.469112 -0.282863 -1.509059 -1.135632\n2013-01-02  1.212112 -0.173215  0.119209 -1.044236\n2013-01-03 -0.861849 -2.104569 -0.494929  1.071804\n\nIn [23]: df['20130102':'20130104']\nOut[23]: \n                   A         B         C         D\n2013-01-02  1.212112 -0.173215  0.119209 -1.044236\n2013-01-03 -0.861849 -2.104569 -0.494929  1.071804\n2013-01-04  0.721555 -0.706771 -1.039575  0.271860<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
Selection by Label<\/h3>\n
See more in\u00a0Selection by Label<\/em><\/a><\/p>\n
For getting a cross section using a label<\/p>\n
\n
\n
In [24]: df.loc[dates[0]]\nOut[24]: \nA    0.469112\nB   -0.282863\nC   -1.509059\nD   -1.135632\nName: 2013-01-01 00:00:00, dtype: float64<\/pre>\n<\/div>\n<\/div>\n
Selecting on a multi-axis by label<\/p>\n
\n
\n
In [25]: df.loc[:,['A','B']]\nOut[25]: \n                   A         B\n2013-01-01  0.469112 -0.282863\n2013-01-02  1.212112 -0.173215\n2013-01-03 -0.861849 -2.104569\n2013-01-04  0.721555 -0.706771\n2013-01-05 -0.424972  0.567020\n2013-01-06 -0.673690  0.113648<\/pre>\n<\/div>\n<\/div>\n
Showing label slicing, both endpoints are\u00a0included<\/em><\/p>\n
\n
\n
In [26]: df.loc['20130102':'20130104',['A','B']]\nOut[26]: \n                   A         B\n2013-01-02  1.212112 -0.173215\n2013-01-03 -0.861849 -2.104569\n2013-01-04  0.721555 -0.706771<\/pre>\n<\/div>\n<\/div>\n
Reduction in the dimensions of the returned object<\/p>\n
\n
\n
In [27]: df.loc['20130102',['A','B']]\nOut[27]: \nA    1.212112\nB   -0.173215\nName: 2013-01-02 00:00:00, dtype: float64<\/pre>\n<\/div>\n<\/div>\n
For getting a scalar value<\/p>\n
\n
\n
In [28]: df.loc[dates[0],'A']\nOut[28]: 0.46911229990718628<\/pre>\n<\/div>\n<\/div>\n
For getting fast access to a scalar (equiv to the prior method)<\/p>\n
\n
\n
In [29]: df.at[dates[0],'A']\nOut[29]: 0.46911229990718628<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
Selection by Position<\/h3>\n
See more in\u00a0Selection by Position<\/em><\/a><\/p>\n
Select via the position of the passed integers<\/p>\n
\n
\n
In [30]: df.iloc[3]\nOut[30]: \nA    0.721555\nB   -0.706771\nC   -1.039575\nD    0.271860\nName: 2013-01-04 00:00:00, dtype: float64<\/pre>\n<\/div>\n<\/div>\n
By integer slices, acting similar to numpy\/python<\/p>\n
\n
\n
In [31]: df.iloc[3:5,0:2]\nOut[31]: \n                   A         B\n2013-01-04  0.721555 -0.706771\n2013-01-05 -0.424972  0.567020<\/pre>\n<\/div>\n<\/div>\n
By lists of integer position locations, similar to the numpy\/python style<\/p>\n
\n
\n
In [32]: df.iloc[[1,2,4],[0,2]]\nOut[32]: \n                   A         C\n2013-01-02  1.212112  0.119209\n2013-01-03 -0.861849 -0.494929\n2013-01-05 -0.424972  0.276232<\/pre>\n<\/div>\n<\/div>\n
For slicing rows explicitly<\/p>\n
\n
\n
In [33]: df.iloc[1:3,:]\nOut[33]: \n                   A         B         C         D\n2013-01-02  1.212112 -0.173215  0.119209 -1.044236\n2013-01-03 -0.861849 -2.104569 -0.494929  1.071804<\/pre>\n<\/div>\n<\/div>\n
For slicing columns explicitly<\/p>\n
\n
\n
In [34]: df.iloc[:,1:3]\nOut[34]: \n                   B         C\n2013-01-01 -0.282863 -1.509059\n2013-01-02 -0.173215  0.119209\n2013-01-03 -2.104569 -0.494929\n2013-01-04 -0.706771 -1.039575\n2013-01-05  0.567020  0.276232\n2013-01-06  0.113648 -1.478427<\/pre>\n<\/div>\n<\/div>\n
For getting a value explicity<\/p>\n
\n
\n
In [35]: df.iloc[1,1]\nOut[35]: -0.17321464905330858<\/pre>\n<\/div>\n<\/div>\n
For getting fast access to a scalar (equiv to the prior method)<\/p>\n
\n
\n
In [36]: df.iat[1,1]\nOut[36]: -0.17321464905330858<\/pre>\n<\/div>\n<\/div>\n
There is one signficant departure from standard python\/numpy slicing semantics. python\/numpy allow slicing past the end of an array without an associated error.<\/p>\n
\n
\n
# these are allowed in python\/numpy.\nIn [37]: x = list('abcdef')\n\nIn [38]: x[4:10]\nOut[38]: ['e', 'f']\n\nIn [39]: x[8:10]\nOut[39]: []<\/pre>\n<\/div>\n<\/div>\n
Pandas will detect this and raise\u00a0IndexError<\/tt>, rather than return an empty structure.<\/p>\n
\n
\n
>>> df.iloc[:,8:10]\nIndexError: out-of-bounds on slice (end)<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
Boolean Indexing<\/h3>\n
Using a single column\u2019s values to select data.<\/p>\n
\n
\n
In [40]: df[df.A > 0]\nOut[40]: \n                   A         B         C         D\n2013-01-01  0.469112 -0.282863 -1.509059 -1.135632\n2013-01-02  1.212112 -0.173215  0.119209 -1.044236\n2013-01-04  0.721555 -0.706771 -1.039575  0.271860<\/pre>\n<\/div>\n<\/div>\n
A\u00a0where<\/tt>\u00a0operation for getting.<\/p>\n
\n
\n
In [41]: df[df > 0]\nOut[41]: \n                   A         B         C         D\n2013-01-01  0.469112       NaN       NaN       NaN\n2013-01-02  1.212112       NaN  0.119209       NaN\n2013-01-03       NaN       NaN       NaN  1.071804\n2013-01-04  0.721555       NaN       NaN  0.271860\n2013-01-05       NaN  0.567020  0.276232       NaN\n2013-01-06       NaN  0.113648       NaN  0.524988<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
Setting<\/h3>\n
Setting a new column automatically aligns the data by the indexes<\/p>\n
\n
\n
In [42]: s1 = pd.Series([1,2,3,4,5,6],index=date_range('20130102',periods=6))\n\nIn [43]: s1\nOut[43]: \n2013-01-02    1\n2013-01-03    2\n2013-01-04    3\n2013-01-05    4\n2013-01-06    5\n2013-01-07    6\nFreq: D, dtype: int64\n\nIn [44]: df['F'] = s1<\/pre>\n<\/div>\n<\/div>\n
Setting values by label<\/p>\n
\n
\n
In [45]: df.at[dates[0],'A'] = 0<\/pre>\n<\/div>\n<\/div>\n
Setting values by position<\/p>\n
\n
\n
In [46]: df.iat[0,1] = 0<\/pre>\n<\/div>\n<\/div>\n
Setting by assigning with a numpy array<\/p>\n
\n
\n
In [47]: df.loc[:,'D'] = np.array([5] * len(df))<\/pre>\n<\/div>\n<\/div>\n
The result of the prior setting operations<\/p>\n
\n
\n
In [48]: df\nOut[48]: \n                   A         B         C  D   F\n2013-01-01  0.000000  0.000000 -1.509059  5 NaN\n2013-01-02  1.212112 -0.173215  0.119209  5   1\n2013-01-03 -0.861849 -2.104569 -0.494929  5   2\n2013-01-04  0.721555 -0.706771 -1.039575  5   3\n2013-01-05 -0.424972  0.567020  0.276232  5   4\n2013-01-06 -0.673690  0.113648 -1.478427  5   5<\/pre>\n<\/div>\n<\/div>\n
A\u00a0where<\/tt>\u00a0operation with setting.<\/p>\n
\n
\n
In [49]: df2 = df.copy()\n\nIn [50]: df2[df2 > 0] = -df2\n\nIn [51]: df2\nOut[51]: \n                   A         B         C  D   F\n2013-01-01  0.000000  0.000000 -1.509059 -5 NaN\n2013-01-02 -1.212112 -0.173215 -0.119209 -5  -1\n2013-01-03 -0.861849 -2.104569 -0.494929 -5  -2\n2013-01-04 -0.721555 -0.706771 -1.039575 -5  -3\n2013-01-05 -0.424972 -0.567020 -0.276232 -5  -4\n2013-01-06 -0.673690 -0.113648 -1.478427 -5  -5<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
Missing Data<\/h2>\n
Pandas primarily uses the value\u00a0np.nan<\/tt>\u00a0to represent missing data. It is by default not included in computations. See the\u00a0Missing Data section<\/em><\/a><\/p>\n
Reindexing allows you to change\/add\/delete the index on a specified axis. This returns a copy of the data.<\/p>\n
\n
\n
In [52]: df1 = df.reindex(index=dates[0:4],columns=list(df.columns) + ['E'])\n\nIn [53]: df1.loc[dates[0]:dates[1],'E'] = 1\n\nIn [54]: df1\nOut[54]: \n                   A         B         C  D   F   E\n2013-01-01  0.000000  0.000000 -1.509059  5 NaN   1\n2013-01-02  1.212112 -0.173215  0.119209  5   1   1\n2013-01-03 -0.861849 -2.104569 -0.494929  5   2 NaN\n2013-01-04  0.721555 -0.706771 -1.039575  5   3 NaN<\/pre>\n<\/div>\n<\/div>\n
To drop any rows that have missing data.<\/p>\n
\n
\n
In [55]: df1.dropna(how='any')\nOut[55]: \n                   A         B         C  D  F  E\n2013-01-02  1.212112 -0.173215  0.119209  5  1  1<\/pre>\n<\/div>\n<\/div>\n
Filling missing data<\/p>\n
\n
\n
In [56]: df1.fillna(value=5)\nOut[56]: \n                   A         B         C  D  F  E\n2013-01-01  0.000000  0.000000 -1.509059  5  5  1\n2013-01-02  1.212112 -0.173215  0.119209  5  1  1\n2013-01-03 -0.861849 -2.104569 -0.494929  5  2  5\n2013-01-04  0.721555 -0.706771 -1.039575  5  3  5<\/pre>\n<\/div>\n<\/div>\n
To get the boolean mask where values are\u00a0nan<\/tt><\/p>\n
\n
\n
In [57]: pd.isnull(df1)\nOut[57]: \n                A      B      C      D      F      E\n2013-01-01  False  False  False  False   True  False\n2013-01-02  False  False  False  False  False  False\n2013-01-03  False  False  False  False  False   True\n2013-01-04  False  False  False  False  False   True<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
Operations<\/h2>\n
See the\u00a0Basic section on Binary Ops<\/em><\/a><\/p>\n
\n
Stats<\/h3>\n
Operations in general\u00a0exclude<\/em>\u00a0missing data.<\/p>\n
Performing a descriptive statistic<\/p>\n
\n
\n
In [58]: df.mean()\nOut[58]: \nA   -0.004474\nB   -0.383981\nC   -0.687758\nD    5.000000\nF    3.000000\ndtype: float64<\/pre>\n<\/div>\n<\/div>\n
Same operation on the other axis<\/p>\n
\n
\n
In [59]: df.mean(1)\nOut[59]: \n2013-01-01    0.872735\n2013-01-02    1.431621\n2013-01-03    0.707731\n2013-01-04    1.395042\n2013-01-05    1.883656\n2013-01-06    1.592306\nFreq: D, dtype: float64<\/pre>\n<\/div>\n<\/div>\n
Operating with objects that have different dimensionality and need alignment. In addition, pandas automatically broadcasts along the specified dimension.<\/p>\n
\n
\n
In [60]: s = pd.Series([1,3,5,np.nan,6,8],index=dates).shift(2)\n\nIn [61]: s\nOut[61]: \n2013-01-01   NaN\n2013-01-02   NaN\n2013-01-03     1\n2013-01-04     3\n2013-01-05     5\n2013-01-06   NaN\nFreq: D, dtype: float64\n\nIn [62]: df.sub(s,axis='index')\nOut[62]: \n                   A         B         C   D   F\n2013-01-01       NaN       NaN       NaN NaN NaN\n2013-01-02       NaN       NaN       NaN NaN NaN\n2013-01-03 -1.861849 -3.104569 -1.494929   4   1\n2013-01-04 -2.278445 -3.706771 -4.039575   2   0\n2013-01-05 -5.424972 -4.432980 -4.723768   0  -1\n2013-01-06       NaN       NaN       NaN NaN NaN<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
Apply<\/h3>\n
Applying functions to the data<\/p>\n
\n
\n
In [63]: df.apply(np.cumsum)\nOut[63]: \n                   A         B         C   D   F\n2013-01-01  0.000000  0.000000 -1.509059   5 NaN\n2013-01-02  1.212112 -0.173215 -1.389850  10   1\n2013-01-03  0.350263 -2.277784 -1.884779  15   3\n2013-01-04  1.071818 -2.984555 -2.924354  20   6\n2013-01-05  0.646846 -2.417535 -2.648122  25  10\n2013-01-06 -0.026844 -2.303886 -4.126549  30  15\n\nIn [64]: df.apply(lambda x: x.max() - x.min())\nOut[64]: \nA    2.073961\nB    2.671590\nC    1.785291\nD    0.000000\nF    4.000000\ndtype: float64<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
Histogramming<\/h3>\n
See more at\u00a0Histogramming and Discretization<\/em><\/a><\/p>\n
\n
\n
In [65]: s = Series(np.random.randint(0,7,size=10))\n\nIn [66]: s\nOut[66]: \n0    4\n1    2\n2    1\n3    2\n4    6\n5    4\n6    4\n7    6\n8    4\n9    4\ndtype: int64\n\nIn [67]: s.value_counts()\nOut[67]: \n4    5\n6    2\n2    2\n1    1\ndtype: int64<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
String Methods<\/h3>\n
See more at\u00a0Vectorized String Methods<\/em><\/a><\/p>\n
\n
\n
In [68]: s = Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'])\n\nIn [69]: s.str.lower()\nOut[69]: \n0       a\n1       b\n2       c\n3    aaba\n4    baca\n5     NaN\n6    caba\n7     dog\n8     cat\ndtype: object<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
Merge<\/h2>\n
\n
Concat<\/h3>\n
Pandas provides various facilities for easily combining together Series, DataFrame, and Panel objects with various kinds of set logic for the indexes and relational algebra functionality in the case of join \/ merge-type operations.<\/p>\n
See the\u00a0Merging section<\/em><\/a><\/p>\n
Concatenating pandas objects together<\/p>\n
\n
\n
In [70]: df = pd.DataFrame(np.random.randn(10, 4))\n\nIn [71]: df\nOut[71]: \n          0         1         2         3\n0 -0.548702  1.467327 -1.015962 -0.483075\n1  1.637550 -1.217659 -0.291519 -1.745505\n2 -0.263952  0.991460 -0.919069  0.266046\n3 -0.709661  1.669052  1.037882 -1.705775\n4 -0.919854 -0.042379  1.247642 -0.009920\n5  0.290213  0.495767  0.362949  1.548106\n6 -1.131345 -0.089329  0.337863 -0.945867\n7 -0.932132  1.956030  0.017587 -0.016692\n8 -0.575247  0.254161 -1.143704  0.215897\n9  1.193555 -0.077118 -0.408530 -0.862495\n\n# break it into pieces\nIn [72]: pieces = [df[:3], df[3:7], df[7:]]\n\nIn [73]: concat(pieces)\nOut[73]: \n          0         1         2         3\n0 -0.548702  1.467327 -1.015962 -0.483075\n1  1.637550 -1.217659 -0.291519 -1.745505\n2 -0.263952  0.991460 -0.919069  0.266046\n3 -0.709661  1.669052  1.037882 -1.705775\n4 -0.919854 -0.042379  1.247642 -0.009920\n5  0.290213  0.495767  0.362949  1.548106\n6 -1.131345 -0.089329  0.337863 -0.945867\n7 -0.932132  1.956030  0.017587 -0.016692\n8 -0.575247  0.254161 -1.143704  0.215897\n9  1.193555 -0.077118 -0.408530 -0.862495<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
Join<\/h3>\n
SQL style merges. See the\u00a0Database style joining<\/em><\/a><\/p>\n
\n
\n
In [74]: left = pd.DataFrame({'key': ['foo', 'foo'], 'lval': [1, 2]})\n\nIn [75]: right = pd.DataFrame({'key': ['foo', 'foo'], 'rval': [4, 5]})\n\nIn [76]: left\nOut[76]: \n   key  lval\n0  foo     1\n1  foo     2\n\nIn [77]: right\nOut[77]: \n   key  rval\n0  foo     4\n1  foo     5\n\nIn [78]: merge(left, right, on='key')\nOut[78]: \n   key  lval  rval\n0  foo     1     4\n1  foo     1     5\n2  foo     2     4\n3  foo     2     5<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
Append<\/h3>\n
Append rows to a dataframe. See the\u00a0Appending<\/em><\/a><\/p>\n
\n
\n
In [79]: df = pd.DataFrame(np.random.randn(8, 4), columns=['A','B','C','D'])\n\nIn [80]: df\nOut[80]: \n          A         B         C         D\n0  1.346061  1.511763  1.627081 -0.990582\n1 -0.441652  1.211526  0.268520  0.024580\n2 -1.577585  0.396823 -0.105381 -0.532532\n3  1.453749  1.208843 -0.080952 -0.264610\n4 -0.727965 -0.589346  0.339969 -0.693205\n5 -0.339355  0.593616  0.884345  1.591431\n6  0.141809  0.220390  0.435589  0.192451\n7 -0.096701  0.803351  1.715071 -0.708758\n\nIn [81]: s = df.iloc[3]\n\nIn [82]: df.append(s, ignore_index=True)\nOut[82]: \n          A         B         C         D\n0  1.346061  1.511763  1.627081 -0.990582\n1 -0.441652  1.211526  0.268520  0.024580\n2 -1.577585  0.396823 -0.105381 -0.532532\n3  1.453749  1.208843 -0.080952 -0.264610\n4 -0.727965 -0.589346  0.339969 -0.693205\n5 -0.339355  0.593616  0.884345  1.591431\n6  0.141809  0.220390  0.435589  0.192451\n7 -0.096701  0.803351  1.715071 -0.708758\n8  1.453749  1.208843 -0.080952 -0.264610\n\nIn [83]: df\nOut[83]: \n          A         B         C         D\n0  1.346061  1.511763  1.627081 -0.990582\n1 -0.441652  1.211526  0.268520  0.024580\n2 -1.577585  0.396823 -0.105381 -0.532532\n3  1.453749  1.208843 -0.080952 -0.264610\n4 -0.727965 -0.589346  0.339969 -0.693205\n5 -0.339355  0.593616  0.884345  1.591431\n6  0.141809  0.220390  0.435589  0.192451\n7 -0.096701  0.803351  1.715071 -0.708758<\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
Grouping<\/h2>\n
By \u201cgroup by\u201d we are referring to a process involving one or more of the following steps<\/p>\n
\n