So, What Is NumPy’s reshape?

#

If you are venturing into data science, you have most likely seen a reshape in some code. Maybe with a mysterious value of -1 in one of the positions, as in reshape(-1, 1). But what does that mean? Why do we sometimes need to transform our data to use model methods in Scikit-Learn? That is what we will see in this article.

To try to reproduce a real situation, we will take a dataset, store it in a pandas dataframe, try to apply a Scikit-Learn model directly, and then use reshape to solve the errors we run into.

Imports and Example Dataset

#

To begin, let’s import the packages we need for our study and choose a dataset to use as an example:

from sklearn import datasets
from sklearn.linear_model import LinearRegression
import numpy as np
import pandas as pd

The dataset chosen for this article is part of the collection of datasets shipped with the Scikit-Learn package: it contains data on patients with diabetes. Let’s get the dataset’s variables:

diabetes_X, diabetes_y = datasets.load_diabetes(return_X_y=True, as_frame=True)

Observe, in the following cells, the first 5 entries of each variable, paying attention to how the results are displayed:

diabetes_X.head()

diabetes_y.head()

0    151.0
1     75.0
2    141.0
3    206.0
4    135.0
Name: target, dtype: float64

Although the goal of the article is not to do a detailed study of the dataset, let’s at least have a description of what each column means. The base features are in the variable diabetes_X and the target values in the variable diabetes_y. Below is a description taken from the Scikit-Learn documentation:

• age - age in years
• sex - the person’s sex
• bmi - body mass index (BMI)
• bp - average blood pressure
• s1 - tc, total serum cholesterol
• s2 - ldl, low-density lipoproteins
• s3 - hdl, high-density lipoproteins
• s4 - tch, total cholesterol / HDL
• s5 - ltg, log of serum triglycerides level
• s6 - glu, blood sugar level
• The target variable is a measure of disease progression one year after baseline.

The Difference Between a Series and a DataFrame

#

reshape is a method of NumPy, which is one of the base libraries of pandas. So let’s start by examining the pandas structures to understand how they are built on top of NumPy.

To understand what reshape means, we need to understand what the dimension of a variable is and how to get it. And how it relates to the fact that the variable is a pandas dataframe or a pandas series.

Let’s start with dimensions, using the ndim attribute:

diabetes_X.ndim

2

diabetes_y.ndim

1

The dimension 2 for our data in diabetes_X is easy to understand: it is the rows and columns. The dimension 1 for diabetes_y is also intuitive, given that we saw a sequence of numbers, a single dimension.

The shape attribute shows the number of items in each dimension:

diabetes_X.shape

(442, 10)

That is, in diabetes_X we have 442 rows and 10 columns.

diabetes_y.shape

(442,)

In diabetes_y we have 442 values.

Let’s look at the type of each variable:

type(diabetes_X), type(diabetes_y)

(pandas.core.frame.DataFrame, pandas.core.series.Series)

We see that diabetes_X is a pandas DataFrame, while diabetes_y is a Series. It is worth describing these pandas entities a little better.

According to the documentation, a DataFrame is a two-dimensional structure with columns of potentially different types. A Series, in turn, is a one-dimensional array capable of holding any type of data. That is, from the documentation alone, we could already know the dimensions of our variables as long as we knew their types.

There is yet another attribute of interest for our variables, size, which returns how many elements there are in our structure:

diabetes_X.size

4420

Since there are 442 rows and 10 columns in diabetes_X, there are 442 x 10 = 4,420 elements. You can think of it as the cells in a spreadsheet with that many rows and columns.

diabetes_y.size

442

For diabetes_y the result is obvious, since it is the number of values in our array.

Creating Series and DataFrames from a Column

#

What happens if we take a column of our dataframe? What will its type be?

bmi_series = diabetes_X["bmi"]
bmi_series.head()

0    0.061696
1   -0.051474
2    0.044451
3   -0.011595
4   -0.036385
Name: bmi, dtype: float64

type(bmi_series)

pandas.core.series.Series

That is, a DataFrame can be thought of as a container of Series. In fact, in the documentation link given earlier, this very statement appears at the end of the definition of DataFrame.

So, when we request the shape we get only one value:

bmi_series.shape

(442,)

Now, pay attention to the next cell. Notice that we put an extra pair of brackets around the column name:

bmi_df = diabetes_X[["bmi"]]
bmi_df.head()

type(bmi_df)

pandas.core.frame.DataFrame

bmi_df.shape

(442, 1)

See that, with just this change, we now have a DataFrame! A DataFrame with a single column, it is true. But the object is no longer just an array of values, it is a DataFrame.

To understand this a little better, let’s look at what is underneath pandas: the NumPy library.

NumPy Underneath pandas

#

The basic structure of NumPy is the ndarray, a name that means N-dimensional array. We can take our previous variables, of pandas types, and ask for them to be expressed in their most basic types, from NumPy, with the to_numpy method:

diabetes_X_array = diabetes_X.to_numpy()
diabetes_X_array

array([[ 0.03807591,  0.05068012,  0.06169621, ..., -0.00259226,
         0.01990842, -0.01764613],
       [-0.00188202, -0.04464164, -0.05147406, ..., -0.03949338,
        -0.06832974, -0.09220405],
       [ 0.08529891,  0.05068012,  0.04445121, ..., -0.00259226,
         0.00286377, -0.02593034],
       ...,
       [ 0.04170844,  0.05068012, -0.01590626, ..., -0.01107952,
        -0.04687948,  0.01549073],
       [-0.04547248, -0.04464164,  0.03906215, ...,  0.02655962,
         0.04452837, -0.02593034],
       [-0.04547248, -0.04464164, -0.0730303 , ..., -0.03949338,
        -0.00421986,  0.00306441]])

With printoptions, we can view it more conveniently, making the structure of our new variable clearer:

with np.printoptions(precision=5, edgeitems=3):
    display(diabetes_X_array)

array([[ 0.03808,  0.05068,  0.0617 , ..., -0.00259,  0.01991, -0.01765],
       [-0.00188, -0.04464, -0.05147, ..., -0.03949, -0.06833, -0.0922 ],
       [ 0.0853 ,  0.05068,  0.04445, ..., -0.00259,  0.00286, -0.02593],
       ...,
       [ 0.04171,  0.05068, -0.01591, ..., -0.01108, -0.04688,  0.01549],
       [-0.04547, -0.04464,  0.03906, ...,  0.02656,  0.04453, -0.02593],
       [-0.04547, -0.04464, -0.07303, ..., -0.03949, -0.00422,  0.00306]])

The ndim and shape attributes seen earlier are, in fact, inherited from NumPy:

type(diabetes_X_array)

numpy.ndarray

diabetes_X_array.ndim

2

diabetes_X_array.shape

(442, 10)

Note, therefore, that the dimension and the shape are derived from the more basic NumPy objects.

We can check the same with our target variable:

diabetes_y_array = diabetes_y.to_numpy()
with np.printoptions(precision=5, edgeitems=3, threshold=10):
    display(diabetes_y_array)

array([151.,  75., 141., ..., 132., 220.,  57.])

type(diabetes_y_array)

numpy.ndarray

diabetes_y_array.ndim

1

diabetes_y_array.shape

(442,)

So we already know that pandas is an extension of NumPy. And reshape? Where does it come into this story?

Scikit-Learn’s Dimension Requirements

#

Many Scikit-Learn methods require us to pass our base variables as a two-dimensional array. Let’s see this in our example.

Let’s recall the Series we created from one of the columns of our DataFrame:

bmi_series = diabetes_X["bmi"]
bmi_series.head()

0    0.061696
1   -0.051474
2    0.044451
3   -0.011595
4   -0.036385
Name: bmi, dtype: float64

type(bmi_series)

pandas.core.series.Series

bmi_series.shape

(442,)

And also our target variable:

diabetes_y.head()

0    151.0
1     75.0
2    141.0
3    206.0
4    135.0
Name: target, dtype: float64

diabetes_y.ndim

1

diabetes_y.shape

(442,)

We can try to check how linear the behavior of the target variable is with respect to the “bmi” Series:

lr = LinearRegression()
lr.fit(bmi_series, diabetes_y)

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

/tmp/ipykernel_471207/2026011743.py in <module>
      1 lr = LinearRegression()
----> 2 lr.fit(bmi_series, diabetes_y)


ValueError: Expected 2D array, got 1D array instead:
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

See that an error occurs, stating Expected 2D array, got 1D array instead, which is already a hint that a Series is not expected, since it is one-dimensional.

The last line of the message is:

Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

Let’s try to follow the instruction:

bmi_series.reshape(-1, 1)

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

/tmp/ipykernel_471207/2203274931.py in <module>
----> 1 bmi_series.reshape(-1, 1)


/opt/anaconda/lib/python3.9/site-packages/pandas/core/generic.py in __getattr__(self, name)
   5573         ):
   5574             return self[name]
-> 5575         return object.__getattribute__(self, name)
   5576 
   5577     def __setattr__(self, name: str, value) -> None:


AttributeError: 'Series' object has no attribute 'reshape'

Notice that the Series type does not have the reshape method. But, as we saw, pandas types are built on top of NumPy types. And reshape is a method of NumPy arrays. So we can do:

bmi_series_numpy_reshape = bmi_series.to_numpy().reshape(-1, 1)
with np.printoptions(precision=5, edgeitems=3, threshold=10):
    display(bmi_series_numpy_reshape)

array([[ 0.0617 ],
       [-0.05147],
       [ 0.04445],
       ...,
       [-0.01591],
       [ 0.03906],
       [-0.07303]])

bmi_series_numpy_reshape.ndim

2

lr.fit(bmi_series_numpy_reshape, diabetes_y)

LinearRegression()

lr.coef_, lr.intercept_, lr.score(bmi_series_numpy_reshape, diabetes_y)

(array([949.43526038]), 152.1334841628967, 0.3439237602253803)

There, we managed to run our regression. But we still do not know exactly what reshape does.

Besides, it would not be very practical, nor Pythonic, to keep resorting to the basic types underneath pandas. There is a better way to approach the question.

The Recommended Way

#

As we saw, using two pairs of brackets, we directly get a DataFrame, which is already a two-dimensional type:

bmi_df = diabetes_X[["bmi"]]
type(bmi_df)

pandas.core.frame.DataFrame

bmi_df.shape

(442, 1)

bmi_df.ndim

2

Now we can pass it directly to the fit method:

lr.fit(bmi_df, diabetes_y)

LinearRegression()

lr.coef_, lr.intercept_, lr.score(bmi_df, diabetes_y)

(array([949.43526038]), 152.1334841628967, 0.3439237602253803)

We get the same result as before, but with much less work.

Great! But what exactly is reshape, and what do the values passed to the method mean?

Understanding the Meaning of (-1, 1) in reshape

#

Let’s create a one-dimensional array and check its shape attribute:

array_1D = np.array((1, 2, 3))
array_1D

array([1, 2, 3])

array_1D.shape

(3,)

We can change the shape of this array explicitly with reshape, passing the number of rows and columns:

array_2D = array_1D.reshape(3, 1)
array_2D

array([[1],
       [2],
       [3]])

array_2D.shape

(3, 1)

array_2D.ndim

2

Now, the same can be done by passing one of the values as -1. That value simply means that NumPy will fill it in based on the total number of elements in the array and the sizes of the other dimensions. That is, if we already say that one of the dimensions has size 1 and there are 3 elements, the remaining dimension can only have size 3:

array_1D.reshape(-1, 1)

array([[1],
       [2],
       [3]])

Likewise, if one dimension has size 3, the other can only have size 1:

array_1D.reshape(3, -1)

array([[1],
       [2],
       [3]])

Thus, the great advantage of using one of the dimensions as -1 is that the code becomes dynamic. You do not need to know the total number of elements in the array; just say the size of one of the dimensions and reshape does the rest of the work. This is very convenient for the Scikit-Learn example, since we only want to add a second dimension, regardless of the size of our dataset.

reshape will raise an error if the dimensions passed do not make sense:

array_1D.reshape(2, 1)

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

/tmp/ipykernel_471207/2649632558.py in <module>
----> 1 array_1D.reshape(2, 1)


ValueError: cannot reshape array of size 3 into shape (2,1)

Let’s take one more example, creating an array with 2 rows and 4 columns:

array = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
array

array([[1, 2, 3, 4],
       [5, 6, 7, 8]])

array.shape

(2, 4)

With what we have seen, it is easy to ask reshape to turn it into an array with two columns:

array.reshape(-1, 2)

array([[1, 2],
       [3, 4],
       [5, 6],
       [7, 8]])

See that reshape correctly interpreted the -1 passed as 4.

Since our array has 8 elements, we can have a 3-dimensional array with 2 elements in each. Below, the -1 is interpreted as 2:

array_3D = array.reshape((-1, 2, 2))
array_3D

array([[[1, 2],
        [3, 4]],

       [[5, 6],
        [7, 8]]])

We need three indices to get a given element. For example, the first element of the array:

array_3D[0][0][0]

1

Other Interesting NumPy Methods

#

Finally, let’s recall one of the arrays we created earlier:

array_2D

array([[1],
       [2],
       [3]])

This array was created by transforming a one-dimensional array into a two-dimensional one, adding a dimension of size 1. We can remove dimensions of size 1 with the squeeze method:

array_2D.squeeze()

array([1, 2, 3])

Recalling our 8-element array:

array

array([[1, 2, 3, 4],
       [5, 6, 7, 8]])

We can turn it into a one-dimensional array in two ways:

array_flatten = array.flatten()
array_flatten

array([1, 2, 3, 4, 5, 6, 7, 8])

array_ravel = array.ravel()
array_ravel

array([1, 2, 3, 4, 5, 6, 7, 8])

However, these two ways are not equivalent. In short:

How can we check whether an array is a copy of another? We can use the base attribute. If the result is None, we have either the original array or a copy; otherwise, the result is the array that served as its base, its reference:

print(array.base)
print(array_flatten.base)
print(array_ravel.base)

None
None
[[1 2 3 4]
 [5 6 7 8]]

As expected, only ravel returned an array, precisely the array that served as its base, showing that the ravel array is just a view of its base array.

Conclusion

#

So, did you understand this reshape thing? We saw not only what it means, with examples from its home library NumPy, but also showed the relationship with pandas and with Scikit-Learn models. Now there is no way to be left with doubts when you find this method in some code.

What other method would you like to see detailed here on the site?

See you next time!