Reducing Pandas Series With Multiple Nan Values To A Set Gives Multiple Nan Values

I'm expecting to get set([nan,0,1]) but I get set([nan, 0.0, nan, 1.0]): >>> import numpy as np >>> import pandas as pd >>> l= [np.nan,0,1,np.nan] >&g

Solution 1:

Not all nans are identical:

In [182]: np.nan is np.nan
Out[182]: True

In [183]: float('nan') isfloat('nan')
Out[183]: False

In [184]: np.float64('nan') is np.float64('nan')
Out[184]: False

Therefore,

In [178]: set([np.nan, np.nan])
Out[178]: {nan}

In [179]: set([float('nan'), float('nan')])
Out[179]: {nan, nan}

In [180]: set([np.float64('nan'), np.float64('nan')])
Out[180]: {nan, nan}

l contains np.nans, which are identical, so

In [158]: set(l)
Out[158]: {nan, 0, 1}

but pd.Series(l).tolist() contains np.float64('nan')s which are not identical:

In [160]: [type(item) for item in pd.Series(l).tolist()]
Out[160]: [numpy.float64, numpy.float64, numpy.float64, numpy.float64]

so set does not treat them as equal:

In [157]: set(pd.Series(l).tolist())
Out[157]: {nan, 0.0, nan, 1.0}

---

If you have a Pandas Series, use it's unique method instead of set to find unique values:

>>>s = pd.Series(l)>>>s.unique()
array([ nan,   0.,   1.])