重复标签#
Index 对象不要求是唯一的；您可以有重复的行或列标签。乍一看，这可能有点令人困惑。如果您熟悉SQL，就会知道行标签类似于表上的主键，并且您永远不会希望在SQL表中出现重复项。但Pandas的角色之一是在混乱的真实世界数据进入下游系统之前清理它们。现实世界中的数据也有重复，即使在本应是唯一的字段中也是如此。
本节介绍重复标签如何更改某些操作的行为，以及如何防止在操作过程中出现重复项，或者在出现重复项时如何检测它们。
In [1]: import pandas as pd
In [2]: import numpy as np
标签重复的后果#
Pandas的一些方法 (Series.reindex() 例如)不要在存在重复项的情况下工作。产量无法确定，因此Pandas增加了产量。
In [3]: s1 = pd.Series([0, 1, 2], index=["a", "b", "b"])
In [4]: s1.reindex(["a", "b", "c"])
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 s1.reindex(["a", "b", "c"])
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/series.py:4813, in Series.reindex(self, *args, **kwargs)
4809 raise TypeError(
4810 "'index' passed as both positional and keyword argument"
4811 )
4812 kwargs.update({"index": index})
-> 4813 return super().reindex(**kwargs)
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/generic.py:4981, in NDFrame.reindex(self, *args, **kwargs)
4978 return self._reindex_multi(axes, copy, fill_value)
4980 # perform the reindex on the axes
-> 4981 return self._reindex_axes(
4982 axes, level, limit, tolerance, method, fill_value, copy
4983 ).__finalize__(self, method="reindex")
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/generic.py:5001, in NDFrame._reindex_axes(self, axes, level, limit, tolerance, method, fill_value, copy)
4996 new_index, indexer = ax.reindex(
4997 labels, level=level, limit=limit, tolerance=tolerance, method=method
4998 )
5000 axis = self._get_axis_number(a)
-> 5001 obj = obj._reindex_with_indexers(
5002 {axis: [new_index, indexer]},
5003 fill_value=fill_value,
5004 copy=copy,
5005 allow_dups=False,
5006 )
5007 # If we've made a copy once, no need to make another one
5008 copy = False
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/generic.py:5047, in NDFrame._reindex_with_indexers(self, reindexers, fill_value, copy, allow_dups)
5044 indexer = ensure_platform_int(indexer)
5046 # TODO: speed up on homogeneous DataFrame objects (see _reindex_multi)
-> 5047 new_data = new_data.reindex_indexer(
5048 index,
5049 indexer,
5050 axis=baxis,
5051 fill_value=fill_value,
5052 allow_dups=allow_dups,
5053 copy=copy,
5054 )
5055 # If we've made a copy once, no need to make another one
5056 copy = False
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/internals/managers.py:636, in BaseBlockManager.reindex_indexer(self, new_axis, indexer, axis, fill_value, allow_dups, copy, only_slice, use_na_proxy)
634 # some axes don't allow reindexing with dups
635 if not allow_dups:
--> 636 self.axes[axis]._validate_can_reindex(indexer)
638 if axis >= self.ndim:
639 raise IndexError("Requested axis not found in manager")
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/indexes/base.py:4323, in Index._validate_can_reindex(self, indexer)
4321 # trying to reindex on an axis with duplicates
4322 if not self._index_as_unique and len(indexer):
-> 4323 raise ValueError("cannot reindex on an axis with duplicate labels")
ValueError: cannot reindex on an axis with duplicate labels
其他方法，如索引，可能会产生非常令人惊讶的结果。通常使用标量进行索引将 降维 。切片为 DataFrame 使用标量将返回一个 Series 。切片为 Series 将返回一个标量。但对于复制品，情况并非如此。
In [5]: df1 = pd.DataFrame([[0, 1, 2], [3, 4, 5]], columns=["A", "A", "B"])
In [6]: df1
Out[6]:
A A B
0 0 1 2
1 3 4 5
我们的栏目里有复制品。如果我们切开 'B' ，我们会得到一个 Series
In [7]: df1["B"] # a series
Out[7]:
0 2
1 5
Name: B, dtype: int64
但切片 'A' 返回一个 DataFrame
In [8]: df1["A"] # a DataFrame
Out[8]:
A A
0 0 1
1 3 4
这也适用于行标签
In [9]: df2 = pd.DataFrame({"A": [0, 1, 2]}, index=["a", "a", "b"])
In [10]: df2
Out[10]:
A
a 0
a 1
b 2
In [11]: df2.loc["b", "A"] # a scalar
Out[11]: 2
In [12]: df2.loc["a", "A"] # a Series
Out[12]:
a 0
a 1
Name: A, dtype: int64
重复标签检测#
您可以检查是否存在 Index (存储行或列标签)与 Index.is_unique ：
In [13]: df2
Out[13]:
A
a 0
a 1
b 2
In [14]: df2.index.is_unique
Out[14]: False
In [15]: df2.columns.is_unique
Out[15]: True
备注
对于较大的数据集，检查索引是否唯一的成本较高。Pandas确实缓存了这个结果，所以重新检查相同的索引非常快。
Index.duplicated() 将返回一个布尔ndarray，指示标签是否重复。
In [16]: df2.index.duplicated()
Out[16]: array([False, True, False])
它可以用作布尔筛选器来删除重复行。
In [17]: df2.loc[~df2.index.duplicated(), :]
Out[17]:
A
a 0
b 2
如果您需要额外的逻辑来处理重复标签，而不仅仅是删除重复标签，请使用 groupby() 在指数上是一个常见的把戏。例如，我们将通过取具有相同标签的所有行的平均值来解决重复问题。
In [18]: df2.groupby(level=0).mean()
Out[18]:
A
a 0.5
b 2.0
不允许重复标签#
1.2.0 新版功能.
As noted above, handling duplicates is an important feature when reading in raw
data. That said, you may want to avoid introducing duplicates as part of a data
processing pipeline (from methods like pandas.concat(),
rename(), etc.). Both Series and DataFrame
disallow duplicate labels by calling .set_flags(allows_duplicate_labels=False).
(the default is to allow them). If there are duplicate labels, an exception
will be raised.
In [19]: pd.Series([0, 1, 2], index=["a", "b", "b"]).set_flags(allows_duplicate_labels=False)
---------------------------------------------------------------------------
DuplicateLabelError Traceback (most recent call last)
Input In [19], in <cell line: 1>()
----> 1 pd.Series([0, 1, 2], index=["a", "b", "b"]).set_flags(allows_duplicate_labels=False)
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/generic.py:428, in NDFrame.set_flags(self, copy, allows_duplicate_labels)
426 df = self.copy(deep=copy)
427 if allows_duplicate_labels is not None:
--> 428 df.flags["allows_duplicate_labels"] = allows_duplicate_labels
429 return df
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/flags.py:105, in Flags.__setitem__(self, key, value)
103 if key not in self._keys:
104 raise ValueError(f"Unknown flag {key}. Must be one of {self._keys}")
--> 105 setattr(self, key, value)
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/flags.py:92, in Flags.allows_duplicate_labels(self, value)
90 if not value:
91 for ax in obj.axes:
---> 92 ax._maybe_check_unique()
94 self._allows_duplicate_labels = value
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/indexes/base.py:743, in Index._maybe_check_unique(self)
740 duplicates = self._format_duplicate_message()
741 msg += f"\n{duplicates}"
--> 743 raise DuplicateLabelError(msg)
DuplicateLabelError: Index has duplicates.
positions
label
b [1, 2]
属性的行标签和列标签都适用 DataFrame
In [20]: pd.DataFrame([[0, 1, 2], [3, 4, 5]], columns=["A", "B", "C"],).set_flags(
....: allows_duplicate_labels=False
....: )
....:
Out[20]:
A B C
0 0 1 2
1 3 4 5
可以使用检查或设置此属性 allows_duplicate_labels ，它指示该对象是否可以具有重复的标签。
In [21]: df = pd.DataFrame({"A": [0, 1, 2, 3]}, index=["x", "y", "X", "Y"]).set_flags(
....: allows_duplicate_labels=False
....: )
....:
In [22]: df
Out[22]:
A
x 0
y 1
X 2
Y 3
In [23]: df.flags.allows_duplicate_labels
Out[23]: False
DataFrame.set_flags() 可用于返回新的 DataFrame 具有如下属性 allows_duplicate_labels 设置为某个值
In [24]: df2 = df.set_flags(allows_duplicate_labels=True)
In [25]: df2.flags.allows_duplicate_labels
Out[25]: True
新的 DataFrame 返回的是与旧数据相同的数据的视图 DataFrame 。或者可以直接在同一对象上设置该属性
In [26]: df2.flags.allows_duplicate_labels = False
In [27]: df2.flags.allows_duplicate_labels
Out[27]: False
在处理原始、杂乱的数据时，最初可能会读入杂乱的数据(可能有重复的标签)，然后删除重复数据，然后不允许重复，以确保您的数据管道不会引入重复数据。
>>> raw = pd.read_csv("...")
>>> deduplicated = raw.groupby(level=0).first() # remove duplicates
>>> deduplicated.flags.allows_duplicate_labels = False # disallow going forward
设置 allows_duplicate_labels=False 在一个 Series 或 DataFrame 使用重复标签，或执行在 Series 或 DataFrame 不允许重复将引发 errors.DuplicateLabelError 。
In [28]: df.rename(str.upper)
---------------------------------------------------------------------------
DuplicateLabelError Traceback (most recent call last)
Input In [28], in <cell line: 1>()
----> 1 df.rename(str.upper)
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/frame.py:5199, in DataFrame.rename(self, mapper, index, columns, axis, copy, inplace, level, errors)
5080 def rename(
5081 self,
5082 mapper: Renamer | None = None,
(...)
5090 errors: IgnoreRaise = "ignore",
5091 ) -> DataFrame | None:
5092 """
5093 Alter axes labels.
5094
(...)
5197 4 3 6
5198 """
-> 5199 return super()._rename(
5200 mapper=mapper,
5201 index=index,
5202 columns=columns,
5203 axis=axis,
5204 copy=copy,
5205 inplace=inplace,
5206 level=level,
5207 errors=errors,
5208 )
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/generic.py:1044, in NDFrame._rename(self, mapper, index, columns, axis, copy, inplace, level, errors)
1042 return None
1043 else:
-> 1044 return result.__finalize__(self, method="rename")
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/generic.py:5560, in NDFrame.__finalize__(self, other, method, **kwargs)
5557 for name in other.attrs:
5558 self.attrs[name] = other.attrs[name]
-> 5560 self.flags.allows_duplicate_labels = other.flags.allows_duplicate_labels
5561 # For subclasses using _metadata.
5562 for name in set(self._metadata) & set(other._metadata):
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/flags.py:92, in Flags.allows_duplicate_labels(self, value)
90 if not value:
91 for ax in obj.axes:
---> 92 ax._maybe_check_unique()
94 self._allows_duplicate_labels = value
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/indexes/base.py:743, in Index._maybe_check_unique(self)
740 duplicates = self._format_duplicate_message()
741 msg += f"\n{duplicates}"
--> 743 raise DuplicateLabelError(msg)
DuplicateLabelError: Index has duplicates.
positions
label
X [0, 2]
Y [1, 3]
此错误消息包含重复的标签，以及 Series 或 DataFrame
重复的标签传播#
一般来说，不允许复制是“棘手的”。它是通过手术保存的。
In [29]: s1 = pd.Series(0, index=["a", "b"]).set_flags(allows_duplicate_labels=False)
In [30]: s1
Out[30]:
a 0
b 0
dtype: int64
In [31]: s1.head().rename({"a": "b"})
---------------------------------------------------------------------------
DuplicateLabelError Traceback (most recent call last)
Input In [31], in <cell line: 1>()
----> 1 s1.head().rename({"a": "b"})
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/series.py:4738, in Series.rename(self, index, axis, copy, inplace, level, errors)
4731 axis = self._get_axis_number(axis)
4733 if callable(index) or is_dict_like(index):
4734 # error: Argument 1 to "_rename" of "NDFrame" has incompatible
4735 # type "Union[Union[Mapping[Any, Hashable], Callable[[Any],
4736 # Hashable]], Hashable, None]"; expected "Union[Mapping[Any,
4737 # Hashable], Callable[[Any], Hashable], None]"
-> 4738 return super()._rename(
4739 index, # type: ignore[arg-type]
4740 copy=copy,
4741 inplace=inplace,
4742 level=level,
4743 errors=errors,
4744 )
4745 else:
4746 return self._set_name(index, inplace=inplace)
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/generic.py:1044, in NDFrame._rename(self, mapper, index, columns, axis, copy, inplace, level, errors)
1042 return None
1043 else:
-> 1044 return result.__finalize__(self, method="rename")
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/generic.py:5560, in NDFrame.__finalize__(self, other, method, **kwargs)
5557 for name in other.attrs:
5558 self.attrs[name] = other.attrs[name]
-> 5560 self.flags.allows_duplicate_labels = other.flags.allows_duplicate_labels
5561 # For subclasses using _metadata.
5562 for name in set(self._metadata) & set(other._metadata):
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/flags.py:92, in Flags.allows_duplicate_labels(self, value)
90 if not value:
91 for ax in obj.axes:
---> 92 ax._maybe_check_unique()
94 self._allows_duplicate_labels = value
File /usr/local/lib/python3.10/dist-packages/pandas-1.5.0.dev0+697.gf9762d8f52-py3.10-linux-x86_64.egg/pandas/core/indexes/base.py:743, in Index._maybe_check_unique(self)
740 duplicates = self._format_duplicate_message()
741 msg += f"\n{duplicates}"
--> 743 raise DuplicateLabelError(msg)
DuplicateLabelError: Index has duplicates.
positions
label
b [0, 1]
警告
这是一个实验性的功能。目前，许多方法都无法传播 allows_duplicate_labels 价值。在未来的版本中，预计每个接受或返回一个或多个DataFrame或Series对象的方法都将传播 allows_duplicate_labels 。