PYTHON基础技能 – Python的10个文件对比与合并高效策略

在日常编程或数据分析工作中，经常需要处理多个文件的对比与合并任务。Python因其强大的文件处理能力和丰富的库支持，成为了处理这类任务的理想选择。下面，我们将逐步探索10种高效的文件对比与合并策略，每一步都配有详细的代码示例和解释。

1. 基础文件读写

首先，了解如何读取和写入文件是基础。

1
<em>#&nbsp;读取文件</em><br>with&nbsp;open('file1.txt',&nbsp;'r')&nbsp;as&nbsp;file1:<br>&nbsp;&nbsp;&nbsp;&nbsp;data1&nbsp;=&nbsp;file1.readlines()<br><br><em>#&nbsp;写入文件</em><br>with&nbsp;open('merged.txt',&nbsp;'w')&nbsp;as&nbsp;merged_file:<br>&nbsp;&nbsp;&nbsp;&nbsp;for&nbsp;line&nbsp;in&nbsp;data1:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;merged_file.write(line)

2. 文件内容对比

使用

库来对比两个文件的差异。

1
import&nbsp;difflib<br><br>with&nbsp;open('file1.txt',&nbsp;'r')&nbsp;as&nbsp;file1,&nbsp;open('file2.txt',&nbsp;'r')&nbsp;as&nbsp;file2:<br>&nbsp;&nbsp;&nbsp;&nbsp;diff&nbsp;=&nbsp;difflib.unified_diff(file1.readlines(),&nbsp;file2.readlines())<br>&nbsp;&nbsp;&nbsp;&nbsp;print('\n'.join(diff))

3. 基于行的合并

当文件基于相同行结构合并时，可以直接遍历追加。

1
data&nbsp;=&nbsp;&#091;]<br><br>for&nbsp;filename&nbsp;in&nbsp;&#091;'file1.txt',&nbsp;'file2.txt']:<br>&nbsp;&nbsp;&nbsp;&nbsp;with&nbsp;open(filename,&nbsp;'r')&nbsp;as&nbsp;file:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;data.extend(file.readlines())<br><br>with&nbsp;open('merged.txt',&nbsp;'w')&nbsp;as&nbsp;merged_file:<br>&nbsp;&nbsp;&nbsp;&nbsp;for&nbsp;line&nbsp;in&nbsp;data:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;merged_file.write(line)

4. 去重合并

利用集合去除重复行后合并。

1
unique_lines&nbsp;=&nbsp;set()<br><br>for&nbsp;filename&nbsp;in&nbsp;&#091;'file1.txt',&nbsp;'file2.txt']:<br>&nbsp;&nbsp;&nbsp;&nbsp;with&nbsp;open(filename,&nbsp;'r')&nbsp;as&nbsp;file:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;unique_lines.update(file.readlines())<br><br>with&nbsp;open('merged_unique.txt',&nbsp;'w')&nbsp;as&nbsp;merged_file:<br>&nbsp;&nbsp;&nbsp;&nbsp;for&nbsp;line&nbsp;in&nbsp;sorted(unique_lines):&nbsp;&nbsp;<em>#&nbsp;排序确保一致的输出顺序</em><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;merged_file.write(line)

5. CSV文件合并

对于CSV文件，可以使用

库。

1
import&nbsp;pandas&nbsp;as&nbsp;pd<br><br>df1&nbsp;=&nbsp;pd.read_csv('file1.csv')<br>df2&nbsp;=&nbsp;pd.read_csv('file2.csv')<br><br><em>#&nbsp;假设合并依据为相同的列名</em><br>merged_df&nbsp;=&nbsp;pd.concat(&#091;df1,&nbsp;df2],&nbsp;ignore_index=True)<br>merged_df.to_csv('merged.csv',&nbsp;index=False)

6. 按列合并CSV

特定列的合并，例如通过共同键连接。

1
merged_df&nbsp;=&nbsp;pd.merge(df1,&nbsp;df2,&nbsp;on='common_key',&nbsp;how='outer')<br>merged_df.to_csv('merged_by_key.csv',&nbsp;index=False)

7. 大文件高效对比

对于大文件，逐行读取对比以节省内存。

1
with&nbsp;open('large_file1.txt',&nbsp;'r')&nbsp;as&nbsp;f1,&nbsp;open('large_file2.txt',&nbsp;'r')&nbsp;as&nbsp;f2:<br>&nbsp;&nbsp;&nbsp;&nbsp;for&nbsp;line1,&nbsp;line2&nbsp;in&nbsp;zip(f1,&nbsp;f2):<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if&nbsp;line1&nbsp;!=&nbsp;line2:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print("Difference&nbsp;found!")<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;break

8. 文本文件的二进制对比

使用

模块比较文件的二进制内容。

1
import&nbsp;filecmp<br><br>if&nbsp;filecmp.cmp('file1.txt',&nbsp;'file2.txt'):<br>&nbsp;&nbsp;&nbsp;&nbsp;print("Files&nbsp;are&nbsp;identical.")<br>else:<br>&nbsp;&nbsp;&nbsp;&nbsp;print("Files&nbsp;differ.")

9. 动态合并多个文件

使用循环动态合并多个文件路径列表中的文件。

1
file_paths&nbsp;=&nbsp;&#091;'file{}.txt'.format(i)&nbsp;for&nbsp;i&nbsp;in&nbsp;range(1,&nbsp;4)]&nbsp;&nbsp;<em>#&nbsp;假设有file1.txt到file3.txt</em><br>with&nbsp;open('merged_all.txt',&nbsp;'w')&nbsp;as&nbsp;merged:<br>&nbsp;&nbsp;&nbsp;&nbsp;for&nbsp;path&nbsp;in&nbsp;file_paths:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;with&nbsp;open(path,&nbsp;'r')&nbsp;as&nbsp;file:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;merged.write(file.read()&nbsp;+&nbsp;'\n')&nbsp;&nbsp;<em>#&nbsp;添加换行符区分不同文件的内容</em>

10. 高级合并策略：智能合并

如果合并依据更复杂，如按日期或ID排序合并，可以先对数据进行排序处理。

1
<em>#&nbsp;假设是CSV且按日期列排序合并</em><br>dfs&nbsp;=&nbsp;&#091;pd.read_csv(f)&nbsp;for&nbsp;f&nbsp;in&nbsp;&#091;'file1.csv',&nbsp;'file2.csv']]<br>sorted_df&nbsp;=&nbsp;pd.concat(dfs).sort_values(by='date_column')&nbsp;&nbsp;<em>#&nbsp;假定'date_column'是日期列</em><br>sorted_df.to_csv('smart_merged.csv',&nbsp;index=False)

进阶技巧和场景

11. 使用正则表达式进行复杂文本处理

在合并或对比前，可能需要对文件内容进行预处理，例如提取特定模式的数据。

1
import&nbsp;re<br><br>pattern&nbsp;=&nbsp;r'(\d{4}-\d{2}-\d{2})'&nbsp;&nbsp;<em>#&nbsp;假设提取日期模式</em><br>lines_with_dates&nbsp;=&nbsp;&#091;]<br><br>with&nbsp;open('source.txt',&nbsp;'r')&nbsp;as&nbsp;file:<br>&nbsp;&nbsp;&nbsp;&nbsp;for&nbsp;line&nbsp;in&nbsp;file:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;match&nbsp;=&nbsp;re.search(pattern,&nbsp;line)<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if&nbsp;match:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;lines_with_dates.append(match.group(0))<br><br><em>#&nbsp;假设你想将提取的信息写入新文件</em><br>with&nbsp;open('dates_extracted.txt',&nbsp;'w')&nbsp;as&nbsp;out_file:<br>&nbsp;&nbsp;&nbsp;&nbsp;for&nbsp;date&nbsp;in&nbsp;lines_with_dates:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;out_file.write(date&nbsp;+&nbsp;'\n')

12. 并行处理大文件对比

对于超大文件，可以利用多线程或多进程提高效率，但需注意文件访问冲突。

1
from&nbsp;multiprocessing&nbsp;import&nbsp;Pool<br>import&nbsp;os<br><br>def&nbsp;compare_lines(line1,&nbsp;line2):<br>&nbsp;&nbsp;&nbsp;&nbsp;return&nbsp;line1&nbsp;==&nbsp;line2<br><br>if&nbsp;__name__&nbsp;==&nbsp;"__main__":<br>&nbsp;&nbsp;&nbsp;&nbsp;with&nbsp;open('file1.txt',&nbsp;'r')&nbsp;as&nbsp;f1,&nbsp;open('file2.txt',&nbsp;'r')&nbsp;as&nbsp;f2:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;lines_f1&nbsp;=&nbsp;f1.readlines()<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;lines_f2&nbsp;=&nbsp;f2.readlines()<br>&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;with&nbsp;Pool(os.cpu_count())&nbsp;as&nbsp;p:&nbsp;&nbsp;<em>#&nbsp;使用CPU核心数作为进程数</em><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;results&nbsp;=&nbsp;p.map(compare_lines,&nbsp;zip(lines_f1,&nbsp;lines_f2))<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;<em>#&nbsp;results是一个布尔值列表，表示对应行是否相同</em>

13. 特殊格式文件的合并

例如XML文件，可以使用

进行解析合并。

1
import&nbsp;xml.etree.ElementTree&nbsp;as&nbsp;ET<br><br>root1&nbsp;=&nbsp;ET.parse('file1.xml').getroot()<br>root2&nbsp;=&nbsp;ET.parse('file2.xml').getroot()<br><br>for&nbsp;child&nbsp;in&nbsp;root2:<br>&nbsp;&nbsp;&nbsp;&nbsp;root1.append(child)<br><br>tree&nbsp;=&nbsp;ET.ElementTree(root1)<br>tree.write('merged.xml')

14. 实时监控文件变化并合并

利用

库监控文件变化，自动执行合并操作。

安装

:

1
pip install watchdog

示例脚本：

1
from&nbsp;watchdog.observers&nbsp;import&nbsp;Observer<br>from&nbsp;watchdog.events&nbsp;import&nbsp;FileSystemEventHandler<br>import&nbsp;time<br><br>class&nbsp;MyHandler(FileSystemEventHandler):<br>&nbsp;&nbsp;&nbsp;&nbsp;def&nbsp;on_modified(self,&nbsp;event):<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if&nbsp;event.is_directory:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<em>#&nbsp;在这里实现你的文件合并逻辑</em><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;print(f'Event&nbsp;type:&nbsp;{event.event_type}&nbsp;&nbsp;path&nbsp;:&nbsp;{event.src_path}')<br><br>if&nbsp;__name__&nbsp;==&nbsp;"__main__":<br>&nbsp;&nbsp;&nbsp;&nbsp;event_handler&nbsp;=&nbsp;MyHandler()<br>&nbsp;&nbsp;&nbsp;&nbsp;observer&nbsp;=&nbsp;Observer()<br>&nbsp;&nbsp;&nbsp;&nbsp;observer.schedule(event_handler,&nbsp;path='.',&nbsp;recursive=False)<br>&nbsp;&nbsp;&nbsp;&nbsp;observer.start()<br>&nbsp;&nbsp;&nbsp;&nbsp;try:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;while&nbsp;True:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;time.sleep(1)<br>&nbsp;&nbsp;&nbsp;&nbsp;except&nbsp;KeyboardInterrupt:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;observer.stop()<br>&nbsp;&nbsp;&nbsp;&nbsp;observer.join()

结语

通过这些高级策略和技巧，你可以更加灵活和高效地处理各种文件对比与合并的需求。