Optimize readme (#209)

* 1、修改加粗异常
2、英文Readme中免责声明改为英文
3、贡献者归类

* 删除重复部分

---------

Signed-off-by: 叨叨 <173192386@qq.com>

README.md

@@ -38,8 +38,7 @@ QQ GROUP：[276392799](https://s.xaoyo.top/gOLUDl) or [276392799](https://s.xaoy
 
 #### 2.1 Core
 
-* (1) Get the **base address offset
+* (1) Get the **base address offset** of WeChat nickname, WeChat account, WeChat phone number, WeChat email, and WeChat KEY
-  ** of WeChat nickname, WeChat account, WeChat phone number, WeChat email, and WeChat KEY
 * (2) Get the WeChat nickname, WeChat account, WeChat phone number, WeChat email, WeChat KEY, WeChat original ID (wxid_******), and WeChat folder path of the currently logged-in WeChat
 * (3) Decrypt WeChat database based on key
 * (4) Combine multiple types of databases for unified viewing
@@ -108,7 +107,7 @@ QQ GROUP：[276392799](https://s.xaoyo.top/gOLUDl) or [276392799](https://s.xaoy
 
 * If you want to modify the UI, clone the [wx_dump_web](https://github.com/xaoyaoo/wxdump_web) and modify it as needed (the UI is developed using VUE+ElementUI)
 
-【注】:
+【note】:
 
 * For obtaining the base address using cheat engine, refer to [CE obtaining base address.md](https://github.com/xaoyaoo/PyWxDump/tree/master/doc/CE获取基址.md)
   (This method can be replaced by the `wxdump bias` command, and is only used for learning principles.)
@@ -118,9 +117,7 @@ QQ GROUP：[276392799](https://s.xaoyo.top/gOLUDl) or [276392799](https://s.xaoy
 
 ### 1. Purpose of use
 
-* This project is only for learning and communication purposes, **please do not use it for illegal purposes**, **please
+* This project is only for learning and communication purposes, **please do not use it for illegal purposes**, **please do not use it for illegal purposes**, **please do not use it for illegal purposes**, otherwise the consequences will be borne by yourself.
-  do not use it for illegal purposes**, **please do not use it for illegal purposes
-  **, otherwise the consequences will be borne by yourself.
 * Users understand and agree that any violation of laws and regulations, infringement of the legitimate rights and interests of others, is unrelated to this project and its developers, and the consequences are borne by the user themselves.
 
 ### 2. Usage Period
@@ -152,45 +149,13 @@ QQ GROUP：[276392799](https://s.xaoyo.top/gOLUDl) or [276392799](https://s.xaoy
 
 * Users are requested to carefully read and understand all contents of this disclaimer, and ensure that they strictly comply with relevant regulations when using this project.
 
-# Ⅳ. 免责声明（非常重要！！！！！！！）
+# Ⅳ. Acknowledgments
 
-### 1. 使用目的
+[![PyWxDump CONTRIBUTORS](https://contrib.rocks/image?repo=xaoyaoo/PyWxDump)](https://github.com/xaoyaoo/PyWxDump/graphs/contributors)  
 
-* 本项目仅供学习交流使用，**请勿用于非法用途**，**请勿用于非法用途**，**请勿用于非法用途**，否则后果自负。
+UI CONTRIBUTORS:    
-* 用户理解并同意，任何违反法律法规、侵犯他人合法权益的行为，均与本项目及其开发者无关，后果由用户自行承担。
 
-### 2. 使用期限
+[![UI CONTRIBUTORS](https://contrib.rocks/image?repo=xaoyaoo/wxdump_web)](https://github.com/xaoyaoo/wxdump_web/graphs/contributors)
-
-* 您应该在下载保存，编译使用本项目的24小时内，删除本项目的源代码和（编译出的）程序；超出此期限的任何使用行为，一概与本项目及其开发者无关。
-
-### 3. 操作规范
-
-* 本项目仅允许在授权情况下对数据库进行备份与查看，严禁用于非法目的，否则自行承担所有相关责任；用户如因违反此规定而引发的任何法律责任，将由用户自行承担，与本项目及其开发者无关。
-* 严禁用于窃取他人隐私，严禁用于窃取他人隐私，严禁用于窃取他人隐私，否则自行承担所有相关责任。
-* 严禁进行二次开发，严禁进行二次开发，严禁进行二次开发，否则自行承担所有相关责任。
-
-### 4. 免责声明接受
-
-* 下载、保存、进一步浏览源代码或者下载安装、编译使用本程序，表示你同意本警告，并承诺遵守它;
-
-### 5. 禁止用于非法测试或渗透
-
-* 禁止利用本项目的相关技术从事非法测试或渗透，禁止利用本项目的相关代码或相关技术从事任何非法工作，如因此产生的一切不良后果与本项目及其开发者无关。
-* 任何因此产生的不良后果，包括但不限于数据泄露、系统瘫痪、侵犯隐私等，均与本项目及其开发者无关，责任由用户自行承担。
-
-### 6. 免责声明修改
-
-* 本免责声明可能根据项目运行情况和法律法规的变化进行修改和调整。用户应定期查阅本页面以获取最新版本的免责声明，使用本项目时应遵守最新版本的免责声明。
-
-### 7. 其他
-
-* 除本免责声明规定外，用户在使用本项目过程中应遵守相关的法律法规和道德规范。对于因用户违反相关规定而引发的任何纠纷或损失，本项目及其开发者不承担任何责任。
-
-* 请用户慎重阅读并理解本免责声明的所有内容，确保在使用本项目时严格遵守相关规定。
-
-# Ⅴ. Acknowledgments
-
-[![PyWxDump CONTRIBUTORS](https://contrib.rocks/image?repo=xaoyaoo/PyWxDump)](https://github.com/xaoyaoo/PyWxDump/graphs/contributors)[![UI CONTRIBUTORS](https://contrib.rocks/image?repo=xaoyaoo/wxdump_web)](https://github.com/xaoyaoo/wxdump_web/graphs/contributors)
 
 otherContributors:
 

doc/CHANGELOG.md

@@ -1,6 +1,59 @@
-## v3.1.35.(待发布)
+## v3.1.46.(待发布)
 
-- 
+- UPDATE CHANGELOG.md
+
+## v3.1.45
+
+- add wx 3.9.12.51
+- UPDATE CHANGELOG.md
+- fix gen_change_log.py
+
+## v3.1.44
+
+- fix #176
+- fix #178
+- update #178
+
+## v3.1.43
+
+- add 3.9.12.45
+- add wx 3.9.12.37
+
+## v3.1.42
+
+- add wx 3.9.12.37
+
+## v3.1.41, tag: v3.1.40, tag: v3.1.39
+
+- 新增消息分类 (#162)
+- fix 修改flask启动方式
+- add wx 3.9.12.31
+- UPDATE CHANGELOG.md
+- Merge remote-tracking branch 'origin'
+
+## v3.1.38
+
+- fix
+- 实时消息增加中文路径支持
+- UPDATE CHANGELOG.md
+
+## v3.1.37
+
+- fix
+- 完善收藏的类型转换体系
+- fix tag查询结果去重
+- modify log fmt
+
+## v3.1.36
+
+- fix #143
+- UPDATE CHANGELOG.md
+
+## v3.1.35
+
+- fix
+- 增加api文档说明
+- UPDATE CHANGELOG.md
 
 ## v3.1.34
 

doc/wx数据库简述.md

@@ -227,23 +227,33 @@ FTS 这一前缀了——这代表的是搜索时所需的索引。
 | 1        | 0            | 文本                                                          |
 | 3        | 0            | 图片                                                          |
 | 34       | 0            | 语音                                                          |
+| 37       | 0            | 打招呼, 加好友的时候输入的 `我是某某某` 这一句话                 |
+| 42       | 0            | 向别人推荐自己的好友                                           |
 | 43       | 0            | 视频                                                          |
 | 47       | 0            | 动画表情（第三方开发的表情包）                                             |
-| 49       | 1            | 类似文字消息而不一样的消息，目前只见到一个阿里云盘的邀请注册是这样的。估计和57子类的情况一样             |
+| 48       | 0            | 地图定位                                                          |
+| 49       | 1            | 类似文字消息而不一样的消息，目前只见到一个阿里云盘的邀请注册是这样的，还有飞书日程。估计和57子类的情况一样             |
+| 49       | 4            | 分享 Bilibili 视频 |
 | 49       | 5            | 卡片式链接，CompressContent 中有标题、简介等，BytesExtra 中有本地缓存的封面路径       |
 | 49       | 6            | 文件，CompressContent 中有文件名和下载链接（但不会读），BytesExtra 中有本地保存的路径    |
 | 49       | 8            | 用户上传的 GIF 表情，CompressContent 中有CDN链接，不过似乎不能直接访问下载           |
 | 49       | 19           | 合并转发的聊天记录，CompressContent 中有详细聊天记录，BytesExtra 中有图片视频等的缓存    |
 | 49       | 33/36        | 分享的小程序，CompressContent 中有卡片信息，BytesExtra 中有封面缓存位置           |
+| 49       | 50           | 微视频 |
+| 49       | 51           | 分享朋友圈动态 |
+| 49       | 53           | 接龙 |
 | 49       | 57           | 带有引用的文本消息（这种类型下 StrContent 为空，发送和引用的内容均在 CompressContent 中） |
 | 49       | 63           | 视频号直播或直播回放等                                                 |
+| 49       | 76           | 分享歌曲 |
 | 49       | 87           | 群公告                                                         |
 | 49       | 88           | 视频号直播或直播回放等                                                 |
 | 49       | 2000         | 转账消息（包括发出、接收、主动退还）                                          |
 | 49       | 2003         | 赠送红包封面                                                      |
+| 50       | 0         | 语音通话                                                      |
+| 65       | 0         | 朋友推荐消息                                                      |
 | 10000    | 0            | 系统通知（居中出现的那种灰色文字）                                           |
 | 10000    | 4            | 拍一拍                                                         |
 | 10000    | 8000         | 系统通知（特别包含你邀请别人加入群聊）                                         |
 
 ## 更多内容查看：
-https://blog.csdn.net/weixin_44495599/article/details/130030359
+https://blog.csdn.net/weixin_44495599/article/details/130030359

pywxdump/WX_OFFS.json

@@ -425,5 +425,33 @@
     93834792,
     0,
     93836256
+  ],
+  "3.9.12.31": [
+    94516904,
+    94518240,
+    94516712,
+    0,
+    94518176
+  ],
+  "3.9.12.37": [
+    94520808,
+    94522144,
+    94522146,
+    0,
+    94522080
+  ],
+  "3.9.12.45": [
+    94503784,
+    94505120,
+    94503592,
+    0,
+    94505056
+  ],
+  "3.9.12.51": [
+    94555176,
+    94556512,
+    94554984,
+    0,
+    94556448
   ]
 }

pywxdump/__init__.py

@@ -5,7 +5,7 @@
 # Author:       xaoyaoo
 # Date:         2023/10/14
 # -------------------------------------------------------------------------------
-__version__ = "3.1.35"
+__version__ = "3.1.45"
 
 import os, json
 
@@ -31,5 +31,4 @@ from .api.export import export_html, export_csv, export_json
 __all__ = ["BiasAddr", "get_wx_info", "get_wx_db", "batch_decrypt", "decrypt", "get_core_db",
            "merge_db", "decrypt_merge", "merge_real_time_db", "all_merge_real_time_db",
            "DBHandler", "MsgHandler", "MicroHandler", "MediaHandler", "OpenIMContactHandler", "FavoriteHandler",
-           "PublicMsgHandler",
+           "PublicMsgHandler", "start_server", "WX_OFFS", "WX_OFFS_PATH", "__version__"]
-           "start_server", "WX_OFFS", "WX_OFFS_PATH", "__version__"]

pywxdump/api/__init__.py

@@ -14,6 +14,7 @@ import mimetypes
 import logging
 from logging.handlers import RotatingFileHandler
 
+from uvicorn.config import LOGGING_CONFIG
 from fastapi import FastAPI, Request, Path, Query
 from fastapi.staticfiles import StaticFiles
 from fastapi.exceptions import RequestValidationError
@@ -28,7 +29,7 @@ from .local_server import ls_api
 from pywxdump import __version__
 
 
-def gen_fastapi_app(handler):
+def gen_fastapi_app(handler, origins=None):
     app = FastAPI(title="wxdump", description="微信工具", version=__version__,
                   terms_of_service="https://github.com/xaoyaoo/pywxdump",
                   contact={"name": "xaoyaoo", "url": "https://github.com/xaoyaoo/pywxdump"},
@@ -36,15 +37,15 @@ def gen_fastapi_app(handler):
                                 "url": "https://github.com/xaoyaoo/PyWxDump/blob/master/LICENSE"})
 
     web_path = os.path.join(os.path.dirname(os.path.dirname(__file__)), "ui", "web")  # web文件夹路径
-
     # 跨域
-    origins = [
+    if not origins:
-        "http://localhost:5000",
+        origins = [
-        "http://127.0.0.1:5000",
+            "http://localhost:5000",
-        "http://localhost:8080",  # 开发环境的客户端地址"
+            "http://127.0.0.1:5000",
-        # "http://0.0.0.0:5000",
+            "http://localhost:8080",  # 开发环境的客户端地址"
-        # "*"
+            # "http://0.0.0.0:5000",
-    ]
+            # "*"
+        ]
     app.add_middleware(
         CORSMiddleware,
         allow_origins=origins,  # 允许所有源
@@ -190,11 +191,25 @@ def start_server(port=5000, online=False, debug=False, isopenBrowser=True,
 
     time.sleep(1)
     server_loger.info(f"启动flask服务，host:port：{host}:{port}")
-    print("[+] 请使用浏览器访问 http://127.0.0.1:5000/ 查看聊天记录")
+    print(f"[+] 请使用浏览器访问 http://127.0.0.1:{port}/ 查看聊天记录")
     global app
-    print("[+] 如需查看api文档，请访问 http://127.0.0.1:5000/docs ")
+    print(f"[+] 如需查看api文档，请访问 http://127.0.0.1:{port}/docs ")
-    app = gen_fastapi_app(file_handler)
+    origins = [
-    uvicorn.run(app=app, host=host, port=port, reload=debug, log_level="info", workers=1, env_file=env_file)
+        f"http://localhost:{port}",
+        f"http://{host}:{port}",
+        f"http://localhost:8080",  # 开发环境的客户端地址"
+        # f"http://0.0.0.0:{port}",
+        # "*"
+    ]
+    app = gen_fastapi_app(file_handler, origins)
+
+    LOGGING_CONFIG["formatters"]["default"]["fmt"] = "[%(asctime)s] %(levelprefix)s %(message)s"
+    LOGGING_CONFIG["formatters"]["access"][
+        "fmt"] = '[%(asctime)s] %(levelprefix)s %(client_addr)s - "%(request_line)s" %(status_code)s'
+    config = uvicorn.Config(app=app, host=host, port=port, reload=debug, log_level="info", workers=1, env_file=env_file)
+    server = uvicorn.Server(config)
+    server.run()
+    # uvicorn.run(app=app, host=host, port=port, reload=debug, log_level="info", workers=1, env_file=env_file)
 
 
 app = None

pywxdump/api/export/exportCSV.py

@@ -30,7 +30,8 @@ def export_csv(wxid, outpath, db_config, my_wxid="我", page_size=5000):
     users = {}
     for i in range(0, chatCount, page_size):
         start_index = i
-        data, users_t = db.get_msg_list(wxid, start_index, page_size)
+        data, users_t = db.get_msgs(wxid, start_index, page_size)
+        print(users, users_t)
         users.update(users_t)
 
         if len(data) == 0:

pywxdump/db/dbFavorite.py

@@ -5,6 +5,8 @@
 # Author:       xaoyaoo
 # Date:         2024/05/18
 # -------------------------------------------------------------------------------
+from collections import defaultdict
+
 from .dbbase import DatabaseBase
 from .utils import timestamp2str, xml2dict
 
@@ -39,7 +41,8 @@ class FavoriteHandler(DatabaseBase):
         """
         return: [(FavLocalID, TagName)]
         """
-        sql = "select A.FavLocalID, B.TagName from FavBindTagDatas A, FavTagDatas B where A.TagLocalID = B.LocalID"
+        sql = ("select DISTINCT  A.FavLocalID, B.TagName "
+               "from FavBindTagDatas A, FavTagDatas B where A.TagLocalID = B.LocalID")
         FavBindTags = self.execute(sql)
         return FavBindTags
 
@@ -125,6 +128,18 @@ class FavoriteHandler(DatabaseBase):
         FavTagsDict = {}
         for FavLocalID, TagName in FavTags:
             FavTagsDict[FavLocalID] = FavTagsDict.get(FavLocalID, []) + [TagName]
+
+        rdata = []
+        for item in FavItemsList:
+            processed_item = {
+                key: item[i] for i, key in enumerate(FavItemsFields.keys())
+            }
+            processed_item['UpdateTime'] = timestamp2str(processed_item['UpdateTime'])
+            processed_item['XmlBuf'] = xml2dict(processed_item['XmlBuf'])
+            processed_item['TypeName'] = Favorite_type_converter(processed_item['Type'])
+            processed_item['FavData'] = FavDataDict.get(processed_item['FavLocalID'], [])
+            processed_item['Tags'] = FavTagsDict.get(processed_item['FavLocalID'], [])
+            rdata.append(processed_item)
         try:
             import pandas as pd
         except ImportError:
@@ -133,7 +148,7 @@ class FavoriteHandler(DatabaseBase):
         pf.columns = FavItemsFields.keys()  # set column names
         pf["UpdateTime"] = pf["UpdateTime"].apply(timestamp2str)  # 处理时间
         pf["XmlBuf"] = pf["XmlBuf"].apply(xml2dict)  # 处理xml
-        pf["TypeName"] = pf["Type"].apply(FavoriteTypeId2Name)  # 添加类型名称列
+        pf["TypeName"] = pf["Type"].apply(Favorite_type_converter)  # 添加类型名称列
         pf["FavData"] = pf["FavLocalID"].apply(lambda x: FavDataDict.get(x, []))  # 添加数据列
         pf["Tags"] = pf["FavLocalID"].apply(lambda x: FavTagsDict.get(x, []))  # 添加标签列
         pf = pf.fillna("")  # 去掉Nan
@@ -141,8 +156,15 @@ class FavoriteHandler(DatabaseBase):
         return rdata
 
 
-def FavoriteTypeId2Name(Type):
+def Favorite_type_converter(type_id_or_name: [str, int]):
-    TypeNameDict = {
+    """
+    收藏类型ID与名称转换
+    名称(str)=>ID(int)
+    ID(int)=>名称(str)
+    :param type_id_or_name: 消息类型ID或名称
+    :return: 消息类型名称或ID
+    """
+    type_name_dict = defaultdict(lambda: "未知", {
         1: "文本",  # 文本 已测试
         2: "图片",  # 图片 已测试
         3: "语音",  # 语音
@@ -154,5 +176,11 @@ def FavoriteTypeId2Name(Type):
         14: "聊天记录",  # 聊天记录 已测试
         16: "群聊视频",  # 群聊中的视频 可能
         18: "笔记"  # 笔记 已测试
-    }
+    })
-    return TypeNameDict.get(Type, "未知")
+
+    if isinstance(type_id_or_name, int):
+        return type_name_dict[type_id_or_name]
+    elif isinstance(type_id_or_name, str):
+        return next((k for k, v in type_name_dict.items() if v == type_id_or_name), (0, 0))
+    else:
+        raise ValueError("Invalid input type")

pywxdump/wx_core/merge_db.py

@@ -161,8 +161,23 @@ def merge_db(db_paths: List[dict], save_path: str = "merge.db", is_merge_data: b
             # 创建包含 NULL 值比较的 UNIQUE 索引
             index_name = f"{table}_unique_index"
             coalesce_columns = ','.join(f"COALESCE({column}, '')" for column in columns)
-            sql = f"CREATE UNIQUE INDEX IF NOT EXISTS {index_name} ON {table} ({coalesce_columns})"
+            sql = f"CREATE UNIQUE INDEX IF NOT EXISTS {index_name} ON {table} ({coalesce_columns})"  # 创建索引
-            out_cursor.execute(sql)
+
+            # ****** 该部分代码来源于 https://github.com/xaoyaoo/PyWxDump/issues/176
+            # 防止数据重复导致索引创建失败
+            sql_if_exists_index = f"SELECT 1 FROM sqlite_master WHERE type='index' AND name='{index_name}' AND tbl_name='{table}';"
+            out_cursor.execute(sql_if_exists_index)
+            ret_if_exists_index = out_cursor.fetchone()
+            if ret_if_exists_index is None:
+                # 之前没创建过索引 先执行删除删除相同数据
+                # DELETE FROM employees WHERE ROWID NOT IN ( SELECT MIN(ROWID) FROM employees GROUP BY name, position);
+                str_columns = ','.join(columns)
+                # sql_clear_same = f"DELETE FROM {table} WHERE ROWID NOT IN (SELECT MIN(ROWID) FROM {table} GROUP BY {str_columns});"
+                sql_clear_same = f'''WITH Ranked AS (SELECT ROWID, ROW_NUMBER() OVER (PARTITION BY {str_columns} ORDER BY ROWID) AS rn FROM {table}) 
+            					DELETE FROM {table} WHERE ROWID IN (SELECT ROWID FROM Ranked WHERE rn > 1);'''
+                out_cursor.execute(sql_clear_same)
+
+            out_cursor.execute(sql)  # 执行创建索引
 
             # 插入sync_log
             sql_query_sync_log = f"SELECT src_count FROM sync_log WHERE db_path=? AND tbl_name=?"

tests/gen_change_log.py

@@ -11,6 +11,7 @@ import time
 
 
 def custom_sort_key(tag):
+    tag = tag.split(',')[0]
     if tag == 'python':
         return "000.000.000"
     elif tag == 'HEAD':