某日,MySQL 数据库实例突然发生异常中断重启。排查错误日志 mysqld.log,发现如下关键报错信息:
16:10:10 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
key_buffer_size=268435456
read_buffer_size=1048576
max_used_connections=1343
max_threads=3000
thread_count=913
connection_count=912
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9518292 K bytes of memory
Hope that’s ok; if not, decrease some variables in the equation.
Thread pointer: 0x2b838c0664e0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 2b80d1601ea8 thread_stack 0x40000
/opt/mysql/app/product/bin/mysqld(my_print_stacktrace+0x35)[0xf6cc45]
/opt/mysql/app/product/bin/mysqld(handle_fatal_signal+0x4a4)[0x7e6eb4]
/lib64/libpthread.so.0(+0x10c10)[0x2b67e7651c10]
/opt/mysql/app/product/bin/mysqld[0x14235d8]
/opt/mysql/app/product/bin/mysqld(_Z17find_item_in_listP4ItemR4ListIS_EPj27find_item_error_report_typeP20enum_resolution_type+0x155)[0xcd7ad5]
/opt/mysql/app/product/bin/mysqld[0xd662e2]
/opt/mysql/app/product/bin/mysqld(_Z11setup_orderP3THD20Bounds_checked_arrayIP4ItemEP10TABLE_LISTR4ListIS2_ES9_P8st_order+0xd5)[0xd668c5]
/opt/mysql/app/product/bin/mysqld(_ZN13st_select_lex7prepareEP3THD+0x747)[0xd6d047]
/opt/mysql/app/product/bin/mysqld(_Z12handle_queryP3THDP3LEXP12Query_resultyy+0x1ca)[0xd7309a]
/opt/mysql/app/product/bin/mysqld[0xd33a43]
/opt/mysql/app/product/bin/mysqld(_Z21mysql_execute_commandP3THxx+0x37aa)[0xd374ba]
/opt/mysql/app/product/bin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x320)[0xd621e0]
/opt/mysql/app/product/bin/mysqld(ZN18Prepared_statement12execute_loopEP6StringbPhS2+0xxx)[0xd6245b]
/opt/mysql/app/product/bin/mysqld(_Z19mysqld_stmt_executeP3THDmmPhm+0x10e)[0xd628ce]
/opt/mysql/app/product/bin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xaff)[0xd39bff]
/opt/mysql/app/product/bin/mysqld(_Z10do_commandP3THD+0x194)[0xd3b154]
/opt/mysql/app/product/bin/mysqld(handle_connection+0x2b4)[0xe0cfb4]
/opt/mysql/app/product/bin/mysqld(pfs_spawn_thread+0x174)[0xfeb4d4]
/lib64/libpthread.so.0(+0x8724)[0x2b67e7649724]
/lib64/libc.so.6(clone+0x6d)[0x2b67e8c95eed]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (2b8991e67d50): SELECT * FROM (SELECT TASK_ORDER_ID , OBJECT_ID , OBJ_CUST_ID , INSTANCE_ID, TASK_ID, APP_INTF_CODE, ACTION_CODE, BE_ID, BE_CODE, SUB_ID, CUST_ID FROM xxxx a WHERE a.APP_CODE = ‘APP_Sub_Suspend’ AND a.APP_INTF_CODE= ‘Sub_Suspend’ AND a.PARTITION_ID = 2 AND a.PLAN_DATE < ‘2026-06-01 16:10:10.236000’ AND (a.STATUS =‘1’ or (a.STATUS = ‘5’ and a.REDO_STATUS = ‘2’)) AND a.AUDIT_STATUS =‘2’ AND a.CANCEL_STATUS IS NULL AND (a.TST_FLAG = 1 or a.TST_FLAG is null) AND a.TASK_ORDER_ID > 0 and a.be_id = 101 ORDER BY a.TASK_ORDER_ID ASC
Connection ID (thread ID): 2684647
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
signal 11 即 SIGSEGV(段错误),表示进程访问了非法内存地址。MySQL Server 在没有前置错误告警的情况下直接崩溃,属于典型的内核级致命故障。
数据库版本:MySQL 5.7.24
二、错误日志分析
2.1 堆栈跟踪(Stack Trace)
崩溃时的调用堆栈如下:
stack_bottom = 2b80d1601ea8 thread_stack 0x40000
/opt/mysql/app/product/bin/mysqld(my_print_stacktrace+0x35)[0xf6cc45]
/opt/mysql/app/product/bin/mysqld(handle_fatal_signal+0x4a4)[0x7e6eb4]
/lib64/libpthread.so.0(+0x10c10)[0x2b67e7651c10]
/opt/mysql/app/product/bin/mysqld[0x14235d8]
/opt/mysql/app/product/bin/mysqld(_Z17find_item_in_listP4ItemR4ListIS_EPj27find_item_error_report_typeP20enum_resolution_type+0x155)[0xcd7ad5]
/opt/mysql/app/product/bin/mysqld[0xd662e2]
/opt/mysql/app/product/bin/mysqld(_Z11setup_orderP3THD20Bounds_checked_arrayIP4ItemEP10TABLE_LISTR4ListIS2_ES9_P8st_order+0xd5)[0xd668c5]
/opt/mysql/app/product/bin/mysqld(_ZN13st_select_lex7prepareEP3THD+0x747)[0xd6d047]
/opt/mysql/app/product/bin/mysqld(_Z12handle_queryP3THDP3LEXP12Query_resultyy+0x1ca)[0xd7309a]
/opt/mysql/app/product/bin/mysqld[0xd33a43]
/opt/mysql/app/product/bin/mysqld(_Z21mysql_execute_commandP3THD+0x37aa)[0xd374ba]
/opt/mysql/app/product/bin/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x320)[0xd621e0]
…
堆栈中的关键函数(从下往上追溯)清晰地指向了问题路径:
| 函数 |
所属模块 |
说明 |
Prepared_statement::execute |
sql/sql_prepare.cc |
预处理语句执行入口 |
mysql_execute_command |
sql/sql_parse.cc |
SQL 命令执行分发 |
handle_query |
sql/sql_select.cc |
查询处理 |
st_select_lex::prepare |
sql/sql_lex.cc |
SELECT 语法树准备 |
setup_order |
sql/sql_base.cc |
ORDER BY 子句解析 |
find_item_in_list |
sql/sql_base.cc |
列表中查找引用项(崩溃点) |
崩溃点定位:find_item_in_list() 函数。
2.2 引发崩溃的 SQL 语句
日志中明确记录了引发崩溃的 SQL(表名已脱敏):
SELECT * FROM (
SELECT
TASK_ORDER_ID, OBJECT_ID, OBJ_CUST_ID, INSTANCE_ID, TASK_ID,
APP_INTF_CODE, ACTION_CODE, BE_ID, BE_CODE, SUB_ID, CUST_ID
FROM xxxx a
WHERE a.APP_CODE = 'APP_Sub_Suspend'
AND a.APP_INTF_CODE = 'Sub_Suspend'
AND a.PARTITION_ID = 2
AND a.PLAN_DATE < '2026-06-01 16:10:10.236000'
AND (a.STATUS = '1' OR (a.STATUS = '5' AND a.REDO_STATUS = '2'))
AND a.AUDIT_STATUS = '2'
AND a.CANCEL_STATUS IS NULL
AND (a.TST_FLAG = 1 OR a.TST_FLAG IS NULL)
AND a.TASK_ORDER_ID > 0
AND a.be_id = 101
ORDER BY a.TASK_ORDER_ID ASC
)
这条 SQL 的结构特征非常明显:
- 派生表(Derived Table):外层
SELECT * FROM ( ... ) 包裹了一个子查询
- ORDER BY 子句:在派生表内部使用了
ORDER BY a.TASK_ORDER_ID ASC
- 预处理语句(Prepared Statement):从堆栈中的
Prepared_statement::execute 可以确认,该 SQL 是通过预处理语句方式执行的
Connection ID(线程ID):2684647,说明崩溃时该连接正在活跃执行中。
三、根因分析
3.1 直接原因
find_item_in_list() 是 MySQL SQL 解析器中用于解析 ORDER BY、GROUP BY 以及列别名引用的核心函数。崩溃发生在此函数中,说明 MySQL 在解析派生表内部的 ORDER BY 子句时,访问了非法或已释放的内存地址。
从调用栈来看,执行路径为:
预处理语句执行 → mysql_execute_command → handle_query →
st_select_lex::prepare → setup_order → find_item_in_list(崩溃)
这表明问题发生在查询编译/优化阶段,而非执行阶段。
3.2 根本原因
MySQL 5.7.24 版本在内核层面存在已知缺陷——当 ORDER BY 子句出现在派生表内部,且该 SQL 通过预处理语句方式执行时,解析器在 find_item_in_list 中对列引用的解析逻辑存在 bug,可能导致访问无效内存地址,触发 SIGSEGV。
该问题在多个 MySQL 5.7 版本中被验证存在(5.7.11 至 5.7.30 均有报告)MySQL Bug #100150:Order by in derived table in prepared statement,https://bugs.mysql.com/bug.php?id=100150。MySQL 官方在 8.0.22 版本中通过 WL#9384 对此进行了修复。
触发条件可归纳为:
- MySQL 版本为 5.7.x(尤其是 5.7.24 及附近版本)
- SQL 中包含派生表(子查询作为 FROM 子句的表)
- 派生表内部包含 ORDER BY 子句
- SQL 通过预处理语句(Prepared Statement)执行
- 实例处于较高并发压力下(本例中活跃连接 912 个)
这几个条件同时满足时,即有可能触发该内核 Bug 导致实例崩溃。
四、解决方案
4.1 根本性解决方案:升级 MySQL 版本
最彻底的解决方式是升级 MySQL 版本:
- 推荐升级到 MySQL 5.7.44(5.7 系列最新稳定版),该版本已修复大量已知缺陷
- 或升级到 MySQL 8.0.22 及以上版本,官方已通过 WL#9384 修复此问题
4.2 临时规避方案:修改 SQL 写法
核心思路:直接去掉外层 SELECT * FROM (...) 的派生表包装,ORDER BY 直接作用于主查询。这样既保留了排序逻辑,又绕过了派生表 + ORDER BY 的解析路径,从根本上规避了触发 Bug 的条件。
原始 SQL:
SELECT * FROM (
SELECT ... FROM xxxx a WHERE ... ORDER BY a.TASK_ORDER_ID ASC
)
修改为:
SELECT
TASK_ORDER_ID, OBJECT_ID, OBJ_CUST_ID, INSTANCE_ID, TASK_ID,
APP_INTF_CODE, ACTION_CODE, BE_ID, BE_CODE, SUB_ID, CUST_ID
FROM xxxx a
WHERE a.APP_CODE = 'APP_Sub_Suspend'
AND a.APP_INTF_CODE = 'Sub_Suspend'
AND a.PARTITION_ID = 2
AND a.PLAN_DATE < '2026-06-01 16:10:10.236000'
AND (a.STATUS = '1' OR (a.STATUS = '5' AND a.REDO_STATUS = '2'))
AND a.AUDIT_STATUS = '2'
AND a.CANCEL_STATUS IS NULL
AND (a.TST_FLAG = 1 OR a.TST_FLAG IS NULL)
AND a.TASK_ORDER_ID > 0
AND a.be_id = 101
ORDER BY a.TASK_ORDER_ID ASC
4.3 运维保障措施
-
强化监控告警
对 MySQL 错误日志中的 signal 11 关键字设置实时告警;监控数据库主备复制状态,确保同步正常;监控 Threads_connected、Threads_running 等连接数指标,关注异常突增。
-
完善应急预案
若再次发生类似崩溃,确保能够快速通过高可用工具完成主备切换;准备全量备份,以防主备出现数据不一致时需要重建恢复。
-
SQL 审核与改造
对业务中所有使用“派生表 + ORDER BY + 预处理语句”组合的 SQL 进行排查和改造。建议业务层统一规范:尽量避免派生表内使用 ORDER BY,将排序逻辑放到最外层。
THE END
更多技术实战与数据库深度解析,欢迎访问 云栈社区 交流探讨。