Yuchen You

Introduction 这篇文章系统性地提出并定义了云计算系统中的"灰色故障"(Gray Failure)问题, 认为它是当前云基础设施高可用性的一大障碍 gray failure component failures whose manifestations are fairly subtle and thus defy quick and definitive detection 系统某部分发生了轻微但持续的问题, 这些问题不易被传统故障检测机制发现, 但已经影响了应用层的性能和可用性;它不同于"失败即停" (fail-stop) 故障, 后者更易被监测和处理 Gray Failure的特征 Differential Observability (差异性可观测性) 对于 gray failure, 不同的组件对于同一个问题的感知是不同的, 尤其是一个实体受到消极影响但是其他实体感受不到这个影响 检测模块可能认为系统还是健康的, 但是实际上应用已经受损 Cloud Anomalies with Gray Failure High redund ...

Introduction 背景问题缺陷 Byzantine 模型假设组件可能出现任意错误甚至恶意行为, 过于复杂难以实施; Fail-stop 模型假设组件要么正常工作, 要么完全停止, 虽然简单易用, 但忽视了现代硬件中常见的"性能故障"; 本文特点 Fail-Stutter 模型: a realistic and yet tractable fault model that accounts for both absolute failure and a new range of performance failures common in modern components. 引入了一个更现实但可操作的故障模型, 结合 fail-stop 和"性能故障"(performance faults); 性能故障指的是组件虽然没完全失效, 但表现不稳定或变慢, 如磁盘速率下降, 缓存命中率降低等; 这种模型认为组件既可以"失败", 也可以"卡顿"或"表现异常"; 需要的模型性质 ( ...

Introduction 这篇论文 Fail-Slow at Scale: Evidence of Hardware Performance Faults in Large Production Systems 由Haryadi S. Gunawi等人发表在第16届USENIX文件与存储技术会议(FAST 2018)上 系统性地揭示了在大规模生产系统中"fail-slow"(故障-缓慢)硬件现象的存在与影响;这类故障指的是硬件仍在运行, 但其性能严重退化, 未达到应有的标准; 硬件类型包括: disk, SSD CPU memory network components 几十年来, 系统领域已经形成了如下成熟的硬件故障模型 (fault model) • Fail-stop(故障停止): 设备完全停止工作并发送明确的错误信号; • Fail-partial(部分失效): 设备部分失效, 部分功能仍然正常; • Fail-transient(瞬时性失效): 偶尔出现错误但会自行恢复; • Corruption fault(数据损坏) 和 Byzantine fau ...

Introduction metastable failures 亚稳态故障 a failure pattern in distributed systems Currently, metastable failures manifest themselves as black swan events(黑天鹅事故) they are outliers(异常事故) because nothing in the past points to their possibility have a severe impact much easier to explain in hindsight(事后) than to predict. Although instances of metastable failures can look different at the surface, deeper analysis shows that they can be understood within the same framework. By reviewing experiences fr ...

Introduction 这篇文章是基于 MetaStable Failures in Distributed Systems 的升级, in the wild 指的是不可控的实际世界 In this work, we study the prevalence(患病率) of such failures in the wild by scouring(冲刷) over publicly available incident reports from many organizations, ranging from hyperscalers to small companies In this paper, we make four contributions that extend the work of Bronson et al. and increase our understanding of metastable failures: A study of metastable failures in the wild that confirms metastable fai ...

When a failure occurs in production systems, the highest priority is to quickly mitigate(缓解) it. Failure Mitigation (FM) is done in a reactive and ad-hoc way, namely taking some fixed actions only after a severe symptom is observed. Propose a preventive and adaptive failure mitigation service, NARYA, that is integraed in a production cloud, Microsoft Azure’s compute platform Narya predicts imminent(迫在眉睫的) host failures based on multi-layer system signals then decides smart mitigation actions go ...

AI for IT Operations (AIOps) aims to automate complex operational tasks, such as fault localization and root cause analysis traditional: ddressing isolated operational tasks LLM and AI agents: enabling end-to-end and multitask automation Target: self-healing cloud systems, a paradigm we term AgentOps AIOpsLab a framework that not only deploys micro-service cloud environments, injects faults, generates workloads, and exports telemetry(遥测得的) data but also orchestrates these components and provid ...

数字取证流程(Digital Forensics Process) 四个核心阶段 识别(Identification) 确定存储关键数据的物理/数字对象(如计算机, 硬盘, 移动设备, 外接媒体等); 收集(Collection) 保护证据完整性, 建立链式监管(Chain of Custody), 记录证据哈希(Hash of Image)以验证来源; 分析(Analysis) 检查文件系统, 日志, 内存等, 恢复删除文件(Deleted Files)或残留数据(Slack Space); 报告(Reporting) 形成专家报告, 提供法律证据(Legal Evidence); 数据收集与保存(Data Collection & Preservation) 数据来源 computer other harddrive monitor keyboard and mouse media (dvd, cd, usb) printer digital forensics did not replace traditional (physical) ...

性质 基本职能 成帧 Framing: encapsulate network layer data 数据链路层会将来自网络层(第三层)的数据进行分组, 添加头部和尾部信息, 以形成一个个帧(Frame) 链路访问 Link Access: Medium access control (MAC) protocol defines when to transmit frames 在多点接入的介质 (例如共享式以太网, 无线网络等) 中, 多个设备可能需要同时传输数据 需要负责决定什么时候由哪一个设备向链路发送数据 可靠交付 Reliable Delivery: Primarily for mediums with high error rates (wireless) 某些网络环境中可能会受到干扰而导致帧丢失或损坏 数据链路层可以提供重传, 确认等机制, 确保帧能可靠地到达接收方 虽然TCP/IP协议栈中大多由传输层(TCP)来保证可靠性, 但一些链路层协议(如PPP)也会提供基本的可靠性功能 差错检测与纠正 Error detection and corre ...

文件系统抽象 File System Abstraction attribute Hardware Reality OS abstraction interface heterogeneous uniform storage objects a few (disks) many (files) name structure simple numeric name (id) rich name (symbolic, hierarchical, unified) access speed slow fast crash resilience unreliable reliable 硬件层到 os 的转变方式 Heterogeneity 原因: many i/o devices, each with its own idiosyncrasy Solution: abstraction build a common interface (Application 层和 File System 层之间设置统一接口 POSIX) write device d ...