说明1：由于发博文的时候总是提示有非法字符，所以全文无法在这里发布出来，如果感兴趣，可通过微信48907959索取文档。

说明2：本研究还在进行之中，后续会将基于一些开源平台和开源项目比如fwaf/Stratosphere IPS等的实践成果分享

人工智能技术应用到网络安全研究与实践

  

2017年3月

 

 

作者:                     王智民

修改历史

 

版本	日期	修订人	说明
1.0	2017-3-20	王智民	初稿

目录

 

 

1        引言... 1

1.1              大数据背景... 1

1.1.1            大数据基本概念.... 1

1.1.2            大数据应用市场.... 2

1.1.3            大数据带来的挑战.... 3

1.1.4            大数据挖掘与算法.... 5

2        斯坦福网络安全数据挖掘课程... 10

2.1              Syllabus教学大纲... 10

2.2              Session1:Data Mining for CyberSecurity. 11

2.2.1            网络安全常识.... 11

2.2.2            安全大数据挖掘需关注的几个方面.... 12

2.3              Session13:Data Mining for CyberSecurity. 13

2.4              Session2:Botnets 14

2.5              Session3: Behavioral Biometrics生物行为识别... 15

2.6              Session4: Web Security. 15

2.6.1            两类web安全检测方法.... 15

2.6.2            基于机器学习的web入侵检测.... 16

2.6.3            机器学习：基于统计学建模.... 17

2.6.4            机器学习：基于文本分析的机器学习模型.... 21

2.6.5            机器学习：基于单分类的机器学习模型.... 24

2.7              Session12:基于payload异常的NIDS. 27

2.8              Session10:多分类器的online NIDS. 29

3        Data-driven IDS by using machine learning. 32

3.1              Abstract 33

3.2              IDS发展现状... 33

3.3              IDS两条技术路线... 34

3.4              Computational Intelligence. 36

3.5              基准数据集DataSets 37

3.6              性能评估... 40

3.7              Artificial Neural Networks 42

3.7.1            Supervised Learning.. 42

3.7.2            Unsupervised Learning.. 47

3.7.3            Summary.. 51

3.8              Fuzzy Sets（模糊集）... 54

3.8.1            Fuzzy Misuse Detection.. 55

3.8.2            Fuzzy Anomaly Detection.. 56

3.8.3            Summary.. 57

3.9              Evolutionary Compution(EC)（进化计算）... 58

3.9.1            The Roles of EC in IDS.. 59

3.9.2            Niching（局部优化）.... 67

3.9.3            Evolutionary Operators（进化算子）.... 68

3.9.4            Fitness Function（适应性函数）.... 68

3.9.5            Summary.. 70

3.10           Artificial Immune Systems（AIS 人工免疫系统）... 71

3.10.1         A Brief Overview of Human Immune System.. 72

3.10.2         Artificial Immune System Models for Intrusion Detection.. 75

3.10.3         A few aspects of AIS.. 78

3.10.4                 Summary.. 79

3.11           Swarm Intelligence (SI，群体智能). 81

3.11.1                 Swarm Intelligence Overview.. 81

3.11.2                 Ant Colony Optimization.. 82

3.11.3                 Particle Swarm Optimization.. 86

3.11.4                 Summary.. 88

3.12           Soft Computing (软计算). 88

3.12.1                 Artificial Neural Networks and Fuzzy Systems.. 89

3.12.2                 Evolutionary Computation and Fuzzy Systems.. 92

3.12.3                 Ensemble Approaches （集成方法）.... 95

3.12.4                 Summary.. 96

3.13           Summary. 97

3.13.1                 Fuzzy sets、ANNs、EC、AISs、SI、SC.. 97

3.13.2                 The Trend of the research in CI.. 98

3.13.3                 The Chanllenges of CI applied in IDS.. 99

3.13.4                 Conclusion.. 102

4        Network IDS by using MLFF-BP. 102

5        IDS by using 18 MLFF algorithms 103

6        IDS by using RBF. 103

7        IDS by using RNN.. 103

8        IDS by using CMAC(belong to RNN) 104

9        IDS by using Multi-layer SOM... 104

10               Visualization analysis of events by using SOMs 105

11               The detection of DoS attacks by using ESOM... 105

12               IDS by using Fuzzy ART.. 106

13               Intrusion suggestion by using fuzzy inference engine. 106

14               Automatic Model Structure Design by using ENN.. 107

15               IDS by using XCS. 107

16               IDS by using LGP. 108

17               Distributed EC models 109

18               IDS by using AIS. 110

19               IDS by using neuro-fuzzy system.. 111

20               Ensemble Approaches （集成方法）... 112

21               adaptation in IDSs. 114

22               需求分析... 114

22.1           应用场景... 114

22.2           价值与技术指标... 115

22.3           关键需求... 116

22.3.1         威胁感知：APT攻击检测.... 116

22.3.2         威胁感知：AET攻击检测.... 116

22.3.3         威胁感知：0 Day攻击检测.... 119

22.3.4         威胁感知：未知木马病毒.... 119

23               实现思路... 119

23.1           方法论... 119

23.1.1         The anomaly IDS.. 120

23.1.2         Training a big data machine to defend.. 121

23.2           “异常”感知建模方法论... 121

23.2.1         Extractor. 122

23.2.2         Trainer. 125

23.2.3         Detector. 129

23.2.4         reTrainer. 129

23.3           “威胁感知”与“威胁预测”建模方法论... 130

23.3.1         异常是否是威胁.... 130

23.3.2         异常是已知或未知威胁（威胁预测）.... 131

23.3.3         异常是哪种威胁.... 133

23.3.4         异常威胁是否成功发生.... 133

23.4           威胁感知系统... 134

23.4.1         DCOS.. 134

23.4.2         DaaS.. 138

23.4.3         如何实现态势感知.... 142

24               工作量初步分析... 143

25               DEMO设计... 144

25.1           移植软件... 144

25.2           移植说明... 144

25.3           性能评估... 144

26               用户界面设计... 144

27               附录... 144

27.1         常见分词算法... 144

27.1.1         常见分词算法.... 144

27.1.2         中文常见分词算法.... 145

27.1.3         N-Gram... 147

27.2         向量相似度计算... 148

27.2.1         马氏距离.... 149

27.2.2         余弦距离.... 149

27.2.3         杰卡德距离.... 150

27.3         Kill Chain. 150

27.3.1         Cyber Kill Chain.. 150

27.3.2         F2T2EA模型.... 151

27.3.3         反杀伤链的基础.... 153

27.4         深度学习... 155

27.5         参考资料... 155