OpenStack Nova VM live migration flow

Original post is:  OpenStack Nova VM migration flow 

nova.api.openstack.compute.contrib.admin_actions._migrate_live()

nova.compute.api.live_migrate()

       - update instance state to MIGRATING state
       - call into scheduler to live migrate (scheduler hint will be set to the host select (which may be none))

nova.scheduler.manager.live_migration()

nova.scheduler.manager._schedule_live_migration()

nova.conductor.tasks.live_migrate.LiveMigrationTask.execute()

        - check that the instance is running

        - check that the instance's host is up

        - if destination host provided, check that it..

              1. is different than the instance's host

              2. is up

              3. has enough memory

              4. is compatible with the instance's host (i.e., hypervisor type and version)

              5. passes live migration checks (call using amqp rpc into nova manager check_can_live_migration_destination)

          - else destination host not provided, find a candidate host and check that it 

              1. is compatible with the instance's host (i.e., hypervisor type and version)

              2. passes live migration checks 

           - call using amqp rpc into nova manager live_migration

             Note: Migration data is initially set by check_can_live_migrate_destionation and can be used for implementation specific parameters from this point.

nova.compute.manager.check_can_live_migrate_destination()

        - driver.check_can_live_migrate_destination()

        - call using amqp rpc into nova manager check_can_live_migrate_source

        - driver.check_ca_live_migrate_destionation_cleanup()

nova.compute.manager.check_can_live_migrate_source()

        - determine if the instance is volume backed and add result to the migration data

        - driver.check_can_live_migrate_source()

nova.compute.manager.live_migration()

        - if block migration request then driver.get_instace_disk_info()

        - call using amqp rpc into nova manager pre_live_migration

              - Error handler: _rollback_live_migraiton

        - driver.live_migration()

nova.compute.manager.pre_live_migration()

        - get the block device information for the instance

        - get the network information for the instance

        - driver.pre_live_migration()

        - setup networks on destination host by calling the network API setup_networks_on_host

        - driver.ensure_filtering_rules_for_instance()

nova.compute.manager._rollback_live_migration()

nova.compute.manager._post_live_migration()

        - driver.get_volume_connector()

        - for each instance volume connection call the volume API terminate_connection

        - driver.unfilter_instance()

        - call into conductor to network_migrate_instance_start which will eventually call the network API migrate_instace_start

        - call using amqp rpc into nova manager post_live_migration_at_destionation

        - if block migration or not shared storage driver.destory()

        - else driver.unplug_vifs()

        - tear down networks on source hosts by calling the network API setup-networks_on_host

nova.compute.manager.post_live_migration_at_destination()

nova.compute.manager.rollback_live_migration_at_destination()

nova.compute.manager.remove_volume_connection()

nova.compute.manager._detach_volume()

IP Routing

[1] introduces IP Subset, IP Routing

Reference
[1] Hyper-V and Networking -- Part 3: IP Routing 

Google Cloud VM Live Migration

Introduction

Heartbleed bug was revealed on April 7th, 2014. On that day, most cloud customers were impacted because patching the system requires VM reboot. At Google, none of the customers were impacted due to the transparent maintenance functionality introduced in Google Compute Engine in Dec 2013.

Through a combination of datacenter topology innovations and live migration technology, they can move their customer running VMs out of the way of planned hardware and software maintenance events, so they keep the infrastructure protected and reliable -- without customers' VMs, applications or workloads noticing that anything happened.

VM Migration Procedure 

The high-level steps are illustrated in the following

• The process begins with a notification that VMs need to be evicted from their current host machine. The notification might start with a file change (e.g., a release engineer indicating that a new BIOS is available), Hardware Operations scheduling maintenance, an automatic signal from an impending hardware failure etc. 
• Once a VM is selected for migration, we provide a notification to the guest that a migration is imminent. After a waiting period, a target host is selected and the host is asked to set up a new, empty "target" VM to receive the migrating "source" VM. Authentication is used to establish a connection between the source and target. 
• There are three stages involved in the VM's migration
• During pre-migration brownout, the VM is still executing on the source, while most state is sent from the source to the target. For instance, we copy all the guest memory to the target, while tracking the pages that have been re-dirtied on the source. The time spent in pre-migration brownout is a function of the size of the guest memory and the rate at which pages are being dirtied.
• During blackout, which is a very brief moment when the VM is not running anywhere, it is paused, and all the remaining state required to being running the VM on the target is sent. 
• During post-migration brownout, the VM executes on the target. The source VM is present, and may be providing supporting functionality for the target. For instance, until the network fabric has caught up the new location of the VM, and source VM provides forwarding services for packets to and from the target VM. 
• Finally, the migration is completed, and the system deletes the source VM.

Reference
[1] Google Compute Engine Users Live Migration Technology to service infrastructure without application downtime 

Hyper-V Live VM Migration Procedure

1. Live migration setup occurs. 

During the live migration setup stage, the source server creates a connection with the destination server. This connection transfers the virtual machine configuration data to the destination server. A skeleton virtual machine is set up on the destination server and memory is allocated to the destination virtual machine.


2. Memory pages are transferred from the source node to the destination node

• In the second stage of a live migration, the memory assigned to the migrating virtual machine is copied over the network to the destination server. This memory is referred to as the "working set" of the migrating virtual machine. A page of memory is 4 KB.
• In addition to copying the working set to the destination server, Hyper-V monitors the pages in the working set on the source server. As memory pages are modified in the source server, they are tracked and marked as being modified. 
• During this phase of the migration, the migrating virtual machine continues to run. Hyper-V iterates the memory copy process several times, with each iteration requiring a small number of modified pages to be copied. 

3. Modified pages are transferred.

• This third phase of the migration is a memory copy process that duplicates the remaining modified memory pages to the destination server. The source server transfers the CPU and device state of the virtual machine to the destination server.
• During this stage, the network bandwidth available between the source and destination servers is critical to the speed of the live migration. Using a 1 Gigabit Ethernet or faster is important. The faster the source server transfers the modified pages from the migrating virtual machines working set, the more quickly the live migration is completed. 
• The number of pages transferred in this stage is determined by how actively the virtual machine accesses and modifies the memory pages. The more modified pages there are, the longer it takes to transfer all pages to the destination server. 

4. The storage handle is moved from the source server to the destination server

• During the fourth stage of a live migration, control of the storage such as any virtual hard disk files or physical storage attached through a virtual Fibre Channel adapter, is transferred to the destination server. 

5. The virtual machine is brought online on the destination server.

• In the fifth stage of a live migration, the destination server now has the up-to-date working set as well as access to any storage used by the virtual machine. At this point, the virtual machine is resumed. 

6. Network cleanup occurs. 

• In the final stage of a live migration, the migrated virtual machine is running on the destination server. At this point, a message is sent to the network switch. This message causes the network switch to obtain the new MAC addresses of the migrated virtual machine so that network traffic to and from the virtual machine can use the correct switch port. 

Reference
[1] Virtual Machine Live Migration Overview

VM Live Migration's Impacts on the Running Applications

1. Will the IP address change after migration?

Both types of live migration exist, including changing and not changing IP address [5].

• Based on Google cloud [1], it can migrate clients' VM without affect the customers. That means the IP address of a VM would not be changed in this case.
• To retain the same IP address, hyper-V requires the source and destination hosts to be within the same subnet. I think Google cloud may not have this requirement.
• I think the virtual network [4] would be able to remove the restrictions on the locations of the destination hosts. "Hyper-V Network Virtualization decouples virtual networks for customer virtual machines from the physical network infrastructure." 

2. Will the migration interrupt the Internet service?

This depends on the implementation. The answer is different regarding different implementation.

• According to google cloud [1], there will be no service interruptions.
• During post-migration brownout, the VM executes on the target. The source VM is present, and may be providing supporting functionality for the target. For instance, until the network fabric has caught up the new location of the VM, and source VM provides forwarding services for packets to and from the target VM. 
• According to hyper-V [2]
• the migration is not downtime-free, the interruption is almost immeasurably brief. Usually the longest delay is the network layer while the virtual machine's MAC address is registered on the new physical switch port and its new location is propagated throughout the network. 
• According to [3], in order to use live migration the VM needs to keep the same IP address across date centers in order to achieve the goal of continuous access from clients to the virtual machine during and after the migration. 

3. How the network is migrated?

The most challenging issue in VM migration is to keep the network working.

In LAN, different hypervisors using different strategies.

• Xen
• It uses ARP to bind the IP address to the new host. 
• The VM sends ARP signal, broadcast that the IP address is moved to a new host.  But this may not be allowed for security reasons. 
• VMware
• VMotion uses VNIC to ensure the network connection. 
• The VNIC will be migrated with VM as well. Every VNIC has a unique MAC address in LAN and is connected to one or multiple NIC. 
• Since VNIC has a MAC address that is irrelevant to the physical network address, the network will be continued as normal using VM live migration. 
• Note due to the restrictions of Ethernet, the source and destination hosts have to be in the same subnet. 

In WAN

• The VM will be given a new IP address in the destination host. In order to ensure the network connection, we can use IP tunnel with combination of dynamic DNS, i.e., we can build a IP tunnel between the source IP and destination IP address, and use it to forward the packets from source host to destination host. Once migration is done, VM can response to the new network. It means the DNS is updated, and the network connection will refer to the new IP address. 

Reference
[1] Google cloud VM live migration
[2] Hyper-V live migration
[3] Live Migration -- Implementation considerations
[4] Hyper-V 网络虚拟化概述 
[5] 虚拟机迁移研究

迈阿密美食 [FL 33139] The Front Porch Cafe

店名：The Front Porch Cafe
Address: 1458 Ocean Dr, Miami Beach, FL 33139

1. Shrimp fritters
有点像海蛎煎，还不错
推荐指数：＊＊＊＊

2. Crab cake Benny

一般般，蟹肉的味道有点腥，不知道是不是不太新鲜
推荐指数：＊＊

迈阿密美食 [FL 33139] Chalan On the Beach

店名：Chalan On the Beach
地址：1580 Washington Ave, Miami Beach, FL 33139
Yelp: https://www.yelp.com/biz/chalan-on-the-beach-miami-beach

本来想去Joe‘s Crab Stone的，但是走路有点远。这家餐馆刚好在我们住的宾馆旁边，而且Yelp上有600多个评价，平均四星，看起来还不错，我们于是就去了。

这是一家秘鲁餐馆，秘鲁的口味其实跟中餐有点相近，店里的服务员看起来都是秘鲁人。我们5点半去的，店里十几张桌子基本都满了，等我们坐下来点完餐，门口就已经开始排起了小长队了。

我们点了一个酸橘汁腌鱼 (Mixed Ceviche al Chalan)，一个打卤红鱼 (Poached Mixed Snapper)，还有一个海鲜炒饭 (Arroz Chaufa de Mariscos)。

1. 酸橘汁腌鱼 (Mixed Ceviche al Chalan)

好吧，这个吃了一点才拍的。这盘里的海鲜有鱼肉，虾，鱿鱼，海虹等，全都是半生的，用酸橘汁腌过了，所以吃起来并不生。但是对我来说，有点酸。朋友很喜欢吃酸，他觉得吃起来有生鱼片的感觉，很喜欢。

推荐指数：＊＊＊

2. 打卤红鱼 (Poached Mixed Snapper)

这条鱼看起来应该有两磅，这家店用量实在太足了，包括等一下要介绍的炒饭，也是非常大盘。这道菜的主角是红绸鱼 (snapper)，应该是稍微煎了之后，加番茄之后勾芡打卤的，有妈妈做的菜的味道，赞！红绸鱼是一种比较难烧的鱼，因为海味有点重，这道菜烧的还算成功，基本掩盖了海味。
推荐指数：＊＊＊

3. 海鲜炒饭 (Arroz Chaufa de Mariscos)

这道菜端出来的时候闻着实在太香了。算是目前在美国吃到的最好吃的炒饭。不同于泰国的湿答答的炒饭，秘鲁的这道炒饭其实有点像闽南的油饭。油饭一般不是用煮熟的米饭炒的，而是用浸泡过的米直接炒熟，所以特别香，并且很Q。大爱这个炒饭。里面用料也很丰富，有虾，鱿鱼等。

推荐指数：＊＊＊＊

[VA 22180] Lotus Garden

1. Beef with broccoli  (Black pepper flavor)

Rate: ☆☆☆☆
Comments: The beef is tender and juicy. The broccoli is fresh and crunchy. I like to have it with the black pepper sauce.

DEF CON 22 in 2014

Hacking Conference Presentation By Weston Hecker Burner Phone DDOS 2 dollars a day 70
Date: 2016-12-18
Relatedness: ＊
Summary: This talk discussed abut phone DDoS attacks.

Blackhat 2016

Investigating DDOS Architecture Actors and Attribution
Date: 2016-12-16
Summary:
The data collection is done through DDoS honeypots in which nodes that mimic UDP services that commonly abused to produce DDoS attacks.
They collect the timestamp, source, destination IPs and ports, attack type.

Blackhat 2013

Lessons from Surviving a 300Gbps Denial of Service Attack
Speaker: Matthew Prince, CEO of Cloudflare 
Date: 2016-12-16
Summary: This talk presents how the attackers use DNS open resolver to conduct a DDoS attack of a size larger than 300Gpbs on Spamhaus. The attack is defeated by 
A similar but less technical talk was given by the speaker in Defcon at the same year. 


Million Browser Botnet
Date: 2016-12-18
Summary:  This talk presents how to hack the browser to crack websites.  The attack is conducted by injecting a javascript to the browser of a client. The secret is to use for-loop to request for resources from the victim website using ftp protocol rather than http to break the ceiling limit in the browser. 

[安全论文阅读笔记］Survey on Web Spam Detection: Principles and Algorithms

Date: 2016-12-16

这篇文章发表在SIGKDD Explorations 2013，作者是来自UIUC的 Nikita Spirin 和 Jiawei Han


这篇文章总结了web spam 检测的主要算法分类。主要针对的spam是搜索引擎spam，而非social media spam。

Spam的分类以及技术
1. Content Spam
因为搜索引擎对网页的内容的排名采用TFIDF模型。因此这些spam会在内容里加入一些popular的词，来提高rank。
2. Link Spam
搜索引擎采用page rank来评估网页排名，因此这些spam会通过提高incoming link数量质量来提高目标页面的排名，他们也会通过购买被抛弃的域名来获取有一定reputation的域名。
3. Cloking and Redirection
对于同一个页面，Spammers会根据不同的clients来展示不同的内容。因此对于搜索引擎爬虫，他们可以放比较有利于rank的内容，而对于普通用户，他们可以展示广告内容。

已有的检测方法大概可以分为三类
1. content-based methods
这些方法主要通过分析word counts, language models, HTML页面的结构，clocking score
2. link-based methods
这些方法主要通过分析link构成的图结构的特性，label propagation，Link pruning and reweighting, graph regularization (建议如果有意通过link结构来做检测的同学可以细读具体内容）
3. data-based methods, e.g., user behavior, clicks, HTTP sessions.
这些方法通过Markov model来分析用户行为等

Defcon 21 in 2013

Evil DoS Attacks and Strong Defenses
Relatedness: ****
Date: 2016-12-15
Summary: This talk presents a DDoS attack based on DNS resolver. It turns out that 0.1% of the DNS resolvers could be able to generate more than 300Gbps attack traffic.

[VA 22030] Hmart Food

Address: 10780 Fairfax Blvd, Fairfax, VA 22030

1. Seafood Pancake

Rate: ☆☆☆☆
Price: $11.99
Comments: The seafood pancake contains a lot of squid and green onions. It is crispy for the outside and very soft inside.


 

DEFCON 24 in 2016

Karyn Benson - Examining the Internet's pollution
Relatedness: ****
Date: 2016-12-08
Summary: 
Analyzing the trash packets on the Internet to find out the attack activity or software bugs. For example, figure out when the attackers are scanning the internet of thing devices. 

Jay Beale and Larry Pesce - Phishing without Failure and Frustration
Relatedness: *
Summary:
It discusses how to conduct phishing test in an enterprise.

Shane Steiger Are You Playing with a Full Deck
Relatedness: *****
Date: 2016-12-08
Summary:
It discusses how the defenders can act to disrupt the attack life-cycle of attackers. This is exactly what the Moving Target Defense (MTD) is. It is important to figure out the attack patterns in order to decide how the defenders can play effectively in breaking the attacks.effectively in breaking the attacks.effectively.effectively


Luke Young - Attacking Network Infrastructure to Generate a 4 Tbs DDoS
Relatedness: ****
Date: 2016-12-09
Summary:
This talk exploits the devices with large uplinks to gain root privileges. Then they can launch huge DDoS attacks by sending HTTP requests to the victims. Note that rather than DNS amplification attack which can be easily blocked by firewall, attack traffics with normal http requests are hard to be blocked.

Lucas Lundgren, Neal Hindocha - Light Weight Protocol: Critical Implications
Relatedness: ***
Date: 2016-12-09
Summary:
This talk discusses the MQTT protocol and how it can be exploited for critical information. In EQTT, each client is subscribed to several queues for different functions. Each client can listen to what other clients are doing. The attackers can easily change the subscriptions of the devices and to control them. For example, the attackers can  issue a software update to the victim devices.
Take away: Please do not expose MTQQ on the internet.

Willa Cassandra Riggins, abyssknight - Esoteric Exfiltration
Relatedness: ****
Date: 2016-12-09
Summary:
This talk discusses how to avoid leaking information via convert channel. It turns out blacklist doesn't work due to business requirement and the criticalness of context.


Kai Zhong - 411: A framework for managing security alerts
Relatedness: *
Date: 2016-12-09
Status: 20 minutes
Summary: This talk discusses how they enrich the elastic search framework to have the function to create and notify the alerts.

Joshua Drake, Steve Christey Coley - Vulnerabilities 101
Relatedness: **
Date: 2016-12-12
Summary: This talk gives advice and guidance for the newbies on how to do the vulnerability research.


Weston Hecker - Hacking Hotel Keys and Point of Sale Systems
Relatedness: **
Interestingness: *****
Date: 2016-12-12
Summary: This talk introduces how they hack the hotel keys.


Zack Fasel, Erin Jacobs - Attacks Against Top Consumer Products
Relatedness: **
Date: 2016-12-13
Summary: This talk introduces the security issues in blue tooth, cameras.

Dr Phil - Mouse Jiggler: Offense and Defense
Relatedness: **
Date: 2016-12-13
Summary: This talk introduces how to defend against mouse jiggler which the attackers use  to keep your computer in active mode.

Greg Norcie - My Usability Goes to 11 A Hackers Guide to Use
Relatedness: **
Date: 2016-12-13
Summary: This talk discusses the difficulty of usable security. In designing the usable software, cognitive walkthrough is a good way to go.  It is important to be in the customers' shoes and think about what customers will react on the functionalities. A user case study would be helpful in evaluating the design.

Grant Bugher - Bypassing Captive Portals and Limited Networks
Relatedness: **
Date: 2016-12-14
Summary: This talk discusses how to bypass to reach the restricted networks. These networks are usually set by the owners and require payment for access.

Delta Zero, KingPhish3r - Weaponizing Data Science for Social Engineering
Relatedness: ****
Date: 2016-12-14
Summary: This talk presents a recurrent neural network that learns to tweet phishing posts targeting specific users. The model is trained using spear phishing pen-testing data. In order to make a click-through more likely, it is dynamically seeded with topics extracted from timeline posts of both the target and the users they retweet or follow. The performance is measured using click-through rate.


Clarence Chio - Machine Duping 101: Pwning Deep Learning Systems
Relatedness: ***
Date: 2016-12-14
Summary: This talk presents attacks on the deep learning model by feeding it with adversarial inputs.
It is important to evaluate the robustness of machine learning system to see what will it turns out when someone tries to bypass it.
It also mentioned that in Facebook spam fighting, the spammers try to alter the spam images to bypass detection. Using fuzzy matching would not be enough to cluster similar images together. However, deep learning is a good way to go. In specific, we can train the images using neural networks and compare the 2nd layers to measure the similarities which allow us to neglect some trivial details on the images.

Alex Chapman and Paul Stone - Toxic Proxies: Bypassing HTTPS
Relatedness: **
Date: 2016-12-14
Summary: This talk presents how the attackers can use new techniques to bypass encryption to monitor your search history and take over your online accounts.
The attackers can extract the information from the users' searching process. Besides, the CDNs' only serve contents without encryption, though the URLs are often complicated and are difficult to be guessed, whoever get those URLs would be able to access the resources.

Salvador Mendoza - Samsung Pay: Tokenized Numbers, Flaws and Issues
Relatedness: **
Date: 2016-12-14
Summary: This talk presents the attack on Samsung pay.
Background knowledge: when you pay using your card, a token number will be generated which contains service code, transaction id, and a random number.
Flaws: If Samsung generates a token which is not used, it has a blank expiration date.
Attach approach:
1. Social engineering to get the token
2. Use the token to do purchase

告一段落

这周刚投完paper，一下子好茫然。开始寻找下一篇paper的idea，之前有一个一直很想研究的问题，但是还没有想出解决方案。同时心里又很抗拒，觉得不应该这样找idea，而是应该知道重要的问题是什么，去解决重要的问题才有意义。理论研究固然有意义，但是这并不是我所追求，也不是我所能匹及的。读这个博士让我明白的最深的一点，就是要知道自己能做什么，不能做什么。能力之外的事情，花再多力气也是白费。

索性听了一天的歌，自从实习回来就没有怎么听了。听了杰伦的新歌，一下子回忆起暑假在F实习的日子，真美好。好幸运地遇到了很棒的同事，好棒的风景。马上就要毕业了，好期待毕业以后的日子。同时也希望自己能够在这剩下的时间里，多学习一些知识和待人处事，在正式进入职场之前。