CloudForms in AWS part 3

This part of the blog series is probably the most interesting one: when you launch a SmartState analysis you will see the following in you evm log files.

[----] I, [2018-03-11T16:12:10.941498 #3004:4f7130]  INFO -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Vm#raw_scan) NAME [Scan from Vm vmtosmartstate] SCAN [nil] [NilClass]
[----] I, [2018-03-11T16:12:11.068516 #3004:4f7130]  INFO -- : Job created: guid: [b0c7c318-2844-436c-bc44-21a29dfa3cce], userid: [admin], name: [Scan from Vm vmtosmartstate], target class: [VmOrTemplate], target id: [99000000000166], process type: [VmScan], server id: [], zone: [default]
[----] I, [2018-03-11T16:12:15.868847 #3014:4f7130]  INFO -- : MIQ(MiqQueue.put) Message id: [99000000000427],  id: [], Zone: [default], Role: [smartstate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout: [600], Priority: [20], State: [ready], Deliver On: [], Data: [], Args: []
[----] I, [2018-03-11T16:12:21.133242 #3004:4f7130]  INFO -- : MIQ(MiqPriorityWorker::Runner#get_message_via_drb) Message id: [99000000000427], MiqWorker id: [99000000000004], Zone: [default], Role: [smartstate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout: [600], Priority: [20], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [5.266320951] seconds
[----] I, [2018-03-11T16:13:05.374585 #3424:4f7130]  INFO -- : MIQ(ManageIQ::Providers::Amazon::AgentCoordinator#find_or_create_keypair) KeyPair smartstate-2ae1670a-2fcc-4bbb-abcc-3fa962895096 will be created!
[----] I, [2018-03-11T16:13:05.877668 #3424:4f7130]  INFO -- : MIQ(ManageIQ::Providers::Amazon::AgentCoordinator#deploy_agent) Smartstate agent will be deployed in vpc: [vpc-076ad862], zone: [us-east-1a] subnet: [subnet-3cc6154b]
[----] I, [2018-03-11T16:13:11.145451 #3424:4f7130]  INFO -- : MIQ(ManageIQ::Providers::Amazon::AgentCoordinator#get_agent_image_id) AMI Image: RHEL-Atomic_7.4_HVM_GA-20180104-x86_64-1-Access2-GP2 [ami-d97120a3] is used to launch smartstate agent.
[----] I, [2018-03-11T16:15:55.976224 #2716:4f7130]  INFO -- : MiqServer: local=Y, master=Y, status= started, id=99000000000001, pid=02716, guid=c5b77187-7fc9-426e-b3cb-6f0ce7860c8e, name=EVM, zone=default, hostname=ip-172-31-44-62.ec2.internal, ipaddress=172.31.44.62, version=5.9.0.22, build=20180221205805_f93a675, active roles=automate:database_operations:datab

There are a few interesting parts here. What is the AgentCoordinator? The code can be found here:

/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-9620e26f4381/app/models/manageiq/providers/amazon/agent_coordinator.rb

It’s out of scope to discuss the entire ruby code, but it will tell you, how we do it.

So when you look at the evm log, you see that it says find_or_create_keypair KeyPair smartstate-2ae1670a-2fcc-4bbb-abcc-3fa962895096

This means that the key used for the ssh login to the RHEL-Atomic_7.4_HVM_GA-20180104-x86_64-1-Access2-GP2 (ami-d97120a3) is generated on the fly.

Let’s explain how you can get that key so you can SSH into the atomic instance and see whats going on.

vmdb
bin/rails c
$evm = MiqAeMethodService::MiqAeService.new(MiqAeEngine::MiqAeWorkspaceRuntime.new)
$evm.vmdb(:ManageIQ_Providers_CloudManager_AuthKeyPair).all.first
$ssa=$evm.vmdb(:ManageIQ_Providers_CloudManager_AuthKeyPair).find_by_resource_id(99000000000001)
$ssa[:auth_key]
irb(main):007:0> $ssa[:auth_key]
=>
"v2:{UOjuc9OSrBm2AcOU5HmnLi1+7XVumlIlfX2SO7fbe1FyAwt9RGqEqyJD5VSRD4QG8RhoFt0yBe0fk5LCq0RGxgdiuIClv50mePsgTyx7f4cOCT94MKM5s3NkfeEq3BdNCzL+CWji+WSXZZn7zE9l/4ezFW7eesfJzsW6npMlH/SNUBdAgiVZzbsKrHZuP4n5agPBYSX+APf+1KvAOGLYaMREGfw4E7OytKGIfOAD/dSKXRx/bhEadCOvnf6sG6XKYGzi82aa+AMzgexszx/93Ce+Pp6fbODxxXJmoDNyzJEoKyuavDiWBEK2SHMV5t2G8U/pb0mjYvavNCl86gtqOGJQ26bZNuUkJlRoSyErOMyrOOhreq0/2RMJvI4Uy6ekCBggyPFS3iPDduQTNM1tPhZPVAW2H+RB8UKZOtf+Oo+eqfkGvQYlnsJt2a4XET+alDxpDT6TEVRCrMpAUUiJt2B1AG3uQ8wL3RDRMvj54uR/uRsvlFMkO9HYrlcKKrQ4myBa3UbVETNcwT844sYwXEP8thmIVMFwIp03oY5ZQOXoPikD1wJJSHQON2bmIbh/xZ3e2ggPwuLkc9ROSBUxkRK4zXhdezwjI3UfP3LVwujHJVmsIBpUDR7oXw2CUwr0be72MFKoWpmvbXf7E8KZaJz1FNZUoOGpMPz5j3eDetlCEwgjJF2UVElT8+2dLosxN2NeLpFY6vpZXm3erOG6WVLM/v5kiX3V2S2JHaUOZdVRrHvk9yoSF5YhQgn72LlpG+Y4DhT3QOf9HMwzg4A4D6EgkRmZfWdr8ABMMoEFonIkPX08ZwJnr3iNAtam8HCvnGGcs49eTPrAzNG/9zjn6ViS4tvGk9EaYYUuHcWgmOySl2WsezmEdE9oHEC7NkX/MbPdSlr1HxfNAkphjdsvmwSABLhOzFeE25zrftTW3Hf1ui7+tsJztDDio00qtXcT+NY8ySupb2siZcywFZFf3WgRzbOX9yjZv4QG8xPzQ80v4+EbKUuDJJkAHd85kAxZYryHQFi/dQO8O39lhJWog6T58gCteMBnYCvLqnYxr40qwXJ6lp44yoi+EBSH7TpLmsY/AM+YvQH1rqUTbi/1o9PvzwkY3C52WpMFeRlh9XDdg/hWWwcHCJc07dNP2kOCtsR1Te5dhTejsJ9ul2MzAGdRnPwuUz9RrMj0Xj/5X8we+4RPgF+/Z3EpjMOFFzrBbykj003n/pRJE0651a5FbdUkO9QeE3jjj4QGaWAb+3dTbVH3OGVIQ0Ebm31po6G1+9SPyfUSVOmmlHAK4XZybPGNQzsW7uorOAtet9++GAR11mBvuO5g9GSOBGw6EEqN76X1CJ3WFaKwfpoQzDXDC3hOMjahB9cZaWu7zXebJPlKUiZPuIfbB93WCOVjS5TU01McrBsjt0rgsSAgPwVzEAyGVInx1knlQ34SiPvgOZYZuvlr9ln2dAdzfmecSR4na4LwjXJd4B3LhBCDofEtTu21ArdxLu5nHjYg1JiaLm1jL2MJeElFkMtJjH8dKFO/e7rbF6Gtq5aMMP1GNFle2EqW53jJsOF7FLZAa5rTNP2qGKBJyWw6dmOPnszIWrMHrSLEJPTpIr4F0V27rwyknCijIDZG/XENDMRtJHLfpjqajVfnlMDp0+1gHX7GhepBN+1b7l9LaRGavJ9iyyBaiw6kHgp7mseRvceDbTkkOUQHEcNL1CYzxt2QHap7noKtfYa0K3GPCa41pXHwaMrG6jeM3Oi0KaoXTBmlfqeDKtqJ1YOzqQ5+qoVd/Um8nWq/HWvjhtoEK7K4zXaEsaVBlPeqrDqUY+sppbvKENXRRfotim9QG+b5ujINRCLl95HSNIQ5J6zDjYPqDToQuRF72L2SdoDHNSotcSzMDf3m91Idoec5oV8M+jPwqjm7BgzCK7RxW8aAAgh0Ht6j+QHYr8Isg00sCDFBrMG9P6nxbZsipeIklTjyCUWFA8rFPBMDfETFhNCkcpZ7TQqZsQtzBgc3gbgIemXw2ORSKvO6RB++b3s81Xj4OPgwG6cH/rvNGvf1ZsE3ONQFXCoVx0ZCXhdiLcvqovSrMctJKFzZdAVTSLKBARc7nc2JQx3Tam+29nn3cTvHH++zx52roTmpO2eI3gWkqpM8TEYAWQVLK8KaK33WCbu7Q6/iCOEKuHPz9v2L1oJBt/7zpU8wxbfafcxOzYuUTbw74EykGzUVaDUuSCbh14oZO9Z1vuBxWO8U}"

In my case its the first key as I did not create another one yet.

Then go ahead and decrypt that key.

MiqPassword.decrypt("v2:{UOjuc9OSrBm2AcOU5HmnLi1+7XVumlIlfX2SO7fbe1FyAwt9RGqEqyJD5VSRD4QG8RhoFt0yBe0fk5LCq0RGxgdiuIClv50mePsgTyx7f4cOCT94MKM5s3NkfeEq3BdNCzL+CWji+WSXZZn7zE9l/4ezFW7eesfJzsW6npMlH/SNUBdAgiVZzbsKrHZuP4n5agPBYSX+APf+1KvAOGLYaMREGfw4E7OytKGIfOAD/dSKXRx/bhEadCOvnf6sG6XKYGzi82aa+AMzgexszx/93Ce+Pp6fbODxxXJmoDNyzJEoKyuavDiWBEK2SHMV5t2G8U/pb0mjYvavNCl86gtqOGJQ26bZNuUkJlRoSyErOMyrOOhreq0/2RMJvI4Uy6ekCBggyPFS3iPDduQTNM1tPhZPVAW2H+RB8UKZOtf+Oo+eqfkGvQYlnsJt2a4XET+alDxpDT6TEVRCrMpAUUiJt2B1AG3uQ8wL3RDRMvj54uR/uRsvlFMkO9HYrlcKKrQ4myBa3UbVETNcwT844sYwXEP8thmIVMFwIp03oY5ZQOXoPikD1wJJSHQON2bmIbh/xZ3e2ggPwuLkc9ROSBUxkRK4zXhdezwjI3UfP3LVwujHJVmsIBpUDR7oXw2CUwr0be72MFKoWpmvbXf7E8KZaJz1FNZUoOGpMPz5j3eDetlCEwgjJF2UVElT8+2dLosxN2NeLpFY6vpZXm3erOG6WVLM/v5kiX3V2S2JHaUOZdVRrHvk9yoSF5YhQgn72LlpG+Y4DhT3QOf9HMwzg4A4D6EgkRmZfWdr8ABMMoEFonIkPX08ZwJnr3iNAtam8HCvnGGcs49eTPrAzNG/9zjn6ViS4tvGk9EaYYUuHcWgmOySl2WsezmEdE9oHEC7NkX/MbPdSlr1HxfNAkphjdsvmwSABLhOzFeE25zrftTW3Hf1ui7+tsJztDDio00qtXcT+NY8ySupb2siZcywFZFf3WgRzbOX9yjZv4QG8xPzQ80v4+EbKUuDJJkAHd85kAxZYryHQFi/dQO8O39lhJWog6T58gCteMBnYCvLqnYxr40qwXJ6lp44yoi+EBSH7TpLmsY/AM+YvQH1rqUTbi/1o9PvzwkY3C52WpMFeRlh9XDdg/hWWwcHCJc07dNP2kOCtsR1Te5dhTejsJ9ul2MzAGdRnPwuUz9RrMj0Xj/5X8we+4RPgF+/Z3EpjMOFFzrBbykj003n/pRJE0651a5FbdUkO9QeE3jjj4QGaWAb+3dTbVH3OGVIQ0Ebm31po6G1+9SPyfUSVOmmlHAK4XZybPGNQzsW7uorOAtet9++GAR11mBvuO5g9GSOBGw6EEqN76X1CJ3WFaKwfpoQzDXDC3hOMjahB9cZaWu7zXebJPlKUiZPuIfbB93WCOVjS5TU01McrBsjt0rgsSAgPwVzEAyGVInx1knlQ34SiPvgOZYZuvlr9ln2dAdzfmecSR4na4LwjXJd4B3LhBCDofEtTu21ArdxLu5nHjYg1JiaLm1jL2MJeElFkMtJjH8dKFO/e7rbF6Gtq5aMMP1GNFle2EqW53jJsOF7FLZAa5rTNP2qGKBJyWw6dmOPnszIWrMHrSLEJPTpIr4F0V27rwyknCijIDZG/XENDMRtJHLfpjqajVfnlMDp0+1gHX7GhepBN+1b7l9LaRGavJ9iyyBaiw6kHgp7mseRvceDbTkkOUQHEcNL1CYzxt2QHap7noKtfYa0K3GPCa41pXHwaMrG6jeM3Oi0KaoXTBmlfqeDKtqJ1YOzqQ5+qoVd/Um8nWq/HWvjhtoEK7K4zXaEsaVBlPeqrDqUY+sppbvKENXRRfotim9QG+b5ujINRCLl95HSNIQ5J6zDjYPqDToQuRF72L2SdoDHNSotcSzMDf3m91Idoec5oV8M+jPwqjm7BgzCK7RxW8aAAgh0Ht6j+QHYr8Isg00sCDFBrMG9P6nxbZsipeIklTjyCUWFA8rFPBMDfETFhNCkcpZ7TQqZsQtzBgc3gbgIemXw2ORSKvO6RB++b3s81Xj4OPgwG6cH/rvNGvf1ZsE3ONQFXCoVx0ZCXhdiLcvqovSrMctJKFzZdAVTSLKBARc7nc2JQx3Tam+29nn3cTvHH++zx52roTmpO2eI3gWkqpM8TEYAWQVLK8KaK33WCbu7Q6/iCOEKuHPz9v2L1oJBt/7zpU8wxbfafcxOzYuUTbw74EykGzUVaDUuSCbh14oZO9Z1vuBxWO8U}")

=> "-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEA4VajXvftedVzvG .......

Copy your RSA private key to cfme.pem and format it correctly:

sed -i 's/\\n/\n/g' cfme.pem

Next you need to fix perms

chmod 400 cfme.pem

If you want you can also compare fingerprints

openssl  pkcs8 -in smartstate-1c6df722-bf49-4e32-ad6c-e0aea84a842a.pem -nocrypt -topk8 -outform DER | openssl sha1 -c
(stdin)= 70:6f:cc:40:10:ea:97:43:26:6e:f5:73:4c:e8:0d:15:e8:59:5a:c1

This should match what you see in the Compute> Cloud -> KeyPairs.

As I assume that you already launched the SmartState analysis . 3 SQS queues were created in the meantime for you as well as an s3 bucket for the heartbeat where it will store the instance id for the agent. If you go to SQS in the AWS console you will see.

Here is the s3 bucket

When running SmartState you will actually have a message in the request queue. This will the CFME to launch the SmartState instance.

As you probably noticed a security group named smartstate was created for you as well.

Now I was wondering if everything on the atomic host is actually kosher. So that’s where our ssh-key comes in

Lets ssh into the atomic host

ssh -i "cfme.pem" ec2-user@ec2-57-88-166-140.compute-1.amazonaws.com

If everything is working correctly you should see the that the cfme-amazon-smartstate image gets pulled.

 

[root@ip-172-31-22-123 ~]# atomic images list
REPOSITORY                                                       TAG    IMAGE ID     CREATED VIRTUAL  SIZE      TYPE 
>registry.access.redhat.com/cloudforms46/cfme-amazon-smartstate  latest 53c8509ae76a 2018-02-22 16:20 311.44 MB docker

The next step, of course, is that CFME will launch it.

[root@ip-172-31-22-123 ~]# atomic containers list
CONTAINER ID IMAGE                NAME COMMAND          CREATED          STATE      BACKEND RUNTIME
60cb45e8e0d4 registry.access.redh silly_nort /bin/sh -c 2018-03-11 20:18 running    docker  docker

If the scan completes successfully you will see vm_scan_complete

[----] I, [2018-03-11T16:20:46.085250 #2978:4f7130]  INFO -- : Q-task_id([b0c7c318-2844-436c-bc44-21a29dfa3cce]) MIQ(MiqEvent#process_evm_event) target = [#<ManageIQ::Providers::Amazon::CloudManager::Vm id: 99000000000166, vendor: "amazon", format: nil, version: nil, name: "vmtosmartstate", description: nil, location: "ec2-54-160-15-44.compute-1.amazonaws.com", config_xml: nil, autostart: nil, host_id: nil, last_sync_on: "2018-03-11 20:20:08", created_on: "2018-03-11 20:08:19", updated_on: "2018-03-11 20:20:42", storage_id: nil, guid: "13f6cdd0-69db-4ce6-b7eb-3f04f38ce739", ems_id: 99000000000001, last_scan_on: "2018-03-11 20:18:45", last_scan_attempt_on: "2018-03-11 20:12:10", uid_ems: "i-0de0a2338f41c40a4", retires_on: nil, retired: nil, boot_time: "2018-03-11 20:03:10", tools_status: nil, standby_action: nil, power_state: "on", state_changed_on: "2018-03-11 20:08:19", previous_state: nil, connection_state: nil, last_perf_capture_on: nil, registered: nil, busy: nil, smart: nil, memory_reserve: nil, memory_reserve_expand: nil, memory_limit: nil, memory_shares: nil, memory_shares_level: nil, cpu_reserve: nil, cpu_reserve_expand: nil, cpu_limit: nil, cpu_shares: nil, cpu_shares_level: nil, cpu_affinity: nil, ems_created_on: nil, template: false, evm_owner_id: nil, ems_ref_obj: nil, miq_group_id: 99000000000001, linked_clone: nil, fault_tolerance: nil, type: "ManageIQ::Providers::Amazon::CloudManager::Vm", ems_ref: "i-0de0a2338f41c40a4", ems_cluster_id: nil, retirement_warn: nil, retirement_last_warn: nil, vnc_port: nil, flavor_id: 99000000000002, availability_zone_id: 99000000000002, cloud: true, retirement_state: nil, cloud_network_id: nil, cloud_subnet_id: nil, cloud_tenant_id: nil, raw_power_state: "running", publicly_available: nil, orchestration_stack_id: nil, retirement_requester: nil, tenant_id: 99000000000001, resource_group_id: nil, deprecated: nil, storage_profile_id: nil, cpu_hot_add_enabled: nil, cpu_hot_remove_enabled: nil, memory_hot_add_enabled: nil, memory_hot_add_limit: nil, memory_hot_add_increment: nil>]
[----] I, [2018-03-11T16:20:46.098293 #2978:4f7130]  INFO -- : Q-task_id([b0c7c318-2844-436c-bc44-21a29dfa3cce]) MIQ(MiqAlert.evaluate_alerts) [vm_scan_complete] Target: ManageIQ::Providers::Amazon::CloudManager::Vm Name: [vmtosmartstate], Id: [99000000000166]
[----] I, [2018-03-11T16:21:01.555276 #2716:4f7130]  INFO -- : MiqServer: local=Y, master=Y, status= started, id=99000000000001, pid=02716, guid=c5b77187-7fc9-426e-b3cb-6f0ce7860c8e, name=EVM, zone=default, hostname=ip-172-31-44-62.ec2.internal, ipaddress=172.31.44.62, version=5.9.0.22, build=20180221205805_f93a675, active roles=automate:database_operations:database_owner:ems_inventory:ems_operations:event:reporting:scheduler:smartproxy:smartstate:user_interface:web_services:websocket

Now you probably wondering, What happens during the SmartState analysis? I’ve created the following diagram for you to understand the magic:

Please, stay tuned for the next part of the series

 

      1. Thx Victor, is it possible that another image is missing after “When running SmartState you will actually have a message in the request queue. This will the CFME to launch the SmartState instance.”?

        Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s