Cloud Platforms Auto-discovery

Device42 Cloud Platform Auto-discovery

Device42 supports auto-discovery of cloud instances (virtual devices) on Amazon AWS, Amazon API, Microsoft Azure, Linode, DigitalOcean, OpenStack, Google Cloud, Alibaba Cloud, Oracle Cloud, and Standalone Kubernetes. Native discovery for new cloud platforms is added according to customer needs [and please do let us know!].

Device42 will auto-discover your cloud virtual machines, databases, and storage as devices. You can then work with your cloud devices just like like you would any other device. You can define application components, store passwords, create custom keys, and so on, just as you would for any other physical or virtual device.

The Discovery menu includes the option for discovery of Cloud instances. The list page shows all of your existing cloud auto-discovery jobs. You’ll typically create one job per account. As with other types of auto-discovery jobs, cloud discovery jobs can be run immediately and/or scheduled. Click Add Cloud Autodiscovery [top right corner] to create a new job, then simply select your cloud type.

Alibaba Cloud Discovery

Alibaba Discovery Items

Click Add Cloud Autodiscovery, and then select Alibaba Cloud as the Cloud Type. Name your job, and then add your Alibaba credentials, including both Access Key ID and your Access Key Secret.

Select one or more Zones for the discovery and select options for Action for Instance not found and Device Name Format. Click Save to add the job to the list of Cloud Autodiscovery jobs. Select the job and click Run Now to run the job immediately or continue editing to schedule runs.

Amazon Web Services Auto-discovery

AWS Discovery Items

API Endpoints Accessed During AWS Discovery

Regular Discovery

• sts.amazonaws.com
• organizations.us-east-1.amazonaws.com (Only if one of any of the available features is enabled.)

• ec2._region_.amazonaws.com

• elasticache._region_.amazonaws.com

• rds._region_.amazonaws.com

S3 – (All regions and all buckets, regardless of the region provided – only if the S3 feature is enabled.)

• *.s3._region_.amazonaws.com
• s3._region_.amazonaws.com
• *.s3.amazonaws.com
• s3.amazonaws.com

Route53 – (Only if the Route53 feature is enabled.)

• route53.amazonaws.com

K8s – (Only if K8s discovery is enabled.)

• eks._region_.amazonaws.com

K8s cluster endpoints access per K8s RBAC setup

• /api/v1/namespaces/kube-system
• /api/v1/nodes?watch=False
• /api/v1/services?watch=False
• /apis/apps/v1/deployments?watch=False OR /apis/extensions/v1beta1/deployments?watch=False (depends on k8s version)
• /apis/networking.k8s.io/v1beta1/ingresses?watch=False OR
  /apis/extensions/v1beta1/ingresses?watch=False (depends on k8s version)

Example of minimum policy (except for K8s cluster endpoints, since it is controlled by K8s RBAC).

{

    "Version": "2012-10-17",
    "Statement": [

        {
                   "Effect": "Allow",
                   "Action": [
                       "organizations:ListRoots",
                       "organizations:ListAccountsForParent",
                       "organizations:ListOrganizationalUnitsForParent",
                       "organizations:DescribeAccount",
                       "autoscaling:Describe*",
                       "logs:DescribeLogStreams",
                       "route53:ListHostedZones",
                       "cloudwatch:GetMetricStatistics",
                       "cloudwatch:Describe*",
                       "route53:ListTagsForResource",
                       "cloudwatch:ListMetrics",
                       "elasticache:Describe*",
                       "ec2:Describe*",
                       "rds:Describe*",
                       "rds:ListTagsForResource",
                       "s3:ListAllMyBuckets",
                       "route53:ListResourceRecordSets",
                       "logs:GetLogEvents",
                       "elasticloadbalancing:Describe*",
                       "eks:ListClusters",
                       "eks:DescribeCluster"
            ],
            "Resource": "*"
        }
    ]
}

Prerequisites

To create an AWS Autodiscovery job, you will need to:

1. 1. Prepare your AWS Account – you can use the policy example shown above.
   2. Device42 utilizes your AWS Access Key and Secret Key to perform discovery; please have these handy.

Note: Device42 encourages customers to follow AWS best practices for managing your IAM credentials, including using strong passwords, regular password rotation, applying the principle of least privilege to users and their passwords, etc.

For more information, see the article Best Practices for Managing AWS Access Keys at https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html.

• Select Discovery > Cloud from the main menu and then click Add Cloud Autodiscovery at the top right of the Cloud Autodiscovery list page.
• Enter a Name for the job.
• Select the Cloud Type > Amazon AWS from the drop-down menu.

• Select the Remote Collector for the job.
• Add your Amazon Access Key ID and your Secret Key for the account(s) to be discovered. (Do this by clicking the magnifying glass, and then clicking Add Password in the upper right corner. Enter your Access Key ID or Secret Key in the field labeled Password.  Device42 will store the keys encrypted.)
• Select Discover Main Account to have the job discover the main AWS account in addition to any AWS Roles accounts you select.
• Select the Available AWS Roles whose account(s) you want to discover and use the arrow to add them to the Chosen AWS Roles list.

Note: See Defining AWS Roles below for instruction for creating the AWS Roles that Devices42 displays for AWS cloud autodiscovery jobs.

• Choose one or more Amazon regions to search.
  
• You can also select options for adding vendor metadata, choose how to  handle instances not found in subsequent discovery vendor, select device name format options, add tags for discovered devices, etc.
• Check Kubernetes Discovery to discover Kubernetes clusters hosted on your cloud platform.
• Add object categories, tags, and a customer for discovered devices, etc.

• Scroll down the page and click the Advanced Features (Show) tab to select the different types of resources you want the job to discover (EBS, Databases, Load Balancer, etc.)

• Add an Auto Discovery Schedule to schedule the job if wanted or add Admin Groups for the job.
• Click Save or Save and continue editing to save the discovery job.
• When you return to the Cloud Discovery list page, you can click Run Now to run the job immediately.

Add or Edit AWS Roles

Device42 includes an editor you can use to define or edit the AWS Roles displayed for Amazon AWS cloud autodiscovery jobs. Follow the steps below to view and add AWS Roles.

• Select Resources > Secrets > AWS Roles from the main menu.

• Device42 displays the AWS Roles list page. Use the AWS Role drop-down to select a role to display or click Advanced to construct more specific searches. See the Advanced Search Feature documentation page for instructions.

• Click Add at the top right to add a new role – click a role Name to view or edit that role.

• Enter a Name for the role.
• Enter the AWS Role label and an optional AWS Role Description.
• In the Account ID and External ID section, click + Add More.

• Add the role Account ID and the External ID – click the eye icon to show or hide the field. Click the trash can icon to remove  the entries.
• Click Save at the top right of the page to save the role.

Device42 adds the new AWS Role to the roles list; it will also appear in the Available AWS Roles list when you create or edit an Amazon AWS cloud autodiscovery job.

Google Cloud Discovery

Google Cloud Discovery Items

Pre-requisites

For Google Cloud Discovery you will need a user account with the following roles:

• compute.viewer
• compute.networkViewer
• container.clusterViewer (For Kubernetes)
• cloudsql.viewer (For Databases)

Configuring Google Cloud Discovery

Device42 can now discover your inventory on the Google Cloud Platform. Simply create a new job, add your credentials, and you’ll be off discovering all of your GCE VMs. Begin by creating a new Google cloud discovery job:

1) Create a new “Cloud Autodiscovery” job from the Discovery, and choose Google Cloud.

2) Browse to your Google Cloud Engine JSON keyfile. Open it in a text editor and copy the contents:

3) Paste the copied JSON in its entirety into the password field:

4) Save and run your job! Optionally, create a schedule to run it automatically!

Data discovered on the Google Cloud Platform is similar to what you might be used to on AWS EC2 instances, namely:

• Discovered Google Cloud VMs are added as virtual devices
• Cloud information is added inline in Device42 for each CI

Options for GCE are as follows:

• Select Kubernetes Discovery to discover Kubernetes clusters hosted on your cloud platform.

• Strip Domain Name: Strip domain name from discovered name (everything after the first period)
• Object category for discovered devices: Choose a category to assign to discovered devices
• Overwrite existing object categories: Select this option to overwrite any previously assigned categories with current selection

Microsoft Azure Auto-discovery

Microsoft Azure Discovery Items

Minimum Permissions for Resource Group Discovery

Microsoft Azure Discovery is similar to Amazon; you will need to:

1. Choose and enter a Name for the job.
2. Select the Cloud Type (choose Microsoft Azure from the drop-down).
3. Enter your Subscription ID.
4. That’s it! Azure will now recognize the Device42 instance as a secure client that is authorized to interact with your Azure account.

Devices > Virtual Devices will now show cloud devices as well. The Vendor shows up as the Device Host for these devices. There is a new Virtual Subtype filter on the right-hand side filter bar.

Optionally, you can also:

• Choose the vendor [note that all vendors are user-defined – Device42 does not ship with a list of vendors].
• Choose a VRF Group. If one is selected, all discovered IPs will be placed in subnets in that VRF Group. This is useful if you have duplicate IPs in your internal network.
• Choose a remote collector to run the job (ensure the chosen remote collector can reach the target network).
• Select the Device Name Format for discovered cloud instances.

• Select Kubernetes Discovery to discover Kubernetes clusters hosted on your cloud platform.

• Check Add tags as custom fields to add discovered tags to Device42 custom fields.
• Check Strip domain name to have Device42 strip the discovered domain suffix (everything after the first period) from the device instance name.
• Choose a category for discovered devices (note that categories are user-defined).

Next, you should “Save and Continue”. Then you can click ‘Run’ to run the job immediately. Or you can save it or save it and have it run on a regular schedule.

OpenStack Auto-discovery

OpenStack Discovery Items

1. When you add an OpenStack job, you will be prompted for a User, Password, and Project Name.

Minimum Permission Requirements for OpenStack Discovery are as follows:

• User / User Group should be attached to the project and have the following permissions described in policy.json:
  • “os_compute_api:os-hypervisors”
  • “os_compute_api:os-extended-server-attributes”

1. The User and Password are the ones you enter into the Openstack authentication screen…
2. When you log into Openstack, the Overview screen shows a list of projects.
3. Enter the project name you would like to access into the Device42 Project Name field.

Optionally, you can also:

• Choose the vendor. Please note that all vendors are user-defined. Device42 does not ship with a list of vendors.
• Choose a VRF Group. If you select a VRF Group, then all IPs found will be placed in subnets in that VRF Group. This is useful if you have duplicate IPs in your internal network.
• Check the “Remove unfound instances from Device42″ box. If you check this box, then each time this auto-discovery job is run, any devices that were previously created for this account but were not found by the auto-discovery job will be deleted. By checking this box, you can ensure that Device42 will remain in sync with OpenStack. If you leave it unchecked, then you may end up with Device42 Cloud Instances (cloud devices) that no longer exist in OpenStack.

Next, you can click Save and Continue. Then you can click Run Now to run the job immediately. Or you can save it and set up a schedule to run the discovery job.

Oracle Cloud Auto-discovery

Oracle Cloud Discovery Items

Click Add Cloud Discovery on the Cloud Discovery page, and then select Oracle Cloud as the Cloud Type.

Enter the following information:

• Name for the discovery job.
• User ID
• Fingerprint (if applicable)
• Key File
• Tenant ID
• Zones

You can also:

• Choose the vendor. Please note that all vendors are user-defined. Device42 does not ship with a list of vendors.
• Choose a VRF Group. If you select a VRF Group, then all IPs found will be placed in subnets in that VRF Group.
• Select a Remote Collector.

Scroll down the page to see additional options.

Click Save and Continue; then you can click Run Now to run the job immediately. Or you can save it and set up a schedule to run the Oracle discovery job.

DigitalOcean Auto-discovery

DigitalOcean Discovery Items

Click Add Cloud Discovery on the Cloud Discovery page, and then select DigitalOcean as the Cloud Type.

Enter a Token Key, and then select any other options you want for the discovery job.

Click Save and Continue; then you can click Run Now to run the job immediately. Or you can save it and set up a schedule to run the Oracle discovery job.

Amazon API Auto-discovery

Amazon API Discovery Items

Use the Cloud Type: “Amazon API” selection to discover your AWS EC2 instances via the Amazon Elastic Compute Cloud API.

When discovering your Amazon Cloud via the Amazon API, Device42 authenticates against the API URL with your AWS API Access Key and Secret Key. To create a discovery job, please ensure you have these available. You can find or generate new AWS API Access keys via the AWS Console -> UserName Menu –> “My Security Credentials”. Expand the “Access keys (access key ID and secret access key)” item, and “Create New Access Key” (or reference an existing one):

1. Begin by setting Cloud Type: ‘Amazon AWS’ via the dropdown [pictured].
2. Enter a ‘Name’ for your Amazon AWS API discovery job.
3. Enter the ‘URL’ to of the AWS API endpoint you are targeting, including the port if necessary – for URLs and other information on AWS API endpoints, reference the “Endpoints” section of the AWS API documentation.
4. Add your AWS API Key ID to the “Account ID” field, followed by the corresponding Amazon Secret Key in the “Secret Key” field for the account to be discovered:
   
   You’ll add both your API Key ID & Secret Key to Device42 as separate ‘password’ entries, and the procedure is the same as adding a new password:
   • Click the magnifying glass to bring up the credential selection screen
   • Click the ‘Add Password’ button in the upper right-hand corner
   • Enter your Account ID in the field labeled “Password:” – The USERNAME FIELD IS NOT USED by cloud discoveries!, & click “Save”

   Repeat the process & add a second entry for your Secret Key, as well. Note that Device42 stores these values encrypted; If desired, you may also set access permissions on your AWS credentials.

5. In the Region: box, enter the region you are targeting, e.g. us-east-1.
6. Set a discovery schedule if desired; Save and run your AWS API discovery!

Options for AWS API Discovery:

• Action for Instance not found: Choose how Device42 will handle the situation of an instance that was previously discovered not being found on subsequent discovery runs. Change Status will update the instance’s status, while “Delete Instance” will delete the missing instance. The best choice for you depends on how you manage your infrastructure.
• Strip Domain Name: Strips the domain name (everything after the first period) from the name as discovered before storing in Device42
• Object category for discovered devices: Choose a category to assign to discovered devices
• Overwrite existing object categories: Select this option to overwrite any previously assigned categories with the current selection

Linode Auto-discovery

Linode Discovery Items

For access to Linode, you will need your API Key. To create a Linode API Key, go to your Linode console…

Select “My Profile” and navigate to “API Keys”

From here, you can create your API Key that Device42 needs to gain access.

Optionally, you can also:

• Choose the vendor. Please note that all vendors are user-defined. Device42 does not ship with a list of vendors.
• Choose a VRF Group. If you select a VRF Group, then all IPs found will be placed in subnets in that VRF Group. This is useful if you have duplicate IPs in your internal network.
• Check the “Remove unfound instances from Device42″ box. If you check this box, then each time this auto-discovery job is run, any devices that were previously created for this account but were not found by the auto-discovery job will be deleted. By checking this box, you can ensure that Device42 will remain in sync with Linode. If you leave it unchecked, then you may end up with Device42 Cloud Instances (cloud devices) that no longer exist in Linode.

Next, you should “Save and Continue”. Then you can click ‘Run’ to run the job immediately. Or you can save it or save it and have it run on a regular schedule.

Standalone Kubernetes Auto-Discovery

Standalone Kubernetes Discovery Items

To add a Standalone Kubernetes discovery job, you’ll need either a Bearer Token or Basic Credentials. You’ll also need to enter a URL and select an Action for Container not found.

Optionally, you can also choose a Vendor and a VRF Group. Please note that all Vendors and VRF Groups are user-defined.

Note: Kubernetes Discovery is also available as an option for Amazon AWS, Google Cloud, and Microsoft Azure cloud autodiscovery jobs. Scroll down the Add Cloud Discovery page and select the Kubernetes Discovery option.

Viewing Cloud Instance Information

Select Devices > Virtual Devices to view your cloud instances as virtual devices.

The Device view and edit pages will now show “Cloud Instance Information” under the Properties tab.