Lucidchart Cloud Insights

The Lucidchart Cloud Insights add-on allows you to import architecture from AWS, Azure, or GCP and generate diagrams from that data.

Note: The examples and screenshots in this article are specific to AWS imports and our Azure and GCP import and diagram layout capabilities are currently in beta. The functionality with Azure and GCP imports is similar to that of AWS, but there may be some discrepancies.

You can import data about your AWS cloud infrastructure using one of three methods: providing a cross-account IAM role with the proper policies; providing IAM user credentials; or running a script from your CLI, which pulls the data locally, then providing the generated JSON file.

Note: If you import using the cross-account role method, you can import from multiple accounts at once by using multiple roles. Lucidchart will save your roles so that you can use them in the future without having to re-enter the information.

To import, open a document and click File > Import Data, then select the Cloud Insights - AWS tile.

Import_.png

Screen_Shot_2020-05-04_at_3.13.58_PM.png

When importing your AWS architecture, you can select the resource types and the metadata you’d like to include in your diagram. If you are working with a very large environment, using Cross account and IAM can help to reduce the amount of data you are importing or reduce the number of API calls Lucidchart makes to AWS. 

After selecting which accounts/regions to import, you will have the chance to select which resource types you want to import. 

Resource_Type_and_Metadata.png

This can be helpful in reducing the amount of data that you are importing or in reducing the number of API calls Lucidchart makes to AWS, if you are working with a very large environment. 

All supported resource types are checked by default. Some resource types are required and cannot be unchecked (including Security Groups, Subnets, and VPCs)

Some resource types also allow an option for unchecking ‘Additional Metadata’. These resource types require additional API calls to AWS to get metadata for each resource of that type. So, unchecking this box will reduce the number of API calls made to AWS (and reduce the metadata imported for that resource type), but will not reduce the number of resources that you import.

Note: Selecting the resource types and metadata is only available on a Cross-account role and IAM import. The additional metadata may include tags (depending on the resource) and metadata for some lines, so unchecking them could result in missing tag or line data.

To import your AWS infrastructure via cross-account role, follow these steps:
  1. File > Import Data > Cloud Insights - AWS
  2. Select Cross-Account Role > Next

    cross_account_role.png
  3. Create a new role in the AWS IAM Console or click Add Previously Imported Account and click Add Account.


    Screen_Shot_2019-12-20_at_1.32.33_PM.png

    Screen_Shot_2019-12-20_at_1.33.21_PM.png
  4. From the panel on the right, select the region(s) from the dropdown that you would like to import, then click Finish.

    Screen_Shot_2019-12-20_at_1.33.41_PM.png

    Your data will be imported to Lucidchart. Lucidchart will automatically lay out your architecture on the canvas.
In order for Lucidchart to access your AWS infrastructure, you’ll need to give Lucidchart the credentials for a new cross-account role. See below for instructions on how to create these in your AWS console.

Note: If you have used the legacy AWS import in the past, any cross-account roles that you have set up will show up in Lucidchart Cloud Insights. However, the permissions for those roles will be based on the old policy, so it will not import everything that Lucidchart Cloud Insights now supports until you update the role with the new permission set.
To create a cross-account role for Lucidchart Cloud Insights, follow these steps:
  1. In a Lucidchart document, click Import Data > Cloud Insights - AWS
  2. Select Cross-Account Role > Next
  3. Create a new role in the AWS IAM Console by clicking on the link provided. You will be navigated to your AWS account.

    Link_to_create_role.png
  4. In AWS, click “Next: Permissions.”
  5. Click “Create Policy.”
  6. Copy the json policy file here: 
    {
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Action": [
                "apigateway:GetIntegration",
                "apigateway:GetResources",
                "apigateway:GetRestApis",
                "apigateway:GetTags",
                "apigateway:GetVpcLinks",
                "apigatewayv2:GetApis",
                "apigatewayv2:GetRoutes",
                "apigatewayv2:GetTags",
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeLaunchConfigurations",
                "cloudfront:ListDistributions",
                "cloudfront:ListTagsForResource",
                "dynamodb:DescribeTable",
                "dynamodb:ListTables",
                "dynamodb:ListTagsOfResource",
                "ec2:DescribeInstances",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeNatGateways",
                "ec2:DescribeNetworkAcls",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeTransitGateways",
                "ec2:DescribeTransitGatewayPeeringAttachments",
                "ec2:DescribeTransitGatewayRouteTables",
                "ec2:DescribeTransitGatewayVpcAttachments",
                "ec2:DescribeVolumes",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcEndpoints",
                "ec2:DescribeVpcEndpointConnections",
                "ec2:DescribeVpnGateways",
                "ec2:DescribeVpcPeeringConnections",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeTags",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetHealth",
                "iam:GetGroupPolicy",
                "iam:GetPolicy",
                "iam:GetPolicyVersion",
                "iam:GetRolePolicy",
                "iam:GetUserPolicy",
                "iam:ListAttachedGroupPolicies",
                "iam:ListAttachedRolePolicies",
                "iam:ListAttachedUserPolicies",
                "iam:ListGroupPolicies",
                "iam:ListGroups",
                "iam:ListGroupsForUser",
                "iam:ListRolePolicies",
                "iam:ListRoles",
                "iam:ListUserPolicies",
                "iam:ListUsers",
                "lambda:ListFunctions",
                "redshift:DescribeClusters",
                "rds:DescribeDBClusters",
                "rds:DescribeDBInstances",
                "rds:ListTagsForResource",
                "route53:ListHostedZones",
                "route53:ListResourceRecordSets",
                "route53:ListTagsForResource",
                "s3:GetBucketLocation",
                "s3:GetBucketTagging",
                "s3:ListAllMyBuckets",
                "sns:GetTopicAttributes",
                "sns:ListTopics",
                "sns:ListTagsForResource",
                "sqs:GetQueueAttributes",
                "sqs:ListQueues",
                "sts:GetCallerIdentity"
            ],
            "Resource": ["*"]
        }]
    }
    
  7. Paste the json policy file into the JSON tab, then click “Review Policy.”
  8. Give your policy a name and description, then click “Create policy.”
  9. In the previous tab where you were creating a new cross-account role, click the refresh button, then search for the name of the policy you just created. Select the policy and click “Next:Tags.”
  10. Add IAM tags to your role if you’d like, then click “Next: Review.”
  11. Give your role a name and description, then click “Create role.”
  12. Click on the role’s name to navigate to your role’s summary page.
  13. Copy the string next to “Role ARN.” You will copy this into Lucidchart to create your Cross-Account Role.
Security note: Lucidchart will not store your AWS IAM credentials after performing the initial scan of your AWS infrastructure. Your credentials will be transferred to our servers using standard encryption methods. Clients may negotiate encryption protocols up to AES-256. We can store a Cross-Account Role that only gives us “describe and list” access to your environment. To import your AWS infrastructure via IAM user credentials, follow these steps:
  1. File > Import Data > Cloud Insights - AWS
  2. Select IAM User > Next

    iam_user.png
  3. Create or use an IAM user with at least the policy permissions given in the IAM Permissions section attached.

    IAM_copy_to_clipboard.png
  4. Enter the IAM user credentials to Lucidchart or upload a CSV for AWS credentials file, then click Finish. Your data will be imported to Lucidchart. Lucidchart will automatically lay out your architecture on the canvas.
In order for Lucidchart to access your AWS infrastructure, you’ll need to give Lucidchart the credentials for a new IAM user. See below for instructions on how to create these in your AWS console.

To create an IAM user with an inline policy for Lucidchart’s AWS import, follow these steps:
  1. Go to the AWS console and navigate to the “Users” section of Identity Access and Management.
  2. Click “Add User” and enter a name for the user. Be sure to check the box next to “Programmatic access” so Lucidchart can use the access key to retrieve the information about your infrastructure.
  3. Click “Next: Permissions” and select “Attach existing policies directly.” Click “Create policy” and give your policy a name and description.
  4. Click “Select” next to “Create your own policy.”
  5. In a Lucidchart document, click Import Data > Cloud Insights - AWS
  6. Select IAM User > Next.
  7. Copy the json policy file here: 
    {
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Action": [
                "apigateway:GetIntegration",
                "apigateway:GetResources",
                "apigateway:GetRestApis",
                "apigateway:GetTags",
                "apigateway:GetVpcLinks",
                "apigatewayv2:GetApis",
                "apigatewayv2:GetRoutes",
                "apigatewayv2:GetTags",
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeLaunchConfigurations",
                "cloudfront:ListDistributions",
                "cloudfront:ListTagsForResource",
                "dynamodb:DescribeTable",
                "dynamodb:ListTables",
                "dynamodb:ListTagsOfResource",
                "ec2:DescribeInstances",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeNatGateways",
                "ec2:DescribeNetworkAcls",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeTransitGateways",
                "ec2:DescribeTransitGatewayPeeringAttachments",
                "ec2:DescribeTransitGatewayRouteTables",
                "ec2:DescribeTransitGatewayVpcAttachments",
                "ec2:DescribeVolumes",
                "ec2:DescribeVpcs",
                "ec2:DescribeVpcEndpoints",
                "ec2:DescribeVpcEndpointConnections",
                "ec2:DescribeVpnGateways",
                "ec2:DescribeVpcPeeringConnections",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeTags",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeTargetHealth",
                "iam:GetGroupPolicy",
                "iam:GetPolicy",
                "iam:GetPolicyVersion",
                "iam:GetRolePolicy",
                "iam:GetUserPolicy",
                "iam:ListAttachedGroupPolicies",
                "iam:ListAttachedRolePolicies",
                "iam:ListAttachedUserPolicies",
                "iam:ListGroupPolicies",
                "iam:ListGroups",
                "iam:ListGroupsForUser",
                "iam:ListRolePolicies",
                "iam:ListRoles",
                "iam:ListUserPolicies",
                "iam:ListUsers",
                "lambda:ListFunctions",
                "redshift:DescribeClusters",
                "rds:DescribeDBClusters",
                "rds:DescribeDBInstances",
                "rds:ListTagsForResource",
                "route53:ListHostedZones",
                "route53:ListResourceRecordSets",
                "route53:ListTagsForResource",
                "s3:GetBucketLocation",
                "s3:GetBucketTagging",
                "s3:ListAllMyBuckets",
                "sns:GetTopicAttributes",
                "sns:ListTopics",
                "sns:ListTagsForResource",
                "sqs:GetQueueAttributes",
                "sqs:ListQueues",
                "sts:GetCallerIdentity"
            ],
            "Resource": ["*"]
        }]
    }
    
  8. Paste the json policy file into the “Policy Document” section of AWS. Click “Create Policy.”
  9. In the previous tab where you were creating a new user, click the refresh button, then search for the name of the policy you just created. Select the policy and click “Next:Review.”
  10. Review the information, then click “Create user.”
  11. Click “Download .csv” to store the access key ID and secret access key, and copy these into Lucidchart to finish your import.
To import your AWS infrastructure via CLI, follow these steps:
  1. File > Import Data > Cloud Insights (AWS)
  2. Select AWS CLI Script > Next

    aws_cli_script.png
  3. Run the provided script using your terminal’s command line interface to generate a JSON file
  4. Upload the resulting JSON to Lucidchart, then click Finish. Your data will be imported to Lucidchart. Lucidchart will automatically lay out your architecture on the canvas.
Once you import your AWS architecture, Lucidchart will automatically lay out its components on the canvas. To the left of the diagram you will see the cloud insights panel.

Navigate__first_diagram_shown_.png

Note: Your AWS diagram will exist as a grouped shape that you can click in and out of. When you are not in edit mode of the shape, the cloud insights panel will be hidden. Double-click anywhere within the diagram to enter the group and re-open the cloud insights panel.

You can show/hide entire classes of resources with the hierarchy checkboxes. For example, if you uncheck VPCs, then Lucidchart will hide the VPC containers from your diagram, but continue to display the resources that were inside of them.

show_hide_classes.png
When you select a shape on the canvas, you can see the metadata associated with that shape in the data panel to the right of the canvas. You can access this panel by clicking on the database icon in the dock. Lucidchart pulled this metadata down with your cloud infrastructure.

data_panel.png
You can filter out shapes based on metadata about the resource.

Filtering.png

Please note the following about filters:
  • You can save filters to views or shared across views. Once you’ve applied filters, you can go to the “View” dropdown and click “Save as New View” to save your filter changes. 

    Filter.png
  • Filters can be checked or unchecked to apply/unapply.
  • If you filter out a container shape, this will also filter out the contents of that container.
  • Filters work as OR relationships. If you have two filters, eg. Filter1 and Filter2, the diagram will display anything that matches the criteria in Filter1 OR Filter2.
  • When you apply a filter, you must select an entity type to apply it to, and the filter will only apply to the entity type that you selected. For example, if you apply the filter of “Name contains ‘doc’” to the EC2 entity type, this will not apply to any VPCs, regardless of whether their names contain ‘doc.’
You can save settings of your architecture diagram as a “view” which can be selected from a list of saved views to replace your current settings with the settings associated with that view.

Views.png

Note: If you make any changes to a view, make sure to save these changes before switching to a new view. Otherwise, you will lose any unsaved changes.
You can summarize shapes for the same resource type into a single shape with a badge identifying the number of shapes summarized.

You can find this feature in the "Layout" tab on the left panel. Once you click on the entity dropdown, you would see two categories “Containers” and “Resources”. The “Containers” section will display the default layout manipulation elements, the “Resources” sections enable the summary shape functionality. Turn on the "Summary Shapes" toggle to summarize the shapes of the selected resources.

image1.png

Tip: You can summarize your instances at a VPC level and thus obtain a count of the instances at that level. You can accomplish this by unchecking "Subnets" in the hierarchy on the "Explore Tab".

image2.png
You can modify the layout of your diagram in the Layout section of the cloud insights panel.

Layout_Panel.png

Select a container type from the dropdown at the top to modify the layout settings for that type. The changes you make will only apply to the contents of the container that you select.

Entity.png

You can make the following adjustments to your diagram’s layout:
  • Determine whether containers in the same row will expand to match the height of the tallest container or resize to fit around their contents. Note: this setting will apply to the entity selected, rather than the contained items.
  • Group resources within a container by type
  • Adjust the number of columns that shapes within a container are arranged in
  • Adjust the horizontal and vertical spacing between shapes within a container
  • Sort contained items by alphabetical order (A to Z or Z to A)

    image4.png

In this menu, you can customize what metadata you would like to have displayed under a resource’s icon.

Layout_Menu.png

Once you click on the entity dropdown, you would see two categories “Containers” and “Resources”. 

  • The “Containers” section will display the default layout manipulation elements.
  • The “Resources” section will enable the configurable metadata functionality. 

To configure the metadata, select the target resource in the "Entity" dropdown. Then, under the “Text Fields” section, you can either type or select a metadata field.  

A selected field will appear with a checkbox under the input field. You can then check or uncheck the fields you would like to display under your resource icon. 

Some of the fields may be empty. If you would like to hide the empty field created when the text field is checked, click on the menu next to the field, and then choose the “Hide Empty Field” option.

Hide_Empty_Field.png

If you click on the database icon at the top right of your cloud insights panel, you will see several options: Refresh Architecture, Replace Architecture, Convert to Regular Shapes, Export As JSON and Export As Anonymized JSON.

image3.png

Clicking Refresh Architecture will update your diagram based on any changes in the connected architecture data. Note: All changes made to the diagram will persist after a refresh except filtering.

Clicking Replace Architecture allows you to replace the data on the canvas with a new data source. This would enable you to bring in new architectural data without needing to do a new import.

Clicking Convert to Regular Shapes will convert your diagram from the Cloud Insights experience to a regular diagram that you can do standard, manual diagramming with. You will no longer be able to make changes via the Cloud Insights panel, but will be able to freely move shapes around the canvas and apply stylistic changes to them (like color and size).

Clicking Export As JSON will generate and download a JSON file containing all the resources and metadata of your infrastructure. The exported file can then be used to perform a CLI import. 

Clicking Export As Anonymized JSON will generate and download a JSON file containing all the resources and metadata of your infrastructure with sensitive fields anonymized. We anonymize all string values, including CIDR blocks, IP addresses, account IDs, etc. This feature enables you to keep a code version of your infrastructure without exposing confidential data.

Note: The options displayed above will vary depending on the type of import you are doing. The refresh option is only available with AWS Cross-account role import. IAM, Azure or CLI imports do not support that option. Export As Anonymized  JSON is currently only available with AWS imports.

You can show lines between shapes that represent the relationships between resources. You can choose to turn on/off specific relationships. 

To display customization options, click on a generated line. From there you can modify the line’s color, type, and style in the expanded menu.


image5.png
Lucidchart Cloud Insights supports the following AWS resources:
  • ALB (Load Balancers v2, Target Groups, Target Health Description)
  • Autoscaling (Groups, Launch Configurations)
  • CloudFront (Distributions)
  • DynamoDB (Tables)
  • EC2 (Instances, NAT Gateways, Network ACLs, Route Tables, Security Groups, Subnets, Transit Gateway, Volumes, VPC Endpoints, VPC Peering Connections, VPCs, VPN Gateways)
  • ELBs (Load Balancers)
  • IAM (Attached Policies, Role Policies, Roles)
  • Lambda (Functions)
  • RDS (Clusters, Instances)
  • Redshift (Clusters)
  • S3 (Buckets - Additional Metadata: roles, groups, etc.)
  • SNS (Topics, met)
  • SQS (Queues)
  • Route 53
  • API Gateway 
  • Customer Gateway
Lucidchart Cloud Insights supports the following Azure resources:
  • Compute (Virtual Machines, Virtual Machine Scale Sets)
  • MySQL (Databases)
  • Network (Application Gateways, Load Balancers, Network Security Groups, Subnets, Virtual Networks)
  • PostgreSQL (Databases)
  • SQL (Databases)
  • Storage (Storage Accounts)
Lucidchart Cloud Insights supports the following GCP resources:
  • Compute (Autoscalers, Instances, Instance Groups)
  • Database (SQL Instances)
  • Manage (Project, Zone)
  • Network (Firewall, Network, Subnetwork)
  • Storage (Bucket)
Cross-Account Roles
Cross-Account roles are the preferred method from AWS for granting 3rd party access to your account. This method allows Lucidchart to securely store credentials that allow you to easily refresh your diagram. See these resources for more information:

Secure, Limited Access for IAM Users

We request limited, “describe”-level permission for the IAM user you create. An IAM user created with these permissions cannot change settings in your AWS architecture or read data in your databases. We only use the IAM user to read the structural metadata of your AWS infrastructure. Please check out this article for information on how to create an IAM user.

CLI Script Alternative

If you wish to review and control the actions we take during our AWS infrastructure scan, you can download and use our provided Python script instead of creating an IAM user. In this scenario, your IAM credentials will never be passed to Lucidchart, and you can review both the code that will run in your environment and the resulting metadata before uploading the metadata to Lucidchart.

Safe Storage of Documents

Lucidchart stores the AWS imported documents and metadata using industry standard protections for confidential data. Imported AWS data is embedded as part of the Lucidchart document, so you can control access to the imported data using Lucidchart’s standard sharing permissions. For additional information regarding how we protect your documents, please refer to our Content Security page or contact our sales team.

No Storage of Access Keys

Lucidchart will not store your AWS IAM credentials after performing the initial scan of your AWS infrastructure. Your credentials will be transferred to our servers using standard encryption methods. Clients may negotiate encryption protocols up to AES-256. We can store a Cross-Account Role that only gives us “describe and list” access to your environment. 

How do I get access to Lucidchart Cloud Insights?
Lucidchart Cloud Insights is an add-on that can be purchased by accounts on the Enterprise plan. You can contact our sales team to get a quote.

Once Lucidchart Cloud Insights in purchased, will every user on the account have access?
Yes, all users on the account will have access to it and do not need a special license.

How does Lucidchart Cloud Insights get access to our AWS environment? What level of access does Lucidchart have?
The recommended method from Lucidchart and AWS is through a Cross-Account Role. We provide the policy which gives us “describe and list” access to your environment so we can get an inventory of resources.

We also have a method using a python script you can run through the Command Line Interface (CLI) which generates a JSON file. Users can then upload this JSON file to Lucidchart and not enter any credential information into Lucidchart.

What APIs does Lucidchart Cloud Insights access?
We primarily use describe and list API endpoints.

My data needs to stay in the EU. Can we still use Lucidchart Cloud Insights? Yes, Lucidchart Cloud Insights will work in our EU data center. Please contact our sales team to learn more.

Can you choose which metadata (the info shown in the shape data panel) is imported? If not, what metadata is imported?
No, you can't currently choose what metadata is imported. However, we are looking to create some ability to do that in the future. All metadata from a describe API call is imported.

What relationships/lines can Lucidchart Cloud Insights draw?
Lines are based on resources that can send traffic from resources to other resources. Here are some examples of connections you can visualize:
  • Elastic Load Balancers (ELBs) to Auto Scaling Groups (ASGs) [ELB→ ASG]
  • ELBs to EC2 instances [ELB → EC2]
  • CloudFront Distributions to S3 Buckets [CloudFront → S3]
  • Application Load Balancers (ALBs) to EC2 [ALB → EC2]
  • VPC Peering Connections 
  • Route 53 Hosted Zones to ELBs, S3 Buckets and CloudFront Distributions [Route 53 Hosted Zones → ELBs, S3, CloudFront]
  • Transit Gateway to Peered Transit Gateway and Virtual Private Cloud (VPC) attachments [Transit Gateways → Transit Gateways, VPCs]
How do you handle instances in AutoScalingGroups?
We display the AutoScaling Group icon, which includes metadata about the range of instances possible with the group, as well as the individual instances in the autoscaling group.

Can I restrict access to Cloud Insights for specific users?
Yes! If you would like to restrict users that have access to Cloud Insights within your organization, please contact your Lucidchart Account Manager or Customer Success Manager.

I just received this error message: “The format of the JSON you uploaded didn’t match what we were expecting. Please rerun the script and try again.” How do I proceed?

Error_Message_Screenshot.png

If you encounter this error message, the most common cause is not using the correct botocore version. Ensure that you have installed the version 1.13.0 of the botocore library using the command: pip install botocore==1.13.0