[{"content":"Steyn Huizinga started his career in 1999, at the age of 14. Initially as a webdeveloper for e-commerce sites and later as Linux and Network Engineer. He continued his career as a self-employed Solution Architect with a strong focus on security and hosting. Steyn joined Oblivion Cloud Control in 2015 as a Cloud Consultant. In January 2025 he joined Cloudar as Managing Partner.\nCloudar and AWS # He is passionate about AWS. In 2021 he was welcomed by Amazon Web Services (AWS) as an APN Ambassador, a very select group of individuals who are passionate about sharing their AWS technical expertise. Blessed to be working with like-minded AWS professionals at AWS Premier Consulting Partner Cloudar. Within Cloudar he has the opportunity to fully focus on AWS and work with innovative and fashionable projects on a daily basis.\nAWS Certifications # Soon after joining Oblivion Cloud Control he managed to get his first AWS Certifications. Within a year he was able to get the all-5 (the maximum number of available certifications). Now it\u0026rsquo;s a balancing act between learning new stuff and the recertification train.\nCertification Valid from State AWS Certified Developer - Associate (DVA) Oct 27, 2015 Active AWS Certified Solutions Architect - Associate (SAA) Oct 06, 2015 Active AWS Certified SysOps Administrator - Associate (SOA) Jan 26, 2016 Active AWS Certified Cloud Practitioner (CLF) Dec 29, 2017 Active AWS Certified DevOps Engineer - Professional (DOP) Jul 28, 2016 Active AWS Certified Solutions Architect - Professional (SAP) May 17, 2016 Active AWS Certified Advanced Networking - Specialty (ANS) Dec 01, 2016 Active AWS Certified Security - Specialty (SCS) Feb 06, 2018 Active AWS Certified Big Data - Specialty (Retired) (BDS) Aug 01, 2017 Retired Personal life # Father, sim racer, tech geek.\n","date":"May 26, 2021","externalUrl":null,"permalink":"/page/about_me/","section":"Pages","summary":"","title":"About me","type":"page"},{"content":"","date":"February 11, 2025","externalUrl":null,"permalink":"/categories/aws/","section":"Categories","summary":"","title":"AWS","type":"categories"},{"content":"","date":"February 11, 2025","externalUrl":null,"permalink":"/tags/aws/","section":"Tags","summary":"","title":"AWS","type":"tags"},{"content":"","date":"February 11, 2025","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"February 11, 2025","externalUrl":null,"permalink":"/tags/cloudar/","section":"Tags","summary":"","title":"Cloudar","type":"tags"},{"content":" Introduction # The Fourth Industrial Revolution, or Industry 4.0, is transforming the way industries operate by integrating advanced technologies such as IoT, AI, and big data analytics. Companies need expert guidance to navigate this shift efficiently. Cloudar, an AWS Premier Consulting Partner, plays a crucial role in helping businesses leverage cloud technologies to build scalable, secure, and efficient Industry 4.0 applications.\nWhy Cloudar for Industry 4.0? # Cloudar specializes in AWS solutions tailored for Industry 4.0 needs, ensuring seamless cloud adoption and integration. With deep expertise and a customer-centric approach, with Cloudars 100% focus on AWS, Cloudar helps businesses harness the power of AWS to drive innovation and operational excellence.\nHow Cloudar Enhances Industry 4.0 Transformation # Cloudar offers a comprehensive suite of services, 100% customer obsessed, including:\nIoT \u0026amp; Edge Computing Implementation: Cloudar enables businesses to connect industrial devices seamlessly, leveraging AWS IoT Core and AWS Greengrass for real-time data processing. Industries benefiting from these solutions include: Manufacturing: Real-time monitoring of production lines and predictive maintenance. Agriculture: Smart farming solutions for soil monitoring and precision agriculture. Healthcare: Connected medical devices for remote patient monitoring. Energy \u0026amp; Utilities: Smart grids and predictive maintenance for power plants. Automotive \u0026amp; Transportation: Fleet management and connected vehicle solutions. AI \u0026amp; Predictive Maintenance Solutions: By integrating AWS AI and machine learning, Cloudar helps organizations implement predictive maintenance, reducing downtime and operational costs. Big Data \u0026amp; Analytics Optimization: Cloudar ensures efficient data processing and analytics by utilizing AWS services such as Amazon Kinesis, AWS Glue, and AWS Redshift. Security \u0026amp; Compliance Advisory: With extensive expertise in cloud security, Cloudar ensures that Industry 4.0 applications meet stringent compliance requirements. Cloud Cost Optimization \u0026amp; FinOps: Cloudar helps businesses optimize their AWS spending while ensuring maximum performance and scalability through FinOps best practices. Cloudar\u0026rsquo;s Fundamental Capabilities (our job zeroes) # Cloudar’s expertise goes beyond traditional cloud adoption, offering foundational capabilities essential for Industry 4.0 success. Cloudar prioritizes reliability, automation, and operational excellence as the first principle, ensuring continuous availability and efficiency in Industry 4.0 environments. Key in Industry 4.0 is the short time to market. Without proper support most transformations will come to a big stall, because of budgets are exceeded, security incidents have been reported, etc. Cloudar\u0026rsquo;s experience will help you avoiding the mistakes that others have made before. Think of:\nCloud Financial Management (or FinOps): Cloudar enables businesses to optimize cloud spending, maximize cost efficiency, and ensure ROI through tailored cloud financial strategies. Security First Approach: Cloudar integrates security from sprint zero, at every stage, ensuring robust protection, compliance adherence, and risk mitigation for Industry 4.0 applications. Overcoming Industry 4.0 Challenges with Cloudar # Despite the benefits of Industry 4.0, companies face challenges such as:\nKickstart Cloud Infrastructure: Cloudar kickstarts your cloud infrastructure setup and management. Legacy System Integration: Cloudar enables seamless integration of legacy systems with modern cloud technologies. Security \u0026amp; Compliance Concerns: Cloudar can assist you in ensuring data protection and regulatory compliance at every stage of cloud adoption. Scalability \u0026amp; Cost Efficiency: Cloudar’s expertise ensures businesses achieve optimal scalability while controlling costs. Cloudar\u0026rsquo;s experience will guide you through the challenges.\nCloudar’s Role in Your Industry 4.0 Journey # As an AWS Premier Consulting Partner, Cloudar provides end-to-end support for Industry 4.0 transformation:\nStrategic Cloud Professional Services: With Cloudar works closely with businesses to develop tailored Industry 4.0 applications that can support your digital strategies. Seamless Cloud Adoption: Cloudar ensures a smooth transition to AWS cloud solutions, with workshops, training, best practices and expertise. Ongoing Support \u0026amp; Optimization: Cloudar offers continuous optimization to enhance efficiency and cost-effectiveness. Including Managed Services that will allow you to laser focus on your business value. Conclusion # Industry 4.0 presents significant opportunities for innovation and efficiency, but navigating the complexities requires expertise. Cloudar provides the necessary guidance and technical support to ensure a seamless, secure, and cost-effective transition to the cloud.\nAre you ready to accelerate your Industry 4.0 transformation? Contact Cloudar today to get started!\nAbout Cloudar # Cloudar is an Amazon Web Services (AWS) Premier Consulting Partner and Managed Services Provider with a 100% focus on AWS. They specialize in designing, building, and managing AWS infrastructure for businesses across a range of industries. With a deep commitment to innovation and security, Cloudar helps organizations fully leverage the power of the AWS cloud, providing tailor-made solutions for high availability, scalability, and cost efficiency.\nAs a trusted AWS partner, Cloudar offers end-to-end cloud solutions, from migration to continuous management and improvement, allowing businesses to focus on their core activities while Cloudar handles the complexity of the cloud.\nWhether you’re looking to modernize your infrastructure, enhance your security posture, or accelerate your digital transformation journey, Cloudar is there to guide you every step of the way.\n","date":"February 11, 2025","externalUrl":null,"permalink":"/2025/02/how-cloudar-drives-industry-4.0-innovation/","section":"Posts","summary":"","title":"How Cloudar Drives Industry 4.0 Innovation","type":"posts"},{"content":"","date":"February 11, 2025","externalUrl":null,"permalink":"/tags/industry-4.0/","section":"Tags","summary":"","title":"Industry 4.0","type":"tags"},{"content":"","date":"February 11, 2025","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"February 11, 2025","externalUrl":null,"permalink":"/","section":"Steyn Huizinga","summary":"","title":"Steyn Huizinga","type":"page"},{"content":"","date":"February 11, 2025","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"September 28, 2023","externalUrl":null,"permalink":"/tags/iam/","section":"Tags","summary":"","title":"IAM","type":"tags"},{"content":" Security is a shared responsibility # As you might have read in my previous posts, public cloud itself should be considered as very secure. For major cloud providers such as AWS security is key. Security incidents would destroy AWS\u0026rsquo; business so they are fully committed to prevent this from happening. Their almost unlimited access to security talent, extensive knowledge, years of experience, enormous budget, benefit of building things from scratch etc etc are indicators that security is serious business. And looking at the reported incidents compared to the size and scope of their services is impressive. If you are wondering which incidents have been reported, see here. No doubt about security of the cloud.\nThis is not preventing security incidents of applications running in the cloud to happen. Despite the number of tools, controls, capabilities and solid security architecture provided by AWS incidents on cloud applications still happen. These type of incidents are caused by human-error or so called security in the cloud. Those security incidents are caused by configuration errors in the space what the customer is responsible for. Compare it to the security of your house. You can have all controls in place, such as proper locks, alarms etc. But if you don\u0026rsquo;t use them and leave the frontdoor wide open\u0026hellip; don\u0026rsquo;t be surprised if things will go wrong. The same can happen in cloud (or any other IT enviroment) if the implementation is not done correctly. Think of permissive IAM policies, open security groups, etc.\nFigure 1: The AWS Shared Responsibility Model\nIn this blog we will focus on improving IAM policies.\nImprove knowledge # Invest in security knowledge Perhaps an open door, but it all starts with knowledge and talent. To level-up your level of knowledge in AWS security, you can explore various resources such as AWS documentation, online courses, certification programs (e.g., AWS Certified Security – Specialty), and industry-specific books and articles. Staying up-to-date with the latest security threats and AWS security features is an ongoing process that can help you adapt to the ever-changing landscape of cloud security.\nRun analysis on your IaC # Scan your Infrastructure as Code artifacts. Examples of tools: cdk-nag, AWS CDK plugin – CfnGuardValidator or the scanning capability of Trend Micro Cloud One Conformity.\nThose tools typically validate your Infrastructure as Code template against frameworks such as the best practices from the AWS Well-Architected Framework\nUse AWS Security Hub, AWS Trusted Advisor and other tools # Insights might change over time and changes can be made unnoticed (e.g. by attackers). Therefore it is important to continously monitory your enviroment on configuration errors. This is called Cloud Security Posture Management and is provided by AWS SecurityHub or 3rd party tools such as Trend Micro Cloud One Conformity. In addition AWS Trusted Advisor can give recommendations on security (available for customers on Business or Enterprise Support plans). Those tools are only powerful when you actually use them.\nBe strict on implementing least-privileged policies # Implementing least-privileged policies is a fundamental principle in cybersecurity and access control that helps minimize potential security risks and limit the potential damage that can occur in the event of a security breach. Being strict about implementing least-privileged policies is crucial for maintaining a secure and well-protected environment. When building an application it\u0026rsquo;s tempting to stop optimizing your IAM policies when your application is working and focus on the next feature of your app. We\u0026rsquo;ve all been there and got a stack of t-shirts.\nBy design don\u0026rsquo;t accept Resource: \u0026quot;*\u0026quot; or Principal: \u0026quot;*\u0026quot;. Don\u0026rsquo;t attach policies such as AdministratorAccess. Let CDK do heavy lifting by granting access to resources and generating the policies. Regularly Test and Assess implemented policies. What you consider as safe today might be unsecure tomorrow. Your knowledge will improve, but also AWS will continuously add new features such as additional fields that can be referenced in policy conditions etc. Don\u0026rsquo;t forget to improve resources policies (encryption keys, data resources etc) and attach/improve IAM Permissions Boundaries. Consider Amazon Verified Permissions for fine-grained application-level policies when developing new application. Be alert on creating privilege escalation, for example by allowing users to assume other roles or attach wider-policies to their own identities. Enable AWS account or organization wide security controls # Use AWS Organizations to enable organizational wide controls such as Service Control Policies (SCPs). These controls have priority above IAM controls. SCPs will allow you to define high-level guardrails or disable services/actions that should never be used. SCPs are even applicable to the root user in your AWS account.\nAWS Organizations can also be used to deploy and operate AWS Security Hub, CloudTrail and AWS Config etc at scale. These services are essential in monitoring your AWS enviroment. When AWS Organization is used properly all your accounts are managed and monitored centrally. With this you have full visibility and control over what is happening in your accounts.\nMonitor your enviroment # Although this is more a detective control it is equally important as preventive controls. Defining IAM policies is still a user responsibility and, in contrast with computers, we humans tend to be error-prone. And our attackers are creative.\nMonitoring is essential for:\nImproving hygiene by removing unused IAM entities Detecting unexpected behaviour Detecting (attempts) of attacks Staying in control your cloud security Examples of AWS services that are essential for monitoring are: AWS Config, AWS Security Hub, AWS CloudTrail, Amazon GuardDuty.\nVerdict # In conclusion, the responsibility for security in cloud computing is a shared responsibility, with cloud providers like AWS offering highly secure infrastructure, but organizations also playing a crucial role in safeguarding their assets. AWS demonstrates its commitment to security through extensive resources, talent, and experience.\nDespite the robust security measures provided by AWS, incidents can still occur due to human errors and misconfigurations, often referred to as \u0026ldquo;security in the cloud.\u0026rdquo; Organizations can enhance their cloud security by adopting several key practices:\nInvest in knowledge: Continuous learning and staying informed about AWS security features and evolving threats are fundamental to maintaining a secure cloud environment.\nAnalyze Infrastructure as Code: Scan Infrastructure as Code artifacts for compliance with best practices, using tools like cdk-nag and AWS CDK plugins.\nLeverage security tools: Utilize AWS Security Hub, AWS Trusted Advisor, and other monitoring tools to detect and prevent configuration errors and security threats.\nImplement least-privileged policies: Strictly adhere to the principle of least privilege in access control to minimize security risks and continually test and assess policies for improvements.\nEnable AWS Account-wide Controls: Use AWS Organizations to establish Service Control Policies (SCPs) that define high-level security guardrails across all AWS accounts, ensuring consistent security measures.\nMonitor Continuously: Employ monitoring and detective controls to detect unusual behavior, potential attacks, and improve overall cloud security hygiene\nBy following these practices, organizations can contribute to a secure cloud environment and effectively mitigate security risks, while benefiting from AWS\u0026rsquo;s robust security infrastructure and expertise. Remember that security is an ongoing process that requires vigilance and adaptation to evolving threats and technologies.\n","date":"September 28, 2023","externalUrl":null,"permalink":"/2023/09/improving-iam-policies/","section":"Posts","summary":"","title":"Improving IAM policies","type":"posts"},{"content":"","date":"September 28, 2023","externalUrl":null,"permalink":"/tags/security/","section":"Tags","summary":"","title":"Security","type":"tags"},{"content":" How does High-Performance Computing differs from regular computing? # Today\u0026rsquo;s server hardware is powerfull enough to execute most compute tasks. With common compute resources most (serial) computing challenges can be solved. However, some tasks are very complex and require a different approach. Think of cases that require improved speed and efficiency, ability to handle large datasets and flexibility etc. For this HPC will bring massive parallel computing, cluster and workload managers and high-performance components to the table.\nWhy HPC and cloud are a good fit? # Setting up HPC on cloud doesn\u0026rsquo;t require upfront (CAPEX) investments. No aging infrastructure. With public cloud you have access to the latest components, that will not only give a better performance but also give a better price/performance ratio. Reduced ongoing costs. Enviroments can be terminated once the job has been executed, with this running and management costs will be reduced. Resources are available on-demand, no ordering/waiting time for the deployment of resources. Security features provided in the cloud can be leveraged to allow job-level (policies, encryption, etc) security controls HPC basics # Modern HPC consists of the following components:\nHead nodes - to configure jobs and orchestrate distributed jobs Compute nodes - to do the heavy lifting GPUs - to accelerate calculations Optimized storage - either low-latency local/shared storage or blob storage (S3) Network - for consistent latency and throughput Containers - for bundeling application and dependencies Linux - scientific software is often based on Linux distributions In the next chapters this will be translated to AWS resources.\nHPC services on AWS # Compute # Technically you could design and build your own HPC cluster on AWS, it will work but you will spend time on plumbing and undifferentiated heavy lifting. The real value of cloud is to use higher-level abstraction services. AWS has two services to support your HPC workload.\nAWS Batch # AWS Batch is a fully managed service that simplifies the running of batch computing workloads across AWS services. It provides a powerful and scalable platform for executing large-scale batch jobs with minimal setup and management overhead. With AWS Batch, there\u0026rsquo;s no need to install or manage batch computing software, so you can focus your time on analyzing results and solving problems.\nKey features of AWS Batch # Efficient Resource Management: AWS Batch automatically provisions the required resources, such as compute instances and storage, based on job requirements. It dynamically scales resources up and down, ensuring optimal utilization and cost-efficiency. Job Scheduling and Prioritization: With AWS Batch, you can define job queues, priorities, and dependencies, allowing for efficient scheduling and execution of jobs. This ensures that critical tasks are prioritized and executed in the most effective manner. Integration with AWS Services: AWS Batch seamlessly integrates with other AWS services, such as Amazon S3, AWS Lambda, and Amazon DynamoDB. This enables you to build end-to-end workflows that leverage the full range of AWS capabilities for data processing, storage, and analytics. AWS Batch consists of the following components:\nJobs # A job is a unit of work that will be executed as an containerized application on AWS Fargate or Amazon EC2 in your compute enviroment. Jobs can be artifacts such as Docker container images, shell scripts or regular Linux executables.\nJob Definitions # AWS Batch job definitions specify how jobs are to be run. Each job references to a job definition. The definition includes configuration such as CPU, memory, orchestration type, multi-node parallel execution, execution timeouts, priorities, retry conditions, target nodes, storage volumes, (EKS) container configuration, secrets etc. Last but not least configure IAM roles and policies. Many of the parameters can be overwritten at runtime.\nJob Queues # Initially jobs will be submitted to a job queue. Each AWS account can host multiple queues, which for example can be used to configurate high-priority and low-prority queues, each with own characteristics. Jobs will reside in the queue until they can be schedule to run in the compute enviroment.\nCompute Environment # Compute enviroments are associated to job queue (one or more enviroments per queue and/or one or more queues per environment). Each environment can have different specifications/configuration. Preferably managed compute enviroments are used. With this AWS Batch will manage capacity and instance types. It can also manage EC2 On-Demand Instances and EC2 Spot Instances on your behalf, selects the right AMI as it uses GPU optimized AMIs for instance types with GPU acceleration. It is also possible to use your own ECS cluster by configuring unmanaged compute enviroments.\nScheduling policies # Scheduling policies can be used to configure how jobs in the queue are allocated and how resources are shared between jobs/users.\nAWS ParallelCluster # AWS ParallelCluster is an open-source cluster management tool that simplifies the creation and management of high-performance computing (HPC) clusters. It allows users to set up clusters with a few simple steps, automating the process of provisioning and configuring the necessary resources.\nKey Features of AWS ParallelCluster # Flexibility and Customization: AWS ParallelCluster supports a wide range of applications and frameworks, including popular ones like MPI, TensorFlow, and Apache Hadoop. This flexibility enables you to build tailored environments that match your specific requirements. It supports Slurm Workload Manager and AWS Batch as scheduler. Scalability: With AWS ParallelCluster, you can easily scale your clusters up or down based on workload demands. The service integrates with AWS Auto Scaling, allowing you to dynamically adjust the cluster size to handle varying computational requirements efficiently. Cost Optimization: AWS ParallelCluster helps optimize costs by allowing you to choose from various instance types and pricing options. It enables you to leverage spot instances for cost-effective computing, while ensuring high availability and fault tolerance. Slurm Workload Managers # Existing HPC might be familiar with Slurm, a highly configurable open-source workload manager. Slurm is used to manage and submit batch jobs to the cluster. It uses CLI commands like sbatch (submit jobs), salloc (obtain a job allocation) and srun (run a parallel job on cluster). Slurm also manages the compute resources for your cluster. The exact configuration is described in the cluster configuration.\nAWS ParallelCluster UI # AWS ParallelCluster UI is a web-based user interface to manage clusters. It\u0026rsquo;s built on serverless services (API Gateway / Lambda) and provides the same functionality as the CLI tool pcluster.\nHPC specific resources # Although most common AWS services can be used in a HPC system, AWS has a few complementary purpose\nAmazon FSx for Lustre # Lustre is an open-source parallel distributed file system, designed for large-scale cluster computing. It uses different layers: Management Servers, Metadata Servers, Object Storage Servers and Object Storage Targets. By decoupling tasks and providing a set of servers (up to hundred hosts) the architecture can provide terabyte-per-second scale consumed by thousands of hosts. It provides more scale than traditional network filesystems. Lustre is POSIX-compliant. AWS provides Lustre as a managed service (Amazon FSx for Lustre). Setting up a new cluster and maintaining it requires minimal effort. Data is protected with AWS KMS, encryption in transit, IAM and daily backups to S3 and has AWS Backup Support. Lustre supports Linux clients only.\nElastic Fabric Adapter # Elastic Fabric Adapter (EFA) is a type of network interface for Amazon EC2 instances. It uses OS-bypass capabilities and enhances the performance of inter-instance communication that is critical for scaling HPC and machine learning applications. It is optimized to work on the existing AWS network infrastructure and it can scale depending on application requirements. With an EFA, HPC applications use MPI or NCCL to interface with the Libfabric API. The Libfabric API bypasses the operating system kernel and communicates directly with the EFA device to put packets on the network. This reduces overhead and enables the HPC application to run more efficiently. EFA is not a replacement for normal interfaces as it has limitations, for example, it cannot be used to send to outside the subnet.\nRunning your first HPC application # For this blog post we will be using AWS ParallelCluster to manage the workload. See solution diagram below for a high-level overview.\nLike most other systems that are designed to run on AWS you first need to have some basics in place. Both AWS Batch and AWS ParallelCluster rely on EC2 and networking. This requires a VPC with public and private subnets, access to Internet (routes/NAT gateway). Import to note is that AWS ParallelCluster requires the VPC options DNS Resolution = yes and DNS Hostnames = yes to be enabled. Generate cluster configuration file # It is recommended to run the commands below in a virtualenv. You can run the commands on your local machine or on EC2 in AWS. Before you start make sure you have configured your VPC configuration and your SSH key pair. The pcluster configure command will guide you through a number of questions, such as the VPC subnet id for your head node, maximum size of your cluster, instance types etc. It is self-explanatory of you are aware of the AWS basics. For more information about installing the tools, click here.\n$ pip install -U aws-parallelcluster setuptools-rust $ export AWS_PROFILE=myprofile $ pcluster configure --config cluster-config.yaml This will generate a configuration file:\nRegion: eu-west-1 Image: Os: alinux2 HeadNode: InstanceType: t2.micro Networking: SubnetId: subnet-08c245b78a82c3924 Ssh: KeyName: steyn-mbp Scheduling: Scheduler: slurm SlurmQueues: - Name: queue1 ComputeResources: - Name: t2micro Instances: - InstanceType: t2.micro MinCount: 0 MaxCount: 10 Networking: SubnetIds: - subnet-0b864625631add41e - subnet-0053337e5ceeea8f3 - subnet-035761b17a12851ac The configuration above is a description of your AWS ParallelCluster and input for your new cluster. For production workloads you need to consider configuring the SharedStorage section.\nCreating the cluster # Now create the cluster\n$ export AWS_PROFILE=myprofile $ pcluster create-cluster --cluster-name steyn --cluster-configuration cluster-config.yaml With the command list-clusters you can view the progress:\n$ export AWS_PROFILE=myprofile $ pcluster list-clusters { \u0026#34;clusters\u0026#34;: [ { \u0026#34;clusterName\u0026#34;: \u0026#34;steyn\u0026#34;, \u0026#34;cloudformationStackStatus\u0026#34;: \u0026#34;CREATE_IN_PROGRESS\u0026#34;, \u0026#34;cloudformationStackArn\u0026#34;: \u0026#34;arn:aws:cloudformation:eu-west-1:123456789012:stack/steyn/173732e0-16b1-11ee-a432-0ab77f79d7f7\u0026#34;, \u0026#34;region\u0026#34;: \u0026#34;eu-west-1\u0026#34;, \u0026#34;version\u0026#34;: \u0026#34;3.6.0\u0026#34;, \u0026#34;clusterStatus\u0026#34;: \u0026#34;CREATE_IN_PROGRESS\u0026#34;, \u0026#34;scheduler\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;slurm\u0026#34; } } ] } Now wait for the clusterStatus to change to CREATE_COMPLETED. This might take a while. Once completed verify that you can signin to your head node:\n$ export AWS_PROFILE=myprofile $ pcluster ssh --cluster-name steyn [ec2-user@ip-10-0-0-149 ~]$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST queue1* up infinite 10 idle~ queue1-dy-t2micro-[1-10] Your cluster is now up \u0026amp; running and waiting for jobs to be submitted to the queue. In your EC2 console you now see one active node, the head node.\nSubmitting your first job # Let\u0026rsquo;s start with a simple job to verify that everything is working. Create a simple script named hellojob.sh.\n$ cat \u0026gt; hellojob.sh #!/bin/bash sleep 30 echo \u0026#34;Hello World from $(hostname)\u0026#34; Now submit this script to the queue:\n$ sbatch hellojob.sh $ sinfo Now AWS ParallelCluster will create a compute node for this job. It might take a moment before the compute node is pulling jobs from the queue.\nYou can view the queue by running squeue:\n$ squeue JOBID PARTITION NAME USER ST\tTIME NODES NODELIST(REASON) 1 queue1 hellojob ec2-user R\t0:08\t1 queue1-dy-t2micro-2 This job, with jobid 1, is in running state for 0:08 seconds. The column ST is showing the state. When completed it will write the output to the current working directory in a file named after the job id: slurm-1.out.\nScheduling parallel jobs # There are multiple strategies to schedule parallel jobs, the most convenient way is to use job arrays in sbatch. In the next example we will schedule 64 jobs in one array.\n$ sbatch --array=0-64 hellojob.sh $ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 20_[40-64] queue1 hellojob ec2-user PD 0:00 1 (Resources) 20_30 queue1 hellojob ec2-user R 0:03 1 queue1-dy-t2micro-1 20_31 queue1 hellojob ec2-user R 0:03 1 queue1-dy-t2micro-7 20_32 queue1 hellojob ec2-user R 0:03 1 queue1-dy-t2micro-2 20_33 queue1 hellojob ec2-user R 0:03 1 queue1-dy-t2micro-8 20_34 queue1 hellojob ec2-user R 0:03 1 queue1-dy-t2micro-6 20_35 queue1 hellojob ec2-user R 0:03 1 queue1-dy-t2micro-9 20_36 queue1 hellojob ec2-user R 0:03 1 queue1-dy-t2micro-3 20_37 queue1 hellojob ec2-user R 0:03 1 queue1-dy-t2micro-4 20_38 queue1 hellojob ec2-user R 0:03 1 queue1-dy-t2micro-5 20_39 queue1 hellojob ec2-user R 0:03 1 queue1-dy-t2micro-10 The results will be written again to your local working directory, but with a different name. The jobid will now be appended by the array index.\nls | tail slurm-20_5.out slurm-20_60.out slurm-20_61.out slurm-20_62.out slurm-20_63.out slurm-20_64.out slurm-20_6.out slurm-20_7.out slurm-20_8.out slurm-20_9.out Cleaning up # Compute nodes will be terminated by default after 10 minutes of idle time. Watch the EC2 console to check the status. After 10 minutes only the head node should be running.\nNow terminate the cluster. These commands have to be executed from your pcluster enviroment.\n$ export AWS_PROFILE=myprofile $ pcluster delete-cluster --cluster-name steyn $ pcluster list-clusters { \u0026#34;clusters\u0026#34;: [ { \u0026#34;clusterName\u0026#34;: \u0026#34;steyn\u0026#34;, \u0026#34;cloudformationStackStatus\u0026#34;: \u0026#34;DELETE_IN_PROGRESS\u0026#34;, \u0026#34;cloudformationStackArn\u0026#34;: \u0026#34;arn:aws:cloudformation:eu-west-1:123456789012:stack/steyn/173732e0-16b1-11ee-a432-0ab77f79d7f7\u0026#34;, \u0026#34;region\u0026#34;: \u0026#34;eu-west-1\u0026#34;, \u0026#34;version\u0026#34;: \u0026#34;3.6.0\u0026#34;, \u0026#34;clusterStatus\u0026#34;: \u0026#34;DELETE_IN_PROGRESS\u0026#34;, \u0026#34;scheduler\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;slurm\u0026#34; } } ] } Repeat the last command until the status changes. The output will be something like:\n$ export AWS_PROFILE=myprofile $ pcluster list-clusters { \u0026#34;clusters\u0026#34;: [] } Managing the cluster # For more convenient access to the cluster, jobs and queues it is recommended to install AWS ParallelCluster UI. This is a serverless web UI that mirrors the pcluster functionality.\nOther resources # The Well-Architected Framework is a strong foundation when designing and building systems on AWS. This framework provides guidance and the possibility for self-assessment. There is a dedicated lens (focus) for HPC workloads, see here.\n","date":"June 29, 2023","externalUrl":null,"permalink":"/2023/06/high-performance-computing-on-aws/","section":"Posts","summary":"","title":"High-performance computing on AWS","type":"posts"},{"content":"","date":"June 29, 2023","externalUrl":null,"permalink":"/tags/hpc/","section":"Tags","summary":"","title":"HPC","type":"tags"},{"content":"","date":"April 7, 2023","externalUrl":null,"permalink":"/tags/amazon-linux/","section":"Tags","summary":"","title":"Amazon Linux","type":"tags"},{"content":"Earlier this month AWS has released Amazon Linux 2023, in short AL2023. Amazon Linux is a Linux distribution maintained by AWS. It is no suprise that the main purpose of this distribution is an optimized experience for running on AWS as it comes with features and intergration with AWS-specific tools. Besides an Amazon Linux Image the distribution is also offered as an container image.\nAL2023 is the last generation of Amazon Linux, the successor of the initial version Amazon Linux and Amazon Linux 2. With the release of AL2023 AWS will release a new major version every two years. According to the cadence we can expect AL2025, AL2027 and so on. With every upcoming release the current release will go from Standard to Maintanance support (security patches only) to be retired after 3 years. This means that a major version has a supported lifespan of 5 years, which is fairly long in cloud. The new versioning and cadence is new for those that are familiar with the versions Amazon Linux and Amazon Linux 2. The previous versions were a rolling release, while from AL2023 (or in hindsight AL2) onwards a new major version will be released every 2 years.\nIn this blog the most significant updates will be highlighted.\nWhat\u0026rsquo;s new in Amazon Linux 2023? # New base and kernels # AL2023 is based on Fedora Linux / CentOS Stream. Previous versions were based on Red Hat Enterprise Linux. Amazon customized the available packages and based the sources directly on stable kernels from kernel.org.\nRelease cycle and support # Starting with this version Amazon will release a new major version every two years. Next version will be launched in 2025 and will be named Amazon Linux 2025. Minor releases will be released every quarter and will include security enhancements, bug fixes etc. The lifecycle for each version (e.g. 2023.0, 2023.1, etc.) will span five years: two years of standard support followed by three years of maintenance support. Also repositories can be locked which will give customers control over updates and to lock major and minor versions if needed.\nSecurity # Compared to previous versions security made a big step. AL2023 sets the bar higher with features such as new developments on pre-configured SELinux and updated policies, live kernel patching, updated crypto policies, secure-boot, kernel module signing etc. For cloud-users the live kernel patching is very useful. During the initial boot of an EC2 instance all pending security patches are installed automatically. Without live kernel patching a reboot was required before the patches were effective, which isn\u0026rsquo;t desired nor very useful in cloud setup with features such as autoscaling. With live kernel patches the security fixes will be applied without a reboot. Also IMDSv2 is now enabled by default.\nSSM parameters # Amazon Linux 2023 images can be referenced via SSM parameters. The name of the parameter is /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-6.1-arm64. The alias part can be replaced with one of the available aliasses:\nal2023-ami-kernel-6.1-arm64 for arm64 architecture al2023-ami-minimal-kernel-6.1-arm64 for arm64 architecture (minimal AMI) al2023-ami-kernel-6.1-x86_64 for x86_64 architecture al2023-ami-minimal-kernel-6.1-x86_64 for x86_64 architecture (minimal AMI) Container image # Amazon Linux 2023 images are available on ECR (most efficient when you are running on AWS) and on Docker Hub. The latest version is tagged with 2023. Subsequent versions will have the minor version as suffix.\nThe image can be pulled from Docker Hub:\n$ docker pull amazonlinux:2023 Or pull from ECR:\n$ docker pull public.ecr.aws/amazonlinux/amazonlinux:2023 Compared with Amazon Linux 2 # AWS published a comprensive comparision between Amazon Linux 2 and Amazon Linux 2023. You can find it here.\n","date":"April 7, 2023","externalUrl":null,"permalink":"/2023/04/amazon-linux-2023/","section":"Posts","summary":"","title":"Amazon Linux 2023","type":"posts"},{"content":"","date":"April 7, 2023","externalUrl":null,"permalink":"/tags/amazon-linux-2023/","section":"Tags","summary":"","title":"Amazon Linux 2023","type":"tags"},{"content":"","date":"April 7, 2023","externalUrl":null,"permalink":"/tags/ec2/","section":"Tags","summary":"","title":"EC2","type":"tags"},{"content":"Sustainability is an important topic. This is not without a reason, since sustainability is key in preserving our planet. The combined world of sustainability and cloud is getting more and more traction. The investements pay off: we get more capabilities to work with and now it\u0026rsquo;s time to start harvesting. The urgency is also there. ICT, including cloud, is responsible for 3% of the global greenhouse gasses emissions. We, as consumers of cloud resource, easily have impact since making changes to your cloud environment can be done easily (e.g. no hardware to write off).\nThis blog is written to provide some practical tips to get started on sustainability. Before we start it\u0026rsquo;s important to become familiar with some terminology.\nScopes # While talking about it won\u0026rsquo;t take long before anyone will start about scope 1, 2 or 3. The brief definition of the scopes:\nScope 1: All direct emissions.\nScope 2: Indirect emissions associated with, for example, the generation of electricity which has been purchased for your own consumption.\nScope 3: All other indirect emissions from activities of an organization from sources it doesn’t control. AWS examples include emissions related to data center construction, and the manufacture and transportation of IT hardware deployed in data centers.\nGreenhouse gas # Any gas that absorbs infrared radiation in the atmosphere. CHG is the abbreviation for greenhouse gas.\nMetric tons of carbon dioxide equivalents (MTCO2e) # The unit of measurement for calculating impact. The unit \u0026ldquo;CO2e\u0026rdquo; represents an amount of a GHG whose atmospheric impact has been standardized to that of one unit mass of carbon dioxide (CO2), based on the global warming potential (GWP) of the gas.\nLet\u0026rsquo;s dive into the details.\n1. If you can\u0026rsquo;t measure it, you can\u0026rsquo;t improve it. # Last year AWS launched the AWS Customer Carbon Footprint Tool. This tool is available for AWS customers at no cost. It can be launched via the AWS console and provides insights in scope 1 and 2 CHG emissions, expressed in MTCO2e. These insights can be used to measure and report via data visualizations. Drill-downs can be made based on summary, by geography, by service, by statistics over time and by the path to 100% renewable energy.\nFirst start is to zoom in on the summary, geography (note: these don not map 1:1 with AWS Regions) and finally the AWS services. Once you understand where you can improve you can start planning and applying changes. The progress of your improvements can be tracked by viewing the statistics over time. Side note here: AWS currently updates the data/reports after three months. So measuring impact requires patience. Rumor has it that this will be improved (anytime soon ofcourse).\n2. Reduce waste and usage # For directly reducing emissions it\u0026rsquo;s important to start reducing waste and usage. Remove any unused and right-size underutilized resources. Less powerful underlying hardware requires less power and cooling to run and therefore will lower your emissions immediately.\n3. Culture # For long-term improvements companies need to change their culture and add sustainability as a non-function requirement. With adding this to the list of requirements the goal can be met on the long-term and will scale across the entire company.\nAlso establish a GreenOps practice. My collegue Mudit Gupta wrote a blog about GreenOps.\n4. Architecture # Region selection # AWS regions are running datacenters on local grids. Each grid is different and not all grids are equally using renewable energy. Taking sustainability into account as non-functional requirement for region selection can drastically reduce your emissions. AWS Architecture Blog has a blog post on How to select a Region for your workload based on sustainability goals.\nAlign SLA with sustainability goals # More performance always feels better. The more responsive your website is the better the experience. But often the last bits of performance optimizations come with a big penalty when it comes to resource consumption. Sometimes it can be worth to consider optimizing performance instead of maximizing performance.\nSoftware architecture # Asynchronuous and decoupled applications allow more precise scaling and reduce idle time. This will reduce emissions.\nInstance types # Select the right instance types by not only selecting on dimensions (cores, memory, etc.), but also consider different architectures. ARM-based processor such as AWS\u0026rsquo; Graviton allow better performance per watt of energy use compared to x86 processors. Doing this during projectime will save you from rebuilding your application (ARM and x86 are not compatible and requires recompiling).\nStorage patterns # Use storage solutions that best supports your workloads. SSD drives are consuming more energy and are usefull for frequently accessed data, but less efficient for data archives. Use magnetic storage for archival-class storage.\nUse Trusted Advisor to identify under-utilized instances. The tool provides insights and recommendations that can be followed up with much effort.\nProgramming languages # Not all programming languages are written to be compute effective. Choosing the right language can reduce the usage of compute resources. The study Energy Efficiency across Programming Languages is a nice read and giving more information about the most popular programming languages and their memory and energy consumption. Please be aware that selecting the right language alone is not enough to make your software architecture efficient. It is the combination of the language and the overall design. Even with the most efficient languages one can write very inefficient software.\n5. Cost-efficiency # Sustainability is on the intersection with FinOps. They mutually support each other and have a lot of the same thinking in common. Improving costs will most likely improve your cost-effiency and visa versa. For example: reducing waste and usage will reduces cost, but obviously also improve sustainability. Emissions for something you didn\u0026rsquo;t needed in the first place are always 100% useless and at the same time right-sizing will bring the emissions closer to the minimum you can do.\nThis blog is just a brief introduction to the topic and far from compleet. So expect more to come! If you have any questions, feel free to reach out.\nMore sources # Sustainability Pillar - AWS Well-Architected Framework\nFinOps Foundation\n","date":"March 20, 2023","externalUrl":null,"permalink":"/2023/03/getting-started-with-sustainability/","section":"Posts","summary":"","title":"Getting started with sustainability","type":"posts"},{"content":"","date":"March 20, 2023","externalUrl":null,"permalink":"/tags/greenops/","section":"Tags","summary":"","title":"GreenOps","type":"tags"},{"content":"","date":"March 20, 2023","externalUrl":null,"permalink":"/tags/sustainability/","section":"Tags","summary":"","title":"Sustainability","type":"tags"},{"content":"","date":"July 26, 2022","externalUrl":null,"permalink":"/tags/s3/","section":"Tags","summary":"","title":"S3","type":"tags"},{"content":" In general the cloud object store Amazon S3 is pretty straightforward to use, but mistakes are easily made. The service itself is proven to be secure (\u0026ldquo;security of the cloud\u0026rdquo;), reliable and performant. However, misconfiguration by the owner of the S3 buckets can give a totally different experience (\u0026ldquo;security in the cloud\u0026rdquo;). In this blog the five most common mistakes will be addressed as a learning experience.\n1. Leaky S3 buckets # Never configure your S3 bucket to allow direct public access to your S3 buckets. The best-known breaches for applications running on AWS were caused by misconfigured S3 buckets (known cases: Verizon, Securitas, Reindeer, etc). Terabytes of confidential data was exposed by accident, just because of one misconfigured configuration switch. This has led to serious damage while it is easy to avoid. The same applies to wrongly configured resource policies and ACLs, e.g. when using wildcards to grant access.\nDo: enable the public access block setting on account level,preventing any buckets from becoming publicly exposed. If you do need to expose data hosted in S3 publicly: configure the S3 bucket as the origin for a CloudFront-distribution, configure origin access identity and only giving access to the bucket for your CloudFront-distribution. This way you will have full control over the security controls such as TLS-configuration, logging, DDoS-protection and WAF. It will also give you superb performance. And last but not least use Cloud Security Posture Management to monitor the configuration state of your AWS envirnoment.\n2. Not using encryption # As a best-practice all data shall be encrypted at rest and in transit. This principle also applies to data stored in S3. There is no excuse to not encrypt your data, since S3 is providing transparent encryption. Unencrypted data is sensitive to interception and unauthorized modifications.\nDo: protect your data using server-side encryption. Preferably use Server-Side Encryption with AWS KMS keys (SSE-KMS) and use your customer managed KMS keys. This will give you full control over the encryption key (resource policy, key rotation, etc). For cost reduction or performance increase, consider configuring the S3 Bucket Key to reduce the number of calls to the KMS-service. For more info read the blog Reduce AWS Costs with S3 Bucket Keys by coworker Vikas Bange.\n3. High costs and poor performance # Poorly designed access patterns can lead to high costs and low performance. The most significant cost metric for S3 is retrieving data. For example when synchronizing data to S3 it is best-practice to use metadata to determine if objects have to be updated, instead of downloading the entire object to compare the source and destination.\nDo: use metadata when synchronizing data, use prefixes, implement S3 gateway endpoints, avoid overwriting existing data to S3, configure the right storage classes, use the region closest to the consumers etc. More can be found on the Optimizing performance Guidelines page.\n4. Using bucket replication as backup # Replication should not be used for disaster recovery. Replication will not protect against accidentally deleting files, file corruption, ransomware, etc. All actions on the source bucket will be replicated to the destination bucket. This makes it impossible to recover after an incident.\nDo: Use AWS Backup to backup non-repreducible data. In return you get typical disaster recovery capabilities such as Point-in-Time Recovery, backup plans and data protection with encryption.\n5. Not knowing what data is stored in S3 # When your organization grows it will be more difficult to know the classification of objects stored in S3. Teams can easily make mistakes and for example, upload sensitive data to a bucket that has not been configured as a bucket for that data classification. At a large scale it is difficult and time-consuming to exactly know everything.\nDo: Use Amazon Macie to discover sensitive data at scale. As a big bonus, Amazon Macie will also report on your security posture (misconfigurations).\nThe points above are just examples of common mistakes and not intended as a guideline for proper use of Amazon S3. The intent of this blog is to make you aware of the common pitfalls on S3.\n","date":"July 26, 2022","externalUrl":null,"permalink":"/2022/07/the-five-common-mistakes-on-s3/","section":"Posts","summary":"","title":"The five common mistakes on S3","type":"posts"},{"content":"","date":"March 15, 2022","externalUrl":null,"permalink":"/tags/ci/cd/","section":"Tags","summary":"","title":"CI/CD","type":"tags"},{"content":"We do encounter a mix of DevOps tools being used in cloud project. For reasons we do prefer to work with AWS native tools and services. One of the reasons (but certainly not limited to that) for that opinion is that native tools provide seamless integration with the fundamentals of AWS itself. Think of tight integration with AWS Identity and Access Management (IAM) or AWS CloudTrail. In AWS it\u0026rsquo;s common to assign roles to resources. For exampl,e steps in CodeBuild (\u0026lsquo;build runners\u0026rsquo; in CodePipeline) have an IAM role with least-privileged policies assigned to grant access to the platform. Roles are using short-lived credentials and are provided natively by the platform. The time to live varies per service, but mostly anything between 15 minutes and 6 hours. Way shorter than static credentials, which are most likely to be rotated every 90 days in theory. The reality is that rotating static credentials manually is a big hassle and (almost) nobody does it.\nCredentials are a recurring topic in third party integrations. Most vendors have support for AWS Access key ID and Secret access key, which are static credentials that require manual rotation. Those credentials are typically stored in a vault and used by the 3rd party tool when authenticating API calls to AWS. Every 90 days an engineer has to create new credentials, update the configuration of the application and then finally deactivate and delete the previous set of credentials.\nBut there is hope. More and more vendors are adding OpenID Connect (OIDC) support to their software. OIDC is an authentication layer on top of the OAuth 2.0 framework. It is used to authenticate identities with an external identity provider. AWS has support for OIDC using external identity providers and assuming IAM roles with temporary credentials.\nGitLab # For this blog we take GitLab as an example. GitLab is a complete DevOps platform, including CI/CD capabilities. It is in use by many companies, has proper AWS support and can be consumed as a SaaS-service. GitLab has OIDC support and can act as the identity provider. In this section I will describe how GitLab can be used in conjunction with AWS, without the usage of static credentials.\nIt is good to have an understanding of how things work, before diving into the configuration. In this example we want to use GitLab for Continuous Delivery, so we want to deploy and provision AWS infrastructure from a pipeline running in GitLab.\nPipeline is triggered by a change in the version control repository. Job is created and executed by a GitLab-runner instance. The runner is provisioned with context, including a JSON Web Token (JWT). This authentication token is used to call AssumeRoleWithWebIdentity on AWS and in exchange AWS provides a set of temporary credentials. Uses the temporary credentials to access AWS services and provision resources. After provisioning is done the runner will be terminated and ultimately the temporary credentials will expire (default 3600 seconds). In this setup GitLab acts as the Identity Provider, GitLab Runners as the client and AWS as the app.\nConfigure AWS # The first step is to let AWS know that we have an external identity provider. This is done by creating a new Identity Provider in the IAM console. Make sure to select Open ID Connect as the Provider type. The Provider URL is the URL of your GitLab instance. For SaaS hosted GitLab the URL is https://gitlab.com. For self-hosted versions use https://gitlab.yourdomain.tld (must start with https:// and not include a trailing \u0026ldquo;/\u0026rdquo;). Click on Get thumbprint to fetch the thumbprint of the provider\u0026rsquo;s certificate. Finally specify the Audience or client ID. This is again the same value as used for the Provider URL. Click on Add provider to create the new provider.\nNow that the provider has been created the role can be created. Go to the create role section in the IAM console and create a new role with Web identity as the trusted entity. Select the Identity provider and Audience created in the previous step. Click Next and create a policy. Copy/paste the policy from the JSON snippet below: { \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Sid\u0026#34;: \u0026#34;GitLabPolicy\u0026#34;, \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: \u0026#34;sts:GetCallerIdentity\u0026#34;, \u0026#34;Resource\u0026#34;: \u0026#34;*\u0026#34; } ] }\nClick next until you see the \u0026ldquo;Review policy\u0026rdquo; page. Now enter a name, for example gitlab-policy and finish by clicking Create policy. Now go back to the tab/window where we started creating the role. Hit the refresh button and select the policy created in the previous step. Click Next. Now provide a name for the role and click Create role. The role will be created. Now search for the role on the role overview page. Open the role and copy the ARN. This should be in the syntax of arn:aws:iam::\u0026lt;account-id\u0026gt;:role/\u0026lt;name of the role\u0026gt;.\nNow we\u0026rsquo;ve created a role, but basically any GitLab customer would be able to assume the role. We need to lock it down by adding a condition in the assume-role policy document. Go to the tab Trust relationships and replace the existing condition with:\n{ \u0026#34;Condition\u0026#34;: { \u0026#34;StringEquals\u0026#34;: { \u0026#34;gitlab.com:sub\u0026#34;: \u0026#34;project_path:mygroup/myproject:ref_type:branch:ref:main\u0026#34; } } } The final trust policy should look like: { \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Principal\u0026#34;: { \u0026#34;Federated\u0026#34;: \u0026#34;arn:aws:iam::\u0026lt;account-id\u0026gt;:oidc-provider/gitlab.com\u0026#34; }, \u0026#34;Action\u0026#34;: \u0026#34;sts:AssumeRoleWithWebIdentity\u0026#34;, \u0026#34;Condition\u0026#34;: { \u0026#34;StringEquals\u0026#34;: { \u0026#34;www.gitlab.com:sub\u0026#34;: \u0026#34;project_path:mygroup/myproject:ref_type:branch:ref:main\u0026#34; } } } ] } More fields to be used in the condition can be found here.\nConfigure GitLab # After configuring AWS you can now configure your GitLab project to retrieve temporary credentials from AWS Security Token Service (STS). First add a new variable to your project. The name of this variable is ROLE_ARN and the value should be the exact ARN from the role created previously. The CI_JOB_JWT_V2 is predefined by GitLab and contains the current JWT token.\nNow add the following step to your jobs in .gitlab.yml: assume_role: script: - \u0026gt; STS=($(aws sts assume-role-with-web-identity --role-arn ${ROLE_ARN} --role-session-name \u0026#34;gitlab-${CI_PROJECT_ID}-${CI_PIPELINE_ID}\u0026#34; --web-identity-token ${CI_JOB_JWT_V2} --query \u0026#39;Credentials.[AccessKeyId,SecretAccessKey,SessionToken]\u0026#39; --output text)) - export AWS_ACCESS_KEY_ID=\u0026#34;${STS[0]}\u0026#34; - export AWS_SECRET_ACCESS_KEY=\u0026#34;${STS[1]}\u0026#34; - export AWS_SESSION_TOKEN=\u0026#34;${STS[2]}\u0026#34; - aws sts get-caller-identity\nAfter the steps above you can trigger your commands to provision AWS resources, for example cdk deploy or terraform apply. You might want to tweak the optional \u0026ndash;duration-seconds parameter to match the timeout of your job in GitLab. Most CD jobs on AWS will be finished within 3600, but sometimes a longer expiration period is required. In addition to static credentials a third key/value pair named AWS_SESSION_TOKEN is added for dynamic credentials. It is included in the example above. But do not forget this variable. For debugging purposes the command - aws sts get-caller-identity is added. You can remove this if you like. In a real-world scenario you probably need to extend the policy. Now trigger your pipeline by pushing a change to your version control repository. The get-caller-identity should return a result like: { \u0026#34;UserId\u0026#34;: \u0026#34;AROA\u0026lt;user-id\u0026gt;\u0026#34;, \u0026#34;Account\u0026#34;: \u0026#34;\u0026lt;account-id\u0026gt;\u0026#34;, \u0026#34;Arn\u0026#34;: \u0026#34;arn:aws:sts::\u0026lt;account-id\u0026gt;:assumed-role/\u0026lt;role-name\u0026gt;/\u0026lt;gitlab-project-id\u0026gt;-\u0026lt;gitlab-pipeline-id\u0026gt;\u0026#34; }\nYou\u0026rsquo;ve now successfully established an identity flow between GitLab and AWS using short-lived credentials. This way you don\u0026rsquo;t longer have to secure and rotate static AWS credentials.\nIn the steps above we\u0026rsquo;ve manually created the resources to get a good understanding of all the required steps. In production environments you want to provision the resources using Infrastructure as Code. Putting it all together in AWS CDK (v2): import { App, Stack, StackProps, CfnOutput } from \u0026#39;aws-cdk-lib\u0026#39;; import { Construct } from \u0026#39;constructs\u0026#39;; import { aws_iam as iam } from \u0026#39;aws-cdk-lib\u0026#39;; interface GitLabOidcProps extends StackProps { gitLabURI: string; gitLabProjectPath: string; }; export class GitLabOidc extends Stack { constructor(scope: Construct, id: string, props: GitLabOidcProps) { super(scope, id, props); // Configure OIDC provider // See https://docs.gitlab.com/ee/ci/cloud_services/aws/#add-the-identity-provider const oidcProvider = new iam.OpenIdConnectProvider(this, \u0026#39;Provider\u0026#39;, { url: props.gitLabURI, clientIds: [ props.gitLabURI, ], }); // Create the role const role = new iam.Role(this, \u0026#39;Role\u0026#39;, { assumedBy: new iam.OpenIdConnectPrincipal(oidcProvider).withConditions({ \u0026#39;StringEquals\u0026#39;: { [ props.gitLabURI.substring(8) + \u0026#39;:sub\u0026#39; ]: \u0026#39;project_path:\u0026#39; + props.gitLabProjectPath, } }), description: \u0026#39;GitLab pipeline role\u0026#39;, }); // Create the policy to attach to the role // Note: this is an example, needs const policy = new iam.ManagedPolicy(this, \u0026#39;Policy\u0026#39;, { statements: [ new iam.PolicyStatement({ sid: \u0026#39;GitLabPolicy\u0026#39;, actions: [ \u0026#39;sts:GetCallerIdentity\u0026#39;, ], resources: [ \u0026#39;*\u0026#39;, ], }), ], }); role.addManagedPolicy(policy); new CfnOutput(this, \u0026#39;pipelineRoleArn\u0026#39;, { value: role.roleArn, description: \u0026#39;Role ARN to assume by pipeline\u0026#39;, }); } } // for development, use account/region from cdk cli const devEnv = { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION, }; const app = new App(); new GitLabOidc(app, \u0026#39;MyGitLabIntegration\u0026#39;, { env: devEnv, gitLabURI: \u0026#39;https://gitlab.com\u0026#39;, gitLabProjectPath: \u0026#39;mygroup/myproject:ref_type:branch:ref:main\u0026#39;, });\nThis post is about configuring GitLab for deployments to AWS, but something similar can be done for other DevOps-tools. For example using GitHub and Azure DevOps.\n","date":"March 15, 2022","externalUrl":null,"permalink":"/2022/03/configure-openid-connect-for-gitlab-and-aws/","section":"Posts","summary":"","title":"Configure OpenID Connect for GitLab and AWS","type":"posts"},{"content":"","date":"March 15, 2022","externalUrl":null,"permalink":"/tags/gitlab/","section":"Tags","summary":"","title":"GitLab","type":"tags"},{"content":"The world is on fire. We\u0026rsquo;re heading towards - or exactly it already is - a humanitarian disaster in Ukraine. We\u0026rsquo;ve all seen the heartbreaking footage from the war. Thousands of homeless people fighting and fearing for their lives. I\u0026rsquo;ve written this blog to help. I do know that a large number of government websites are hosted on AWS.\nThe conflict between Ukraine and Russia is expanding with cyber warfare. There is fighting on the ground, but also online. Government websites are taken down, broadcasting companies are being hacked, etc. Everything is done to manipulate and disrupt communication technologies. Although most attacks will be sophisticated, there are some simple measurements in AWS you can take to make it more difficult for attackers. It will not be 100% waterproof. Hackers often use Tor-networks and Command and Control-machines. But every bit helps.\nI can\u0026rsquo;t really support the people in Ukraine, but I can try to help with bringing in AWS knowledge and hope that the people will keep access to the information they need.\nGeographic restrictions # Communication between clients-and-servers is done based on IP addresses. Clients are connecting to a server using a TCP-session. The server can see the client\u0026rsquo;s IP address and uses that as the source address. The IP address is just a number and not really useful if you want to restrict access to your resources by excluding countries or continents. The IP address itself will not provide you sufficient context to filter traffic.\nIP addresses are part of a bigger network, divided into blocks. Blocks are owned by providers (ISPs) and registered at a Regional Internet Registry (RIR). World-wide five RIRs are managing and allocating IP addresses:\nThe African Network Information Center (AFRINIC) serves Africa. The American Registry for Internet Numbers (ARIN) serves Antarctica, Canada, parts of the Caribbean, and the United States. The Asia-Pacific Network Information Centre (APNIC) serves East Asia, Oceania, South Asia, and Southeast Asia. The Latin America and Caribbean Network Information Centre (LACNIC) serves most of the Caribbean and all of Latin America. The Réseaux IP Européens Network Coordination Centre (RIPE NCC) serves Europe, Central Asia, Russia, and West Asia. Netblocks owned by ISPs (and other parties) are registered at the local RIR. The RIRs provide services to link netblocks to their owners, via whois services. This information holds the company data of the ISP, including the country. With this information you can more or less determine from which country a connection is initiated. This information is not always 100% accurate, for various reasons. Common reason is that ISPs can operate in multiple countries and are running a global infrastructure. For geo services this data is enriched with data based on feedback (deducting information from mobile devices, data mining and routing information for example), to provide a reliable source of geo information.\nThis information is made available as a database and can be used for geolocation technology, such as websites and CDNs.\nAmazon CloudFront # Amazon CloudFront is a content delivery network (CDN) operated by AWS. It is used to serve data from globally-distributed networks, locally for end-users. This provides access speed (low proximity), but also scalability. In 2013 AWS added the Geo Restriction feature. This feature can be used to restrict access to content based on the geographical location of the viewer. It can be used in three modes: no restrictions, allow list (implicit deny, explicit allows) and deny list (implicit allow, explicit denies). For the two list modes, country codes (NL for the Netherlands, US for the United States, etc.) can be provided to control traffic from the countries. CloudFront then evaluates each request against the list. Denied countries will have an HTTP status code 403 (Forbidden) response. Traffic from allowed countries will be passed to the origin (and ultimately serve the content (if no other deny rules have been hit).\nFor the use-case in the introduction the block list type will give the best result. In this case we want the website to be accessible for everyone, except from a number of countries. We use Russia and their best known allies as entries in the deny list:\nCountry code RU: Russian Federation Country code BY: Belarus Country code CN: China If you are using AWS CDK and TypeScript, add the following line to your CloudFront distribution object: geoRestriction: cloudfront.GeoRestriction.denylist(\u0026#39;RU\u0026#39;, \u0026#39;BY\u0026#39;, \u0026#39;CN\u0026#39;),\nIf you use native CloudFormation add the following to your AWS::CloudFront::Distribution resource: Restrictions: GeoRestriction: Locations: - RU - BY - CN RestrictionType: blacklist\nAWS WAF # AWS Web Application Firewall (WAF) is a web application firewall. It can inspect content, but it can also be used to apply rules based on session context. Such as source IP address. AWS WAF integrates with CloudFront or with regional services such as Application Load Balancer (ALB), an Amazon API Gateway REST API, or an AWS AppSync GraphQL API.\nFor this use-case we use the geographic match rule statement feature. With this feature you can write geo-based rules to be included in a WebACL. With this type of rule statement you configure a list of country codes which you want to deny or allow (depending on the selected default action). For now we want to allow all countries, except three specific countries. The best choice is default action allow and include a geoMatchStatement to block access from the three countries.\nFor WAF it is recommended to, in addition to blocking countries, block anonymous networks (networks that are known to be part of) and sources with a bad reputation. AWS provides manage rule lists that are useful for our case:\nAWSManagedIPReputationList: Inspects for a list of IP addresses that have been identified as bots by Amazon threat intelligence. AnonymousIPList: Inspects for a list of IP addresses of sources known to anonymize client information, like TOR nodes, temporary proxies, and other masking services. Now put the custom rules, two managed rule lists together in a WebACL using AWS CDK:\nImport WAFv2 module: import * as wafv2 from \u0026#39;aws-cdk-lib/aws-wafv2\u0026#39;;\nCreate WebACL, combined with our own custom country block rule and two AWS managed rule groups: const webacl = new wafv2.CfnWebACL(this, \u0026#39;GeoRestriction\u0026#39;, { description: \u0026#39;Block suspicious traffic\u0026#39;, defaultAction: { allow: {}, }, scope: \u0026#39;CLOUDFRONT\u0026#39;, visibilityConfig: { sampledRequestsEnabled: true, cloudWatchMetricsEnabled: true, metricName: \u0026#39;GeoRestrictionWebACL\u0026#39;, }, rules: [{ priority: 0, action: { block: {}, }, visibilityConfig: { sampledRequestsEnabled: true, cloudWatchMetricsEnabled: true, metricName: \u0026#39;GeoRestrictionCountryRule\u0026#39;, }, name: \u0026#39;GeoRestrictionCountry\u0026#39;, statement: { geoMatchStatement: { countryCodes: [ \u0026#39;RU\u0026#39;, \u0026#39;BY\u0026#39;, \u0026#39;CN\u0026#39;, ], }, }, }, { priority: 1, overrideAction: { none: {}, }, visibilityConfig: { sampledRequestsEnabled: true, cloudWatchMetricsEnabled: true, metricName: \u0026#39;AWS-AWSManagedRulesAmazonIpReputationList\u0026#39;, }, name: \u0026#39;AWS-AWSManagedRulesAmazonIpReputationList\u0026#39;, statement: { managedRuleGroupStatement: { vendorName: \u0026#39;AWS\u0026#39;, name: \u0026#39;AWSManagedRulesAmazonIpReputationList\u0026#39;, }, }, }, { priority: 2, overrideAction: { none: {}, }, visibilityConfig: { sampledRequestsEnabled: true, cloudWatchMetricsEnabled: true, metricName: \u0026#39;AWS-AWSManagedRulesAnonymousIpList\u0026#39;, }, name: \u0026#39;AWS-AWSManagedRulesAnonymousIpList\u0026#39;, statement: { managedRuleGroupStatement: { vendorName: \u0026#39;AWS\u0026#39;, name: \u0026#39;AWSManagedRulesAnonymousIpList\u0026#39;, }, }, },], });\nOr in plain CloudFormation: GeoRestriction: Properties: DefaultAction: Allow: {} Description: \u0026#34;Block suspicious traffic\u0026#34; Rules: - Action: Block: {} Name: GeoRestrictionCountry Priority: 0 Statement: GeoMatchStatement: CountryCodes: - RU - BY - CN VisibilityConfig: CloudWatchMetricsEnabled: true MetricName: GeoRestrictionCountryRule SampledRequestsEnabled: true - Name: AWS-AWSManagedRulesAmazonIpReputationList OverrideAction: None: {} Priority: 1 Statement: ManagedRuleGroupStatement: Name: AWSManagedRulesAmazonIpReputationList VendorName: AWS VisibilityConfig: CloudWatchMetricsEnabled: true MetricName: AWS-AWSManagedRulesAmazonIpReputationList SampledRequestsEnabled: true - Name: AWS-AWSManagedRulesAnonymousIpList OverrideAction: None: {} Priority: 2 Statement: ManagedRuleGroupStatement: Name: AWSManagedRulesAnonymousIpList VendorName: AWS VisibilityConfig: CloudWatchMetricsEnabled: true MetricName: AWS-AWSManagedRulesAnonymousIpList SampledRequestsEnabled: true Scope: CLOUDFRONT VisibilityConfig: CloudWatchMetricsEnabled: true MetricName: GeoRestrictionWebACL SampledRequestsEnabled: true Type: \u0026#34;AWS::WAFv2::WebACL\u0026#34;\nFinally attach WAF to CloudFront by adding the following to your CloudFront distribution object. Please note that the webAclId expects the arn of the WebACL and not the physical or logical ID. webAclId: webacl.attrArn, To attach the WebACL to regional resources, please use the construct CfnWebACLAssociation.\nYou\u0026rsquo;ve now configured your CloudFront distribution to use the WAF ACL, blocking traffic from the offending countries, from anonymous nodes and from addresses with a bad reputation.\nIn the example above the country filtering is redundant. Both CloudFront Geo Restriction feature and AWS WAF GeoMatchStatement have been used. To simplify management and reduce costs you can remove the rule with geoMatchStatement from the WebACL. It is included in this example if you want to use AWF for regional services and thus cannot rely on CloudFront\u0026rsquo;s Geo Restriction feature. Don\u0026rsquo;t forget the scope-property when using this WAF for regional services.\nNote: Global WAF WebACLs are managed in us-east-1 only. Trying to deploy them in different regions will throw an error. In CDK the target region for a stack can be enforced by specifying the region in your app\u0026rsquo;s environment: env: { region: \u0026#39;us-east-1\u0026#39; },\nAmazon Route 53 # Amazon Route 53 supports geolocation-based routing. Based on the country where the DNS request originated from, the answer will/can be different. We can use this in order to respond with a bogus value for the offending countries, by pointing them to 127.0.0.1 (localhost). This way resolving your resources is made a bit more complex. However, if this DNS request has been made via a global DNS-provider the results may vary. This is certainly not waterproof.\nconst banned_country_codes = [\u0026#39;RU\u0026#39;, \u0026#39;BY\u0026#39;, \u0026#39;CN\u0026#39;]; for (let banned_country_code of banned_country_codes) { new route53.CfnRecordSet(this, \u0026#39;CountryCodeRecordSet\u0026#39; + banned_country_code, { name: \u0026#39;www.domain.tld\u0026#39;, type: \u0026#39;A\u0026#39;, geoLocation: { countryCode: banned_country_code, }, hostedZoneId: domain.hostedZone.hostedZoneId, resourceRecords: [ \u0026#39;127.0.0.1\u0026#39;, ], ttl: \u0026#39;600\u0026#39;, setIdentifier: \u0026#39;banned_country_\u0026#39; + banned_country_code, }); }; Finally add a recordset with countryCode '*' which is pointing to your actual resource. This way all countries not listed in banned_country_codes will get the response of your Internet-facing resource.\nAWS Shield # AWS Shield is a managed Distributed Denial of Service (DDoS) protection service that safeguards applications running on AWS. AWS Shield comes in two tiers: Standard and Advanced. AWS Shield Standard is automatically enabled on CloudFront, load balancers and Amazon Route 53, without additional costs. AWS Shield Advanced is a paid service and provides tailored detection, advanced mitigation, proactive event response and DDoS cost protection (almost like an insurance), 24x7 response team etc. This service, both levels, can be used at global resources such as CloudFront and with regional resources such as load balancers.\nPutting it all together # Like any other security challenge, the best results can be achieved by applying security-in-depth. Bulk of the malicious traffic can be dropped at the global infrastructure of AWS. At this level you can benefit from the scale of AWS. Amazon CloudFront and AWS WAF both provide options to deny traffic based on Geo Restriction. This is low maintenance and quite effective. But not waterproof. Consider using the WAF managed rules to deny traffic from anonymous networks. And finally make your application robust by applying Auto Scaling groups or using serverless solutions.\nSome tips:\nBy blocking traffic early in the process, you reduce the risk for your regional - and perhaps less scalable or less cost-effective - resources. Don\u0026rsquo;t forget to monitor and apply alarms based on metrics In your VPC only allow ingress traffic from CloudFront, reducing your attack vector. Consider AWS Shield Advanced. It might look expensive, but once activated the service can be used (this is included in the pricing) across all your AWS accounts. If you also want to apply geo statements on your regional WAF, please make sure that you are using the X-Forwarded-For header to determine the source IP address. Deploy/manage your global resources from us-east-1-region. This simplifies your IaC (less cross-region resources). This diagram should give an high-level overview of how an (classic tiered) application could be protected by Geo Restrictions. Note: Geo restriction of course can also be used for less political reasons such as DRM or licensing cases.\n","date":"March 3, 2022","externalUrl":null,"permalink":"/2022/03/limiting-access-using-geographic-restrictions/","section":"Posts","summary":"","title":"Limiting access using geographic restrictions","type":"posts"},{"content":"","date":"January 3, 2022","externalUrl":null,"permalink":"/tags/amazon-inspector/","section":"Tags","summary":"","title":"Amazon Inspector","type":"tags"},{"content":"","date":"January 3, 2022","externalUrl":null,"permalink":"/tags/amazon-inspector-2/","section":"Tags","summary":"","title":"Amazon Inspector 2","type":"tags"},{"content":"Six years after the initial launch AWS announced the new Amazon Inspector. The launch took place during re:Invent 2021. With this launch the previous version has been renamed to Amazon Inspector Classic. The new Amazon Inspector is rearchitected to support a wider range of workloads and simplified workflow management.\nScale with simplified management # The management of Inspector 2 has been simplified. Enabling and configuring the service can be done with a few API calls or, if you persist on manual labor, a few clicks in ClickOps. To give an idea of the steps needed to enable Inspector for both existing and new member accounts, the steps are listed below.\nEnabling Inspector in a single account is now as easy as running the command below: $ aws inspector2 enable \\ --resource-types EC2 ECR\nIf you want to enable Inspector in your AWS Organization, run the previous command in your Organizations master account and then enable this account as the delegated admin account. The account ID can be any account in your Organizations (preferably a dedicated security account). In this case we use 123456789012: $ aws inspector2 enable-delegated-admin-account \\ --delegated-admin-account-id 123456789012\nNow configure Inspector for all members by associating all member accounts. You only have to do this for existing member accounts and you can do this by running this command from your delegated administrator account: $ aws inspector2 associate-member \\ --account-id 111111111111 $ aws inspector2 associate-member \\ --account-id 222222222222 $ aws inspector2 associate-member \\ --account-id 333333333333 $ aws inspector2 associate-member \\ --account-id 444444444444 Note: this can take a while before all accounts are enabled. You can view the status by running: $ aws inspector2 list-members Wait for all accounts to get the status \u0026quot;relationshipStatus\u0026quot;: \u0026quot;ENABLED\u0026quot; before running the next command.\nFinal step is enabling EC2 and ECR scanning in all member accounts: $ aws inspector2 enable \\ --resource-types EC2 ECR \\ --account-ids 111111111111 222222222222 333333333333 444444444444 Inspector is now enabled for all existing member accounts. Turn auto-enable on so that scanning is enabled for new member accounts: $ aws inspector2 update-organization-configuration \\ --auto-enable \u0026#39;{\u0026#34;ec2\u0026#34;: true, \u0026#34;ecr\u0026#34;: true}\u0026#39;\nYou need to repeat the steps above for each Region in which you want to enable scans for your members.\nInspector is now ready to be used and unlike the previous version there is no need to create schedules via Amazon CloudWatch Events. Inspector will immediately start scanning any existing EC2 instance and ECR repository. Scans for EC2 are automatically initiated when launching new instances, installing new software, Inspector database updates, et cetera. Scans for ECR can be done with continual scanning (with Enhanced scanning enabled on ECR) or on-push.\nTo scan EC2 instances with full visibility make sure you provision your EC2 instances with the AWS Systems Manager (SSM) agent configured. Make sure the SSM agent is running and AmazonSSMManagedInstanceCore policy added to the instance profile). Without the SSM agent, configure Inspector will only present open network paths, but not CVEs. You don\u0026rsquo;t have to install the separate Inspector agent any longer.\nGiven the simplicity of the steps illustrated earlier it won\u0026rsquo;t be very hard to do the same within your favorite Infrastructure as code (IaC) tool. If you are a CloudFormation user you currently have to build custom resources to make it happen. For AWS CDK users we have a construct available on Construct Hub.\nGain centralized visibility # Amazon Inspector 2 uses AWS Organizations features to allow central management and visibility. This makes it more convenient to manage a multi-account environment. The delegated master account will host all findings from the member accounts, allowing simplified management by the security department and hassle-free automation. Findings are also pushed to Amazon EventBridge. With this it is a lot easier to write event-based automation tasks to do, for example, event analysis and auto-remediation using Lambda or Step Functions.\nThe visibility has also been rebuilt. It is now possible to select multiple views. Findings can be viewed by vulnerability, account, instance, container image and repository.\nTime for some screenshots. The general overview of Inspector: If you then click the kernel package you will get an overview of all CVEs detected in this package: By selecting findings by vulnerability you will see this screen: Also new is the integration with Security Hub. If both Inspector and Security Hub are enabled findings will be sent automatically to Security Hub to give a comprehensive view of your security state. This will also allow easier integration with 3rd party solutions such as Splunk.\nScreenshot from Inspector findings reported in Security Hub: The finding in Security Hub holds all the details required for triage and analysis. This is a snippet (just the Vulnerabilities field): { \u0026#34;Vulnerabilities\u0026#34;: [ { \u0026#34;Id\u0026#34;: \u0026#34;CVE-2021-4002\u0026#34;, \u0026#34;VulnerablePackages\u0026#34;: [ { \u0026#34;Name\u0026#34;: \u0026#34;kernel\u0026#34;, \u0026#34;Version\u0026#34;: \u0026#34;4.14.252\u0026#34;, \u0026#34;Epoch\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;Release\u0026#34;: \u0026#34;195.483.amzn2\u0026#34;, \u0026#34;Architecture\u0026#34;: \u0026#34;X86_64\u0026#34; } ], \u0026#34;Cvss\u0026#34;: [ { \u0026#34;Version\u0026#34;: \u0026#34;3.1\u0026#34;, \u0026#34;BaseScore\u0026#34;: 5.1, \u0026#34;BaseVector\u0026#34;: \u0026#34;CVSS:3.1/AV:L/AC:L/PR:N/UI:N/S:U/C:L/I:L/A:N\u0026#34; }, { \u0026#34;Version\u0026#34;: \u0026#34;3.1\u0026#34;, \u0026#34;BaseScore\u0026#34;: 5.1, \u0026#34;BaseVector\u0026#34;: \u0026#34;CVSS:3.1/AV:L/AC:L/PR:N/UI:N/S:U/C:L/I:L/A:N\u0026#34;, \u0026#34;Source\u0026#34;: \u0026#34;REDHAT_CVE\u0026#34;, \u0026#34;Adjustments\u0026#34;: [] } ], \u0026#34;Vendor\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;REDHAT_CVE\u0026#34;, \u0026#34;Url\u0026#34;: \u0026#34;https://access.redhat.com/security/cve/CVE-2021-4002\u0026#34;, \u0026#34;VendorSeverity\u0026#34;: \u0026#34;Moderate\u0026#34;, \u0026#34;VendorCreatedAt\u0026#34;: \u0026#34;2021-11-25T00:00:00Z\u0026#34; }, \u0026#34;ReferenceUrls\u0026#34;: [] } ], \u0026#34;FindingProviderFields\u0026#34;: { \u0026#34;Severity\u0026#34;: { \u0026#34;Label\u0026#34;: \u0026#34;MEDIUM\u0026#34; }, \u0026#34;Types\u0026#34;: [ \u0026#34;Software and Configuration Checks/Vulnerabilities/CVE\u0026#34; ] } }\nAutomate discovery and continual scanning # When using Inspector Classic it was required to specify which EC2 instances to scan, based on tags. With Inspector 2 this has been changed. Inspector 2 will start the discovery of supported resources automatically, depending on which services you enable scans for. If you\u0026rsquo;ve used EC2 resource tags in Inspector Classic to limit the scope of Inspector, you can now do that by specifying ResourceTags in the FilterCriteria when listing all findings reported by Inspector.\nThis automated discovery will make it easier to demonstrate compliance, since the scope will be \u0026ldquo;all EC2 instances with the SSM Agent working are included\u0026rdquo;. With this you\u0026rsquo;re no longer dependent on manual and error-prone work such as tagging.\nPrioritize with contextualized scoring # Findings now include more actionable detail compared to Amazon Inspector Classic, including improved contextual risk scoring that clarifies how to prioritize and respond to the most critical findings. This will make it easier to do triage and analysis. You can focus on the most important findings first.\nAutomate workflows and take actions # As I said, findings are now automatically pushed to EventBridge and Security Hub. This simplifies the automated response workflows and security event management. In addition, Inspector now comes with new management API calls which allow much easier organization-wide configuration.\nVulnerability scanning has never been this easy. :-)\nAmazon Inspector Classic (v1) will be deprecated. A 12-month depreciation notice will be put on the service soon.\nRead more about this announcement and the User Guide.\nThis blog can also be found on our company website\n","date":"January 3, 2022","externalUrl":null,"permalink":"/2022/01/amazon-inspector-2-whats-new/","section":"Posts","summary":"","title":"Amazon Inspector 2 - What's new?","type":"posts"},{"content":"","date":"January 3, 2022","externalUrl":null,"permalink":"/tags/amazon-inspector-classic/","section":"Tags","summary":"","title":"Amazon Inspector Classic","type":"tags"},{"content":"","date":"January 3, 2022","externalUrl":null,"permalink":"/tags/reinvent/","section":"Tags","summary":"","title":"ReInvent","type":"tags"},{"content":"","date":"December 27, 2021","externalUrl":null,"permalink":"/page/","section":"Pages","summary":"","title":"Pages","type":"page"},{"content":"","date":"November 5, 2021","externalUrl":null,"permalink":"/tags/cloud-financial-management/","section":"Tags","summary":"","title":"Cloud Financial Management","type":"tags"},{"content":"","date":"November 5, 2021","externalUrl":null,"permalink":"/tags/club-cloud-2021/","section":"Tags","summary":"","title":"Club Cloud 2021","type":"tags"},{"content":"Michel Zitman (Cloud Financial Management Practice Lead at Oblivion) shares his best practices on how to build efficient FinOps KPIs and demonstrates the importance they have on your FinOps Governance. Cloud cost management appears to be a challenging subject in the majority of organizations. From our experience we have concluded that 35% of cloud spend is actually wasted as a result of over-provisioned resources alone. And given the fact that, on a global scale, we are only at the beginning of cloud adoption, it’s time for action. Guests: Sidney Epner (CloudHealth by VMware) and Michel Zitman (Oblivion)\n","date":"November 5, 2021","externalUrl":null,"permalink":"/2021/11/exploring-finops-kpis-to-measure-the-true-value-of-the-cloud/","section":"Posts","summary":"","title":"Exploring FinOps KPI's to measure the true value of the cloud","type":"posts"},{"content":"","date":"November 5, 2021","externalUrl":null,"permalink":"/tags/finops/","section":"Tags","summary":"","title":"FinOps","type":"tags"},{"content":"Livestream session to talk with Maheshwar, Developer Evangelist at AWS, about how to keep up to date with your knowledge on AWS. Guests: Sohan Maheshwar (AWS)\n","date":"November 5, 2021","externalUrl":null,"permalink":"/2021/11/keeping-up-with-your-cloud-knowledge/","section":"Posts","summary":"","title":"Keeping up with your cloud knowledge","type":"posts"},{"content":"","date":"November 5, 2021","externalUrl":null,"permalink":"/tags/knowledge/","section":"Tags","summary":"","title":"Knowledge","type":"tags"},{"content":"Why is it important to optimize costs? TCO has two main components: infrastructure cost and software licenses cost. The usual optimization scenario is when a customer wants to find compute, storage and licensing configuration that reaches the desired performance level at minimum cost. When you run SQL Server workloads, licensing makes up the majority of the TCO, so this is where you want to optimize. In this session we are going to cover basics of Microsoft licensing on AWS and best practices for compute and licensing optimization of SQL Server workloads on AWS. And finally we will discuss how AWS Optimization and Licensing Assessment (AWS OLA) can help you to see potential cost savings when migrating your Windows Workloads to AWS. Guests: Alexander Dragunov (AWS) and Michel Zitman (Oblivion)\n","date":"November 5, 2021","externalUrl":null,"permalink":"/2021/11/best-practices-for-optimizing-cost-and-performance-of-your-microsoft-workloads-on-aws/","section":"Posts","summary":"","title":"Best practices for optimizing cost and performance of your Microsoft workloads on AWS","type":"posts"},{"content":"","date":"November 5, 2021","externalUrl":null,"permalink":"/tags/microsoft-workloads/","section":"Tags","summary":"","title":"Microsoft Workloads","type":"tags"},{"content":"","date":"November 2, 2021","externalUrl":null,"permalink":"/tags/cloud-native-development/","section":"Tags","summary":"","title":"Cloud-Native Development","type":"tags"},{"content":"Together with Mark Boon (Senior Digital Engineer @ DSM), Andrès Koetsier (cloud consultant at Oblivion) goes in-dept into how DSM adopted a cloud-native way of working to rapidly develop new applications to solve sustainable challenges. Guests: Mark Boon (DSM) and Andrès Koetsier (Oblivion)\n","date":"November 2, 2021","externalUrl":null,"permalink":"/2021/11/dsms-cloud-native-journey/","section":"Posts","summary":"","title":"DSM's cloud native journey","type":"posts"},{"content":"","date":"November 2, 2021","externalUrl":null,"permalink":"/tags/data/","section":"Tags","summary":"","title":"Data","type":"tags"},{"content":"Martijn Doedens (Cloud Security Consultant at Oblivion) and Niels Zeilemaker (CTO at GoDataDriven) summarise the current state of secure data processing in the (AWS) cloud. We cover the specific services related to data and the services that are used to standardise secure landing zone, actual risks and threats that our customers face when processing big data on AWS and which configurations (aka best practices) you should always implement. Guests: Niels Zeilemaker (GoDataDriven) and Martijn Doedens (Oblivion)\n","date":"November 2, 2021","externalUrl":null,"permalink":"/2021/11/data-security-in-the-cloud/","section":"Posts","summary":"","title":"Data \u0026 Security in the cloud","type":"posts"},{"content":"Have you spotted access keys in code? Putting it directly in your code might not be a good idea. Maybe you saw that Kubernetes offers secrets for this. Have you found your Kubernetes secrets to be readable by everyone? So where do you put your secrets in a secure way? In this session we will look at various solutions, such as Hashicorp Vault, Kubernetes, and solutions offered by AWS. Guests: Jeroen Willemsen (Xebia)\n","date":"November 2, 2021","externalUrl":null,"permalink":"/2021/11/securing-your-secrets-in-the-cloud/","section":"Posts","summary":"","title":"Securing your secrets in the cloud","type":"posts"},{"content":"In this session we talk about how to do detection and response in the AWS public cloud and what native AWS services can be used to do this. What options are available and what (if any) 3rd party solutions play a role here? If it was up to us, what are the top use cases a security operations. Guests: Christiaan van Tienhoven (TKP) - Martijn Doedens (Oblivion)\n","date":"November 2, 2021","externalUrl":null,"permalink":"/2021/11/threat-detection-and-response-in-the-cloud/","section":"Posts","summary":"","title":"Threat detection and response in the cloud","type":"posts"},{"content":"Jeroen van der Leer discusses five main reasons modern organizations choose to adopt the public cloud - and why you should too. Guests: Jeroen van der Leer (Oblivion)\n","date":"November 1, 2021","externalUrl":null,"permalink":"/2021/11/compelling-reasons-to-adopt-the-public-cloud/","section":"Posts","summary":"","title":"Compelling reasons to adopt the public cloud","type":"posts"},{"content":"","date":"November 1, 2021","externalUrl":null,"permalink":"/tags/strategy/","section":"Tags","summary":"","title":"Strategy","type":"tags"},{"content":"Yesterday AWS CloudFormation Guard version 2.0 was introduced. Guard is an open source tool that can be used to validate CloudFormation templates against certain rules. You can use it for linting your templates both on syntax and semantics. Linting tools are essential in CI/CD pipelines and an powerful addition when it comes to validating structured and human-readable files such as JSON and YAML. In your delivery pipelines you would preferably run linting tests as early as possible, before proceeding to steps such as compile and deploy steps et cetera(fail-fast).\nIf you zoom out a little bit and look to the entire governance process linting tools can help you making big steps in shifting-left. This means that you want to report your findings as early in the development phase as possible. In this case linting tests should be run as soon as possible. You want to enforce your requirements before actual deployments are made.\nLinting tests can include a large number of validation steps. Most obvious ones are security related, to enforce encryption and enabled auto-patching for example. But you can also think of enforcing best practices, such as mandatory tagging. Last but not least you can also include company specific requirements such as naming conventions.\nAs per usual AWS doesn’t provide a single tool for a purpose. For linting CloudFormation another, maybe better known, tool is CloudFormation Linter (cfn-lint). There is a lot of overlap between the two tools, but also some noticeable differences. I believe the biggest differences are that cfn-lint can be solely used to lint CloudFormation templates while cfn-guard can be used to lint any JSON or YAML file. With cfn-guard you can potentially validate Helm Charts (Kubernetes container definitions) or maybe even Terraform JSON files. Another noticeable difference is that cfn-guard doesn’t come with pre-built rules, while cfn-lint is pretty usable of the shelf.\nLooking at maturity levels: I find the Unit testing framework in cfn-guard pretty valuable. Nothing is as annoying as linting rules causing false-positives or unpredictable behaviour. Also the fact that you can group rules together. This can help, for example, to distinguish between security and best practices rules. Writing and using unit tests can really help to get that consistent experience among your users.\nWhat I find a bit clumsy is that the tool itself is written in Rust. If you opensource a tool and give it to the community it might be a logical choice to use a common language like Python or JavaScript, so that a large group of developers can contribute. Also many other AWS projects/tools are written in Python or JavaScript (or even Go).\nFinally checkout my GitHub repo for some examples on how to use cfn-guard in a Docker container. It also includes an example rule and Unit test.\nThis blog can also be found on our company website\n","date":"May 18, 2021","externalUrl":null,"permalink":"/2021/05/first-glance-at-aws-cloudformation-guard-2.0/","section":"Posts","summary":"","title":"First glance at AWS CloudFormation Guard 2.0","type":"posts"},{"content":"","date":"May 18, 2021","externalUrl":null,"permalink":"/tags/open-source/","section":"Tags","summary":"","title":"Open Source","type":"tags"},{"content":"Recently a nasty vulnerability in AWS CloudShell was reported by Google’s Project Zero. Due to a bug in the library handling interaction with the shell and the browser an attacker could trigger remote code execution. AWS CloudShell is a browser based shell which gives a pre-authenticated web-based shell. The service most-often used by a user with administrator privileges on the AWS console.\nVia remote shell the credentials of the user starting AWS CloudShell can be obtained and can be used to call AWS services. Needless to say what the potential impact could have been. It would be easy to blame the developers for this vulnerability, but that would be too short sighted. In general we all know that software is written by humans and that humans are prone to error. We need a more comprehensive approach to protect our digital assets and reputation. Control the users having access # Very obvious, but too often I do come across environments with local users. In the first period the users are probably still aligned with the actual user requiring access to your environment, but over time drift will occur. This will happen because the environment is not aware of joiner/leavers/movers within your organization. Best way to mitigate this is to implement federation with your corporate directory, so that changes at the HR-side automatically are reflected in your authorization logic as RBAC assignment changes.\nAlso consider providing task-specific roles to your users. For example a user, who is allowed to make changes to the platform, does not need to assume a role with write access when he/she is only examining the platform.\nMinimize blast radius # It’s naive to think that your environment will never be compromised: at some point it will happen. We also know that when we get compromised is the next question will be: “what is the impact of this breach?” and possibly how painful the answer shall be: “it affects all our production apps”. In AWS it is very common to apply a multi-account strategy. An AWS account provides the highest level of isolation and a customer can have as many AWS accounts as needed. In the early days running multiple AWS accounts was adding quite some overhead to operational tasks, but with the recent developments around AWS Organizations, AWS Control Tower, AWS Transit Gateway, AWS Resource Access Manager, et cetera. most of the challenges are solved. At least to the level at which they cannot be an excuse to apply a multiple account strategy anymore.\nCommon applied strategies are:\nBusiness units oriented strategy (account per business unit, multiple applications and owners per account). Application lifecycle oriented strategy (account per stage in D, T, A and P, multiple applications and owners per account). Cost-center oriented strategy (account per cost-center, multiple applications and potentially per account). Application oriented strategy (account per application, single owner). Owner oriented strategy (single owner per account, multiple applications per account). User oriented strategy (one account per user, mainly used for sandboxing purposes). From our experience I can tell that, in the end, I prefer an application oriented strategy above the other strategies. It gives the smallest possible blast radius, fine-grained insights and flexibility in ownership/access.\nReduce the effective time that you are vulnerable # Losing or leaking passwords is a very common mistake. It happens either by mistake or careless way of working. Think of passwords are exposed over unencrypted channels, are committed in source code repositories, left on a sticky note under the keyboard or on the monitor, easily guessable, etc. Once the credentials are in the hand of an attacker they can be abused without any limitations. Adding OTP (2FA) and/or temporary credentials provided by an identity provider reduces the time when the credentials are valid to a minimum.\nFor this threat I would recommend to implement OTP on the corporate identity provider (with reasonable expiration period for STS credentials) or enforce the presence of MFA authentication on the IAM user (e.g. don’t provide write/read access to the platform without valid MFA token).\nEnhance your IAM policies with additional attributes # IAM policies on AWS can be written very fine grained. In addition to specifying which actions are allowed the policies can be enhanced by using conditions. These conditions have to be met before an action is allowed. This can be used to control who has access to what, based on context-aware information, such as source IP address or information about the authentication process. Multiple conditions can be combined to form a solid policy statement.\nInteresting conditions are:\naws:MultiFactorAuthPresent: Use this to enforce MFA authentication before allowing access to AWS services or resources. aws:SourceIp: Use this condition to specify a source address to allow actions from. For example only from your company’s known locations. aws:RequestedRegion: Use this condition to limit the action to a specific AWS region. E.g. most attackers will use the most popular AWS Region us-east-1. You might only want to allow an authenticated and authorized user access to a specific region. We are based in Europe and therefore mainly are using the regions eu-{west,central}-1. An exhaustive list of condition keys can be found here. Some services have specific condition keys, those are listed here.\nReduce IAM privileges and implement Service Control Policies # Administrator users regularly having too much privileges to the platform. It is very common that those users have the AdministratorAccess managed IAM policy attached, while most of the operations are read-only and write-access operations are limited to a very small subset of calls. For most tasks it is sufficient to have write-access via Infrastructure As Code. You can enforce calls to CloudFormation to use a dedicated role with write access. This way making changes to the platform requires some thinking and requires specific knowledge of the platform. I know, it’s not foolproof. But everything to slow down the attacker helps.\nAlso implementing Service Control Policies is recommended. I prefer to have Service Control Policies applied in a coarse-grained way. This way you can, Organization-wide, restrict access for actions that should ‘never happen’.\nProtect landing zone assets # Prevent attackers (and users) to disable foundation services such as your VPC, GuardDuty, CloudTrail, etc. I assume you’re using a pipeline to deploy your landing zone assets. In your SCPs you can specify policies to allow your pipeline roles access to foundation services and deny write-access to all other identities.\nRestricting access to subscription calls # Some API calls are expensive and have impact on the long term. Think of API calls for buying (C)RIs, enabling AWS Shield Advanced and other long running subscriptions. This is something you, also in regular situations, want to prevent and only grant to specific roles. It works best when restrictions are applied at Organization root-level.\nDenying access to unused services and regions # Attackers will try to use whatever service they can. The less services are allowed, the lower their success rate will be. Same goes for regions. If you know that your workloads are only going to use a subset of available AWS regions it might be a good idea to deny all other regions (be careful, some global services/resources require access to us-east-1 region).\nTip: use the NotAction statement to whitelist services\nProactively monitor behaviour # AWS provides a number of services that can be used to analyze behaviour on the AWS platform. As a starting point I recommend to configure CloudTrail to record activity in all accounts and regions, preferably on Organization-level and enable Amazon GuardDuty. And don’t wait with gathering insights on activity until you have been compromised.\nPeriodically review IAM policies # Use IAM Access Analyzer to discover resources that are (unintentionally) shared with resources outside your zone of trust. It will give you an overview of resources that are remotely accessible.\nTo improve your IAM policies use the IAM Access Advisor to collect information about which services are being used by your users. This is valuable input to start optimizing/tweaking IAM policies. You want to consider removing unused services from the list of allowed actions.\nMonitor activity # One of the most difficult questions to answer is: “are you currently under attack?”. Observability is mandatory to answer this question. Logging all activity is easy, but dividing events in two categories: expected and unexpected is a lot more difficult. How do you define expected events? It requires a lot of tuning, large number of false-positives etc. And what to do with new types of events? For example due to new features and service launches etc.\nAmazon GuardDuty can help here. This is a fully managed service by AWS and has a very straight forward configuration. You simply enable the services and it will start monitoring your environment. It uses CloudTrail, VPC Flow Logs and Route 53 logs (DNS) to detect anomalies. Detection is done by Machine Learning models. It supports a large number of events and the list is still growing. Best example is for example the use of your AWS credentials. GuardDuty will build up a baseline of your behaviour. If you, for example, use your credentials from your (home) office and suddenly your credentials are being used at a different geographical location GuardDuty will generate a finding. This finding can be forwarded to your favorite monitoring system. By consuming events from GuardDuty you gain insights on what is happening on the platform. For some events you can even decide to have auto-remediation in place by triggering a Lambda-function when an event has occurred.\nConfigure budget alerts # Cloud is mainly based on pay as you go consumption models. So significant changes in service usage will lead to financial changes. Most obvious is a sudden raise in costs. e.g. the attacker uses your environment to start crypto mining or data leakage is happening (data transfer out is expensive in cloud). But an unexpected drop in costs is also something that might happen. An attacker might stop or terminate resources to wipe his/her traces or when he wants to hurt you by deleting data. Monitor changes on the budget closely to be gain insight on behaviour. Setting a fixed budget alert might help, but it is probably not the smartest way. AWS and cloud management platforms provide forecasting and cost anomaly detection. Leverage those features to keep an close eye on your costs and to reduce the number of false-positives generated by budget alerts.\nFinal thoughts # With this blog I didn’t had the intention to come up with an exhaustive list of security controls. I do had the intention to give some insights on how to reduce the impact of a breach. We all know that that software (and hardware) have their weak spots when it comes to security. And we also know that we might get faced by an attacker. There is a lot we can do to reduce the impact in case something happens. Hopefully this blog contributes to make your cloud environment a safe place.\nNote: The issue is solved by the time of releasing this blog #noworries\nThis blog can also be found on our company website\n","date":"May 12, 2021","externalUrl":null,"permalink":"/2021/05/defense-in-depth-matters/","section":"Posts","summary":"","title":"Defense in depth matters","type":"posts"},{"content":"In the last few years the costs for running security went skyrocketing. We’ve seen an explosion of security solutions, all fighting for their own place in cybersecurity. I’m pretty sure each solution has its purpose, but from a CISO perspective the landscape is getting more complex and more expensive each year, or friendly phrased: less cost-effective.\nPublic cloud is the main driver for innovation. With the arrival of new thinking, new capabilities and reinventing how we run IT things have changed tremendously.\nI would like to take the opportunity and mix two hot topics in a series of blogs: Security and Cost Optimization. And yes (spoiler), the two can strengthen each other. In this and coming blogs I’m sharing our experience gained over the last years and tens of migrations/implementations.\nDrop Active Directory integration # One of the most painful topics during a migration is Active Directory integration. Not only from a financial perspective, but also from a technical perspective. For decades we’ve been implementing Active Directory as our primary user directory and subsequently let all our resources join the directory. This way we had the feeling that we had some kind of fine grained control over who had access to which resource. Familiar error message?\nBeen there, done that (got the t-shirt) # The main technical problem with AD is tightly integrated with infrastructure. Another example it replaces your DNS-resolver and it relies on IP communication. This forces you to create an attack surface which more or less spans all your networks, since you have to expose AD to different types of networks. For example: exposing your private AD to resources hosted in a public subnet. And also, if you have multiple domain masters, they all need to be reachable by every resource. This will lead to a very complex routing setup as your environment starts to grow and especially when running hybrid mode. This is something you want to avoid when using a platform with very strong isolation capabilities.\nAnother challenge is the pets versus cattle analogy. Resources in public cloud (if done the correct way) are dynamic and short-living. They come and go as the demand changes. I’m aware that you can script domain joining, but when looking at AD’s design you notice that this is never built to facilitate such a dynamic environment. After a while you end up with thousands of orphaned resources. Nor it is highly scalable. Don’t forget that AD preferably wants all clients to use AD as their resolvers for DNS-resolving.\nProbably I can talk for hours on why extending your AD setup for administrative purposes to a capable platform is not a very good idea in general. Let’s focus on how to solve this in a more effective way. One of the key features of AD is to provide RDP/SSH access to instances for remote desktop/shell access. AD is taking care of the authentication and authorization process. So for each connection attempt the user is prompted (or passed via Kerberos) to login. Based on policies a user/group is allowed to enter a instance. This is not a scalable model, since each instance or group of instances needs to have a Group Policy Object allowing the user identity for remote access. Also the process of domain joining is cumbersome: think of hacky PowerShell scripts and very odd Linux configuration (different configuration per distribution).\nIntroduce AWS Systems Manager Session Manager # To simplify remote management AWS launched two features: SSM Session Manager (EC2 instances) and ECS Exec (containers). It works for Linux, Windows and MacOS. Both features have strong integration with the platform and are easy to implement and maintain. It works via the SSM Agent, which is provided via recent AWS provided AMIs. In AWS it’s best practice to attach an instance profile to each compute instance. This profile can be granted to access AWS resources. In order to allow the SSM Agent to ‘call home’ and register itself at the SSM service, simply attach the AWS managed policy AmazonSSMManagedInstanceCoreto the instance profile. By default it will use the public API endpoints of SSM to connect to, but of course you can deploy VPC endpoints to avoid traffic traveling across the Internet. Now launch your instance and the preparations on the platform are completed. Easy, isn’t it?\nOverview of the AWS Systems Manager architecture # Now it’s time to allow your IAM users/roles to access the instances. Depending on your needs you can decide yourself who is granted access to call thessm:StartSession action. This can be combined with your AWS Account Structure, federation, and IAM conditions such as tags, time, source IP, etc. to make it as fine-grained as needed.\nNote: this also works for hybrid environments. You can install the SSM Agent on your on-premises resources and manage your fleet of instances via AWS Session Manager. It is also compatible with outbound/forward proxies. So it technically fits almost any environment.\nFrom an audit perspective all actions are recorded in AWS CloudTrail, session data to S3 or Amazon CloudWatch Logs. If you want real-time event response you can use Amazon EventBridge in conjunction with Amazon Simple Notification Service to forward events to your Slack, SIEM, ITSM, or Lambda ( for auto-remediation) etc.\nGoodbye RDP and SSH # As a final step you shouldn’t forget to remove the ingress rules in your Security Group, since direct access via RDP and SSH is no longer required. If, for any reason, you use SSH for port-forwarding: don’t worry. This is supported by SSM. Read more about it here.\nAWS Systems Manager is providing access via the web console and via AWS Command Line Interface, so you can choose which one to use.\nNote: ECS Exec currently only supports access via awscli.\nFancy. What about the costs? # The running costs for the solution above are probably way below the price of your lunch (even when working from home). SSM Session Manager is free of charge, for CloudWatch Logs and S3 storage costs are charged and for VPC endpoint there is an hourly charge.\nFinal thoughts # Introducing AWS Systems Manager will a unleash a number of new capabilities while the security footprint such as operational management, running costs, number of tools etc. will be reduced to a minimum. The beauty of this solution is also that you don’t have to expose ports or establish cross network/environment connectivity which normally would harm your level of isolation and increase the blast radius. I strongly advise reading about the other capabilities provided by SSM here, as it will give you a lot of powerful features. Due to the lightweight nature of this agent it is also possible to run this on architectures with less resources, such as IoT devices.\nThis blog can also be found on our company website\n","date":"March 31, 2021","externalUrl":null,"permalink":"/2021/03/optimizing-your-security-footprint-in-aws-part-2/","section":"Posts","summary":"","title":"Optimizing your security footprint in AWS — part 2","type":"posts"},{"content":"In the last few years the costs for running security went skyrocketing. We’ve seen an explosion of security solutions, all fighting for their own place in cybersecurity. I’m pretty sure each solution has its purpose, but from a CISO perspective the landscape is getting more complex and more expensive each year, or friendly phrased: less cost-effective.\nPublic cloud is the main driver for innovation. With the arrival of new thinking, new capabilities and reinventing how we run IT things have changed tremendously.\nI would like to take the opportunity and mix two hot topics in a series of blogs: Security and Cost Optimization. And yes (spoiler), the two can strengthen each other. In this and coming blogs I’m sharing our experience gained over the last years and tens of migrations/implementations.\nDrop Active Directory integration # One of the most painful topics during a migration is Active Directory integration. Not only from a financial perspective, but also from a technical perspective. For decades we’ve been implementing Active Directory as our primary user directory and subsequently let all our resources join the directory. This way we had the feeling that we had some kind of fine grained control over who had access to which resource. Familiar error message?\nBeen there, done that (got the t-shirt) # The main technical problem with AD is tightly integrated with infrastructure. Another example it replaces your DNS-resolver and it relies on IP communication. This forces you to create an attack surface which more or less spans all your networks, since you have to expose AD to different types of networks. For example: exposing your private AD to resources hosted in a public subnet. And also, if you have multiple domain masters, they all need to be reachable by every resource. This will lead to a very complex routing setup as your environment starts to grow and especially when running hybrid mode. This is something you want to avoid when using a platform with very strong isolation capabilities.\nAnother challenge is the pets versus cattle analogy. Resources in public cloud (if done the correct way) are dynamic and short-living. They come and go as the demand changes. I’m aware that you can script domain joining, but when looking at AD’s design you notice that this is never built to facilitate such a dynamic environment. After a while you end up with thousands of orphaned resources. Nor it is highly scalable. Don’t forget that AD preferably wants all clients to use AD as their resolvers for DNS-resolving.\nProbably I can talk for hours on why extending your AD setup for administrative purposes to a capable platform is not a very good idea in general. Let’s focus on how to solve this in a more effective way. One of the key features of AD is to provide RDP/SSH access to instances for remote desktop/shell access. AD is taking care of the authentication and authorization process. So for each connection attempt the user is prompted (or passed via Kerberos) to login. Based on policies a user/group is allowed to enter a instance. This is not a scalable model, since each instance or group of instances needs to have a Group Policy Object allowing the user identity for remote access. Also the process of domain joining is cumbersome: think of hacky PowerShell scripts and very odd Linux configuration (different configuration per distribution).\nIntroduce AWS Systems Manager Session Manager # To simplify remote management AWS launched two features: SSM Session Manager (EC2 instances) and ECS Exec (containers). It works for Linux, Windows and MacOS. Both features have strong integration with the platform and are easy to implement and maintain. It works via the SSM Agent, which is provided via recent AWS provided AMIs. In AWS it’s best practice to attach an instance profile to each compute instance. This profile can be granted to access AWS resources. In order to allow the SSM Agent to ‘call home’ and register itself at the SSM service, simply attach the AWS managed policy AmazonSSMManagedInstanceCoreto the instance profile. By default it will use the public API endpoints of SSM to connect to, but of course you can deploy VPC endpoints to avoid traffic traveling across the Internet. Now launch your instance and the preparations on the platform are completed. Easy, isn’t it?\nOverview of the AWS Systems Manager architecture # Now it’s time to allow your IAM users/roles to access the instances. Depending on your needs you can decide yourself who is granted access to call thessm:StartSession action. This can be combined with your AWS Account Structure, federation, and IAM conditions such as tags, time, source IP, etc. to make it as fine-grained as needed.\nNote: this also works for hybrid environments. You can install the SSM Agent on your on-premises resources and manage your fleet of instances via AWS Session Manager. It is also compatible with outbound/forward proxies. So it technically fits almost any environment.\nFrom an audit perspective all actions are recorded in AWS CloudTrail, session data to S3 or Amazon CloudWatch Logs. If you want real-time event response you can use Amazon EventBridge in conjunction with Amazon Simple Notification Service to forward events to your Slack, SIEM, ITSM, or Lambda ( for auto-remediation) etc.\nGoodbye RDP and SSH # As a final step you shouldn’t forget to remove the ingress rules in your Security Group, since direct access via RDP and SSH is no longer required. If, for any reason, you use SSH for port-forwarding: don’t worry. This is supported by SSM. Read more about it here.\nAWS Systems Manager is providing access via the web console and via AWS Command Line Interface, so you can choose which one to use.\nNote: ECS Exec currently only supports access via awscli.\nFancy. What about the costs? # The running costs for the solution above are probably way below the price of your lunch (even when working from home). SSM Session Manager is free of charge, for CloudWatch Logs and S3 storage costs are charged and for VPC endpoint there is an hourly charge.\nFinal thoughts # Introducing AWS Systems Manager will a unleash a number of new capabilities while the security footprint such as operational management, running costs, number of tools etc. will be reduced to a minimum. The beauty of this solution is also that you don’t have to expose ports or establish cross network/environment connectivity which normally would harm your level of isolation and increase the blast radius. I strongly advise reading about the other capabilities provided by SSM here, as it will give you a lot of powerful features. Due to the lightweight nature of this agent it is also possible to run this on architectures with less resources, such as IoT devices.\nThis blog can also be found on our company website\n","date":"March 23, 2021","externalUrl":null,"permalink":"/2021/03/optimizing-your-security-footprint-in-aws-part-1/","section":"Posts","summary":"","title":"Optimizing your security footprint in AWS — part 1","type":"posts"}]