Untitled Publication

Kubernetes 101: Building Scalable Applications - ConfigMaps & Secrets

Joeri JM Smissaert — Tue, 01 Feb 2022 00:00:00 GMT

{{

}}

Providing Variables to Kubernetes Applications

While we shouldn't run naked Pods, we've already seen we can pass environment variables when creating a Pod:
kubectl run mydb --image=mysql --env="MYSQL_ROOT_PASSWORD=password"

When creating a Deployment, however, there's no command line option to provide variables. We'll need to create the Deployment first, then set the environment variables:

kubectl create deploy mydb --image=mysql
kubectl set env deploy mydb MYSQL_ROOT_PASSWORD=password

Obviously you could generate the Deployment YAML file first and add your variables to the YAML file before creating the Deployment.

student@minikube:~$ kubectl create deployment mydb --image=mariadb
deployment.apps/mydb created

student@minikube:~$ kubectl get pods
NAME                   READY   STATUS   RESTARTS   AGE
mydb-fb7ff4d78-kqbvj   0/1     Error    0          40s

student@minikube:~$ kubectl logs mydb-fb7ff4d78-kqbvj
2022-04-16 07:41:41+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.7.3+maria~focal started.
2022-04-16 07:41:41+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2022-04-16 07:41:41+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.7.3+maria~focal started.
2022-04-16 07:41:41+00:00 [ERROR] [Entrypoint]: Database is uninitialized and password option is not specified
    You need to specify one of MARIADB_ROOT_PASSWORD, MARIADB_ALLOW_EMPTY_ROOT_PASSWORD and MARIADB_RANDOM_ROOT_PASSWORD

student@minikube:~$ kubectl set env deploy mydb MYSQL_ROOT_PASSWORD=password
deployment.apps/mydb env updated

student@minikube:~$ kubectl get pods
NAME                    READY   STATUS              RESTARTS      AGE
mydb-6df85bcdbb-thm2h   0/1     ContainerCreating   0             5s
mydb-fb7ff4d78-kqbvj    0/1     Error               3 (47s ago)   108s

student@minikube:~$ kubectl get pods
NAME                    READY   STATUS    RESTARTS   AGE
mydb-6df85bcdbb-thm2h   1/1     Running   0          13s

student@minikube:~$ kubectl get deploy mydb -o yaml > mydb.yml
...

student@minikube:~$ kubectl create deploy mynewdb --image=mariadb --dry-run=client -o yaml > mynewdb.yaml
student@minikube:~$ kubectl create -f mynewdb.yaml 
deployment.apps/mynewdb created

student@minikube:~$ kubectl set env deploy mynewdb MYSQL_ROOT_PASSWORD=password --dry-run=client -o yaml > mynewdb.yaml 
student@minikube:~$ grep -i password mynewdb.yaml 
        - name: MYSQL_ROOT_PASSWORD
          value: password

student@minikube:~$ kubectl describe deploy mynewdb | grep -i password
student@minikube:~$ kubectl apply -f mynewdb.yaml 
deployment.apps/mynewdb configured

student@minikube:~$ kubectl describe deploy mynewdb | grep -i password
      MYSQL_ROOT_PASSWORD:  password

ConfigMaps

Code should be static, which makes it portable so that it can be used in other environments. To achieve this we need to separate site-specific information, like environment variables, from the code. These should not be provided in the Deployment configuration.

ConfigMaps are the solution to this issue, we can define variables and have our Deployment point to the ConfigMap. ConfigMaps are created in a different way depending what it will be used for:

Variables
Configuration Files
Command line arguments

Providing Variables with ConfigMaps

We can create a ConfigMap for variables in two ways:

By passing a file that contains the variables in a key=value format:
kubectl create cm mycm --from-env-file=myfile
By passing the variables directly:
kubectl create cm mycm--from-literal=MYSQL_ROOT_PASSWORD=password

Once you have the ConfigMap, you can update your deployment so that it points to the ConfigMap: kubectl set env --from=configmap/mycm deploy/mydeployment

student@minikube:~$ cat dbvarsfile 
MYSQL_ROOT_PASSWORD=password
MYSQL_USER=joeri

student@minikube:~$ kubectl create cm mydbvars --from-env-file=dbvarsfile 
configmap/mydbvars created

student@minikube:~$ kubectl create deploy mydb --image=mariadb
deployment.apps/mydb created

student@minikube:~$ kubectl set env deploy mydb --from=configmap/mydbvars
deployment.apps/mydb env updated

student@minikube:~$ kubectl describe deploy mydb | grep MYSQL_
      MYSQL_ROOT_PASSWORD:    Optional: false
      MYSQL_USER:                      Optional: false

student@minikube:~$ kubectl get deploy mydb -o yaml > mydb.yaml
...

Providing Configuration Files with ConfigMaps

In addition to providing variables, we can provide configuration files to our application by making use of ConfigMaps:
kubectl create cm myconf --from-file=/my/file.conf

If a ConfigMap is created from a directory instead of a file, all files in that directory will be included in the ConfigMap. When using ConfigMap for configuration files the ConfigMap must be mounted in the application, it behaves similarly to a Volume.

From a high level, we need to:

Generate the base YAML code, then add the ConfigMap mount to it later
Define a Volume of the ConfigMap type in the application manifest
Mount this volume on a specific directory, the configuration file will appear inside that directory.

In the below example we'll provide an index.html file to Nginx via a ConfigMap:

student@minikube:~$ echo "Hello World!" > index.html
student@minikube:~$ kubectl create cm myindex --from-file=index.html
configmap/myindex created

student@minikube:~$ kubectl describe cm myindex
Name:         myindex
Namespace:    default
Labels:       
Annotations:  

Data
====
index.html:
----
Hello World!


BinaryData
====

Events:  

student@minikube:~$ kubectl create deploy myweb --image=nginx
deployment.apps/myweb created

We'll edit the deployment and add volumes and volumeMounts to spec.template.spec:

student@minikube:~$ kubectl edit deploy myweb
...
    spec:
      volumes:
      - name: cmvol
        configMap:
          name: myindex
      containers:
        volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: cmvol
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        resources: {}
...

Let's verify our changes:

student@minikube:~$ kubectl describe deploy myweb
Pod Template:
  Labels:  app=myweb
  Containers:
   nginx:
    Image:        nginx
    Port:         
    Host Port:    
    Environment:  
    Mounts:
      /usr/share/nginx/html from cmvol (rw)
  Volumes:
   cmvol:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      myindex
...

student@minikube:~$ kubectl get all --selector app=myweb
NAME                        READY   STATUS    RESTARTS   AGE
pod/myweb-ff8bf9988-287n2   1/1     Running   0          13m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/myweb   1/1     1            1           19m

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/myweb-8764bf4c8   0         0         0       19m
replicaset.apps/myweb-ff8bf9988   1         1         1       13m

student@minikube:~$ kubectl exec pod/myweb-ff8bf9988-287n2 -- cat /usr/share/nginx/html/index.html
Hello World!

Understanding Secrets

Secrets allow you to store sensitive data such as passwords, authentication tokens and SSH keys, outside of a Pod to reduce the risk of accidental expose. Some Secrets are automatically created by Kubernetes while others can be created by the user. System-created Secrets are important for Kubernetes resources to connect to other cluster resources.

Secrets are Base64 encoded and not encrypted.

Three types of Secret types are offered:

docker-registry: Used for connecting to a Docker registry.
TLS: Used to store TLS key material.
generic: Creates a secret from a local file, directory or literal value

You need to specify the type when defining the Secret: kubectl create secret generic ...

How Kubernetes Uses Secrets

All Kubernetes resources need access to TLS keys in order to access the Kubernetes API. These keys are provided by Secrets and used through ServiceAccounts. By using the ServiceAccount, the application has access to its Secret.

Let's inspect one of the secrets Kubernetes uses. As mentioned previously, Secrets are used through ServiceAccounts, so we need to find out the ServiceAccount first before we can inspect the details of the Secret:

student@minikube:~$ kubectl get pods -n kube-system
NAME                               READY   STATUS    RESTARTS        AGE
coredns-64897985d-lhqq6            1/1     Running   1 (6m49s ago)   25m
etcd-minikube                      1/1     Running   1 (6m49s ago)   25m
kube-apiserver-minikube            1/1     Running   1 (6m49s ago)   25m
kube-controller-manager-minikube   1/1     Running   1 (6m49s ago)   25m
kube-proxy-khgjl                   1/1     Running   1 (6m49s ago)   25m
kube-scheduler-minikube            1/1     Running   1 (6m49s ago)   25m
storage-provisioner                1/1     Running   2 (6m49s ago)   25m

student@minikube:~$ kubectl get pods -n kube-system coredns-64897985d-lhqq6 -o yaml | grep serviceAccount
  serviceAccount: coredns
  serviceAccountName: coredns
      - serviceAccountToken:

student@minikube:~$ kubectl get sa -n kube-system coredns -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-02-01T15:48:54Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "299"
  uid: 519a806e-35c0-45be-a5a0-495d9f7c7586
secrets:
- name: coredns-token-j6qdj

student@minikube:~$ kubectl get secret -n kube-system coredns-token-j6qdj -o yaml
apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCakNDQWU2Z0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwdGFXNXAKYTNWaVpVTkJNQjRYRFRJeU1ETXdOekUxTlRrd05Wb1hEVE15TURNd05URTFOVGt3TlZvd0ZURVRNQkVHQTFVRQpBeE1LYldsdWFXdDFZbVZEUVRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS0pDCnhPam5XYXRKSW9tdUcrSGtsM3J0aFhGV0NwUGhic3FXTkhsbGlqeTlWWVRvYlYwdHVrUW1sRkRvZGc2N1RULzQKWmlYNVFvUXdyV0NOSTYrYmtPMGpGMUhPRXJQNUF2S3ZJMEpabzliSTZzN1NPVmVsNHJsRGtRUGFScjBWajhrZwpGZTZNb2tUZGswQlBmQ1l5c2hhNmNBUGNaaHl1Wjl3clJRYi83dnZkS3BzZ2tLZ1ZOMmVEQnNqRzNGWFc1M2JvCkx6azJsT1NORHRxNndVSTdlZzIrNjR2UEQ5YkdWU09IU3JraVNMTVdtU3ZWL0d3SlV3dFd6YVhtZWhJZ1NLRVAKY3ZxMWtRN0dvUEVzTUF6TUtMb2F4bXdpZlUxQ0xISE93akhWTlZvVXcvVmNOQlZCOGlnRGd4cmJMSjg3bU9pOQpqbzJpck1BNTZqZExPVk1rUFlzQ0F3RUFBYU5oTUY4d0RnWURWUjBQQVFIL0JBUURBZ0trTUIwR0ExVWRKUVFXCk1CUUdDQ3NHQVFVRkJ3TUNCZ2dyQmdFRkJRY0RBVEFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVcKQkJTODl5UHdEYzJxZG13VGFlbWxZcndvclRqVTdqQU5CZ2txaGtpRzl3MEJBUXNGQUFPQ0FRRUFlOTdNNjV3WQpxUU5nR2NzT3A4Tm4rbzdGdXQ0cWMyWldjdll5bEZKUnFURjFIVjhwZDIzTFR0V3VoRkQraVk5SDJuLzFNdzdwCnVFcHdVUjAzVHpIUUVpL1JjTUJPV0JBakFGVzJHck5RelhVbzdyOE03a3FHdEN3MVd4WXduQVBhNGJ1SG41SWcKT0lhQTA4V25udW4rcFFRMW5WL25aU04yV2xwRzRrblhGcHAzcjhTQ21uVkd1L296VjV3bGZ3WU9Ea3prZExSMgp4bjA5SHhTWkJsclpDdFZqWUxDaVRYbkN3Q3pTVXZSNjhYWkNZVWRWTHF5ZzZyZXBYb0dsSkJzY0ZMZURtKzZrCnVZR1ZvY0Ezd0FpWktoazJPV2EzcGdnTFJod2xTdTRqaFZZNk82WFpvQXMzOHcvYzFTeWY1WDBqcFB1OVdPRW8KSDdsTEpCOTdhamhYdVE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  namespace: a3ViZS1zeXN0ZW0=
  token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklsWlplWEJ4UkdoUVNUQnJaM1JTTjJGNVdUQTRlakJZUjBWS2VVaHdNRlJSYzFoQ1JFOXJPWGRGYmxFaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpyZFdKbExYTjVjM1JsYlNJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZqY21WMExtNWhiV1VpT2lKamIzSmxaRzV6TFhSdmEyVnVMV28yY1dScUlpd2lhM1ZpWlhKdVpYUmxjeTVwYnk5elpYSjJhV05sWVdOamIzVnVkQzl6WlhKMmFXTmxMV0ZqWTI5MWJuUXVibUZ0WlNJNkltTnZjbVZrYm5NaUxDSnJkV0psY201bGRHVnpMbWx2TDNObGNuWnBZMlZoWTJOdmRXNTBMM05sY25acFkyVXRZV05qYjNWdWRDNTFhV1FpT2lJMU1UbGhPREEyWlMwek5XTXdMVFExWW1VdFlUVmhNQzAwT1RWa09XWTNZemMxT0RZaUxDSnpkV0lpT2lKemVYTjBaVzA2YzJWeWRtbGpaV0ZqWTI5MWJuUTZhM1ZpWlMxemVYTjBaVzA2WTI5eVpXUnVjeUo5LmN6cGp6SUM5NG9jSV81N21vZU5wZ2xXLWtVZnpHdUlUUktfa09qbkw3M0xuN1p2M2tLMWU2TjNqbUpPSW95d2RMcms5NWNwZC1pT1VjQWdpQVcxN3dJRUZ1THR4WnVkbmsyNnBwWU1sdDNLWHpBMkJycjdkYzZGM0xjdG9RNTdPMEY0MnEybXpJS0dnVDBVYkhmYTNwTjd4ZDY0Zk04RVFpZUc2bEZBSlNuYjlBTGVqSjd6X1JjeWdkLU1SOE9Qc2gtd05KMW1RSlVrUktzenVwTHdZcERKSXVCSGx6a093Rm04YXJ5ODZ3Y0pGdzNSbm5mcFo4ZTF0aWwtWUVSTmV3aDdMdzhvTGRrSzJNUnVVSnBKZmtGZ1kteWhWejdwa3MtNW53U05BUWVpTEk2RG9oUFBqd3BYa3hzWWVCbUhZVWo1b3JMNlZ5NG9Xb1ZZTnJIU0JvUQ==
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: coredns
    kubernetes.io/service-account.uid: 519a806e-35c0-45be-a5a0-495d9f7c7586
  creationTimestamp: "2022-02-01T15:48:55Z"
  name: coredns-token-j6qdj
  namespace: kube-system
  resourceVersion: "294"
  uid: d5b84f10-e6fa-46d0-92ec-2b7895a799eb
type: kubernetes.io/service-account-token

Notice how the values in the Secret Yaml output above are base64 encoded, e.g. for the namespace:

student@minikube:~$ echo a3ViZS1zeXN0ZW0= | base64 -d
kube-system

Configuring Applications to Use Secrets

There are different use cases for using Secrets in applications:

Providing TLS keys to the application:
kubectl create secret tls my-tls-keys --cert=pathto/my.crt --key=pathto/my.key
Provide security to passwords:
kubectl create generic my-secret-pw --from-literal=password=verysecret
Provide access to an SSH private key:
kubectl create generic my-ssh-key --from-file=ssh-private-key=.ssh/id_rsa
Provide access to sensitive files which would be mounted in the application with root access only:
kubectl create secret generic my-secret-file --from-file=/my/secretfile

Secrets are used in a similar way to using ConfigMaps in applications:

If your Secret contains variables (like a password), use kubectl set env.
If it contains files (like keys), mount the Secret. Consider using defaultMode: 0400 permissions when mounting the Secret in the Pod spec.

Mounted Secrets are automatically updated in the application when the Secret is updated.

Let's demonstrate this:

student@minikube:~$ kubectl create secret generic dbpw --from-literal=ROOT_PASSWORD=password
secret/dbpw created

student@minikube:~$ kubectl describe secret dbpw
Name:         dbpw
Namespace:    default
Labels:       
Annotations:  

Type:  Opaque

Data
====
ROOT_PASSWORD:  8 bytes

student@minikube:~$ kubectl get secret dbpw -o yaml
apiVersion: v1
data:
  ROOT_PASSWORD: cGFzc3dvcmQ=
kind: Secret
metadata:
  creationTimestamp: "2022-02-01T16:30:07Z"
  name: dbpw
  namespace: default
  resourceVersion: "1661"
  uid: 6aff6adf-73e1-4ffd-b99a-fd036a034c6b
type: Opaque

student@minikube:~$ echo cGFzc3dvcmQ= | base64 -d
password

Now, let's deploy our Secret to an app:

student@minikube:~$ kubectl create deployment mynewdb --image=mariadb
deployment.apps/mynewdb created

Remember that mariadb is expecting at the very least a MYSQL_ROOT_PASSWORD environment variable. But since we created our Secret with ROOT_PASSWORD instead of MYSQL_ROOT_PASSWORD we would need to set a prefix when attaching the Secret to the application. This can come in handy in case I have other applications that could potentially use the same Secret.

student@minikube:~$ kubectl set env deployment mynewdb --from=secret/dbpw --prefix=MYSQL_
deployment.apps/mynewdb env updated

Now, while our password is base64 encoded, this isn't the case inside the Pod where it's in clear text:

student@minikube:~$ kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
mynewdb-7cc5fb9c55-58wkz   1/1     Running   0          13m

student@minikube:~$ kubectl exec mynewdb-7cc5fb9c55-58wkz -- env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=mynewdb-7cc5fb9c55-58wkz
MYSQL_ROOT_PASSWORD=password

Configuring Docker Registry Access Secret

The docker-registry Secret type stores container registry (Docker Hub, Quay.io, self hosted, ...) authentication credentials. While you don't need to authenticate, it's recommended to prevent pull rate errors in case you're running a busy cluster.

There's two ways to create the Secret: Either by directly passing the credentials, or by passing an existing Docker Config file which contains the credentials:

student@minikube:~$ kubectl create secret docker-registry -h
Examples:
  # If you don't already have a .dockercfg file, you can create a dockercfg secret directly by using:
  kubectl create secret docker-registry my-secret --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER
--docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL

  # Create a new secret named my-secret from ~/.docker/config.json
  kubectl create secret docker-registry my-secret --from-file=.dockerconfigjson=path/to/.docker/config.json

Kubernetes 101: Building Scalable Applications - Storage

Joeri JM Smissaert — Mon, 03 Jan 2022 00:00:00 GMT

{{

}}

Storage Options

Files stored in a container will only live as long as the container itself: they are ephemeral. To solve this problem we can use Pod Volumes, they outlive containers and stay available during the Pod lifetime. The Pod Volume is a property of the Pod, not the container.

Pod Volumes can directly bind to any specific storage type, e.g. Cephs, emptyDir, fibre channel, NFS, ... By using Persistent Volume Claims, you can decouple the Pod from site-specific storage: You make the Pod specification more portable since you don't configure the site-specific storage but only describe what's needed from the storage: Size and permissions.

The Persistent Volume Claim connects to a Persistent Volume which in turn defines access to external storage available in the cluster. A site administrator must make sure this Persistent Volume exists. So when a Persistent Volume Claim is created, it will search for an available Persistent Volume that matches the requirements of the storage request in the Persistent Volume Claim. If no match is found, there's StorageClass that can automatically create and allocate the storage.

This abstraction allows a developer to create and distribute generic Pod manifest files and leave the storage up to the site where it's being deployed. We'll go over examples of this to make the concept more clear.

Configuring Volume Storage

Pod local volumes are defined in pod.spec.volumes, they point to a specific volume type but for testing purposes emptyDir and hostPath are common. This volume is mounted through pod.spec.containers.volumeMounts.

apiVersion: v1
kind: Pod
metadata: 
  name: volpod
spec:
  volumes: 
    - name: test
      emptyDir: {}
  containers:
  - name: centos1
    image: centos:7
    command:
      - sleep
      - "3600" 
    volumeMounts:
      - mountPath: /centos1
        name: test
  - name: centos2
    image: centos:7
    command:
      - sleep
      - "3600"
    volumeMounts:
      - mountPath: /centos2
        name: test

In the above Pod Spec, we've defined a volume named test with the volume type emptyDir. This volume is mounted in two containers on the /centos1 and /centos2 path inside the container. Both containers can share data via this volume:

student@minikube:~$ kubectl create -f volpod.yaml 
pod/volpod created
...
student@minikube:~$ kubectl exec -it volpod -c centos1 -- bash -c 'echo "Hi there!" > /centos1/hello'
student@minikube:~$ kubectl exec -it volpod -c centos2 -- cat /centos2/hello
Hi there!

Persistent Volume Storage

A Persistent Volume is a resource that exists independently from any Pod, it ensures that data is kept during container or Pod restarts. We'll use a Persistent Volume Claim to connect to a Persistent Volume. The Persistent Volume Claim is what actually talks to the backend storage provider and it will use volumes available on that storage type: It will search for available volumes depending on the requested capacity and access mode.

kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-volume
  labels:
      type: local
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mydata"

In the above PersistentVolume we've created a Persistent Volume resource with the name pv-volume, a capacity of 2GB, an accessMode of ReadWriteOnce and a hostPath of /mydata. The hostPath is created on the worker-node where the Pod that will use this PersistentVolume is running. ReadWriteOnce makes sure that only one Pod can read/write data at the same time, ReadWriteMany or ReadOnly can be used as well.

student@minikube:~$ kubectl create -f pv.yaml 
persistentvolume/pv-volume created

student@minikube:~$ kubectl get pv
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pv-volume   2Gi        RWO            Retain           Available                                   70s

student@minikube:~$ kubectl describe pv pv-volume
Name:            pv-volume
Labels:          type=local
Annotations:     
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    
Status:          Available
Claim:           
Reclaim Policy:  Retain
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        2Gi
Node Affinity:   
Message:         
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /mydata
    HostPathType:  
Events:

Configuring Persistent Volume Claims

To use a Persistent Volume, we need a Persistent Volume Claim which requests access to Persistent Volume. The Pod Volume spec uses the name of the Persistent Volume Claim and in turn the PVC accesses the Persistent Volume. After connecting to a Persistent Volume, the Persistent Volume Claim will show as bound. The bind is exclusive, the Persistent Volume cannot be used by another Persistent Volume Claim.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pv-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

Notice that in the above spec, the PVC does not connect to a specific Persistent Volume. The only thing we see is that we need a volume with a capacity of 1GB and ReadWriteOnce access.

student@minikube:~$ kubectl create -f pvc.yaml 
persistentvolumeclaim/pv-claim created

student@minikube:~$ kubectl get pvc
NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pv-claim   Bound    pvc-d5f02edb-3d71-4a69-b977-71fd5bfa020e   1Gi        RWO            standard       22s

student@minikube:~$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM              STORAGECLASS   REASON   AGE
pv-volume                                  2Gi        RWO            Retain           Available                                              11m
pvc-d5f02edb-3d71-4a69-b977-71fd5bfa020e   1Gi        RWO            Delete           Bound       default/pv-claim   standard                39s

Our pv-volume Persistent Volume was not used and instead a new Persistent Volume was created by StorageClass because there was no available match due to the capacity request in the PVC.

Pod Storage with PV and PVC

The purpose of configuring a Pod with a Persistent Volume Claim is to decouple from site-specific information: When distributing a Pod spec with a PVC spec we do not need to know anything about site-specific storage. The PVC will find the necessary Persistent Volume storage to bind to:

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
---
kind: Pod
apiVersion: v1
metadata:
   name: nginx-pvc-pod
spec:
  volumes:
    - name: site-storage
      persistentVolumeClaim:
        claimName: nginx-pvc
  containers:
    - name: pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: webserver
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: site-storage

student@minikube:~$ kubectl create -f ckad/pvc-pod.yaml 
persistentvolumeclaim/nginx-pvc created
pod/nginx-pvc-pod created

student@minikube:~$ kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nginx-pvc   Bound    pvc-b7327501-ff4c-4f6d-9c79-d10c6ce771e8   2Gi        RWX            standard       13s

student@minikube:~$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-b7327501-ff4c-4f6d-9c79-d10c6ce771e8   2Gi        RWX            Delete           Bound    default/nginx-pvc   standard                9s

student@minikube:~$ kubectl describe pv pvc-b7
Name:            pvc-b7327501-ff4c-4f6d-9c79-d10c6ce771e8
Labels:          
Annotations:     hostPathProvisionerIdentity: 3d1fa9ec-a297-4851-8782-97e7eb238447
                 pv.kubernetes.io/provisioned-by: k8s.io/minikube-hostpath
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    standard
Status:          Bound
Claim:           default/nginx-pvc
Reclaim Policy:  Delete
Access Modes:    RWX
VolumeMode:      Filesystem
Capacity:        2Gi
Node Affinity:   
Message:         
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp/hostpath-provisioner/default/nginx-pvc
    HostPathType:  
Events:            

student@minikube:~$ kubectl exec -it nginx-pvc-pod -- touch /usr/share/nginx/html/testfile
student@minikube:~$ minikube ssh
docker@minikube:~$ ls /tmp/hostpath-provisioner/default/nginx-pvc/
testfile

StorageClass

Kubernetes StorageClass allows for automatic provisioning of Persistent Volumes when a Persistent Volume Claim request comes in. This must be backed by a Storage Provisioner which ultimately takes care of the volume configuration.
StorageClass can also be used a a selector label with the storageClassName field. Normally, PVC to PV binding is done on best match.

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/mnt/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: task-pv-pod
spec:
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
        claimName: pv-claim
  containers:
    - name: task-pv-container
      image: httpd
      ports:
        - containerPort: 80
          name: "httpd-server"
      volumeMounts:
        - mountPath: "/var/www/html"
          name: task-pv-storage

Kubernetes 101: Building Scalable Applications - Networking

Joeri JM Smissaert — Mon, 06 Dec 2021 00:00:00 GMT

{{

}}

The Kubernetes network model dictates that:

Every Pod has its own IP address
Containers within a Pod share the Pod IP address and can communicate with each other using a loopback interface (localhost).
Pods can communicate with all other Pods in the cluster using the Pod IP addresses and without using NAT.
Isolation is defined by using network policies.

Pod-to-Pod communication is the foundation of Kubernetes. You can look at a Pod like you would look at a VM, the VM has a unique IP address. The containers within the Pods are like processes running within a VM, they run in the same network namespace and share an IP address.

Basic network connectivity is built-in with kubenet but can be extended by using third-party network implementations that plug into Kubernetes using the Container Network Interface API.

The Kubernetes networking model relies heavily on IP addresses. Services, Pods, containers, and nodes communicate using IP addresses and ports:

ClusterIP: The IP address assigned to a Service. This address is stable for the lifetime of the Service.
Pod IP: The IP address assigned to a given Pod. This is ephemeral.
Node IP: The IP address assigned to a given node.

Services

A Service is an API resource that is used to expose a logical set of Pods, determined by a selector (label), to an external network by applying round-robin load balancing that forwards the traffic. kube-controller-manager will continuously scan for Pods that match a selector and include those in the Service. Adding or removing Pods immediately impacts the Service.

Services exist independently from the applications or Pods they provide access to, e.g. removing a Deployment will not remove a Service. This means that one Service can provide access to Pods in multiple Deployments, Kubernetes will automatically load balance between these Pods.

kube-proxy on the nodes watches the Kubernetes API for new Services and endpoints (connected Pods). It opens random ports and listens for traffic to the Service port on the Cluster IP address, then redirects traffic to a Pod that is specified as an endpoint. It typically doesn't require any configuration.

There are different Service Types:

ClusterIP: The default type which exposes the Service on an internal cluster IP address.
NodePort: Opens a specific port on the node that forwards to the Service cluster IP address.
LoadBalancer: Used on public cloud, it will provision a load balancer in the cloud for the Service.
ExternalName: Works with DNS names.

We will focus on ClusterIP and NodePort.

Creating Services

kubectl expose can be used to create Services, providing access to Deployments, ReplicaSets, Pods or other. In most cases it exposes a Deployment which in turn allocates its Pods as the Service Endpoint. If you inspect the Service, you'll see it doesn't actually connect to the Deployment but to the Pods in the Deployment by using the Selector label. The --port argument is required to specify the port that the Service should use.

There are different types of ports in Services:

port: The port on which the Service is accessible.
targetport: The port on the application that the Service addresses. The same value for port will be used if targetport is not specified.
nodeport: The port that is exposed externally while using the nodePort Service type. Required when using the nodePort Service Type but is set automatically.

kubectl create service can be used as an alternative solution to create Services. When creating a NodePort Service type, the port and targetport are specified as a key:value pair in the --tcp argument:

kubectl create service nodeport my-node-port-service --tcp=80:80

Here we are not targeting a Deployment, but because I'm naming the NodePort Service my-node-port-service the service will look for all Pods that have the label selector app=my-node-port-service.

Let's expose a simple Nginx application.

student@minikube:~$ kubectl create deploy nginx-app --image=nginx:latest --replicas=3
deployment.apps/nginx-app created

student@minikube:~$ kubectl expose deploy nginx-app --port=80
service/nginx-app exposed

student@minikube:~$ kubectl get service nginx-app
NAME        TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
nginx-app   ClusterIP   10.97.84.163           80/TCP    8s

We've created a service of the type ClusterIP which is available on the internal IP address 10.97.84.163. The IP address is internal from the point of view of the Kubernetes cluster. Remember, we're not working inside the cluster:

student@minikube:~$ docker ps
CONTAINER ID   IMAGE                                 COMMAND                  CREATED       STATUS       PORTS                                                                                                                                  NAMES
4680f20c93ff   gcr.io/k8s-minikube/kicbase:v0.0.30   "/usr/local/bin/entr…"   2 hours ago   Up 2 hours   127.0.0.1:49157->22/tcp, 127.0.0.1:49156->2376/tcp, 127.0.0.1:49155->5000/tcp, 127.0.0.1:49154->8443/tcp, 127.0.0.1:49153->32443/tcp   minikube

On our minikube machine, we have a minikube Docker container running which runs the Kubernetes cluster and node inside. This means that we cannot reach the ClusterIP address from outside of Docker. In order to achieve that, we need to open a port on our Kubernetes Node using the NodePort service type. Edit the service, change the type and add the nodePort value:

student@minikube:~$ kubectl edit service nginx-app

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: nginx-app
  name: nginx-app
  namespace: default
  resourceVersion: "6160"
  uid: 8d2e2744-328d-4e1f-b8f8-96404515faae
spec:
  clusterIP: 10.97.84.163
  clusterIPs:
  - 10.97.84.163
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - nodePort: 32000
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx-app
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

Save your changes.

student@minikube:~$ kubectl get service nginx-app
NAME        TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
nginx-app   NodePort   10.97.84.163           80:32000/TCP   4m8s

We see that our Service Type has changed and the Service is running on port 80 accessible trough NodePort 32000:

student@minikube:~$ curl http://$(minikube ip):32000



Welcome to nginx!

The minikube ip command shows what IP address your Kubernetes node is using, in the above example I applied command substitution.

kubectl create service nodeport nginx-app --tcp=80:80

As opposed to the kubectl expose deployment, here we are not targeting a Deployment, but because I'm naming the NodePort Service nginx-app the service will look for all Pods that have the label selector app=nginx-app which would be all the Pods in our nginx-app Deployment.

student@minikube:~$ kubectl create service nodeport nginx-app --tcp=80:80
service/nginx-app created

student@minikube:~$ kubectl describe service nginx-app
Name:                     nginx-app
Namespace:                default
Labels:                   app=nginx-app
Annotations:              <none>
Selector:                 app=nginx-app

Using Service Resources in Microservices

In a microservices architecture, different frontend and backend Pods are used to provide the application:

Frontend Pods (e.g. webservers) can be exposed for external access using the NodePort Service type.
Backend Pods (e.g. databases) can be exposed internally only using the clusterIP Service type.

An example would be a frontend Deployment with WordPress and a backend Deployment with MariaDB. You don't want to expose MariaDB to external traffic, only the frontend Pods should be able to communicate with the database. They can do so using the Cluster IP address, or even without IP address by using a headless ClusterIP Service type. We'll cover that later on.

Services and DNS

Exposed Services automatically register with the Kubernetes internal DNS. The internal DNS consists of the kube-dns Service and the coreDNS Pod.
This allows all Pods to address Services using the Service name:

student@minikube:~$ kubectl get service,pods -n kube-system
NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.96.0.10           53/UDP,53/TCP,9153/TCP   3h39m

NAME                                   READY   STATUS    RESTARTS        AGE
pod/coredns-64897985d-2fwlb            1/1     Running   0               3h39m

Notice the Cluster IP address of the kube-dns service above. Now, let's run a Pod and have a look at its DNS settings:

student@minikube:~$ kubectl run testpod --image=busybox -- sleep 3600
pod/testpod created

student@minikube:~$ kubectl exec -it testpod -- cat /etc/resolv.conf 
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The nameserver is set to the Cluster IP address of the kube-dns service. Lookups are also done in the default.svc.cluster.local domain, where default is the name of the Name Space:

student@minikube:~$ kubectl exec -it testpod -- nslookup nginx-app
Server:        10.96.0.10
Address:    10.96.0.10:53

Name:    nginx-app.default.svc.cluster.local
Address: 10.96.165.179

student@minikube:~$ kubectl get service nginx-app
NAME        TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
nginx-app   NodePort   10.96.165.179           80:32000/TCP   8m35s

Ingress

Ingress is a Kubernetes API resource used to provide external access using DNS to internal Kubernetes cluster Services by means of an externally Ingress managed load balancer, also known as an Ingress Controller. Creating an Ingress resource without Ingress Controller has no effect, you need both. The Ingress Controller can be anything you're already familiar with: HAProxy, Nginx, Apache, traefik, kong, ...

{{

}}

To summarize, Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource. Ingress can be configured to do the following:

Give Services externally-reachable URLs
Terminate SSL/TLS
Load balance traffic
Offer name based virtual hosting

Configuring the Minikube Ingress Controller

Minikube provides an easy Ingress integration using a Minikube addon:

student@minikube:~$ minikube addons list
|-----------------------------|----------|--------------|--------------------------------|
|         ADDON NAME          | PROFILE  |    STATUS    |           MAINTAINER           |
|-----------------------------|----------|--------------|--------------------------------|
| ambassador                  | minikube | disabled     | third-party (ambassador)       |
| auto-pause                  | minikube | disabled     | google                         |
| csi-hostpath-driver         | minikube | disabled     | kubernetes                     |
| dashboard                   | minikube | disabled     | kubernetes                     |
| default-storageclass        | minikube | enabled ✅   | kubernetes                     |
| efk                         | minikube | disabled     | third-party (elastic)          |
| freshpod                    | minikube | disabled     | google                         |
| gcp-auth                    | minikube | disabled     | google                         |
| gvisor                      | minikube | disabled     | google                         |
| helm-tiller                 | minikube | disabled     | third-party (helm)             |
| ingress                     | minikube | disabled     | unknown (third-party)          |
| ingress-dns                 | minikube | disabled     | google                         |
| istio                       | minikube | disabled     | third-party (istio)            |
| istio-provisioner           | minikube | disabled     | third-party (istio)            |
| kong                        | minikube | disabled     | third-party (Kong HQ)          |
| kubevirt                    | minikube | disabled     | third-party (kubevirt)         |
| logviewer                   | minikube | disabled     | unknown (third-party)          |
| metallb                     | minikube | disabled     | third-party (metallb)          |
| metrics-server              | minikube | disabled     | kubernetes                     |
| nvidia-driver-installer     | minikube | disabled     | google                         |
| nvidia-gpu-device-plugin    | minikube | disabled     | third-party (nvidia)           |
| olm                         | minikube | disabled     | third-party (operator          |
|                             |          |              | framework)                     |
| pod-security-policy         | minikube | disabled     | unknown (third-party)          |
| portainer                   | minikube | disabled     | portainer.io                   |
| registry                    | minikube | disabled     | google                         |
| registry-aliases            | minikube | disabled     | unknown (third-party)          |
| registry-creds              | minikube | disabled     | third-party (upmc enterprises) |
| storage-provisioner         | minikube | enabled ✅   | google                         |
| storage-provisioner-gluster | minikube | disabled     | unknown (third-party)          |
| volumesnapshots             | minikube | disabled     | kubernetes                     |
|-----------------------------|----------|--------------|--------------------------------|

student@minikube:~$ minikube addons enable ingress
🌟  The 'ingress' addon is enabled

student@minikube:~$ kubectl get ns
NAME              STATUS   AGE
default           Active   73m
ingress-nginx     Active   88s
kube-node-lease   Active   73m
kube-public       Active   73m
kube-system       Active   73m

student@minikube:~$ kubectl get all -n ingress-nginx
NAME                                           READY   STATUS      RESTARTS   AGE
pod/ingress-nginx-admission-create-qz4sr       0/1     Completed   0          115s
pod/ingress-nginx-admission-patch-tbzsw        0/1     Completed   1          115s
pod/ingress-nginx-controller-cc8496874-nrsq6   1/1     Running     0          115s

NAME                                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             NodePort    10.101.67.244           80:30708/TCP,443:31969/TCP   116s
service/ingress-nginx-controller-admission   ClusterIP   10.106.3.163            443/TCP                      116s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   1/1     1            1           116s

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-cc8496874   1         1         1       116s

NAME                                       COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   1/1           12s        116s
job.batch/ingress-nginx-admission-patch    1/1           13s        116s

Using Ingress

The below example continues to build on the nginx-app Deployment and Service.

student@minikube:~$ kubectl create ingress nginx-app-ingress --rule="/=nginx-app:80" --rule="/hello=newdeploy:8080"
ingress.networking.k8s.io/nginx-app-ingress created

We create a new Ingress resource with the name nginx-app-ingress:

The first rule routes traffic from the root / to our nginx-app Service on port 80.
The second rule routes traffic from the URI /hello to a non existing newdeploy Service on port 8080.

student@minikube:~$ kubectl describe ingress nginx-app-ingress
Name:             nginx-app-ingress
Labels:           
Namespace:        default
Address:          192.168.49.2
Default backend:  default-http-backend:80 ()
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /        nginx-app:80 (172.17.0.4:80,172.17.0.5:80,172.17.0.6:80 + 2 more...)
              /hello   newdeploy:8080 ()
Annotations:  
Events:
  Type    Reason  Age                    From                      Message
  ----    ------  ----                   ----                      -------
  Normal  Sync    3m10s (x2 over 3m11s)  nginx-ingress-controller  Scheduled for sync

Notice that the backends or Pods for newdeploy are not found.

Before proceeding, update the /etc/hosts file to associate a domain with the IP address of our minikube container (which is running our K8s cluster). You can find the IP by running the minikube ip command. e.g. 192.168.42.2 nginx-app.demo

Next, let's test our Ingress resource:

student@minikube:~$ kubectl get ingress
NAME                CLASS   HOSTS   ADDRESS        PORTS   AGE
nginx-app-ingress   nginx   *       192.168.49.2   80      12m

student@minikube:~$ curl nginx-app.demo



Welcome to nginx!

student@minikube:~$ curl nginx-app.demo/hello

503 Service Temporarily Unavailable

We should fix the /hello URI by creating the newdeploy Deployment and Service:

student@minikube:~$ kubectl create deployment newdeploy --image=gcr.io/google-samples/hello-app:2.0
deployment.apps/newdeploy created

student@minikube:~$ kubectl expose deployment newdeploy --port=8080
service/newdeploy exposed

student@minikube:~$ curl nginx-app.demo/hello
Hello, world!
Version: 2.0.0
Hostname: newdeploy-698574c958-kvnbc

Configuring Ingress Rules

In the previous example, we've configured the nginx-app-ingress Ingress resource with the rules --rule="/=nginx-app:80" --rule="/hello=newdeploy:8080. Each Ingress Rules contains the following:

An optional host. If no host is specified, the rule applies to all inbound HTTP traffic.
A list of paths, each path has its own backend. Paths can be exposed as regular expressions.
The backend, which consists of either a service or a resource. You can configure a default backend for incoming traffic that doesn't match any of the defined backends. The service backed relates to a Service while a resource backend refers to Cloud based object storage. We'll focus on service backends.

The Ingress pathType specifies how to deal with path requests:

The Exact pathType indicates that an exact match should occur: If the path is set to /foo and the request is /foo/, there is no match.
The Prefix pathType indicates that the requested path should start with:
- If the path is set to /, any requested path will match.
- If the path is set to /foo, then /foo as well as /foo/ and /foo/bar will match.

There are different Ingress Types:

Single Service: kubectl create ingress ingress-name --rule="/hello=hello-service:80"
Simple fanout: kubectl create ingress ingress-name --rule="/hello=hello-service:80" --rule="/goodbye=goodbye-service:80"
Name-based Virtual Hosting: kubectl create ingress ingress-name --rule="my.example.com/hello*=hello-service:80" --rule="my.example.org/goodbye*=goodbye-service:80"

Let's cover this in an example:

student@minikube:~$ kubectl create deploy foo --image=nginx
deployment.apps/foo created

student@minikube:~$ kubectl create deploy bar --image=httpd
deployment.apps/bar created

student@minikube:~$ kubectl expose deploy foo --port=80
service/foo exposed

student@minikube:~$ kubectl expose deploy bar --port=80
service/bar exposed

student@minikube:~$ kubectl create ingress multihost --rule="foo.example.com/=foo:80" --rule="bar.example.com/=bar:80"
ingress.networking.k8s.io/multihost created

Create the necessary /etc/hosts entries for foo.example.com and bar.example.com. Edit the multihost Ingress resource and set the pathType to Prefix for both backends

student@minikube:~$ kubectl edit ingress multihost
ingress.networking.k8s.io/multihost edited

student@minikube:~$ kubectl get ingress multihost
NAME        CLASS   HOSTS                             ADDRESS        PORTS   AGE
multihost   nginx   foo.example.com,bar.example.com   192.168.49.2   80      75s

student@minikube:~$ curl foo.example.com



Welcome to nginx!

student@minikube:~$ curl foo.example.com/lololol

404 Not Found

student@minikube:~$ curl bar.example.com
It works!
student@minikube:~$ curl bar.example.com/lololol


404 Not Found

Network Policies

By default there are no restrictions to network traffic in Kubernetes: Pods can always communicate, even if they're in other Name Spaces. We can limit this by using Network Policies, however, this needs to be supported by the network plugin. Remember that by default Kubernetes only offers basic network connectivity and this can be expanded with third party plugins.

If you don't use a Network Policy, all traffic is allowed. If using a Network Policy and there's no match, traffic is denied. Minikube doesn't automatically start with a network plugin, so let's restart minikube and configure it to use the Calico network plugin:

student@minikube:~$ minikube stop
✋  Stopping node "minikube"  ...
🛑  Powering off "minikube" via SSH ...
🛑  1 node stopped.

student@minikube:~$ minikube delete
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /home/student/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.

student@minikube:~$ minikube start --cni=calico
😄  minikube v1.25.2 on Ubuntu 18.04 (amd64)
✨  Automatically selected the docker driver. Other choices: ssh, none
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=2200MB) ...
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring Calico (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

student@minikube:~$ kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS      AGE
calico-kube-controllers-8594699699-r4rwl   1/1     Running   0             2m7s
calico-node-8qzhj                          1/1     Running   0             2m7s

As with other Kubernetes resources, when defining a Pod- or NameSpace-based NetworkPolicy, a selector label is used to specify what traffic is allowed to and from the Pods that match the selector.

Three different NetworkPolicy Identifiers can be used to match network traffic:

podSelector: Allows access to a Pod with the corresponding selector label.
namespaceSelector: Allows incoming traffic from namespaces with the matching selector label.
ipBlock: Do not confuse with the verb to block - Specify a range of IP addresses that should get access.

Here's an example NetworkPolicy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: access-nginx
spec:
  podSelector:
    matchLabels:
      app: nginx
  ingress:
  - from:
    - podSelector:
        matchLabels:
          access: "true"
...

---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels: 
    app: nginx
spec:
  containers:
  - name: nwp-nginx
    image: nginx:1.17
...

---
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  labels:
    app: sleepy
spec:
  containers:
  - name: nwp-busybox
    image: busybox
    command:
    - sleep
    - "3600"

The above NetworkPolicy can be understood as follows:

Apply the Network Policy to Pods that have the label app: nginx
Allow incoming traffic from Pods that have the label access: "true"

In other words, our nginx Pod will only accept traffic from Pods that have the access: "true" label set:

student@minikube:~$ kubectl create -f ckad/nwpolicy-complete-example.yaml 
networkpolicy.networking.k8s.io/access-nginx created
pod/nginx created
pod/busybox created

student@minikube:~$ kubectl get networkpolicy
NAME           POD-SELECTOR   AGE
access-nginx   app=nginx      2m59s

student@minikube:~$ kubectl describe networkpolicy
Name:         access-nginx
Namespace:    default
Created on:   2021-12-01 18:12:12 +0000 UTC
Labels:       
Annotations:  
Spec:
  PodSelector:     app=nginx
  Allowing ingress traffic:
    To Port:  (traffic allowed to all ports)
    From:
      PodSelector: access=true
  Not affecting egress traffic
  Policy Types: Ingress

student@minikube:~$ kubectl expose pod nginx --port=80
service/nginx exposed

student@minikube:~$ kubectl exec -it busybox -- wget --spider --timeout=1 nginx
Connecting to nginx (10.108.90.255:80)
wget: download timed out
command terminated with exit code 1

student@minikube:~$ kubectl label pod busybox access=true
pod/busybox labeled

student@minikube:~$ kubectl exec -it busybox -- wget --spider --timeout=1 nginx
Connecting to nginx (10.108.90.255:80)
remote file exists
student@minikube:~$

Kubernetes 101: Building Scalable Applications - Deployments

Joeri JM Smissaert — Wed, 03 Nov 2021 00:00:00 GMT

{{

}}

Deployments

Deployments are the standard for running applications in Kubernetes, it protects Pods and will automatically restart them if anything goes wrong. Additionally, it offer features that add to the scalability and reliability of the application:

Scalability: Scaling the number of application instances to meet the demand.
Updates and Update Strategy: Zero-downtime application updates

We use the kubectl create deploy command to create a Deployment:

student@minikube:~$ kubectl create deployment myweb --image=nginx --replicas=3
deployment.apps/myweb created

student@minikube:~$ kubectl describe deploy myweb
Name:                   myweb
Namespace:              default
CreationTimestamp:      Mon, 01 Nov 2021 09:08:57 +0000
Labels:                 app=myweb
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=myweb
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=myweb
  Containers:
   nginx:
    Image:        nginx
    Port:         
    Host Port:    
    Environment:  
    Mounts:       
  Volumes:        
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  
NewReplicaSet:   myweb-8764bf4c8 (3/3 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  2m29s  deployment-controller  Scaled up replica set myweb-8764bf4c8 to 3


student@minikube:~$ kubectl get all
NAME                        READY   STATUS    RESTARTS   AGE
pod/myweb-8764bf4c8-6gxv8   1/1     Running   0          4m23s
pod/myweb-8764bf4c8-6mvn8   1/1     Running   0          4m23s
pod/myweb-8764bf4c8-q72nq   1/1     Running   0          4m23s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1            443/TCP   36m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/myweb   3/3     3            3           4m23s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/myweb-8764bf4c8   3         3         3       4m23s

We created the myweb deployment based on the nginx image with 3 replicas or desired Pods. Notice the Labels and Selector fields.
The Deployment created the ReplicaSet to ensure that a specified number of Pods are always running at any given time, and it created the Pods. Both the ReplicaSet and the Pods are managed by the Deployment.

You cannot manage Pods independently when they are part of a Deployment. When trying to delete a Pod, the Deployment kicks in and uses the ReplicaSet to make sure we have 3 running Pods:

student@minikube:~$ kubectl delete pod myweb-8764bf4c8-6gxv8
kpod "myweb-8764bf4c8-6gxv8" deleted

student@minikube:~$ kubectl get pods
NAME                    READY   STATUS              RESTARTS   AGE
myweb-8764bf4c8-6mvn8   1/1     Running             0          14m
myweb-8764bf4c8-q72nq   1/1     Running             0          14m
myweb-8764bf4c8-qf2vc   0/1     ContainerCreating   0          5s

Deployment Scalability

Before Deployments existed, ReplicaSets were used to manage scalability. In the previous section we saw that our deployment created the necessary ReplicaSet: Manage ReplicaSets only through Deployments. We do not care about managing ReplicaSets individually.

We can use the kubectl scale deployment command to manually scale an existing deployment:
kubectl scale deployment my-deployment --replicas=5

student@minikube:~$ kubectl scale deployment myweb --replicas=5
deployment.apps/myweb scaled

student@minikube:~$ kubectl describe deploy myweb | grep -i replicas
Replicas:               5 desired | 5 updated | 5 total | 5 available | 0 unavailable

student@minikube:~$ kubectl get pods
NAME                    READY   STATUS              RESTARTS   AGE
myweb-8764bf4c8-44zxq   0/1     ContainerCreating   0          3s
myweb-8764bf4c8-6mvn8   1/1     Running             0          36m
myweb-8764bf4c8-7dpnx   0/1     ContainerCreating   0          3s
myweb-8764bf4c8-q72nq   1/1     Running             0          36m
myweb-8764bf4c8-qf2vc   1/1     Running             0          22m

Additionally, there's the kubectl edit deployment command which opens a text-editor for you, similar to systemctl edit for editing Systemd Unit files. This command, however, does not allow you to modify every single setting of a deployment.
In the below example I changed the deployment namespace and replicas:

student@minikube:~$ kubectl edit deploy myweb
A copy of your changes has been stored to "/tmp/kubectl-edit-3283969971.yaml"
error: the namespace from the provided object "secret" does not match the namespace "default". You must pass '--namespace=secret' to perform this operation.

As you can see, Kubernetes isn't happy about changing the namespace.

Deployment Updates

Deployments allow for zero-downtime application updates.
When an update is applied, a new ReplicaSet is created with the new properties: Pods with the new properties are started in the new ReplicaSet. After updating, the old ReplicaSet is no longer used and may be deleted. Or, you can keep it around for rolling-back. The deployment.spec.revisionHistoryLimit is set to keep the last 10 ReplicaSets.

The deployment.spec.strategy.type property defines how to handle updates:

RollingUpdate: The default value. Replaces old Pods with new Pods in such a way to ensure the application remains available to users.
Recreate: Kill all existing Pods before creating new ones. The application will be down. More on the this later...

Let's perform a rolling update of Nginx using the kubectl set command. The command only accepts a limited amount of arguments.


student@minikube:~$ kubectl create deploy mynginx --image=nginx:1.14
deployment.apps/mynginx created

student@minikube:~$ kubectl describe deploy mynginx
Name:                   mynginx
Namespace:              default
CreationTimestamp:      Sat, 02 Apr 2022 10:29:52 +0000
Labels:                 app=mynginx
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=mynginx
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=mynginx
  Containers:
   nginx:
    Image:        nginx:1.14

student@minikube:~$ kubectl get all --selector app=mynginx
NAME                           READY   STATUS    RESTARTS   AGE
pod/mynginx-6b9d85f696-w4wpt   1/1     Running   0          64s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mynginx   1/1     1            1           64s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/mynginx-6b9d85f696   1         1         1       64s

Notice the Image field in the output of the kubectl describe command, the default StrategyType, as well as how the middle part of the Pod name matches the suffix of the ReplicaSet name: pod/mynginx-6b9d85f696-w4wpt => replicaset.apps/mynginx-6b9d85f6960. We can conclude that this Pod belongs to that ReplicaSet.

Now, update the image version to 1.17. The kubectl set command only accepts a limited amount of arguments.

student@minikube:~$ kubectl set 
env             image           resources       selector        serviceaccount  subject 

student@minikube:~$ kubectl set image deploy mynginx nginx=nginx:1.17
deployment.apps/mynginx image updated

student@minikube:~$ kubectl get all --selector app=mynginx
NAME                           READY   STATUS              RESTARTS   AGE
pod/mynginx-6b9d85f696-w4wpt   1/1     Running             0          7m4s
pod/mynginx-6d9cd8f877-g4dkv   0/1     ContainerCreating   0          8s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mynginx   1/1     1            1           7m4s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/mynginx-6b9d85f696   1         1         1       7m4s
replicaset.apps/mynginx-6d9cd8f877   1         1         0       9s

We see that our old ReplicaSet and Pod are still there, our application is still available, while a new Pod with the new Nginx image is being created. Once the new Pod is running, the old Pod will be deleted but the old (empty) ReplicaSet will still be there:

student@minikube:~$ kubectl get all --selector app=mynginx
NAME                           READY   STATUS    RESTARTS   AGE
pod/mynginx-6d9cd8f877-g4dkv   1/1     Running   0          2m4s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mynginx   1/1     1            1           9m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/mynginx-6b9d85f696   0         0         0       9m
replicaset.apps/mynginx-6d9cd8f877   1         1         1       2m5s

The rolling update is complete and the old ReplicaSet is still available in case we need to roll back (covered later on in this article).

Labels, Selectors, and Annotations

Labels are key:value pairs that are defined in resources like Pods, Deployments and Services. They are either set automatically or can be set manually by an administrator. Each label key that is attached to a single object resource must be unique, though different objects can have the same label key:value pairs. This allows us to group objects, or map a specific structure onto objects, and query only the objects with a specific label.

If we look back at our previous deployment, we can see that each object in the deployment has the app=mynginx label set:

student@minikube:~$ kubectl describe pod mynginx-6d9cd8f877-g4dkv | grep Labels:
Labels:       app=mynginx

student@minikube:~$ kubectl describe rs mynginx | grep Labels:
Labels:         app=mynginx

student@minikube:~$ kubectl describe deploy mynginx | grep Labels:
Labels:                 app=mynginx

So using a label selector, we can target the related objects of a specific application.
e.g.:

student@minikube:~$ kubectl get all --selector app=mynginx
NAME                           READY   STATUS    RESTARTS   AGE
pod/mynginx-6d9cd8f877-g4dkv   1/1     Running   0          29m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mynginx   1/1     1            1           36m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/mynginx-6b9d85f696   0         0         0       36m
replicaset.apps/mynginx-6d9cd8f877   1         1         1       29m

Our kubectl create deployment command automatically set the app=appname label, where appname is the name of the deployment.

Example:

student@minikube:~$ kubectl create deploy mylabel --image=nginx
deployment.apps/mylabel created

student@minikube:~$ kubectl label deploy mylabel state=demo
deployment.apps/mylabel labeled

student@minikube:~$ kubectl get deploy --show-labels
NAME      READY   UP-TO-DATE   AVAILABLE   AGE   LABELS
mylabel   1/1     1            1           45s   app=mylabel,state=demo

student@minikube:~$ kubectl get deploy --selector state=demo
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
mylabel   1/1     1            1           70s

Notice that while we've given the deployment mylabel a new label, this new label is not inherited by the resources or objects created by the deployment:

student@minikube:~$ kubectl get all --show-labels
NAME                           READY   STATUS    RESTARTS   AGE     LABELS
pod/mylabel-566dc5f574-ctkqg   1/1     Running   0          7m28s   app=mylabel,pod-template-hash=566dc5f574

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE    LABELS
service/kubernetes   ClusterIP   10.96.0.1            443/TCP   168m   component=apiserver,provider=kubernetes

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE     LABELS
deployment.apps/mylabel   1/1     1            1           7m28s   app=mylabel,state=demo

NAME                                 DESIRED   CURRENT   READY   AGE     LABELS
replicaset.apps/mylabel-566dc5f574   1         1         1       7m28s   app=mylabel,pod-template-hash=566dc5f574

student@minikube:~$ kubectl get all --selector state=demo
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mylabel   1/1     1            1           8m58s

We can also remove a label. Let's remove the label with the key app from the Pod mylabel-566dc5f574-ctkqg:

student@minikube:~$ kubectl label pod mylabel-566dc5f574-ctkqg app-
pod/mylabel-566dc5f574-ctkqg unlabeled

student@minikube:~$ kubectl get all
NAME                           READY   STATUS              RESTARTS   AGE
pod/mylabel-566dc5f574-ctkqg   1/1     Running             0          12m
pod/mylabel-566dc5f574-pxkdz   0/1     ContainerCreating   0          5s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1            443/TCP   174m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mylabel   0/1     1            0           12m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/mylabel-566dc5f574   1         1         0       12m

student@minikube:~$ kubectl get all --selector app=mylabel
NAME                           READY   STATUS    RESTARTS   AGE
pod/mylabel-566dc5f574-pxkdz   1/1     Running   0          3m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mylabel   1/1     1            1           15m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/mylabel-566dc5f574   1         1         1       15m

Our deployment could no longer find the Pod which is supposed to have the app=mylabel label, so it created a new Pod: mylabel-566dc5f574-pxkdz.
Since the Pod with the removed label is no longer managed by our deployment, we can delete it without our deployment (or rather ReplicaSet) recreating it.

Annotations can't be used in queries, but are useful to provide detailed non-identifying metadata in an object: maintainer, author, license, ...

Update Strategy

When a Deployment changes, the Pods are immediately updated according to the Update Strategy:

RollingUpdate: Updates Pods one at a time to guarantee availability of the application.
Recreate: All Pods are killed and new Pods are created. This leads to temporary unavailability of the application which can be useful when different versions of an application cannot run simultaneously (e.g. a database).

The task of the Deployment is to ensure that enough Pods are running at all times. When a change is made, the changed version is deployed in a new ReplicaSet. The old ReplicaSet is scaled to 0 (deactivated) once the update was confirmed as successful. We can use kubectl rollout history to get details about recent transactions, and kubectl rollout undo to undo a previous change.

The RollingUpdate options guarantee a certain minimal and maximum number of Pods to be always available:

maxUnavailable: Determines the maximum number of Pods that are upgraded at the same time.
maxSurge: The number of Pods that can run beyond the desired number of Pods specified in the ReplicaSet to guarantee minimal availability.

student@minikube:~$ kubectl get deploy mylabel -o yaml
...
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: mylabel
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
...

Deployment History

At this point we know that Deployment updates create a new ReplicaSet with new properties, the old ReplicaSet is kept but is scaled down to 0 Pods. Since the old ReplicaSet is kept around, we can easily undo a change. We can use kubectl rollout history to get details about recent roll outs, and kubectl rollout undo to undo a previous change.

Let's start by updating our mylabel deployment. We'll give all the Pods a new environment variable: foo=bar:

kubectl set env deploy mylabel foo=bar
deployment.apps/mylabel env updated

student@minikube:~$ kubectl rollout history deploy mylabel
deployment.apps/mylabel 
REVISION  CHANGE-CAUSE
1         
2         

student@minikube:~$ kubectl rollout history deploy mylabel --revision=1
deployment.apps/mylabel with revision #1
Pod Template:
  Labels:    app=mylabel
    pod-template-hash=566dc5f574
  Containers:
   nginx:
    Image:    nginx
    Port:    
    Host Port:    
    Environment:    
    Mounts:    
  Volumes:    


student@minikube:~$ kubectl rollout history deploy mylabel --revision=2
deployment.apps/mylabel with revision #2
Pod Template:
  Labels:    app=mylabel
    pod-template-hash=57f55bcb47
  Containers:
   nginx:
    Image:    nginx
    Port:    
    Host Port:    
    Environment:
      foo:    bar
    Mounts:    
  Volumes:

We can see that we added the environment variable in revision 2. So let's roll back revision 1:

student@minikube:~$ kubectl rollout undo deploy mylabel --to-revision=1
deployment.apps/mylabel rolled back

Deployment Alternatives

There are two additional Deployments alternatives:

StatefulSets: the workload API object used to manage stateful applications. We'll cover these once we know more about Networking and Storage.
DaemonSet: ensures that all (or some) Nodes run a copy of a Pod (1 Pod, no replicas). As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

A simple use case for a DaemonSet is for example the need to run some sort of Agent on every worker-node.

The YAML code for DaemonSets needs to be created from scratch, you can't use kubectl create to generate the YAML :( Example YAML code:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginxdaemon
  namespace: default
  labels:
    k8s-app: nginxdaemon
spec:
  selector:
    matchLabels:
      name: nginxdaemon
  template:
    metadata:
      labels:
        name: nginxdaemon
    spec:
      containers:
      - name: nginx
        image: nginx

student@minikube:~$ kubectl create -f daemon.yaml 
daemonset.apps/nginxdaemon created

student@minikube:~$ kubectl get ds,pods
NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nginxdaemon   1         1         1       1            1                     13s

NAME                    READY   STATUS    RESTARTS   AGE
pod/nginxdaemon-5nn27   1/1     Running   0          13s

Kubernetes 101: Kubernetes Essentials

Joeri JM Smissaert — Sat, 02 Oct 2021 00:00:00 GMT

{{

}}

Managing Basic Pod Features

Deployments are the standard for running applications in Kubernetes. For the sake of getting familiar with Kubernetes and understanding the essentials, we'll be creating and running native Pods.

Understanding Pods

A Pod is an abstraction of a server which can run multiple containers within a single namespace, exposed by a single IP address. The Pod is the smallest entity that can be created and managed by Kubernetes: Kubernetes does not manage containers, it manages Pods.

Managing Pods with kubectl

Typically managed Pods are started through a Deployment resource.
Naked Pods are started using the kubectl run option: kubectl run mynginx --image=nginx
Naked Pods cannot be scaled, are not rescheduled in case of failure, cannot be replaced automatically and can't have rolling updates.

kubectl run -h: Show all options for creating a Pod.
kubectl run mynginx --image=nginx: Start a Pod with the name mynginx from the nginx Dockerhub image.
kubectl get pods: Show the parameters of all Pods
kubectl get pods mynginx: Show the parameters of a specific Pod
kubectl get pods mynginx -o yaml: Show the output in YAML format.
kubectl describe pods: Show all details about all pods
kubectl describe pods mynginx: Show all details about a specific Pod

YAML

YAML is a human-readable data-serialization language which uses indentation to identify relations.

Basic YAML Manifest Ingredients

All of the YAML manifest ingredients are defined in the API. You can use kubectl explain to get more information about the YAML fields or properties:

student@minikube:~$ kubectl explain pods
KIND:     Pod
VERSION:  v1

DESCRIPTION:
     Pod is a collection of containers that can run on a host. This resource is
     created by clients and scheduled onto hosts.

FIELDS:
   apiVersion    
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

   kind    
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

   metadata    <Object>
     Standard object's metadata. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

   spec    
     Specification of the desired behavior of the pod. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

   status    
     Most recently observed status of the pod. This data may not be up to date.
     Populated by the system. Read-only. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

student@minikube:~$ kubectl explain pods.spec
...
student@minikube:~$ kubectl explain pods.spec.containers
...

The kubectl explain pods.spec.containers command shows us that the container spec has multiple fields of which the below are the most important ones:

FIELDS:

  name  -required-
    Name of the container specified as a DNS_LABEL.

  image 
    Docker image name.  

  command <[]string>
    Entrypoint array. Not executed within a shell. The docker image's
    ENTRYPOINT is used if this is not provided.

  args <[]string>
    Arguments to the entrypoint. The docker image's CMD is used if this is not
    provided

  env  <[]Object>
    List of environment variables to set in the container. Cannot be updated.

If you have a YAML file with a Pod spec you can create a Pod from it:

student@minikube:~$ cat busybox.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: busybox2
  namespace: default
spec:
  containers:
  - name: busy
    image: busybox
    command:
      - sleep
      - "3600" 
student@minikube:~$ kubectl create -f busybox.yaml 
pod/busybox2 created

student@minikube:~$ kubectl get pods
NAME       READY   STATUS              RESTARTS   AGE
busybox2   0/1     ContainerCreating   0          7s

Similarly you can delete or update (apply Spec changes) the Pod using the same YAML file:

student@minikube:~$ kubectl delete -f busybox.yaml 
pod "busybox2" deleted

student@minikube:~$ kubectl apply -f busybox.yaml 
pod/busybox2 created

student@minikube:~$ kubectl apply -f busybox.yaml 
pod/busybox2 unchanged

Generating YAML files

By using YAML files we use Kubernetes in a declarative way where the files are typically stored in a git repository and which fits well into a DevOps strategy. The imperative way of working with Kubernetes is where you create everything from the command line.

We can write YAML files but we should generate them instead and modify it to suit our specific needs:
kubectl run mynginx --image=nginx --dry-run=client -o yaml > mynginx.yaml
The --dry-run option prevents Kubernetes from actually running the Pod.

Understanding and Configuring Multi-Container Pods

The one-container Pod is the standard, they are easier to build and maintain. Typically, to create applications that consists of multiple containers, micro-services should be used. In a microservice, different independently managed Pods are connected by resources provided by Kubernetes.

There are some use cases where you might want to run multiple containers in a single Pod:

Sidecar container: A container that enhances the primary application, for example logging.
Ambassador container: A container that represents the primary container to the outside world, for example a proxy.
Adapter container: Used to adopt the traffic or data pattern to match the traffic or data pattern in other applications in the cluster.

These containers are not defined by specific Pod properties, you won't find information on their specs in kubectl explain pod.spec.

Sidecar Containers

A sidecar container is providing additional functionality to the main container where it makes no sense to run this functionality in a separate Pod. The essence is that the main container and sidecar container have access to shared resources in order to exchange information.
e.g. Istio service mesh injects sidecar containers in Pods to enable traffic management.

Here's a basic example of a multi-container Pod:

student@minikube:~$ cat sidecar.yaml 
kind: Pod
apiVersion: v1
metadata:
  name: sidecar-pod
spec:
  volumes:
  - name: logs
    emptyDir: {}

  containers:
  - name: main
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "while true; do date >> /var/log/date.txt; sleep
10;done"]
    volumeMounts:
    - name: logs
      mountPath: /var/log

  - name: sidecar
    image: centos/httpd
    ports:
    - containerPort: 80
    volumeMounts:
    - name: logs
      mountPath: /var/www/html

The shared resource in the above example is the volume with the name logs and the emptyDir: {} property. An emptyDir volume is initially empty and can be mounted at different paths in different containers as we can see in the above YAML by looking at the container volumeMounts.

The main container writes the current date and time to /var/log/date.txt every 10 seconds, while the sidecar container will be able to read and present the file to a user since it has the same volume mounted albeit on a different path from the container perspective.

Let's create the Pod, open a shell session in the sidecar container and run cURL to check the output created by the main container:

student@minikube:~$ kubectl create -f sidecar.yaml 
pod/sidecar-pod created

student@minikube:~$ kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
sidecar-pod   2/2     Running   0          10s

student@minikube:~$ kubectl exec -it sidecar-pod -c sidecar -- /bin/bash
[root@sidecar-pod /]# yum install curl -y
....
[root@sidecar-pod /]# curl http://localhost/date.txt
....

Managing Init Containers

An init container is an additional container in a Pod that needs to complete a task before the "regular" container is started. As long as the init container hasn't completed its job, the regular container is not started.

Have a look at this official example YAML file fo init containers. We'll work with a more simplified version here:

apiVersion: v1
kind: Pod
metadata:
  name: init-demo
spec:
  containers:
  - name: nginx
    image: nginx
  initContainers:
  - name: init-box
    image: busybox
    command:
    - sleep
    - "3600"

In the above example our init-box container will sleep for 1 hour and only once the sleep command finishes our nginx container will spin up. We can see our Pod is in the Init status:

student@minikube:~$ kubectl create -f init-demo.yaml 
pod/init-demo created

student@minikube:~$ kubectl get pods
NAME        READY   STATUS     RESTARTS   AGE
init-demo   0/1     Init:0/1   0          4s

We can use the describe command to get more information about the Pod:

student@minikube:~$ kubectl describe pod init-demo
...
Init Containers:
  init-box:
    Container ID:  docker://c23ec32c3ba19d43417c730117b6319b0c57d6c8938c76ae641b1afad0e08c11
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:caa382c432891547782ce7140fb3b7304613d3b0438834dce1cad68896ab110a
    Port:          
    Host Port:     
    Command:
      sleep
      3600
    State:          Running
...
Containers:
  nginx:
    Container ID:   
    Image:          nginx
    Image ID:       
    Port:           
    Host Port:      
    State:          Waiting
      Reason:       PodInitializing
...

The Events section in the output of the describe command shows us what containers have been started:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  78s   default-scheduler  Successfully assigned default/init-demo to minikube
  Normal  Pulling    77s   kubelet            Pulling image "busybox"
  Normal  Pulled     65s   kubelet            Successfully pulled image "busybox" in 11.886228471s
  Normal  Created    65s   kubelet            Created container init-box
  Normal  Started    64s   kubelet            Started container init-box

Using NameSpaces

Kubernetes leverages Linux kernel-level resource isolation: NameSpaces. Different NameSpaces can be used to strictly separate between customer resources and to apply different security-related settings such as Role-Based Access Control and Quotas.

Let's demonstrate this the imparative way:

# Show all available namespaces
student@minikube:~$ kubectl get ns
NAME              STATUS   AGE
default           Active   14d
kube-node-lease   Active   14d
kube-public       Active   14d
kube-system       Active   14d

# Show all resources per namespace
student@minikube:~$ kubectl get all -A
NAMESPACE     NAME                                   READY   STATUS    RESTARTS       AGE
kube-system   pod/coredns-64897985d-sj5lw            1/1     Running   3 (108s ago)   14d
...

#  Create a new namespace
student@minikube:~$ kubectl create ns secret
namespace/secret created

# Start a new Pod in the new namespace
student@minikube:~$ kubectl run secretnginx --image=nginx -n secret
pod/secretnginx created

student@minikube:~$ kubectl get pods
No resources found in default namespace.

# List all Pods in the secret namespace
student@minikube:~$ kubectl get pods -n secret
NAME          READY   STATUS              RESTARTS   AGE
secretnginx   0/1     ContainerCreating   0          10s

We can do the same thing the declarative way by defining namespace under the Pod metadata:

apiVersion: v1
kind: Pod
metadata:
  name: busyboxPod
  namespace: secret

Check properties of the namespace using the describe command:

student@minikube:~$ kubectl describe ns secret
Name:         secret
Labels:       kubernetes.io/metadata.name=secret
Annotations:  <none>
Status:       Active

No resource quota.

No LimitRange resource.

Lastly, let's declaratively combine the creation of a namespace and a pod inside the same namespace:

student@minikube:~$ kubectl create ns production --dry-run=client -o yaml > nginx_prod.yml
student@minikube:~$ cat nginx_prod.yml 
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: null
  name: production
spec: {}
status: {}

Notice that kind is Namespace.
Now, we add the Pod to the same namespace inside the same Yaml file:

student@minikube:~$ kubectl run nginx-prod -n production --image=nginx --dry-run=client -o yaml >> nginx_prod.yml 
student@minikube:~$ cat nginx_prod.yml 
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: null
  name: production
spec: {}
status: {}
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: nginx-prod
  name: nginx-prod
  namespace: production
spec:
  containers:
  - image: nginx
    name: nginx-prod
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

We should modify the Yaml file in such a way that it's clear we're dealing with 2 Yaml list items in a single file. We'll add the --- lines to indicate the start of a new list item:

student@minikube:~$ cat nginx_prod.yml 
---
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: null
  name: production
spec: {}
status: {}
---
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: nginx-prod
  name: nginx-prod
  namespace: production
spec:
  containers:
  - image: nginx
    name: nginx-prod
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

... and we can now create the actual resources from the Yaml file:

student@minikube:~$ kubectl create -f nginx_prod.yml 
namespace/production created
pod/nginx-prod created

student@minikube:~$ kubectl get all -n production
NAME             READY   STATUS              RESTARTS   AGE
pod/nginx-prod   0/1     ContainerCreating   0          15s
student@minikube:~$

Managing Advanced Pod Features

Exploring Pod State

kubectl describe pod podname is a human-readable way to see all Pod parameters and settings as currently stored in the etcd database. You can use the offical documentation for more information about these settings and parameters.

While we can describe the Pod externally, we can also connect to the Pod and run commands on the primary container in the Pod:

Connect using kubectl exec -it podname -- sh
Disconnect by executing the exit command. (or CTR+P CTRL+Q if the shell is running as process ID 1)

student@minikube:~$ kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
mynginx   1/1     Running   0          44s

student@minikube:~$ kubectl get pods mynginx -o json | less
...
student@minikube:~$ kubectl get pods mynginx -o yaml | less
...
student@minikube:~$ kubectl describe pods mynginx
...
student@minikube:~$ kubectl exec -it mynginx -- sh
# pwd
/
# ps aux
sh: 1: ps: not found
# cd /proc
# ls
1   acpi       cmdline     diskstats    filesystems  irq          kmsg       locks    mounts      sched_debug  softirqs       sysvipc       version
34  asound     consoles  dma          fs       kallsyms   kpagecgroup  mdstat   mtrr      schedstat    stat          thread-self  version_signature
35  buddyinfo  cpuinfo     driver       interrupts   kcore      kpagecount   meminfo  net          scsi           swaps          timer_list   vmallocinfo
53  bus        crypto     execdomains  iomem       key-users  kpageflags   misc     pagetypeinfo  self           sys          tty       vmstat
60  cgroups    devices     fb          ioports       keys       loadavg       modules  partitions      slabinfo     sysrq-trigger  uptime       zoneinfo
# cat 1/cmdline
nginx: master process nginx -g daemon off;
# cat 53/cmdline
sh
# cat 35/cmdline
nginx: worker process
# exit
student@minikube:~$

Most containers run minimal images where not all commands may be available, in the above example the ps command is not available. In this case we can make advantage of the proc pseudo filesystem.

Using Pod Logs

The Pod entrypoint application does not connect to any STDOUT, instead, application output is sent to the Kubernetes cluster. We can use kubectl logs to see this output and help us in troubleshooting:

student@minikube:~$ kubectl run mydb --image=mariadb
pod/mydb created

student@minikube:~$ kubectl get pods
NAME      READY   STATUS              RESTARTS   AGE
mydb      0/1     ContainerCreating   0          5s
...
student@minikube:~$ kubectl get pods
NAME      READY   STATUS             RESTARTS      AGE
mydb      0/1     CrashLoopBackOff   1 (15s ago)   76s

student@minikube:~$ kubectl describe pod mydb
...
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
...

student@minikube:~$ kubectl logs mydb
[ERROR] [Entrypoint]: Database is uninitialized and password option is not specified
    You need to specify one of MARIADB_ROOT_PASSWORD, MARIADB_ALLOW_EMPTY_ROOT_PASSWORD and MARIADB_RANDOM_ROOT_PASSWORD

Looking at the log output, we needed to specify one or more specific environment variables. Let's fix this, but since we can't update a Pod (only deployments which we'll see later) we need to delete our Pod first:

student@minikube:~$ kubectl delete pod mydb
pod "mydb" deleted

student@minikube:~$ kubectl run mydb --image=mariadb --env="MARIADB_ROOT_PASSWORD=myrootpassword"
pod/mydb created

student@minikube:~$ kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
mydb      1/1     Running   0          40s

student@minikube:~$ kubectl logs mydb
[Note] mariadbd: ready for connections.
Version: '10.7.3-MariaDB-1:10.7.3+maria~focal'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

Port Forwarding

A simple way of accessing a Pod is by using Port Forwarding: Expose a port on the host running the Pod that forwards to the Pod. This is useful for testing Pod accessibility on a specific cluster node but isn't used to expose the Pod to external users. Regular user access to applications in the Pod is provided via Services and Ingress.

When you run kubectl get pods -o wide or kubectl describe pod podname you'll see the Pod has an IP address. This IP address is accessible only from within the cluster, you cannot use it to address the Pod from outside the cluster.

student@minikube:~$ kubectl get pods mynginx -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
mynginx   1/1     Running   0          63m   172.17.0.3   minikube              <none>

student@minikube:~$ ping 172.17.0.3
PING 172.17.0.3 (172.17.0.3) 56(84) bytes of data.
^C
--- 172.17.0.3 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2053ms

student@minikube:~$ curl 172.17.0.3
curl: (7) Failed to connect to 172.17.0.3 port 80: No route to host

So if we need to test network accessibility to our Pod, we use Port Forwarding:

student@minikube:~$ kubectl port-forward mynginx 8080:80 &
[1] 19855
student@minikube:~$ Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
student@minikube:~$

This command starts a port forwarding process in the foreground, so we add the & at the end of the command to start it in the background.

student@minikube:~$ curl localhost:8080
Handling connection for 8080



Welcome to nginx!

To stop port forwarding, we bring the process back to the foreground and stop it using CTRL+C:

student@minikube:~$ fg
minikube kubectl -- port-forward mynginx 8080:80
^C
student@minikube:~$

Configuring securityContext

A securityContext defines privileges and access control settings for a Pod and/or container, and includes:

Discretionary Access Control
SELinux or AppArmor
Running as privileged or unprivileged user
AllowPrivilegeEscalation to control if a process can gain more privileges than its parent process

kubectl explain can give you a complete overview.

Let's work with examples:

student@minikube:~$ kubectl explain pod.spec.securityContext
...
student@minikube:~$ kubectl explain pod.spec.containers.securityContext
...
student@minikube:~/ckad$ cat securitycontextdemo2.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: security-context-demo
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  volumes:
  - name: sec-ctx-vol
    emptyDir: {}
  containers:
  - name: sec-ctx-demo
    image: busybox
    command: [ "sh", "-c", "sleep 1h" ]
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
    securityContext:
      allowPrivilegeEscalation: false

student@minikube:~/ckad$ kubectl create -f securitycontextdemo2.yaml 
pod/security-context-demo created

student@minikube:~/ckad$ kubectl get pods security-context-demo -o yaml
...
spec:
  containers:
  - command:
    - sh
    - -c
    - sleep 1h
    image: busybox
    imagePullPolicy: Always
    name: sec-ctx-demo
    resources: {}
    securityContext:
      allowPrivilegeEscalation: false
...

student@minikube:~/ckad$ kubectl exec -it security-context-demo -- sh
/ $ cd data/demo
/data/demo $ echo "Hello" > test
/data/demo $ ls -l
total 4
-rw-r--r--    1 1000     2000             6 Mar 24 17:05 test
/data/demo $ id
uid=1000 gid=3000 groups=2000

When we create a new file in the Pods primary container, we see that the owner of the file is id 1000 (runAsUser) and group owner is 2000 (fsGroup) as specified in the Yaml securityContext. The id command reveals our runAsUser ID, our primary group id 3000 and our secondary group id 2000.

Managing Jobs

Pods are the essence of Kubernetes, when your Pod goes down then Kubernetes will start a new Pod. In that sense, Pods are normally created to run forever. There can be cases where you want a Pod to execute a one-shot task, like backup jobs, a calculation or batch processing. This is were you can use Jobs: The Pod will run until it finishes its task then stops.

We can set ttlSecondsAfterFinished to clean up completed Jobs automatically so that we don't keep both the Job and the Pod (created by the Job) around forever.

There are 3 different Job types specified by the completion and parallelism parameters:

Non-parallel Jobs: 1 Job - 1 Pod
- completions=X
- paralellism=1
Parallel Jobs with a fixed completion count: the Job is completed after successfully running as many times as specified by jobs.spec.completions. The number of parallel or concurrent Pods that are started by the Job are specified by jobs.spec.parallelism.
- completions=X
- paralellism=Y
Parallel Jobs with a work queue: Multiple Jobs are started, when one completes the Job is done.
- completions=1
- parallelism=X

Here's an example:

student@minikube:~$ kubectl create job onejob --image=busybox --dry-run=client -o yaml -- date > onejob.yml
student@minikube:~$ cat onejob.yml 
apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: null
  name: onejob
spec:
  template:
    metadata:
      creationTimestamp: null
    spec:
      containers:
      - command:
        - date
        image: busybox
        name: onejob
        resources: {}
      restartPolicy: Never
status: {}

Notice that kind is Job and that restartPolicy is set to Never. In this example the container just executes the date command and then is done.

student@minikube:~$ kubectl create -f onejob.yml 
job.batch/onejob created

student@minikube:~$ kubectl get jobs
NAME     COMPLETIONS   DURATION   AGE
onejob   0/1           4s         4s

student@minikube:~$ kubectl get jobs,pods
NAME               COMPLETIONS   DURATION   AGE
job.batch/onejob   0/1           7s         7s

NAME               READY   STATUS              RESTARTS   AGE
pod/onejob-zjgd9   0/1     ContainerCreating   0          7s

Once the Job is done, the Job COMPLETIONS and Pod STATUS is updated:

student@minikube:~$ kubectl get jobs,pods
NAME               COMPLETIONS   DURATION   AGE
job.batch/onejob   1/1           10s        41s

NAME               READY   STATUS      RESTARTS   AGE
pod/onejob-zjgd9   0/1     Completed   0          41s

student@minikube:~$ kubectl delete -f onejob.yml 
job.batch "onejob" deleted

Now let's create a parallel Job:

student@minikube:~$ kubectl create job paralleljob --image=busybox --dry-run=client -o yaml -- sleep 5 > paralleljob.yml

student@minikube:~$ cat paralleljob.yml 
apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: null
  name: paralleljob
spec:
  completions: 6
  parallelism: 3
  ttlSecondsAfterFinished: 60
  template:
    metadata:
      creationTimestamp: null
    spec:
      containers:
      - command:
        - sleep
        - "5"
        image: busybox
        name: paralleljob
        resources: {}
      restartPolicy: Never
status: {}

After generating the YAML file we've added the completions, parallelism and ttlSecondsAfterFinished values.

Until the Job has completed 6 times, the Job will make sure that 3 Pods are running the Job at all times. When one Pod finished a new Pod is started. At completion of the Job, 6 Pods will have been created by the Job. The Job and Pods are deleted after 60 seconds.

student@minikube:~$ kubectl create -f paralleljob.yml 
job.batch/paralleljob created

student@minikube:~$ kubectl get jobs,pods
NAME                    COMPLETIONS   DURATION   AGE
job.batch/paralleljob   6/6           29s        29s

NAME                    READY   STATUS      RESTARTS   AGE
pod/paralleljob-6s9l4   0/1     Completed   0          11s
pod/paralleljob-7swk6   0/1     Completed   0          19s
pod/paralleljob-8pzgp   0/1     Completed   0          14s
pod/paralleljob-ldmtf   0/1     Completed   0          29s
pod/paralleljob-tsk4q   0/1     Completed   0          29s
pod/paralleljob-x6p8k   0/1     Completed   0          29s

student@minikube:~$ kubectl get jobs,pods
No resources found in default namespace.

Managing Cronjobs

While Jobs are used to run a task a specific number of times, CronJobs are used for tasks that are recurrent or that need to run on a regular basis. In that sense they are very similar to Linux cronjobs.
When running a CronJob, a Job is scheduled and in turn the Job will start a Pod.

Let's go over this in detail:

student@minikube:~$ kubectl create cronjob -h | less
  # Create a cron job with a command
  kubectl create cronjob my-job --image=busybox --schedule="*/1 * * * *" -- date
...
student@minikube:~$ kubectl create cronjob runme --image=busybox --schedule="*/1 * * * *" -- echo Hello there!
cronjob.batch/runme created

student@minikube:~$ kubectl get cronjobs,jobs,pods
NAME                  SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/runme   */1 * * * *   False     0                  15s

student@minikube:~$ kubectl get cronjobs,jobs,pods
NAME                  SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/runme   */1 * * * *   False     1        8s              28s

NAME                       COMPLETIONS   DURATION   AGE
job.batch/runme-27480120   0/1           8s         8s

NAME                       READY   STATUS              RESTARTS   AGE
pod/runme-27480120-xv5l4   0/1     ContainerCreating   0          8s

student@minikube:~$ kubectl get cronjobs,jobs,pods
NAME                  SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/runme   */1 * * * *   False     1        2s              82s

NAME                       COMPLETIONS   DURATION   AGE
job.batch/runme-27480120   1/1           13s        62s
job.batch/runme-27480121   0/1           2s         2s

NAME                       READY   STATUS              RESTARTS   AGE
pod/runme-27480120-xv5l4   0/1     Completed           0          62s
pod/runme-27480121-nljcn   0/1     ContainerCreating   0          2s

As you can see above, the cronjob.batch/runme cronjob will create a new Job at the top of each minute and each Job will create a new Pod which will run until completion of the task.

student@minikube:~$ kubectl delete cronjob runme
cronjob.batch "runme" deleted
student@minikube:~$ kubectl get cronjobs,jobs,pods
No resources found in default namespace.

Resource Requests and Limits

By default, a Pod will consume as much CPU and memory as necessary.
We can however use pod.spec.containers.resources to limit usage of those on a per container basis. CPU and memory limitations are the most common, but there are others.
Each container can has its CPU and memory usage restricted by:

Request: kube-scheduler will look for a worker-node that has this amount of resources available and schedule the Pod to run there. It's allowed for a container to use more resources than defined here. If no suitable worker-node is found, the Pod status remains in Pending.
Limit: This is a hard limit. If configured, the container runtime prevents the container from using more than the configured resource limit. For the memory resource type, this could result in an out of memory error if the container attempts to consume more memory than allowed.

a Pod resource request/limit is the sum of the resource requests/limits for each resource type and for each container in the Pod.

CPU limits are expressed in millicore or millicpu: 1/1000 of a CPU core.
So 500m is 0.5 CPU and 2000m is 2 CPU.

Example:

---
apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: db
    image: mariadb
    env:
    - name: MYSQL_ROOT_PASSWORD
      value: "password"
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  - name: wordpress
    image: wordpress
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

Kubernetes 101: Understanding Kubernetes

Joeri JM Smissaert — Wed, 08 Sep 2021 00:00:00 GMT

{{

}}

What is Kubernetes?

https://kubernetes.io/

Kubernetes is an open-source ecosystem for automating deployment, scaling and managing of containerized applications. It provides a core solution with many third-party add-ons focusing on different areas:

Networking
Ingress
Monitoring
Packaging
...

Kubernetes has its origins at Google where it was known as Borg. It's currently owned by the Cloud Native Computing Foundation, an open-source foundation within the Linux Foundation.

Vanilla Kubernetes is Kubernetes directly created from the source code hosted by the CNCF. Different Kubernetes distributions exist that add specific functionality and a selection of solutions from the ecosystem:

Google Anthos
Red Hat OpenShift
Suse Rancher
Canonical Kubernetes
...

A new release of Kubernetes is published every 3 months. When a new release is published, new versions of the API (more on that later) may become available and old features may get deprecated. If a feature is deprecated it's important to adopt the new method: because of the 3 month release cycle, the feature will go away within the next 2 releases.

Kubernetes Architecture

Kubernetes has the following main components:

Control Plane and worker nodes
Operators (aka "control loop", "watch-loops" or "controller")
Services
Pods of containers
Namespaces and quotas
Network and policies
Storage.

A Kubernetes cluster is made of a Control Plane node and a set of worker nodes. The cluster is driven via API calls to operators.

{{

}}

The Control Plane Node

The various components responsible for ensuring that the current state of the cluster matches the desired state are called the Control Plane.

kube-apiserver

The kube-apiserver is central to the operation of the Kubernetes cluster and exposes the Kubernetes API. You can communicate with the API using a local client called kubectl or you can write your own client and use curl commands.All actions are accepted and validated by this component, and it is the only connection to the etcd database.

kube-scheduler

The kube-scheduler determines which node will host a Pod of containers. The scheduler will try to view available resources and then try to deploy the Pod based on availability and success.

etcd database

The state of the cluster, networking, and other persistent information is kept in an etcd database. etcd is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. This database is only accessible by kube-apiserver.

kube-controller-manager

Orchestration is managed through a series of watch-loops or control loops, also called controllers or operators. A control loop is a non-terminating loop that regulates the state of a system. Each controller interrogates the kube-apiserver for a particular object state, then modifies the object until the declared state matches the current state. These controllers are compiled into the kube-controller-manager, but others can be added using custom resource definitions.

The kube-controller-manager is a core control loop daemon which interacts with the kube-apiserver to determine the state of the cluster. If the state does not match, the manager will contact the necessary controller to match the desired state.

Worker Nodes

A Worker Node consists of components that maintain running pods.

kubelet

The kubelet systemd process interacts with the underlying container engine. It accepts the API calls for Pod specifications and it will configure the local node until the specification has been met by passing requests to the local container engine.

kube-proxy

The kube-proxy creates and manages networking rules to expose the container on the network to other containers or the outside world.

Container runtime

The container runtime or container engine is responsible for running containers.
Each Worker Node could run a different engine if needed: Docker, containerd, CRI-O, podman, ...

The Most Essential API Resources

Deployment

The default operator for containers is a Deployment. A Deployment does not directly work with pods, instead it manages ReplicaSets. The ReplicaSet is an operator which will create or terminate pods according to a podSpec. The podSpec is sent to the kubelet, which then interacts with the container engine to download and make the required resources available, then spawn or terminate containers until the status matches the spec.

Pod

Containers are not managed individually, instead, they are part of a larger object called a Pod. A Pod consists of one or more containers which share an IP address, access to storage and namespace. Typically, one container in a Pod runs an application, while other containers support the primary application.

Service

The service operator requests existing IP addresses and information from the endpoint operator, and will manage the network connectivity based on labels. A service is used to communicate between pods, namespaces, and outside the cluster.

Creating a Lab Environment

The Kubernetes 101 series of articles that I will be publishing over time are meant to provide a basic introduction to Kubernetes. As such, we'll not be using a full blown Kubernetes cluster but we'll be relying on Minikube instead.

With Minikube we can quickly and easily setup a local Kubernetes cluster and focus on learning the basics. In a later series, we'll deep dive into a full blown Kubernetes cluster with multiple worker nodes.

We will be installing Minikube in an Ubuntu virtual machine with 4GiB of RAM and 2vCPUs and we'll be using Docker as the container engine, so make sure you install Docker as well. Once your virtual machine is ready, head over to the Minikube installation instructions. Make sure you start a cluster, install kubectl and create an alias for it to make life easier.

Verifying Minikube is working

The minikube command has different options, here's an overview of the commonly used ones:

minikube status: Gets the status of a local Kubernetes cluster.
minikube start: Starts a local Kubernetes cluster.
minikube stop: Stops a running local Kubernetes cluster.
minikube ssh: Log into the minikube environment (for debugging)
minikube dashboard: Opens the Kubernetes dashboard in the local browser.
minikube delete: Deletes a local Kubernetes cluster.
minikube ip: Retrieves the IP address of the specified node.
minikube version: Print the version of minikube.

You can see all available options by using the minikube --help command.

These will come in handy as well:

kubectl get all: Display all resources.
docker ps: List containers.

Bash Completion

Bash completion for kubectl will come in handy. The kubectl completion -h command has instructions for different shells like zsh and fish. Below are the instructions for bash:

apt install bash-completion -y
echo "source <(kubectl completion bash)" >> ~/.bashrc
source ~/.bashrc

Running an application

Let's go over the steps of starting our cluster and launching a simple Nginx Pod:

# We start our Minikube cluster
student@minikube:~$ minikube start
...

# Install kubectl
student@minikube:~$ minikube kubectl -- get pods -A
...

# Verify the status
student@minikube:~$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

# List all Docker containers - See how Minikube is running a Kubernetes cluster inside a single Docker container
student@minikube:~$ docker ps
...

# Have a look at the different Kubernetes components which are running in Pods inside the kube-system namespace.
student@minikube:~$ kubectl get pods -n kube-system
NAME                               READY   STATUS    RESTARTS      AGE
coredns-64897985d-sj5lw            1/1     Running   0             11m
etcd-minikube                      1/1     Running   0             11m
kube-apiserver-minikube            1/1     Running   0             11m
kube-controller-manager-minikube   1/1     Running   0             11m
kube-proxy-mgcrk                   1/1     Running   0             11m
kube-scheduler-minikube            1/1     Running   0             11m
storage-provisioner                1/1     Running   1 (10m ago)   11m

# Let's run an Nginx Pod
student@minikube:~$ kubectl run nginx --image=nginx
pod/nginx created

student@minikube:~$ kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          24s

student@minikube:~$ kubectl get all
NAME        READY   STATUS    RESTARTS   AGE
pod/nginx   1/1     Running   0          37s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1            443/TCP   13m

Play around with the different minikube commands, and, once done head over to the next article.

Podman 102: Building a WordPress multi-service container with Nginx, PHP-FPM and MariaDB

Joeri JM Smissaert — Tue, 13 Apr 2021 00:00:00 GMT

{{< figure class="center" src="/img/redhat-8-logo.png" alt="Red Hat logo">}} {{< figure class="center" src="/img/podman.png" alt="Podman logo">}}

Typically an application container runs a single service, but instead of breaking apart existing multi-serivce applications into microservices (and connecting them with e.g. Kubernetes or OpenShift), we can use Podman (in contrast to Docker) to run multi-service containers using Systemd. Basically we would achieve something similar to LXD system containers but with Podman.

Podman understands what Systemd needs to do to run in a container. When Podman starts a container that is running init or systemd as its initial command, Podman automatically sets up the tmpfs and cgroups so that Systemd can start succesfully.

Systemd attempts to write to the cgroup file system. By default, containers cannot write to the cgroup file system when SELinux is enabled. The container_manage_cgroup boolean must be enabled for this to be allowed on a SELinux enforced system: setsebool -P container_manage_cgroup true

In this post I'll create a rather basic multi-service container based on the Fedora container image which will be running Nginx, MariaDB and PHP-FPM to serve up a WordPress site with persistent storage both for the document root and the database.

I've pushed the final version of the image I've build below to my Quay.io repository.

Step 1 - Test Nginx

[student@server1 ~]$ cat Dockerfile
FROM fedora
MAINTAINER Joeri Smissaert

RUN dnf -y upgrade; dnf -y install nginx; dnf clean all; systemctl enable nginx
RUN mkdir -p /var/www/worpdress.server1.local/public
RUN mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.backup
ADD https://gist.githubusercontent.com/smissaertj/9d02fd974b64fd1a30fd905bc730a098/raw/dee50eb0bea7b93acb6ad0ddb6894cefb74c9d45/nginx.conf /etc/nginx/nginx.conf

EXPOSE 80

CMD ["/sbin/init"]

Let's build the image:

podman build -t fedora_wordpress .

...start the container:

[student@server1 ~]$ podman run -d --name test -p 8080:80 -v /home/student/html:/var/www/wordpress.server1.local/public:Z fedora_wordpress
...

...create a test file in the bind mounted document root and test using cURL:

[student@server1 ~]$ mkdir html
[student@server1 ~]$ echo "JOERI" > html/index.html
[student@server1 ~]$ curl localhost:8080
JOERI
[student@server1 ~]$ echo "TEST 123" > html/index.html
[student@server1 ~]$ curl localhost:8080
TEST 123

So far so good :)

Step 2 - Test PHP-FPM

In this step we only install and enable PHP-FPM. If the test fails, then I need to revise my Nginx and/or PHP-FPM pool configuration. My Nginx configuration file is custom, while I left the default PHP-FPM configuration file in place.

[student@server1 ~]$ cat Dockerfile
FROM fedora
MAINTAINER Joeri Smissaert

RUN dnf -y upgrade; dnf -y install nginx php-fpm php-mysqlnd php-pdo php-json; dnf clean all; systemctl enable nginx; systemctl enable php-fpm
RUN mkdir -p /var/www/worpdress.server1.local/public
RUN mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.backup
ADD https://gist.githubusercontent.com/smissaertj/9d02fd974b64fd1a30fd905bc730a098/raw/dee50eb0bea7b93acb6ad0ddb6894cefb74c9d45/nginx.conf /etc/nginx/nginx.conf

EXPOSE 80

CMD ["/sbin/init"]

Adjust original Dockerfile with the modifications above, then rebuild the image and run the container:

[student@server1 ~]$ podman build -t fedora_wordpress .
...
[student@server1 ~]$ podman run -d --name test -p 8080:80 -v /home/student/html:/var/www/wordpress.server1.local/public:Z fedora_wordpress
...

Remove the html/index.html file and create an html/index.php file with the following content:

[student@server1 ~]$ cat html/index.php

 <head>
  <title>PHP Testtitle>
 head>
 <body>
 <?php echo '<p>Hello Worldp>'; ?>
 body>

When we run a cURL test, we should not be seeing the and ?> tags, indicating that our php code was succesfully parsed by PHP-FPM:


[student@server1 ~]$ curl localhost:8080

 <head>
  <title>PHP Testtitle>
 head>
 <body>
 <p>Hello Worldp>
 body>

Yaay! :D
Step 3 - Test MariaDB
I'll create persistent storage for the database by means of a podman volume:
[student@server1 ~]$ podman volume create wordpress_db
wordpress_db

[student@server1 ~]$ podman volume ls
DRIVER      VOLUME NAME
local       wordpress_db
Again, we adjust our Dockerfile and rebuild our custom image:
FROM fedora
MAINTAINER Joeri Smissaert

RUN dnf -y upgrade; dnf -y install nginx php-fpm php-fpm php-mysqlnd php-pdo php-json mariadb-server; dnf clean all; systemctl enable nginx; systemctl enable php-fpm; systemctl enable mariadb
RUN mkdir -p /var/www/worpdress.server1.local/public
RUN mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.backup
ADD https://gist.githubusercontent.com/smissaertj/9d02fd974b64fd1a30fd905bc730a098/raw/dee50eb0bea7b93acb6ad0ddb6894cefb74c9d45/nginx.conf /etc/nginx/nginx.conf

EXPOSE 80

CMD ["/sbin/init"]
We run the container:
[student@server1 ~]$ podman run -d --name test -v wordpress_db:/var/lib/mysql:Z fedora_wordpress
...
Next, we create the database and configure the database user and password:
[student@server1 ~]$ podman exec test mysql -e "create database wordpressdb;"
[student@server1 ~]$ podman exec test mysql -e "grant all privileges on wordpressdb.* to 'wordpress'@'localhost' identified by 'password';"
We can now move on to the next step and install WordPress.
Step 4 - Install WordPress
In this step I'll move away from a bind mounted directory (used during Step 1 and Step 2) to a podman volume to persistently store the WordPress files.
[student@server1 ~]$ podman volume create wordpress_files
wordpress_files
[student@server1 ~]$ podman volume ls
DRIVER      VOLUME NAME
local       wordpress_files
local       wordpress_db
[student@server1 ~]$ podman volume inspect wordpress_files
[
     {
          "Name": "wordpress_files",
          "Driver": "local",
          "Mountpoint": "/home/student/.local/share/containers/storage/volumes/wordpress_files/_data",
          "CreatedAt": "2021-04-14T00:25:06.906021271+04:00",
          "Labels": {

          },
          "Scope": "local",
          "Options": {

          },
          "UID": 0,
          "GID": 0,
          "Anonymous": false
     }
]
From the last command we can see where exactly the data will be stored:
Mountpoint": "/home/student/.local/share/containers/storage/volumes/wordpress_files/_data"
I'll go ahead and extract WordPress inside that directory:
[student@server1 ~]$ cd ~/.local/share/containers/storage/volumes/wordpress_files/_data/
[student@server1 _data]$ wget https://wordpress.org/latest.tar.gz
...
[student@server1 _data]$ tar xf latest.tar.gz --strip-components 1
[student@server1 _data]$ rm -rf latest.tar.gz
Let's start the container and test our installation:
[student@server1 ~]$ podman run -d --name wordpress_test_container -p 8080:80 -v wordpress_db:/var/lib/mysql:Z -v wordpress_files:/var/www/wordpress.server1.local/public:Z fedora_wordpress
bc006160ad6b74b81fa3fc353bc0cbb1cec3b394365dc98259984a86c971cd9f
[student@server1 ~]$
Testing with cURL seems to go fine:
[student@server1 ~]$ curl -I localhost:8080
HTTP/1.1 302 Found
Server: nginx/1.18.0
Date: Tue, 13 Apr 2021 20:34:46 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Powered-By: PHP/7.4.16
Location: http://localhost:8080/wp-admin/setup-config.php
So at this point we have a working WordPress multi-service container :D
Below is the final version of our Dockerfile:
FROM fedora
MAINTAINER Joeri Smissaert

RUN dnf -y upgrade; dnf -y install nginx php-fpm php-fpm php-mysqlnd php-pdo php-json mariadb-server; dnf clean all; systemctl enable nginx; systemctl enable php-fpm; systemctl enable mariadb
RUN mkdir -p /var/www/worpdress.server1.local/public
RUN mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.backup
ADD https://gist.githubusercontent.com/smissaertj/9d02fd974b64fd1a30fd905bc730a098/raw/dee50eb0bea7b93acb6ad0ddb6894cefb74c9d45/nginx.conf /etc/nginx/nginx.conf

EXPOSE 80

CMD ["/sbin/init"]
I've pushed the final version of the image I've build to my Quay.io repository.
As long as we keep the wordpress_files and wordpress_db volumes, I can destroy the running container and recreate it without any effect on the data:
podman run -d --name container_name -p 8080:80 -v wordpress_files:/var/www/wordpress.server1.local/public:Z -v wordpress_db:/var/lib/mysql:Z quay.io/smissaertj/fedora_wordpress
Finally, I want my WordPress site to start at boot even when I'm not logging in to my machine as the user which created the container:
[student@server1 ~]$ podman ps
CONTAINER ID  IMAGE                              COMMAND     CREATED        STATUS            PORTS                 NAMES
d4fb97ba659d  localhost/fedora_wordpress:latest  /sbin/init  6 minutes ago  Up 6 minutes ago  0.0.0.0:8080->80/tcp  wordpress_test

[student@server1 ~]$ mkdir -p ~/.config/systemd/user
[student@server1 ~]$ cd .config/systemd/user/

[student@server1 user]$ podman generate systemd --name wordpress_test --files --new
/home/student/.config/systemd/user/container-wordpress_test.service

[student@server1 user]$ su - root
Password:
[root@server1 ~]# loginctl enable-linger student
[root@server1 ~]# exit
logout

[student@server1 user]$ systemctl --user daemon-reload
[student@server1 user]$ systemctl --user enable container-wordpress_test.service
Created symlink /home/student/.config/systemd/user/multi-user.target.wants/container-wordpress_test.service → /home/student/.config/systemd/user/container-wordpress_test.service.
Created symlink /home/student/.config/systemd/user/default.target.wants/container-wordpress_test.service → /home/student/.config/systemd/user/container-wordpress_test.service.

[student@server1 user]$ reboot
The --new option passed to the podman generate systemd command will make sure that the container is destroyed when the service stops and recreated when the service starts.



Podman 101: Managing and Running Containers
Joeri JM Smissaert — Sun, 14 Mar 2021 00:00:00 GMT
{{< figure class="center" src="/img/redhat-8-logo.png" alt="Red Hat logo">}}
{{< figure class="center" src="/img/podman.png" alt="Podman logo">}}
Understanding Containers
For a data center to operate efficiently, its machines and running components on those machines must become as generic and as much automated as possible. We can partly achieve this by seperating the applications from the operating system. This means not just packaging applications into things we install (like RPM or Deb packages), but also putting together sets of software into packages that themselves can run in ways that keep them independent and seperate from the operating system. Virtual Machines and Containers are two ways of packaging sets of software and their dependencies in a way which is separated from the host operating system they are running on.
A virtual machine is a complete operating system that runs on another operating sytem, you can have many virtual machines on one physical computer. Everything an application or service needs to run can be stored inside that virtual machine or in attached storage. A virtual machine has its own kernel, file system, process table, network interfaces and other operating system features separate from the host, while sharing CPU and RAM with the host system. A VM sees an emulation of the computer hardware and not the host hardware directly, hence the term virtual machine.
A Container is similar to a virtual machine, except that it doesn't have its own kernel. It remains separate from the host system by using its own set of namespaces. Just like a VM, you can move it from one host to another to run it wherever it is convenient. Typically you would build your own container images by getting a secure base image and then adding your own layers of software on top of that image to create a new image. To share your image, you push them to shared container registries from where others are allowed to pull them. 
Containers run on top of a container engine, like Docker, CRI-O (which is the default on RHEL 8), Moby or rkt, and typically a container runs a single application or service (which can be connected in microservices using OpenShift or Kubernetes for example), although there are systemd images from which you can build multiservice containers. 
Podman is a daemonless container engine that is compatible with Docker, for developing, managing, and running Open Container Initiative (OCI) containers and container images on Linux.
Namespaces
Linux support for namespaces is what allows containers to be contained. With namespaces, the Linux kernel can associate one or more processes with a set of resources. Normal processes, not run in a container, use the same host namespaces. By default, processes in a container can only see the container's namespaces and not those of the host. 

Process table - A container has its own set of process IDs and, by default, can only see processes running inside the container. While PID 1 on the host is the init (systemd) process, in a container PID 1 is the first process run inside the container.   

Network interfaces - By default, a container has a single network interface and is assigned an IP address when the container runs. A service run inside a container is not exposed outside of the host system, by default. You can have hundreds of webservers running on the same host without conflict, but you need to manage how those ports are exposed outside of the host.  

Mount table - By default, a container can't see the host's root file system or any other mounted file system listed in the host's mount table. Files or directories needed from the host can be selectively bind-mounted inside the container.  

User IDs - Containerized processes run as some UID within the host's namespace, and, with another set of UIDs nested within the container. This can, for example, let a process run as root within the container but not have any special privileges to the host system.  

UTS - The UNIX Time Sharing namespace allows a containerized process to have a different host and domain name from the host. 

Control Group - A containerized process runs within a selected cgroup and cannot see the other cgroups available on the host system. Similarly, it cannot see the identify of its own cgroup. Control Groups are used for resource management.

Interprocess Communications - A containerized process cannot see the IPC namespace of the host.



Although access to any host namespace is restricted by default, privileges to host namespaces can be opened selectively. In that way, you can do things like mount configuration files or data inside the container and map container ports to host ports to expose services outside of the host.

Container Registries
Permanent storage for containers is done in what is referred to as a container registry. When you create a container image that you want to share, you can push that image to a public or private (which you maintain yourself) container registry. Someone who wants to use your container image will then pull it from the registry. 
Large public container image registries are, for example, Docker hub and Quay Registry. 
Base Images and Layers
Although you can create containers from scratch, most often a container is built by starting with a well-known base image and adding software to it. Linux distributions offer base images in different forms, like standard and minimal versions. But there are also base images you can build on that offer runtimes for PHP, Java and other development environments.
Red Hat offers freely available Universal Base Images (UBIs) for standard, minimal and a variety of runtime containers. You can find those by searching the Red Hat Container Catalog.
You can add software to a base image by defining the build using yum commands to install software from software repositories into the new container. When you add software to an image, it creates a new layer that becomes part of the new image. You can reuse the same base image for all container you build, only one copy of the base image is needed on the host. If you're running 10 different containers based on the same base image, you only need to pull and store the base image once. For each new image you build, you only add the data that differs from the base image.
Running and Managing Containers with Podman
Pulling and Running Containers
In order to start using containers with podman, we need to install the container-tools module:
[root@server1 student]# yum module install container-tools
...
Let's choose a reliable image to try out, one that comes from an official project, is up to date and has been scanned for vulnerabilities:
[student@server1 ~]$ podman pull registry.access.redhat.com/ubi8/ubi
Trying to pull registry.access.redhat.com/ubi8/ubi...
Getting image source signatures
Copying blob 64607cc74f9c done  
Copying blob 13897c84ca57 done  
Copying config 9992f11c61 done  
Writing manifest to image destination
Storing signatures
9992f11c61c5fa38a691f80c7e13b75960b536aade4cce8543433b24623bce68
[student@server1 ~]$
We can verify that the image is on our system using the podman images command:
[student@server1 ~]$ podman images
REPOSITORY                               TAG     IMAGE ID      CREATED       SIZE
registry.access.redhat.com/ubi8/ubi      latest  9992f11c61c5  11 days ago   213 MB
Next, let's start an interactive shell from this base image. We use the podman run command, specify the -i (interactive) and -t (terminal) options, followed by the name of the image (ubi) and the command we wish to start once the container is up and running (bash):
[student@server1 ~]$ podman run -it ubi bash
[root@888b3cbea5cc /]#
We are in an interactive session within the container from the bash shell. Notice the container is using the host kernel:
[root@888b3cbea5cc /]# ls
bin  boot  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

[root@888b3cbea5cc /]# cat /etc/os-release  | grep -i ^NAME
NAME="Red Hat Enterprise Linux"

[root@888b3cbea5cc /]# uname -r
4.18.0-240.el8.x86_64
We can add software to the container:
[root@888b3cbea5cc /]# yum install procps -y
...

[root@888b3cbea5cc /]# ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 13:13 pts/0    00:00:00 bash
root          39       1  0 13:20 pts/0    00:00:00 ps -ef
Notice that form within the container, we only see two running processes: the shell and the ps command. PID 1 is the bash shell.
We can exit the container by using the exit command. The container is now no longer running, but it's still available on the host in a stopped state. The podman ps --all command shows all available containers:
[student@server1 ~]$ podman ps -a
CONTAINER ID  IMAGE                                       COMMAND               CREATED         STATUS                    PORTS                                                                  NAMES                                                    bold_aryabhata
888b3cbea5cc  registry.access.redhat.com/ubi8/ubi:latest  bash                  9 minutes ago   Exited (0) 3 seconds ago                                                                                    musing_almeida
Managing Container State
Unless you specifically set a container to be removed when it's stopped (--rm option), paused or fails, the container is still on your system. You can see the status of all containers on the system, running or stopped, using the podman ps command:
[student@server1 ~]$ podman run -d nginx
e968c7e569cbe60d909b2108ba5a2067bb3e771327f4729b85566280efe944a6

[student@server1 ~]$ podman ps
CONTAINER ID  IMAGE                           COMMAND               CREATED        STATUS            PORTS   NAMES
e968c7e569cb  docker.io/library/nginx:latest  nginx -g daemon o...  4 seconds ago  Up 3 seconds ago          loving_swartz

[student@server1 ~]$ podman stop e968
e968c7e569cbe60d909b2108ba5a2067bb3e771327f4729b85566280efe944a6

[student@server1 ~]$ podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

[student@server1 ~]$ podman ps -a
CONTAINER ID  IMAGE                                       COMMAND               CREATED         STATUS                    PORTS   NAMES
e968c7e569cb  docker.io/library/nginx:latest              nginx -g daemon o...  27 seconds ago  Exited (0) 5 seconds ago          loving_swartz
The podman stop command sends a SIGTERM signal and if the container doesn't stop after 10 seconds it will send a SIGKILL signal. 
You can also send the SIGKILL signal immediately using the podman kill command.
Just like the podman stop command stops a container, you can start a container using podman start or simply restart a container using podman restart.
Lastly, we can delete the container permanently by using the podman rm command:
[student@server1 ~]$ podman rm e968
e968c7e569cbe60d909b2108ba5a2067bb3e771327f4729b85566280efe944a6

[student@server1 ~]$ podman ps -a
CONTAINER ID  IMAGE                                       COMMAND  CREATED        STATUS                    PORTS   NAMES
[student@server1 ~]$
Note that the podman rm command only deletes the container and not the image.
Running commands in a container
When we are detached from a container we can still execute commands inside the container using podman exec:
[student@server1 ~]$ podman exec cd87 cat /etc/os-release | grep ^NAME
NAME="Debian GNU/Linux"
Or, we can attach to the container:
[student@server1 ~]$ podman exec -it cd87 /bin/bash
root@cd87164b978f:/#
...and detach using the CTRL-P+Q sequence.
Managing Container Ports
We can map a host port to the container application port to make the application in the container reachable from the host machine:
[student@server1 ~]$ podman run -d -p 8000:80 nginx
965fe32d0b4b96d469ddb5638edaa5ac18fe41fc083082844bc8ddae0f6a9a33

[student@server1 ~]$ podman ps
CONTAINER ID  IMAGE                           COMMAND               CREATED        STATUS            PORTS                 NAMES
965fe32d0b4b  docker.io/library/nginx:latest  nginx -g daemon o...  3 seconds ago  Up 2 seconds ago  0.0.0.0:8000->80/tcp  musing_mclaren

[student@server1 ~]$ podman port -a
965fe32d0b4b  80/tcp -> 0.0.0.0:8000

[student@server1 ~]$ podman port 965
80/tcp -> 0.0.0.0:8000
In the example above, we mapped the host port 8000 to port 80 of the container.
Note that you can only map container ports to non privileged (>1024) ports on the host when running rootless containers.
With the above done, we can curl the host port and see Nginx serving its default content:
[student@server1 ~]$ curl localhost:8000

<html>
<head>
<title>Welcome to nginx!title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
style>
head>
<body>
<h1>Welcome to nginx!h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.orga>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.coma>.p>

<p><em>Thank you for using nginx.em>p>
body>
html>
Now, if we would want access from outside of the host machine, we should not forget to configure the host machine's firewall:
[student@server1 ~]$ su - root
Password: 
[root@server1 ~]# firewall-cmd --add-port=8000/tcp --permanent && firewall-cmd --reload
success
success
[root@server1 ~]# exit
logout
[student@server1 ~]$

By default podman runs rootless containers.
Rootless containers cannot bind to a privileged port and do NOT have an IP address, you would need port forwarding instead. If you need a container with an IP address, you need a root container: sudo podman run -d nginx

Attaching Storage to Containers
Storage in containers is ephemeral: modifications are written to the container writeable layer and stay around for the container lifetime.
For persistent storage needs, we use bind mounts to connect a directory inside the container to a directory on the host machine.
We start preparing on the hostmachine, creating directories, setting basic permissions and changing the SELinux file context type to container_file_t. 
SELinux is very important when using root containers, as without, the root container will have access to the entire host file system.
I'll run through an example where we set the document root of the nginx image to the /home/student/html directory on the host machine. Inside that directory we'll create a basic html file that the nginx container is going to serve.
Preparing Host Storage
[root@server1 student]# pwd
/home/student

[student@server1 ~]$ ls -l
total 0
drwxrwxr-x. 2 student student 6 Apr 12 21:29 html

[root@server1 student]# semanage fcontext -a -t container_file_t "/home/student/html(/.*)?"
[root@server1 student]# restorecon -Rv /home/student/html
Relabeled /home/student/html from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:container_file_t:s0
Mounting Storage Inside the Container.
At this point we can delete the container from the previous example, start a new container and bind mount the host directory /home/student/html to the default document root of Nginx in the container: /usr/share/nginx/html
If the container user is owner of the host directory, the :Z (SELinux) option can be used:
podman run -d --name web1 -p 8000:80 -v /home/student/html:/usr/share/nginx/html:Z nginx

--d we run the container in detached mode.
--name we set a name for our new container.
-p we map the host port to the container port.
-v we bind a host directory to a directory inside the container.
nginx the name of the image we use to start our container from. 

[student@server1 ~]$ podman run -d --name web1 -p 8000:80 -v /home/student/html:/usr/share/nginx/html:Z nginx
1988217288c55050a2820881ccf75e4436097d8128f9d2dec8a08af6674c6f88

[student@server1 ~]$ podman ps
CONTAINER ID  IMAGE                           COMMAND               CREATED        STATUS            PORTS                 NAMES
1988217288c5  docker.io/library/nginx:latest  nginx -g daemon o...  4 seconds ago  Up 4 seconds ago  0.0.0.0:8000->80/tcp  web1

[student@server1 ~]$ curl localhost:8000

<head><title>403 Forbiddentitle>head>
<body>
<center><h1>403 Forbiddenh1>center>
<hr><center>nginx/1.19.9center>
body>
html>
After starting the container, you'll see that the curl test now returns a 403 Forbidden status.
This is because the Nginx document root is bound to an empty directory on our host machine. Let's create an html file for Nginx to serve:
[student@server1 ~]$ echo "TEST NGINX
" > html/index.html
[student@server1 ~]$ curl localhost:8000
TEST NGINX
[student@server1 ~]$
At this point we can manage the content that Nginx is serving directly from the host machine.
Environment Variables
Podman allows us to set arbitrary environment variables that will become available to processes running in the container:  
podman run -d --name mydb -e MYSQL_ROOT_PASSWORD=password -e MYSQL_USER=student -e MYSQL_PASSWORD=password -e MYSQL_DATABASE=studentdb -p 3306:3306 mariadb
Using the -e option, in the above example, we set the MySQL root password, user, password and database name. If we don't specify a value for a variable, then podman will look for the value in the host environment and only set it if that variable has a value.
Similarly, instead of passing the environment variables one by one, we can define them in a file and then pass the filename to podman using the --env-file option:
podman run -d --name mydb --env-file=variables.txt -p 9999:3306 mariadb
[student@server1 ~]$ cat variables.txt 
MYSQL_ROOT_PASSWORD=password
MYSQL_USER=student
MYSQL_PASSWORD=password
MYSQL_DATABASE=studentdb
We can now connect from the host machine to the MariaDB instance in the container:
[student@server1 ~]$ podman run -d --name mydb --env-file=variables.txt -p 3306:3306 mariadb
bd08dcbd3eef3907423ee2e55164e1e222a511f58a96d2c4e474f4ea8d56235b

[student@server1 ~]$ mysql -u student -h 127.0.0.1 -p
Enter password: 

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.5.5-10.5.9-MariaDB-1:10.5.9+maria~focal mariadb.org binary distribution

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| studentdb          |
+--------------------+
2 rows in set (0.00 sec)

Some containers require environment variables to run them. If a container fails because of this requirement, use podman logs container_name to see the application log.
Alternatively, use podman inspect | grep -i usage.

Managing Containers as Services
Now that we have a running container, we can auto start it in a stand-alone situation. The container would start running even though the user that is running the container is not logged in.
For this we can create systemd user unit files (for rootless containers), and manage them with systemctl.
Systemd user services start when a user session is opened, and close when the user session is stopped.
We need to use the loginctl enable-linger command to start systemd user services at boot without requiring the user to login:
[root@server1 ~]# loginctl enable-linger student
[root@server1 ~]# loginctl show-user student | grep -i ^linger
Linger=yes
[root@server1 ~]#
Next, we use podman generate systemd to generate a user systemd unit file. This will create the file in the working directory.
We need to create the ~/.config/systemd/user directory (for a root container what would be in /etc/systemd/system), and move the user unit file into this directory.
[student@server1 ~]$ mkdir -p ~/.config/systemd/user
[student@server1 ~]$ podman generate systemd --name mydb --files
/home/student/container-mydb.service

[student@server1 ~]$ mv container-mydb.service ~/.config/systemd/user/
[student@server1 ~]$ systemctl --user daemon-reload 
[student@server1 ~]$ systemctl --user enable container-mydb.service 
Created symlink /home/student/.config/systemd/user/multi-user.target.wants/container-mydb.service → /home/student/.config/systemd/user/container-mydb.service.
Created symlink /home/student/.config/systemd/user/default.target.wants/container-mydb.service → /home/student/.config/systemd/user/container-mydb.service.
[student@server1 ~]$
When we reboot our host machine, the mydb container will automatically start even though the student user is not logged in.
To have systemd create the container when the service starts, and delete the container when the service stops, add the --new option. Keep in mind you'll lose all changes if you didn't configure persistent storage for the container:  
[student@server1 ~]$ podman generate systemd --name mydb --files --new
Working with Images
An image is a read-only but runnable instance of a container that can be used to build new images.
They are obtained from registries which are configured in /etc/containers/registries.conf:
[student@server1 ~]$ grep -ia1 ^registries /etc/containers/registries.conf 
[registries.search]
registries = ['registry.access.redhat.com', 'registry.redhat.io', 'docker.io']

--
[registries.insecure]
registries = []

--
[registries.block]
registries = []
Under the [registries.search] value we find an array of registries that will be searched for a specific image in the order they appear in. 
For example, if you do podman pull nginx, podman will look for the nginx image on registry.access.redhat.com, registry.redhat.io, docker.io subsequently until it finds the image.
Registries that do not use TLS when using images, or which are using self-signed certificates need to be placed under [registries.insecure].
You can block specific registries under [registries.block], or, if you specify a wildcard ("*") then all registries are blocked except those that were specified under [registries.search].
You can also verify what regestries are in used by issueing the podman info command.
Searching for images
We use the podman search command to search for images on either all configured registries or only on specific registries. The search results can be filtered using different options as well. 
A few examples below:
[student@server1 ~]$ podman search docker.io/nginx --limit 1
INDEX       NAME                      DESCRIPTION                STARS   OFFICIAL   AUTOMATED
docker.io   docker.io/library/nginx   Official build of Nginx.   14707   [OK] 

[student@server1 ~]$ podman search registry.redhat.io/nginx --limit 1
INDEX       NAME                                 DESCRIPTION                                       STARS   OFFICIAL   AUTOMATED
redhat.io   registry.redhat.io/rhel8/nginx-116   Platform for running nginx 1.16 or building ...   0 

[student@server1 ~]$ podman search docker.io/mariadb --filter is-official=true
INDEX       NAME                        DESCRIPTION                                       STARS   OFFICIAL   AUTOMATED
docker.io   docker.io/library/mariadb   MariaDB Server is a high performing open sou...   4043    [OK]
Inspecting Images
Now that we have an idea of what nginx images are available to us, we can inspect them remotely (without pulling them) using skopeo:
[student@server1 ~]$ skopeo inspect docker://docker.io/nginx
{
    "Name": "docker.io/library/nginx",
    "Digest": "sha256:6b5f5eec0ac03442f3b186d552ce895dce2a54be6cb834358040404a242fd476",
    "RepoTags": [
        "1-alpine-perl",
        "1-alpine",
...
Note that the skopeo inspect command always takes the docker:// prefix regardless of what registry the image you're inspecting is located on:
[student@server1 ~]$ skopeo inspect docker://registry.redhat.io/rhel8/mariadb-103
{
    "Name": "registry.redhat.io/rhel8/mariadb-103",
    "Digest": "sha256:c6f117263e36880af79bba1de2018462126d226439d28d074f30bcfaf57dabe1",
    "RepoTags": [
        "1-116",
        "1-116-source",
...
If we have a local image we wish to inspect, we can use podman inspect instead:
[student@server1 ~]$ podman images
REPOSITORY                           TAG     IMAGE ID      CREATED      SIZE
docker.io/library/nginx              latest  519e12e2a84a  3 days ago   137 MB
docker.io/library/mariadb            latest  e76a4b2ed1b4  10 days ago  407 MB
registry.access.redhat.com/ubi8/ubi  latest  9992f11c61c5  13 days ago  213 MB
[student@server1 ~]$ podman inspect registry.access.redhat.com/ubi8/ubi
[
    {
        "Id": "9992f11c61c5fa38a691f80c7e13b75960b536aade4cce8543433b24623bce68",
        "Digest": "sha256:17ff29c0747eade777e8b9868f97ba37e6b8b43f5ed2dbf504ff9277e1c1d1ca",
        "RepoTags": [
            "registry.access.redhat.com/ubi8/ubi:latest"
...
Removing Images
When new images become available, the old version of the image is kept on your system.
We can remove images using the podman rmi command:
[student@server1 ~]$ podman images
REPOSITORY                           TAG     IMAGE ID      CREATED      SIZE
docker.io/library/nginx              latest  519e12e2a84a  3 days ago   137 MB
docker.io/library/mariadb            latest  e76a4b2ed1b4  10 days ago  407 MB
registry.access.redhat.com/ubi8/ubi  latest  9992f11c61c5  13 days ago  213 MB

[student@server1 ~]$ podman rmi ubi
Untagged: registry.access.redhat.com/ubi8/ubi:latest
Deleted: 9992f11c61c5fa38a691f80c7e13b75960b536aade4cce8543433b24623bce68

[student@server1 ~]$ podman images
REPOSITORY                 TAG     IMAGE ID      CREATED      SIZE
docker.io/library/nginx    latest  519e12e2a84a  3 days ago   137 MB
docker.io/library/mariadb  latest  e76a4b2ed1b4  10 days ago  407 MB
Creating Images from a Dockerfile
We can use podman and buildah to create new images from a Dockerfile. The resulting images are OCI compliant, so they will work on any runtime that meets the OCI Runtime Specification (such as Docker and CRI-O).
In the below example we prepare a Dockerfile to install the Apache webserver onto a Fedora image and later use podman build to create a new image from this Dockerfile.
[student@server1 ~]$ cat Dockerfile 
# Base on the Fedora image
FROM fedora:latest
MAINTAINER Joeri Smissaert

# Update image and install Nginx
RUN dnf -y update; dnf -y clean all
RUN dnf -y install httpd

# Expose the default port 80
EXPOSE 80

# Run Nginx
CMD ["/usr/sbin/httpd","-DFOREGROUND"]

[student@server1 ~]$ podman build -t fedora-apache .
...

[student@server1 ~]$ podman images
REPOSITORY                           TAG     IMAGE ID      CREATED         SIZE
localhost/fedora-apache              latest  cb083eb46577  15 minutes ago  483 MB

[student@server1 ~]$ podman run -d --name myweb1 -p 8080:80 fedora-apache
2f8f1ef6c484f2825f7a11f30c8601799b0736145917f6428b395b4c599cbd6e

[student@server1 ~]$ podman ps
CONTAINER ID  IMAGE                             COMMAND               CREATED        STATUS            PORTS                   NAMES
2f8f1ef6c484  localhost/fedora-apache:latest    /usr/sbin/httpd -...  3 seconds ago  Up 2 seconds ago  0.0.0.0:8080->80/tcp    myweb1
Tagging and Pushing an Image to a Registry
In this example, I'll tag and push the fedora-apache image to Quay.io.
[student@server1 ~]$ podman login quay.io
Username: ********
Password: 
Login Succeeded!

[student@server1 ~]$ podman tag fedora-apache quay.io/smissaertj/fedora-apache:v1.0
[student@server1 ~]$ podman push quay.io/smissaertj/fedora-apache:v1.0
Getting image source signatures
Copying blob 7ddfcddbaf0e done  
Copying blob dcbc36c2ed7d done  
Copying blob 6d668c00f3f1 done  
Copying config cb083eb465 done  
Writing manifest to image destination
Copying config cb083eb465 [--------------------------------------] 0.0b / 1.9KiB
Writing manifest to image destination
Storing signatures
[student@server1 ~]$
You can find the image here:
https://quay.io/smissaertj/fedora-apache



Configuring and Managing Time Services
Joeri JM Smissaert — Wed, 10 Feb 2021 00:00:00 GMT
{{< figure class="center" src="/img/redhat-8-logo.png" alt="Red Hat logo">}}
Understanding Local Time
When a Linux machine boots, the hardware clock, also referred to as the real-time clock, is read. This clock resides in the computer hardware, it's in an integrated circuit on the system board that is independent of the current state of the operating system. It keeps running when the computer is shutdown, as long as the system board battery or power supply feeds it. The hardware clock value is known as hardware time, the system gets its initial time setting from hardware time. The hardware clock is usually set to Coordinated Universal Time (UTC).
System time is maintained by the operating system, it's independent of the hardware clock. When the system time is changed, the new system time is not automatically synchronized with the hardware clock.  
System time is kept in UTC, applications runing on the operating system convert system time into local time. Local time is the actual time in the current time zone, daylight saving time (DST) is considered so that the system always shows an accurate time.
{{}}
Concept | Explanation
-------|------
Hardware clock | The clock that resides on the main board of a computer system.
Real-time clock | Same as hardware clock.
System time | The time that is maintained by the operating system.
Software clock | Similar to system time.
UTC | Coordinated Universal Time, a worldwide standard time.
Daylight saving time | Calculation that is made to change time automatically when DST changes occur.
local time | The time that corresponds to the time in the current time zone.
{{
}}
Using Network Time Protocol
Since the hardware clock is typically part of the computer's motherboard, it can be potentially unreliable. It's a good idea to use time from a more reliable source. Generally speaking, two solutions are available.  
One option is to buy a more reliable hardware clock. Using an external hardware clock is a common solution in datacenter environments to guarantee reliable time is maintained even if external networks for time synchronization are temporarily not available. An example would be a very accurate atomic clock. 
A more common solution is to configure your machine to use Network Time Protocol (NTP), a method of maintaining system time provided through NTP servers on the Internet. To determine which Internet NTP server should be used, the concept of stratum is introduced. Stratum defines the reliability of an NTP time source, and the lower the stratum value, the more reliable it is. Typically, Internet time servers are using stratum 1 or 2. When you configure a local time server, you can use a higher stratum value. As a consequence, machines configured to use the local time server will only ever use it if Internet time servers (with a lower stratum) are not available. 
Setting up a machine to use NTP on RHEL 8 is easy if the server is already connected to the internet. In this case the /etc/chrony.conf file is prepopulated with a standard list of NTP servers. You would only need to turn on NTP using the timedatectl set-ntp true command (more on this later). 
Managing Time on Red Hat Enterprise Linux
On a Linux system, time is calculated as an offset of epoch time. Epoch time is the number seconds since January 1, 1970, in UTC. You can convert an epoch time stamp to a human readable form using the date --date command, followd by the epoch string starting with an @:
[student@server1 ~]$ date --date @1420987251
Sun Jan 11 06:40:51 PM +04 2015
Using date
The date command enables you to manage the system time. Or you can use it to show the current time in different formats:

date - Shows the current system time.
date +%d-%m-%y - Shows the current system day, month and year.
date -s 16:03 - Sets the current system time to 3 minutes pas 4pm.

Using hwclock
The date command will not change the hardware time. To manage hardware time you can use the hwclock command, which has many options (See hwclock --help).
Some options of interest:

hwclock --systohc - Sync the current system time to the hardware clock.
hwclock --hctosys - Sync the current hardware time to the system clock.

Using timedatectl
The timedatectl command shows detailed information about the current time and date. It also displays the time zone, in addition to information about the use of NTP network time and DST.
The timedatectl command works with the below subcommands to perform time operations:
{{}}
Command | Explanation
-------|------
status | Shows the current time settings.
set-time TIME | Sets the current time.
set-timezone TIMEZONE | Sets the time zone.
list-timezone | Shows a list of all time zones.
set-local-rtc [0|1] | Controls whether the real-time clock (hardware clock) is in local time.
set-ntp [0|1] | Enables or disables NTP.
{{
}}
[root@server1 ~]# timedatectl status
               Local time: Mon 2021-03-15 21:27:17 +04
           Universal time: Mon 2021-03-15 17:27:17 UTC
                 RTC time: Mon 2021-03-15 17:27:17
                Time zone: Indian/Mauritius (+04, +0400)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no


[root@server1 ~]# timedatectl set-time 22:30
[root@server1 ~]# timedatectl
               Local time: Mon 2021-03-15 22:30:03 +04
           Universal time: Mon 2021-03-15 18:30:03 UTC
                 RTC time: Mon 2021-03-15 18:30:03
                Time zone: Indian/Mauritius (+04, +0400)
System clock synchronized: no
              NTP service: inactive
          RTC in local TZ: no
After enabling NTP again, you will have to wait a few minutes for the time to synchronize again:
[root@server1 ~]# timedatectl set-ntp 1
[root@server1 ~]# timedatectl
               Local time: Mon 2021-03-15 21:30:19 +04
               ....

[root@server1 ~]# timedatectl list-timezones | grep -i mauritius
Indian/Mauritius
[root@server1 ~]# timedatectl set-timezone Indian/Mauritius
[root@server1 ~]#
Managing Time Zone Settings
Between Linux servers, time is normally communicated in UTC. This allows servers located in different time zones to use the same time settings, making it easier to manage large organizations. To make it easier for end users, we should set the local time, and for this we would need to configure an appropriate time zone. 
There are 3 approaches to setting the local time zone.

Use timedatectl set-timezone
Use the tzselect command to start an text based interface.
Go the the /usr/share/zoneinfo directory where you'll find different subdirectories containing files for each time zone. To select a time zone, you create a symbolic link with the name /etc/localtime to the relevant time zone file. e.g. ln -sf /usr/share/zoneinfo/America/Los_Angeles /etc/localtime

Configuring Time Service Clients
By default, the chrony service is configured to get the right time from the Internet. In a corporate environment it is not always desirable for clients to go out to the Internet, and instead time servers on the local network are configured.
In the below example we'll configure an NTP server on server2 and we'll configure server1 as the client.
On server1 we comment out the predefined NTP server in /etc/chrony.conf and define the server2 pool:
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#pool 2.rhel.pool.ntp.org iburst
pool server2
On server2 we edit /etc/chrony.conf to allow connections from a specific subnet, we set a stratum value, then configure the firewall and restart the chronyd service:
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#pool 2.rhel.pool.ntp.org iburst

allow 192.168.0.0/16
local stratum 8
[root@server2 ~]# firewall-cmd --add-service=ntp --permanent
success
[root@server2 ~]# firewall-cmd --reload
success
[root@server2 ~]# systemctl restart chronyd
[root@server2 ~]#
Restart the chronyd service on server1 and check if server2 is used as a source:
[root@server1 ~]# systemctl restart chronyd
[root@server1 ~]# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^? server2                       8   6     1     6    +15us[  +15us] +/-   98us
[root@server1 ~]#



Configuring and Auto Mounting Remote File Systems Using fstab and automount: NFS & CIFS
Joeri JM Smissaert — Sun, 10 Jan 2021 00:00:00 GMT
{{< figure class="center" src="/img/redhat-8-logo.png" alt="Red Hat logo">}}
Using NFS Services
The Network File System is a protocol that was developed for UNIX by Sun in the early 1980s. Its purpose is to make mounting of remote file systems in the local file system hierarchy possible. It was often used with Network Information Services (NIS) which provides network-based authentication, all machines connected to the NIS server used the same user accounts and security was handled by the NIS server. NFS security by default is limited to allowing and restricting specific hosts. 
Without NIS, NFS seems to be an unsecure solution: if on server1 the user X has UID 1001 and on server2 user Y has UID 1001, then user X would have the same access to server2 resources as user Y. To prevent situations like this, NFS should be used together with a centralized authentication service like the Lightweight Directory Access Protocol (LDAP) and Kerberos. This solution is not covered in this article. 
On RHEL8, NFSv4 is the default version of NFS wich you can override when mounting using the nfsvers= mount option. Typically, clients will automatically fallback to a previous version of NFS if required.
Offering an NFS Share
To setup an NFS share you would need to go through a few tasks:

Create local directories which you want to share and copy some data into them:
[root@server2 ~]# mkdir -p /nfs_data /nfs_users/user{1..2}
[root@server2 ~]# cp -r /etc/[a-c]* /nfs_data/
[root@server2 ~]# cp -r /etc/[d-f]* /nfs_users/user1/
[root@server2 ~]# cp -r /etc/[g-i]* /nfs_users/user2/

Edit the /etc/exports file to define the NFS shares:
[root@server2 ~]# cat /etc/exports
/nfs_data    *(rw,no_root_squash)
/nfs_users        *(rw,no_root_squash)

Start and enable the NFS server:
[root@server2 ~]# yum install nfs-utils
[root@server2 ~]# systemctl enable --now nfs-server

Configure the firewall to allow incoming NFS traffic


[root@server2 ~]# firewall-cmd --add-service=nfs --permanent
success
[root@server2 ~]# firewall-cmd --add-service=rpc-bind --permanent
success
[root@server2 ~]# firewall-cmd --add-service=mountd --permanent
success
[root@server2 ~]# firewall-cmd --reload
success
Mounting the NFS Share
In order to mount an NFS share we need to know the name of the share. Typically this information is known by the administrator, but you have multiple options to discover what shares are available:

If NFSv4 is used on the server, you can use a root mount. You mount the root directory of the NFS server and you'll see all shares you have access to under your local mount point. 
Use the showmount -e command


The showmount command may have issues with NFSv4 servers that are behind a firewall. The command relies on the portmapper service which uses random UDP ports while the firwall nfs service opens port 2049 only, which doesn't allow portmapper traffic. In these cases you can use the root mount option to discover the shares.

[root@server1 ~]# showmount -e server2
Export list for server2:
/nfs_data   *
/nfs_users *
[root@server1 ~]# mount server2:/ /mnt
[root@server1 ~]# ls /mnt/
nfs_data  nfs_users
Using CIFS Services
Microsoft published the technical specifications of its Server Message Block (SMB) protocol. This protocol is the foundation of all shares that are created in a Windows environment.
Releasing these specifications led to the start of the Samba project. The goal of this project was to provide SMB services on top of other operating systems. Samba has developed into the standard for file sharing between different operating systems and is now often referred to as the Common Internet File System (CIFS).
Setting Up a Samba Server
Before jumping into configuring the samba server, let's clearly define our goals.
Server2, the samba server, should be sharing the following directories:

/var/samba/public_read_share - read only access for guests, mounted on /mnt/public_read_share
/var/samba/public_write_share - read/write permissions for guests, mounted on /mnt/public_write_share
/var/samba/student_share - read permissions for guests, read/write permissions for users in the students group. Mounted on /mnt/students_share.

Installing and Configuring Samba
Install the samba package and create the shared directories:
[root@server2 ~]# yum install samba -y
...
[root@server2 ~]# mkdir -p /var/samba/{public_share,public_write_share,students_share}
[root@server2 ~]# ls /var/samba/
public_share  public_write_share  students_share
We enable the smbd_anon_write SELinux Boolean which allows anonymous users to modify public files labeled with the public_content_rw_t file context.
Next, we set the appropriate SELinux file contexts:

public_content_t - Allows Read Only access to public files.
public_content_rw_t - Allows Read/Write access to public files.
samba_share_t - As samba doesn't have default paths for shares, we make sure SELinux recognizes our share as a standard samba share.

[root@server2 samba]# pwd
/var/samba

[root@server2 samba]# ls -lh
total 0
drwxr-xr-x. 2 root root 6 Mar 11 13:24 public_share
drwxr-xr-x. 2 root root 6 Mar 11 13:24 public_write_share
drwxr-xr-x. 2 root root 6 Mar 11 13:24 students_share

[root@server2 samba]# setsebool -P smbd_anon_write on
[root@server2 samba]# getsebool smbd_anon_write 
smbd_anon_write --> on

[root@server2 samba]# semanage fcontext -a -t public_content_t "/var/samba/public_share(/.*)?"
[root@server2 samba]# semanage fcontext -a -t public_content_rw_t "/var/samba/public_write_share(/.*)?"
[root@server2 samba]# semanage fcontext -a -t samba_share_t "/var/samba/students_share(/.*)?"
[root@server2 samba]# restorecon -Rv /var/samba/
Relabeled /var/samba/public_share from unconfined_u:object_r:var_t:s0 to unconfined_u:object_r:public_content_t:s0
Relabeled /var/samba/public_write_share from unconfined_u:object_r:var_t:s0 to unconfined_u:object_r:public_content_rw_t:s0
Relabeled /var/samba/students_share from unconfined_u:object_r:var_t:s0 to unconfined_u:object_r:samba_share_t:s0

[root@server2 samba]# ls -lhZ
total 0
drwxr-xr-x. 2 root root unconfined_u:object_r:public_content_t:s0    6 Mar 11 13:24 public_share
drwxr-xr-x. 2 root root unconfined_u:object_r:public_content_rw_t:s0 6 Mar 11 13:24 public_write_share
drwxr-xr-x. 2 root root unconfined_u:object_r:samba_share_t:s0       6 Mar 11 13:24 students_share
Create the students group and, add the user student to the group.
Create the smb_user through which we'll be able to write to the public_write_share directory.
Add the Linux user student to samba and set a password. This credential will be used to authenticate and mount the students_share directory.
Set the Linux permissions on the shared directories:
[root@server2 samba]# groupadd students
[root@server2 samba]# usermod -aG students student
[root@server2 samba]# id student
uid=1000(student) gid=1000(student) groups=1000(student),1001(students)

[root@server2 samba]# useradd smb_user --no-create-home --shell /sbin/nologin
[root@server2 samba]# 

[root@server2 samba]# smbpasswd -a student
New SMB password:
Retype new SMB password:
Added user student.

[root@server2 samba]# chgrp smb_user public_write_share
[root@server2 samba]# chmod 0770 public_write_share
[root@server2 samba]# chmod g+s public_write_share
[root@server2 samba]# 

[root@server2 samba]# chgrp students students_share
[root@server2 samba]# chmod 0775 students_share
[root@server2 samba]# chmod g+s students_share
[root@server2 samba]# 

[root@server2 samba]# ls -lhZ
total 0
drwxr-xr-x. 2 root root     unconfined_u:object_r:public_content_t:s0    6 Mar 11 13:24 public_share
drwxrws---. 2 root smb_user unconfined_u:object_r:public_content_rw_t:s0 6 Mar 11 13:24 public_write_share
drwxrwsr-x. 2 root students unconfined_u:object_r:samba_share_t:s0       6 Mar 11 13:24 students_share
Note that we don't change any permissions on public_share, since we only need read access.
Next, we configure the samba shares in /etc/samba/smb.conf:
[root@server2 samba]# cd /etc/samba
[root@server2 samba]# mv smb.conf smb.conf.old
[root@server2 samba]# vim smb.conf
...

[root@server2 samba]# testparm
Load smb config files from /etc/samba/smb.conf
Loaded services file OK.
Server role: ROLE_STANDALONE

Press enter to see a dump of your service definitions

# Global parameters
[global]
    security = USER
    workgroup = SAMBA
    idmap config * : backend = tdb


[public_read]
    comment = Public Read Only Share
    guest ok = Yes
    path = /var/samba/public_share


[public_write]
    comment = Public Read/Write Share
    force user = smb_user
    guest ok = Yes
    path = /var/samba/public_write_share
    read only = No
    write list = smb_user


[students]
    comment = Read/Write access for the students group. Read access for anyone else.
    guest ok = Yes
    path = /var/samba/students_share
    write list = +students
We need to allow samba traffic through our firewall:
[root@server2 samba]# firewall-cmd --add-service=samba --permanent
success
[root@server2 samba]# firewall-cmd --reload
success
The final step before moving on to the client side would be to start and enable the samba service:
[root@server2 samba]# systemctl enable --now smb
Created symlink /etc/systemd/system/multi-user.target.wants/smb.service → /usr/lib/systemd/system/smb.service.
Discovering CIFS Shares
On server1, where the shares are going to be mounted, you discover available shares using the smbclient -L //hostname command. 
Make sure you have the cifs-utils and samba-client packages installed:
[root@server1 ~]# yum install -y cifs-utils samba-client
...
Let's discover the shares we created on server2. When you're prompted for a password, just hit Enter without providing a password.
[root@server1 ~]# smbclient -L //server2
Enter SAMBA\root's password: 
Anonymous login successful

    Sharename       Type      Comment
    ---------       ----      -------
    public_read     Disk      Public Read Only Share
    public_write    Disk      Public Read/Write Share
    students        Disk      Read/Write access for the students group. Read access for anyone else.
    IPC$            IPC       IPC Service (Samba 4.12.3)
SMB1 disabled -- no workgroup available
[root@server1 ~]#
We're ready to move to the next step and mount our shares.
Mounting and Authenticating to Samba Shares
In the previous steps we created two guest shares and one share that needs authentication. 
We can mount these as follows:  

mount -t cifs -o guest //server2/public_read /mnt/public_read_share  
mount -t cifs -o guest //server2/public_write /mnt/public_write_share  
mount -t cifs -o username=student,password=password //server2/students_share /mnt/students_share  

Before you do so, create the local mount points:
[root@server1 ~]# mkdir /mnt/{public_read_share,public_write_share,students_share}
[root@server1 ~]# ls -l /mnt/
total 0
drwxr-xr-x. 2 root root 6 Mar 11 14:41 public_read_share
drwxr-xr-x. 2 root root 6 Mar 11 14:41 public_write_share
drwxr-xr-x. 2 root root 6 Mar 11 14:41 students_share

[root@server1 ~]# mount -t cifs -o guest //server2/public_read /mnt/public_read_share
[root@server1 ~]# mount -t cifs -o guest //server2/public_write /mnt/public_write_share
[root@server1 ~]# mount -t cifs -o username=student,password=password //server2/students /mnt/students_share
Next, test the read/write access to the shares. The outcome should be as expected.

Note that we've mounted the share as root, this means the /mnt/students_share directory will only be writeable for the user root. In the next step we'll cover how to auto mount the share at boot time.

Mounting Remote File Systems Through fstab
As we've seen in an earlier post, the /etc/fstab file can be used to mount file systems automatically at boot time. 
Mounting NFS Shares Through fstab
Mounting NFS Shares through /etc/fstab is pretty straightforward. Add the following line to the fstab file:
server2:/nfs_data    /nfs_data    nfs  sync     0 0
With the sync option we ensure that modified files are committed to the remote file system immediately instead of being placed in a write buffer.
Mounting Samba Shares Through fstab
When mounting Samba file systems through /etc/fstab, you need to consider a specific challenge: The user credentials that are needed to issue the mount. 
These are typically specified as mount options using username= and password=, but it is not a good idea to put these in clear text in the /etc/fstab file.
We can work around this by creating a file in the root home that contains these credentials, and referencing /etc/fstab to that file:
[root@server1 ~]# pwd
/root
[root@server1 ~]# cat cifs.txt 
user=student
pass=password
[root@server1 ~]#
We set strict permissions on the file so only root can read it:
[root@server1 ~]# chmod 0600 cifs.txt
[root@server1 ~]#
Next, for the //server2/students share, we add the following line to /etc/fstab:
//server2/students    /mnt/students_share    cifs    credentials=/root/cifs.txt,gid=students,file_mode=0664,dir_mode=0775 0 0
Let's break down what the line does exactly:

//server2/students - The remote file system we're mounting.
/mnt_students_share - The local mount point of the share.
cifs - The remote file system type.
credentials=/root/cifs.txt - Specifies the file that contains the credentials necessary to mount the remote file system.
gid=students - We set group ownership on the files and directories to the group students .
file_mode=0664 - We set the necessary file permissions: read+write for Owner and Group, read for Others.
dir_mode=0775 -  We set the necessary directory permissions: read+write+execute for Owner and Group, read+execute for Others.
0 0 - We don't need backup support through the dump utility and we don't want fsck to check the disk integrity during boot.

Simarly to the above, the entry for the //server2/public_write_share would look like this:
//server2/public_write_share    /mnt/public_write_share cifs    guest,file_mode=0777,dir_mode=0777    0 0
We autenticate as the user guest aganst the remote file system and we allow everyone read+write access to files and, read+write+execute permissions to directories.
For the last share, //server2/public_share, we don't specify Linux permissions in the /etc/fstab file as this share has been set to read-only by default on the Samba server. 
//server2/public_share    /mnt/public_read_share    cifs    guest    0 0
Here all three /etc/fstab entries:
//server2/public_share    /mnt/public_read_share    cifs    guest    0 0
//server2/public_write_share    /mnt/public_write_share cifs    guest,file_mode=0666,dir_mode=0777    0 0
//server2/students    /mnt/students_share    cifs    credentials=/root/cifs.txt,gid=students,file_mode=0664,dir_mode=0775 0 0
Using Automount to Mount Remote File Systems
As an alternative to using /etc/fstab we can configure automount to mount the shares automatically. The difference is that mounts through automount are "on demand", which ensures that no files systems are mounted when it's not needed. This works completely in user space and no root permissions are required, contrary to mounts using the mount command.
You need to install the autofs package to use automount:
[root@server1 ~]# yum install -y autofs
...
[root@server1 ~]# systemctl enable --now autofs
...
Defining Mounts in Automount
Mounts in automount are defined through a two-step procedure:

Edit the master configuration file in /etc/auto.master where you specify the local mount point and the secondary configuration file.
Edit the secondary configuration file where you specify the subdirectory that will be created in the mount point.

For this exercise, we'll be using the nfs_data NFS share on server2:
[root@server2 ~]# cat /etc/exports
/nfs_users    *(rw,no_root_squash)
/nfs_data    *(rw,no_root_squash)
On server1, open the /etc/auto.master file and add the below line:  
/nfs_data    /etc/auto.nfs_data
On server1, open the /etc/auto.nfs_data file and add the below line:  
files -rw server2:/nfs_data
Restart the autofs service:
[root@server1 /]# systemctl restart autofs
Go to the /nfs_data directory on server1, notice there is no files directory:
[root@server1 nfs_data]# ls
[root@server1 nfs_data]#
Change directory to /nfs_data/files:
[root@server1 nfs_data]# cd files
[root@server1 files]# ls
automount_test
The /nfs_data share on server2 was auto mounted on /nfs_data/files on server1.
Using Wildcards in Automount
In some cases we're better off using dynamic directory names, for example when mounting home directories. The home directory of a user would be automatically mounted when that user logs in.  
We'll be simulating this by using the /nfs_users NFS share on server2:
[root@server2 ~]# cat /etc/exports
/nfs_users    *(rw,no_root_squash)
/nfs_data    *(rw,no_root_squash)
First, unmount the /nfs_users mount point on server1, if you still have it mounted, and delete the directory:
[root@server1 /]# umount /nfs_users 
[root@server1 /]# rm -rf nfs_users
Add the below line to the /etc/auto.master file on server1:
/nfs_users      /etc/auto.nfs_users
Create the /etc/auto.nfs_users file and add the below:
* -rw server2:/nfs_users/&`
Restart the autofs service:
[root@server1 /]# systemctl restart autofs
Go to the /nfs_users directory and notice it's empty:
[root@server1 /]# cd /nfs_users
[root@server1 nfs_users]# ls
[root@server1 nfs_users]#
Change directory to /nfs_users/user1:
[root@server1 nfs_users]# cd user1
[root@server1 user1]# ls
user1_automount_test
[root@server1 user1]#
See how the other user folders are auto-mounted on demand:
[root@server1 nfs_users]# ls
user1
[root@server1 nfs_users]# cd user2
[root@server1 user2]# ls
user2_automount_test
[root@server1 user2]# cd ..
[root@server1 nfs_users]# ls
user1  user2



Managing a Firewall with Firewalld
Joeri JM Smissaert — Sun, 27 Dec 2020 00:00:00 GMT
{{< figure class="center" src="/img/redhat-8-logo.png" alt="Red Hat logo">}}
{{< figure class="center" src="/img/firewalld.png" alt="Firewalld logo" width="200px">}}
Understanding Linux Firewalling
Firewalling is implemented in the Linux kernel by means of the netfilter subsystem to limit traffic coming in to a server or going out of the server. Netfilter allows kernel modules to inspect every incoming, outgoing, or forwarded packet and act upon it by either allowing it or blocking it. In essence, netfilter controls access to and from the network stack at the Linux kernel module level.
Iptables was the default solution to interact with netfilter, it provides a sophisticated way of defining firewall rules but it's also challenging to use due to the complicated syntax and the ordering of rules which can become complex. The iptables service is no longer offered in RHEL8, it has been replaced with nftables, a new solution with more advanced options.
Firewalld
Firewalld is a higher-level netfilter implementation that is more user-friendly compared to iptables or nftables. While administrators can manage the Firewalld rules, applications can also communicate with it using the DBus messaging system: rules can be added or removed without any direct action required from the system administrator. Applications can address the firewall from user space. 

Firewalld applies rules to incoming packets only by default, no filtering happens on outgoing packets.

Firewalld Zones
Firewalld makes management easier by working with zones. A zone is a collection of rules that are applied to incoming packets matching a specific source address or network interface. 
The use of zones is import on servers that have multiple network interfaces. Each interface could be a different zone where different rules would apply. On a machine with only one network interface you can work with one zone, the default zone. 
Every packet that comes into a system is analyzed for its source address, based on the source address Firewalld decides if it belongs to a specific zone. If not, the zone for the incoming network interface is used. If no specific zone is available, the packet is handled by the rules in the default zone. 
{{}}
Zone Name | Description
-----|----
block | Incoming network connections are rejected with the "icmp-host-prohibited" message. Connections that were initiated on this system are allowed.
dmz | For use on computers in the demilitarized zone. Selected incoming connections are accepted, and limited access to the internal network is allowed.
drop | Any incoming packets are dropped and there is no reply.
external | For use on external networks with masquarading (Network Address Translation) enabled, used on routers. Selected incoming connections are accepted.
home | Most computers on the same network are trusted, only selected incoming connections are accepted.
internal| Most computers on the same network are trusted, only selected incoming connections are accepted.
public | Other computers on the same network are not trused, limited connections are accepted. This is the default zone for all newly created network interfaces.
trusted | All network connections are accepted.
work | Most computers on the same network are trusted, only selected incoming connections are accepted.
{{
}}
Firewalld Services
Services are the second key element while working with Firewalld.
A service in Firewalld is not the same as a service in systemd. A Firewalld service defines what exactly should be accepted as incoming traffic in the firewall, it includes ports to be opened and supoorting kernel modules that should be loaded. 
Behind each service is an XML configuration file that explains which TCP or UDP ports are involved and, if required, what kernel modules must be loaded.  Default (RPM installed) XML files are stored in /usr/lib/firewalld/services while custom XML files can be added to the /etc/firewalld/services directory.
[root@localhost ~]# firewall-cmd --get-services
RH-Satellite-6 amanda-client amanda-k5-client amqp amqps ...
...


[root@localhost ~]# cat /usr/lib/firewalld/services/ftp.xml 
"1.0" encoding="utf-8"?>
<service>
  <short>FTPshort>
  <description>FTP is a protocol used for remote file transfer. If you plan to make your FTP server publicly available, enable this option. You need the vsftpd package installed for this option to be useful.description>
  <port protocol="tcp" port="21"/>
  <helper name="ftp"/>
service>
Working with Firewalld
Firewalld provides a command-line interface tool that works with a runtime and permament (on-disk) configuration state: firewall-cmd
Below is an example of how you can use the tool to retrieve current settings and make configuration changes. Always make sure to commit changes to disk using the --permanent flag so that your changes survive a reboot, then --reload to apply the changes to the runtime environment.
[root@localhost ~]# firewall-cmd --get-default-zone
public

[root@localhost ~]# firewall-cmd --get-zones
block dmz drop external home internal libvirt public trusted work

[root@localhost ~]# firewall-cmd --list-all --zone=public
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp1s0
  sources: 
  services: cockpit dhcpv6-client ftp http https ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

[root@localhost ~]# firewall-cmd --get-services
RH-Satellite-6 amanda-client amanda-k5-client amqp amqps apcupsd audit bacula bacula-client bb bgp bitcoin bitcoin-rpc bitcoin-testnet
...

[root@localhost ~]# firewall-cmd --list-services
cockpit dhcpv6-client ftp http https ssh

[root@localhost ~]# firewall-cmd --add-service=vnc-server --permanent
success

[root@localhost ~]# firewall-cmd --list-services
cockpit dhcpv6-client ftp http https ssh

[root@localhost ~]# firewall-cmd --reload
success

[root@localhost ~]# firewall-cmd --list-services
cockpit dhcpv6-client ftp http https ssh vnc-server

[root@localhost ~]# firewall-cmd --add-port=2022/tcp --permanent
success

[root@localhost ~]# firewall-cmd --reload
success

[root@localhost ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp1s0
  sources: 
  services: cockpit dhcpv6-client ftp http https ssh vnc-server
  ports: 2022/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:
Key Commands
firewall-cmd --list-all
firewall-cmd --list-all --zone=public

firewall-cmd --get-default-zone
firewall-cmd --get-zones

firewall-cmd --get-services
firewall-cmd --list-services

firewall-cmd --add-service ftp
irewall-cmd --add-service ftp --permanent
firewall-cmd --reload

firewall-cmd --add-port=2022/tcp --permanent
firewall-cmd --reload



Enhancing Linux Security with SELinux
Joeri JM Smissaert — Wed, 16 Dec 2020 00:00:00 GMT
{{< figure class="center" src="/img/redhat-8-logo.png" alt="Red Hat logo">}}
{{< figure class="center" src="/img/selinux.png" alt="SELinux logo" width="200px">}}
SELinux is a security enhancement module, deployed on top of Linux, which provides improved security via Role Based Access Controls (RBACs) on subjects and objects (processes and resources). Traditional Linux security used Discretionary Access Controls (DACs).
With DAC, a process can access any file, directory, device or other resource that leaves itself open to access. Using RBAC, a process only has access to resources that it is explicitely allowd to access, based on the assigned role. The way that SELinux implements RBAC is to assign an SELinux policy to a process. That process restricts access as follows:

Only let the process access resources that carry the explicit labels
Make potentially insecure features, e.g. write access to a directory, available as Booleans, which can be turned on or off.

SELinux is not a replacement for DAC, it's an additional security layer:

DAC rules are still used when using SELinux;
DAC rules are checked first, if those allow access then SELinux policies are checked;
If DAC rules deny access then SELinux policies are not checked.

In essence, SELinux severaly limits what potentially malicious code may gain access to and generally limits activity on the Linux system.
Understanding How SELinux Works
SELinux provides a combination of Role Based Access Control and either Type Enforcement (TE) or Multi-Level Security (MLS). In RBAC, access to an object is granted or denied based on the subject's assigned role in the organization. It's not based on usernames or process ID. In this post I will focus only on Type Enforcement, which is the default SELinux targeted policy.
Type Enforcement
Type Enforcement is necessary to implement the RBAC model, it secures a system through these methods:

Labeling objects as certain security types;
Assigning subjects to particular domains and roles;
Providing rules to allow certain domains and roles to access certain object types.

Let's look at an example.
The below ls -l command shows the DAC controls on the files. The output shows the file's owner, group and permissions:
[student@localhost my_stuff]$ ls -l
total 0
-rw-rw-r--. 1 student student 0 Jan 19 06:25 test001
We can add the -Z option to display the SELinux RBAC controls too:
[student@localhost my_stuff]$ ls -lZ
total 0
-rw-rw-r--. 1 student student unconfined_u:object_r:user_home_t:s0 0 Jan 19 06:25 test001
The last example displays four items assiciated with the file that are specific to SELinux:

user unconfined_u
role object_r
type user_home_t
level s0

The above four RBAC items are used in the SELinux access control  to determine appropriate access levels. Together, these items are called the SELinux security context or sometimes the security label.
These security contexts are given to to subjects (processes and users). Each security context has a specific name. The name given depends upon what object or subject it has been assigned: Files have a file context, users have a user context, and processes have a process context also referred to as a domain.
The rules allowing access are called allow rules or policy rules. A policy rule is the process SELinux follows to grant or deny access to a particular system security type. Thus, Type Enforcement ensures that only certain "types" of subjects can access certain "types" of objects.
Implementing SELinux Security Models
SELinux implements the RBAC model through a combination of four primary SELinux pieces:

Operational modes
Security contexts
Policy types
Policy rule packages

We already touched on some of these design elements.
Understanding SELinux Operational Modes
SELinux comes with three operational modes: disabled, permissive and enforcing.
Each of these modes offeres different benefits for Linux system security.
Using Disabled Mode
In the disabled mode, SELinux is turned off. The default method of access control, Discretionary Access Control, is used instead.
Using Permissive Mode
In permissive mode, SELinux is turned on, but the security policy rules are not enforced. When a security policy rule should deny access, access will still be allowed. However, a message is sent to a log file denoting that access should've been denied.
SELinux permissive mode is useful for testing and troubleshooting. 
Using Enforcing Mode
In enforcing mode SELinux is turned on and all of the security policy rules are enforced.
Understanding SELinux Security Contexts
An SELinux security context is the method used to classify objects (such as files) and subjects (such as users or programs). A security context consists of four attributes: user, role, type and level.

User - The user attribute is a mapping of a Linux username to an SELinux name. This is not the same as a users's login name, and it's referred to specifically as the SELinux user. The SELinux username ends with a _u, making it easy to identify in the output. Regular unconfined users have an unconfined_u user attribute in the default targeted policy.

Role - The role attribute is assigned to subjects and objects. Each role is granted access to other subjects and objects based on the role's security clearance and the object's classification level. Users are assigned a role and that role is authorized for particular types of domains (or process context). The SELinux role has _r at the end. Processes run by root have a system_r role, while regular users run processes under the unconfined_r role.

Type - The type attribute defines a domain type for processes, a user type for users, and a file type for files. This attribute is also called the security type. Most policy rules are concerned with the security type of a process and what files, ports, devices and other resources that process has access to based on their security types. The SELinux type name ends with a _t.

Level - The level is an attribute of Multi-Level Security (MLS), it's optional in Type Enforcement. 


Users, Files, and Processes Have Security Contexts
To see your SELinux user context, enter the id command at the shell prompt:
[student@localhost ~]$ id
uid=1000(student) gid=1000(student) groups=1000(student) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[student@localhost ~]$
Use the -Z option on the ls command to see an individual file's context:
[student@localhost my_stuff]$ ls -lZ
total 0
-rw-rw-r--. 1 student student unconfined_u:object_r:user_home_t:s0 0 Jan 19 06:25 test001
Use the -Z option on the ps command to see a process's security context:
[student@localhost my_stuff]$ ps -eZ | grep bash
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 2872 pts/0 00:00:00 bash

[student@localhost my_stuff]$ ps -eZ | grep systemd
system_u:system_r:init_t:s0           1 ?        00:00:01 systemd
system_u:system_r:syslogd_t:s0      638 ?        00:00:00 systemd-journal
Understanding SELinux Policy Types
The policy type directly determines what sets of policy rules are used to dictate what an object can access. The policy type also determines what specific security context attributes are needed.
SELinux has different policies:

Targeted (default)
MLS
Minimum

The Targeted policy's primary purpose is to restrict "targeted" daemons, but it can also restrict other processes and users. Targeted daemons are sandboxed, they run in an environment where their access to other objects is tightly controlled so that no malicious attacks launched through those daemons can affect other services or the Linux system as a whole. 
All subjects and objects not targeted are run in the unconfined_t domain. This domain has no SELinux policy restrictions and thus only used traditional Linux security.
SELinux Policy Rule Packages
Policy rules are installed with SELinux and are grouped into packages, also called modules.
There is user documentation on these various policy modules in the form of HTML files. To view this documentation on RHEL, open your browser and enter the following url:
file:///usr/share/doc/selinux-policy/html/index.html
If you don't have the policy documentation you can install it:
yum install selinux-policy-doc
This documentation allows you to review how policy rules are created and packaged.
Configuring SELinux
SELinux comes preconfigured, you can use the SELinux features without any configuration.
The configuration can only be set and modified by root. The primary configuration file is /etc/sysconfig/selinux which is a symlink to /etc/selinux/config:
[root@localhost ~]# ls -lh /etc/sysconfig/selinux 
lrwxrwxrwx. 1 root root 17 Sep 26 09:44 /etc/sysconfig/selinux -> ../selinux/config
[root@localhost ~]# cat /etc/sysconfig/selinux 

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
This file allows you to set the mode and policy type.
Setting the SELinux Mode and Policy Type
We can use the getenforce command to see the current SELinux mode, to see both the current mode and the mode set in the configuration file, use the sestatus command:
[root@localhost ~]# getenforce
Enforcing

[root@localhost ~]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      31
To change the mode setting, you can use the setenforce command with either the 0 or permissive argument, or the 1 or enforcing argument.
This will change the SELinux mode during runtime and leaves the setting in the primary configuration file untouched. Rebooting the system will apply the mode set in the primary configuration file.

You cannot use setenforce to change SELinux to disabled mode.

Switching from disabled to enforcing should be done using the primary configuration file and a reboot. Using the setenforce command may hang your system due to incorrect file labels. Rebooting after changing from disabled may take a while as the filesystem will be relabeled.  
This means that SELinux checks and changes the security context of any files with incorrect security contexts that can cause problems in the new mode, and any file not labeled will be labeled with contexts. This process can take a long time since each file's context is checked.
The policy type you choose determines whether SELinux enforces TE, MLS or Minimum. The default policy type is targeted.
When setting the policy type to MLS or Minimum you need to make sure you have the policy package installed:
yum list selinux-policy-mls selinux-policy-minimum
Managing SELinux Security Contexts
Current SELinux file and process security contexts can be viewed using the secon command:

-u Shows the user of the security context.
-r Shows the role of the security context.
-t Shows the type of the security context.

Without any arguments, the command shows you the current process's security context:
[student@localhost ~]$ secon -urt
user: unconfined_u
role: unconfined_r
type: unconfined_t
To view another process's security context, use the -p option followed by the process id.
e.g. systemd:
[student@localhost ~]$ secon -urt -p 1
user: system_u
role: system_r
type: init_t
To view a file's security context, use the -f option:
[student@localhost ~]$ secon -urt -f /etc/passwd
user: system_u
role: object_r
type: passwd_file_t

The secon command does not show the security context for the current user, instead use the id command.

Setting Security Context Types

Since the RHCSA exam focuses only on context types, I will not be covering the user and role contexts.

To set a context type we can use the semanage command.
semanage writes the new context to the SELinux policy from where it can be applied to the file system.
The semanage command may not be available by default. You can find the RPM containing semanage using yum whatprovides */semanage:
[root@localhost ~]# yum whatprovides */semanage
policycoreutils-python-utils-2.9-9.el8.noarch : SELinux policy core python utilities
Repo        : BaseOS
Matched from:
Filename    : /usr/sbin/semanage
Filename    : /usr/share/bash-completion/completions/semanage
The policycoreutils-python-utils has to be installed in order to use semanage.
To set context using semanage we need to know the appropriate context type. An easy way to find the appropriate context is by looking at the default context settings on already-existing items:
[root@localhost ~]# ls -lZ /var/www
total 0
drwxr-xr-x. 2 root root system_u:object_r:httpd_sys_script_exec_t:s0  6 Jun  8  2020 cgi-bin
drwxr-xr-x. 4 root root system_u:object_r:httpd_sys_content_t:s0     61 Jan  6 12:19 html
/var/www/html is a default location for the Apache HTTP Service. If we would want to add a new folder to /var/www to serve content with Apache, we now know we need the http_sys_content_t context type. 
For demonstration purposes, let's created the my_dir directory in our home folder, then move it to /var/www. The reason why we do this is because if we create the directory in /var/www it will inherit the correct context type from the parent directory.
[root@localhost ~]# ls -lZ /var/www
total 0
drwxr-xr-x. 2 root root system_u:object_r:httpd_sys_script_exec_t:s0  6 Jun  8  2020 cgi-bin
drwxr-xr-x. 4 root root system_u:object_r:httpd_sys_content_t:s0     61 Jan  6 12:19 html
drwxr-xr-x. 2 root root unconfined_u:object_r:admin_home_t:s0         6 Jan 20 11:44 my_dir
The mv command kept the admin_home_t context type on our directory.
We can change the context type as follows:
[root@localhost ~]# semanage fcontext -a -t httpd_sys_content_t "/var/www/my_dir(/.*)?"
[root@localhost ~]# ls -lZd /var/www/my_dir
drwxr-xr-x. 2 root root unconfined_u:object_r:admin_home_t:s0 6 Jan 20 11:44 /var/www/my_dir
The -a option is used to add a context type, then we use -t to specify the context type. The last part of the command indicates the folder we apply the changes to and contains a regular expression, (/.*)?, to refer to the directory my_dir and anything that exists below that directory.
Notice how the semanage command didn't provide any output, and our ls -lZd command still shows the original context type.
This is because using semanage we only applied the context type to the SELinux policy but not to the file system. We need to apply the change to the file system using restorecon:
[root@localhost ~]# restorecon -R -v /var/www/my_dir
Relabeled /var/www/my_dir from unconfined_u:object_r:admin_home_t:s0 to unconfined_u:object_r:httpd_sys_content_t:s0
[root@localhost ~]# ls -lZd /var/www/my_dir
drwxr-xr-x. 2 root root unconfined_u:object_r:httpd_sys_content_t:s0 6 Jan 20 11:44 /var/www/my_dir
The following example changes the SELinux context type on a network port, assuming you would want to make the ssh service available over port 2222.
[root@localhost ~]# semanage port -l | grep ssh
ssh_port_t                     tcp      22
[root@localhost ~]# semanage port -a -t ssh_port_t -p tcp 2222
[root@localhost ~]# semanage port -l | grep ssh
ssh_port_t                     tcp      2222, 22
Finding the Context Type You Need
There are three approaches in finding the context type you need:

Look at the default environment;
Read the configuration files;
Use man -k _selinux to find the SELinux-specific man pages for your service.

The man pages are not installed by default, to install them you need to install the policycoreutils-devel package. Once installed, use the mandb command to update the man page database and issue the sepolicy manpage -a -p /usr/share/man/man8 command to install the SELinux man pages:
[root@localhost ~]# yum whatprovides */sepolicy
policycoreutils-devel-2.9-9.el8.i686 : SELinux policy core policy devel utilities
Repo        : BaseOS
Matched from:
Filename    : /usr/bin/sepolicy
Filename    : /usr/share/bash-completion/completions/sepolicy

[root@localhost ~]# yum install -y policycoreutils-devel
...

[root@localhost ~]# sepolicy manpage -a -p /usr/share/man/man8
...

[root@localhost ~]# mandb
...

[root@localhost ~]# man -k _selinux | grep http
apache_selinux (8)   - Security Enhanced Linux Policy for the httpd processes
httpd_helper_selinux (8) - Security Enhanced Linux Policy for the httpd_helper processes
httpd_passwd_selinux (8) - Security Enhanced Linux Policy for the httpd_passwd processes
...

[root@localhost ~]# man apache_selinux
Restoring Default File Contexts
Previously, we applied the context type from the policy to the file system using the restorecon command. The policy contains the default settings for most files and directories, so if ever a wrong context setting is applied we can use restorecon to reapply the default from the policy to the file system.
Using restorecon this way can be useful to fix problems on new files. There's a specific way context settings are applied:

If a new file or directory is created, it inherits the context type of the parent directory.
If a file or directory is copied, this is considered a new file or directory.
If a file is moved, or copied using cp -a and thus keeping properties, the original context type is applied.

The latter of the above 4 ways can be fixed by using restorecon. It's also possible to relabel the entire file system using restorecon -Rv / or by creating the file /.autorelabel in the root /. The next time you reboot the system will discover the /.autorelabel file and the entire file system will be relabeled.
Managing SELinux via Booleans
SELinux Booleans are provided to easily change the behaviour of a rule. A Boolean is a switch that toggles a setting on or off and it allows you to change parts of a SELinux policy rule without any knowledge of policy writing. These changes are applied during runtime and do not require a reboot. 
You can get a list of Booleans using the getsebool -a command and filtering that down using grep:
[root@localhost ~]# getsebool -a | grep httpd
httpd_anon_write --> off
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_connect_ftp --> off
httpd_can_connect_ldap --> off
httpd_can_connect_mythtv --> off
httpd_can_connect_zabbix --> off
...
The semanage boolean -l command provides more detail, it shows the current setting and the default one.
[root@localhost ~]# semanage boolean -l | head
SELinux boolean                State  Default Description

abrt_anon_write                (off  ,  off)  Allow ABRT to modify public files used for public file transfer services.
abrt_handle_event              (off  ,  off)  Determine whether ABRT can run in the abrt_handle_event_t domain to handle ABRT event scripts.
To set a Boolean we use setsebool and to apply the change permanently we add the -P option:
[root@localhost ~]# getsebool -a | grep ftpd
ftpd_anon_write --> off
...

[root@localhost ~]# setsebool ftpd_anon_write on
[root@localhost ~]# semanage boolean -l | grep ftpd_anon
ftpd_anon_write                (on   ,  off)  Determine whether ftpd can modify public files used for public file transfer services. Directories/Files must be labeled public_content_rw_t.

[root@localhost ~]# setsebool -P ftpd_anon_write on
[root@localhost ~]# semanage boolean -l | grep ftpd_anon
ftpd_anon_write                (on   ,   on)  Determine whether ftpd can modify public files used for public file transfer services. Directories/Files must be labeled public_content_rw_t.
Troubleshooting SELinux Policy Violations
SELinux logs everything it is doing, the primary source to get logging information is the audit log which is in /var/log/audit/audit.log. Message are logged with type=AVC, which stands for Access Vector Cache. 
[root@localhost ~]# grep AVC /var/log/audit/audit.log | tail -1
type=AVC msg=audit(1611246770.937:136): avc:  denied  { getattr } for  pid=4178 comm="httpd" path="/test/index.html" dev="dm-0" ino=35157701 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=file permissive=0
The first relevant part in the output is the text acv: denied { getattr }. This means some process tried to read the attributes of a file and it was denied access.
Further down we can see comm="httpd" which means the command that was trying to issue the getattr request was httpd. Next, we see path="test/index.html", which is the file that this process tried to access.
In the last part we see information about the source context and the target context:
scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:default_t:s0

default_t is used for files that do not match any pattern in the SELinux policy. I created /test/index.html in the root and SELinux doesn't know what security context to give to this file, so it assigned default_t.

We also see that Permissive mode is disabled:
permissive=0
The issue here is that the SELinux policy denies access from the httpd_t security context to the default_t security context.
We can solve this issue by setting the correct target security context:
[root@localhost ~]# semanage fcontext -a -t httpd_sys_content_t "/test(/.*)?"
[root@localhost ~]# restorecon -Rv /test
Relabeled /test from unconfined_u:object_r:default_t:s0 to unconfined_u:object_r:httpd_sys_content_t:s0
Relabeled /test/index.html from unconfined_u:object_r:default_t:s0 to unconfined_u:object_r:httpd_sys_content_t:s0
Analyzing SELinux with Sealert
We can use sealert to easier understand SELinux messages in /var/log/audit/audit.log.
First, you need to install sealert: yum install setroubleshoot-server
Once this is installed, issue the journalctl | grep sealert command:
Jan 21 11:32:57 localhost.localdomain setroubleshoot[4395]: SELinux is preventing httpd from getattr access on the file /test/index.html. For complete SELinux messages run: sealert -l e4fc58ab-c1c0-4525-a955-eff9a5570a7c
Jan 21 11:33:00 localhost.localdomain setroubleshoot[4395]: SELinux is preventing httpd from getattr access on the file /test/index.html. For complete SELinux messages run: sealert -l e4fc58ab-c1c0-4525-a955-eff9a5570a7c
Follow the instructions and run saelert -l UUID.
sealert will analyze what's happened and provide some suggestions what you need to do to fix the problem. Each suggestion will have a confidence score and the higher this score the more likely the suggested solution would be applicable. 
[root@localhost ~]# sealert -l e4fc58ab-c1c0-4525-a955-eff9a5570a7c
SELinux is preventing httpd from getattr access on the file /test/index.html.

*****  Plugin catchall_labels (83.8 confidence) suggests   *******************

If you want to allow httpd to have getattr access on the index.html file
Then you need to change the label on /test/index.html
Do
# semanage fcontext -a -t FILE_TYPE '/test/index.html'
...



Network Services - Managing Apache HTTP
Joeri JM Smissaert — Mon, 09 Nov 2020 00:00:00 GMT
{{< image src="/img/redhat-8-logo.png" alt="Red Hat logo" position="center" >}}
{{< image src="/img/apache.png" alt="Apache logo" position="center" >}}
Managing Apache HTTP Services is not part of the current RHCSA exam objectives, but we need minimal knowledge on this topic in order to master the SELinux-related objectives later on. 
The Apache server is provided through different software packages. The basic packages is httpd which contains everything for an operational but basic website. For a complete overview of all the packages use yum search httpd. 
Understanding the httpd Package
Let's examine the httpd package by downloading it using yumdownloader and running a few rpm commands on it:
[root@localhost ~]# yumdownloader httpd
Last metadata expiration check: 0:00:31 ago on Tue 06 Oct 2020 11:05:52 AM EST.
[root@localhost ~]# ls
anaconda-ks.cfg  httpd-2.4.37-21.module_el8.2.0+382+15b0afa8.x86_64.rpm  initial-setup-ks.cfg
[root@localhost ~]# rpm -qpi httpd-2.4.37-21.module_el8.2.0+382+15b0afa8.x86_64.rpm 
Name        : httpd
Version     : 2.4.37
Release     : 21.module_el8.2.0+382+15b0afa8
Architecture: x86_64
Install Date: (not installed)
Group       : System Environment/Daemons
Size        : 5105105
License     : ASL 2.0
Signature   : RSA/SHA256, Mon 08 Jun 2020 05:08:58 PM EDT, Key ID 05b555b38483c65d
Source RPM  : httpd-2.4.37-21.module_el8.2.0+382+15b0afa8.src.rpm
Build Date  : Mon 08 Jun 2020 04:15:29 PM EDT
Build Host  : x86-02.mbox.centos.org
Relocations : (not relocatable)
Packager    : CentOS Buildsys 
Vendor      : CentOS
URL         : https://httpd.apache.org/
Summary     : Apache HTTP Server
Description :
The Apache HTTP Server is a powerful, efficient, and extensible
web server.
[root@localhost ~]#
We can see the package was created by CentOS Buildsys and that it is indeed the Apache HTTP Server package.
Next, let's have a look at the configuration files:
[root@localhost ~]# rpm -qpc httpd-2.4.37-21.module_el8.2.0+382+15b0afa8.x86_64.rpm 
/etc/httpd/conf.d/autoindex.conf
/etc/httpd/conf.d/userdir.conf
/etc/httpd/conf.d/welcome.conf
/etc/httpd/conf.modules.d/00-base.conf
/etc/httpd/conf.modules.d/00-dav.conf
/etc/httpd/conf.modules.d/00-lua.conf
/etc/httpd/conf.modules.d/00-mpm.conf
/etc/httpd/conf.modules.d/00-optional.conf
/etc/httpd/conf.modules.d/00-proxy.conf
/etc/httpd/conf.modules.d/00-systemd.conf
/etc/httpd/conf.modules.d/01-cgi.conf
/etc/httpd/conf/httpd.conf
/etc/httpd/conf/magic
/etc/logrotate.d/httpd
/etc/sysconfig/htcacheclean
[root@localhost ~]#
The main configuration file is /etc/httpd/conf/httpd.conf. The welcome.conf file defines the default home page for your website, until you add content. The magic file defines rules that the server can use to figure out a file's type when the server tries to open it. The /etc/logrotate.d/httpd file defines how log files produced by Apache are rotated.
Most Apache modules put their configuration files into the  /etc/httpd/conf.d directory but some may drop their configuration files into the /etc/httpd/conf.modules.d/ directory. Any file in those directories that ends with the .conf extension is included in the main httpd.conf file and used to configure Apache.
Setting Up a Basic Web Server
Let's install the httpd package and some of the most commonly used additional packages using the yum module install httpd command:
[root@localhost ~]# yum module install httpd
...
...
Installed:
  apr-1.6.3-9.el8.x86_64                                                    apr-util-1.6.1-6.el8.x86_64                                          apr-util-bdb-1.6.1-6.el8.x86_64                                  
  apr-util-openssl-1.6.1-6.el8.x86_64                                       centos-logos-httpd-80.5-2.el8.noarch                                 httpd-2.4.37-21.module_el8.2.0+382+15b0afa8.x86_64               
  httpd-filesystem-2.4.37-21.module_el8.2.0+382+15b0afa8.noarch             httpd-tools-2.4.37-21.module_el8.2.0+382+15b0afa8.x86_64             mod_http2-1.11.3-3.module_el8.2.0+307+4d18d695.x86_64            
  mod_ssl-1:2.4.37-21.module_el8.2.0+382+15b0afa8.x86_64                   

Complete!
Open the main configuration file, /ect/httpd/conf/httpd.conf, and look for the DocumentRoot parameter.
This parameter specifies the default location where the Apache Web Server looks for content to serve. It should be set to DocumentRoot "/var/www/html".
In the directory /var/www/html, create a file with the name index.html and the content Welcome To My Web Server!. Next, start and enable the httpd service and check if the service is up and running. 
[root@localhost ~]# echo "Welcome To My Webserver!" > /var/www/html/index.html
[root@localhost ~]# 
[root@localhost ~]# systemctl enable --now httpd
Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service → /usr/lib/systemd/system/httpd.service.
[root@localhost ~]# 
[root@localhost ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-10-06 11:28:24 EST; 3s ago
     Docs: man:httpd.service(8)
 Main PID: 33293 (httpd)
   Status: "Started, listening on: port 443, port 80"
    Tasks: 213 (limit: 11323)
   Memory: 17.8M
   CGroup: /system.slice/httpd.service
           ├─33293 /usr/sbin/httpd -DFOREGROUND
           ├─33296 /usr/sbin/httpd -DFOREGROUND
           ├─33298 /usr/sbin/httpd -DFOREGROUND
           ├─33299 /usr/sbin/httpd -DFOREGROUND
           └─33301 /usr/sbin/httpd -DFOREGROUND

Oct 06 11:28:24 localhost.localdomain systemd[1]: Starting The Apache HTTP Server...
Oct 06 11:28:24 localhost.localdomain httpd[33293]: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain. Set the 'ServerName' directive globally to >
Oct 06 11:28:24 localhost.localdomain systemd[1]: Started The Apache HTTP Server.
Oct 06 11:28:24 localhost.localdomain httpd[33293]: Server configured, listening on: port 443, port 80
When the httpd service starts, five httpd daemon processes are launched by default to respond to requests for the web server. You can configure more or fewer daemons to be started based on settings in the main configuration file. 
We can verify it's working by making an http request to localhost using curl:
[root@localhost ~]# curl http://localhost
Welcome To My Webserver!
Creating Apache Virtual Hosts
Apache supports the creation of separate websites within a single server. Individual sites are configured in what we refer to as virtual hosts which is just a way to have the content for multiple domain names available from the same Apache server. The content that is served to a web client is based on the (domain) name used to access the server.
For example, if a client got to the server by requesting the name www.example.org, he would be redirected to a virtual host container that has its ServerName parameter set to www.example.org.
Name-based virtual hosting is the most common solution where virtual hosts use different names but the same IP address.
IP-based virtual hosts are less common but is required if the name of a web server must resolve to a unique IP address. This solution requires multiple IP addresses on the same machine.
In this section we'll be setting up name-based virtual hosts. 

If your Apache server is configured for virtual hosts, all sites it's hosting should be handled by virtual hosts. If someone accesses the server via IP address or a name that is not set in a virtual host then the first virtual host is used as the default location to serve up content.
You can create a catch-all entry for those requests by creating a virtual host for _default:80. 

Create a file named example.org.conf in /etc/httpd/conf.d/ using the following template:
80>

    ServerAdmin    webmaster@example.org
    ServerName    example.org
    ServerAlias www.example.org
    DocumentRoot /var/www/html/example.org/

DirectoryIndex index.php index.html index.htm

This example includes the following settings:

The *:80 specification indicates to what address and port this virtual host applies. If your machine has multiple IP addresses, you can replace the * with an IP. The port is optional but should always be used to prevent interference with SSL virtual hosts (which use port 443).
The ServerName and ServerAlias lines tell Apache which names this virtual host should be recognized as. You can either leave out ServerAlias or specify more than one name on the same line, space separated. 
The DocumentRoot specifies where the content for this virtual host is stored.
The DirectoryIndex directive sets the list of files to look for and serve when the web server receives a request.

Create the index.html file inside the DocumentRoot with the following content: Welcome To Example.org
[root@localhost conf.d]# mkdir /var/www/html/example.org
[root@localhost conf.d]# echo "Welcome To Example.org" > /var/www/html/example.org/index.html
Create a second virtual host with different values, e.g.:
[root@localhost conf.d]# cat foobar.com.conf 
80>

    ServerAdmin    webmaster@foobar.com
    ServerName    foobar.com
    ServerAlias    www.foobar.com
    DocumentRoot     /var/www/html/foobar.com/

DirectoryIndex index.php index.html index.htm


[root@localhost conf.d]# mkdir /var/www/html/foobar.com
[root@localhost conf.d]# echo "Welcome to Foobar.com!" > /var/www/html/foobar.com/index.html
[root@localhost conf.d]#
Next we want to make sure that the domains used in our virtual hosts resolve to our local machine and not to the internet. Edit your hosts file and add the domains to the line that starts with the local loopback address:
[root@localhost conf.d]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 foobar.com www.foobar.com example.org www.example.org
After we restarted the httpd service, we can test if our setup is working correctly:
[root@localhost conf.d]# systemctl restart httpd
[root@localhost conf.d]# curl http://foobar.com
Welcome to Foobar.com!
[root@localhost conf.d]# curl http://example.org
Welcome To Example.org
This covered some Apache basics which we will need for testing advanced topics like firewall configuration and SELinux.



Network Services - Configuring SSH
Joeri JM Smissaert — Fri, 02 Oct 2020 00:00:00 GMT
{{< image src="/img/redhat-8-logo.png" alt="Red Hat logo" position="center" >}}
Hardening the SSH Server
SSH is a convenient and important solution to establish remote connections to servers. If your SSH server is visible directly from the internet, you can be sure that sooner or later intruders will try to connect to it, intending to do harm.
Dictionary attacks are common against an SSH server. SSH servers usually offer their services on port 22, and every Linux servers has a root account. Based on this information it's easy for an attacker to try to log in as root by guessing the password if the password has limited complexity and no additional security measures are in place. Sooner or later the intruder will be able to connect.
We can protect ourselves against these kind of attacks by:

Disabling root login.
Disabling password login and using key-based authentication.
Configuring a non default port for SSH to listen on.
Allowing only specific users to log in on SSH.

Limiting Root access
SSH servers have root login enabled by default, which is a big security concern. Disabling root login is easy: Modify the PermitRootLogin parameter in /etc/ssh/sshd_config and reload or restart the service:
# Authentication:

#LoginGraceTime 2m
PermitRootLogin no
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10
Configuring Alternative Ports
Security problems on Linux servers start with a port scan issued by an attacker. There are 65,535 ports that can potentially be listening, and scanning all those ports takes a lot of time so most port scans focus on well known ports only. Port 22 is always among these ports.
To protect against port scans we can configure the SSH server to listen on another port. You can choose a completely random port, as long as the port is not already in use by another service.
# If you want to change the port on a SELinux system, you have to tell
# SELinux about this change.
# semanage port -a -t ssh_port_t -p tcp #PORTNUMBER
#
Port 39860
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

To avoid being locked out of the server after making changes to the SSH listing port, it's a good idea to open two sessions. Use one session to apply the port change and test, use the other sessions to keep your current connection open. Active sessions will not be disconnected after restarting the SSH server (unless the restart fails), so if something is wrong with the configuration and you're not longer able to connect you still have the second session to fix the problem.

Modifying SELinux to Allow for Port Changes
After changing the SSH port you also need to configure SELinux to allow this change. Network ports are labeled with SELinux security labels to prevent services from accessing ports they shouldn't.
Use the semanage port command to change the label on the target port. Before doing so, it's a good idea to check if the port already has a label: semanage port -l, e.g. semanage port -l | grep ssh
If the port doesn't have a label, use the -a option to add a label, if it does have a label use -m to modify the current security label.
semanage port -a -t ssh_port_t -p tcp 39860
semanage port -m -t ssh_port_t -p tcp 443 
Limiting User Access
The AllowUsers option takes a space separated list of usernames that will be allowed to login through SSH. If the user root still needs to be able to log in you'll have to include it as well in the list. 

This option does not appear anywhere in the /etc/ssh/sshd_config file by default.

Another interesting option is MaxAuthTries. It specifies the maximum number of authentication attempts permitted per connection.MaxAuthTries is also useful for analyzing security events, it logs failed login attempts once the number of failures reaches half this value. The higher the number of attempts, the more likely it is an intruder is trying to get in.
SSH writes log information about failed login attempts to the AUTHPRIV syslog facility. This facility is by default configured to write information to /var/log/secure.
Other Useful sshd Options
Apart from security-related options, there are some useful miscellaneous options you can use to streamline performance.
Session Options
On RHEL8, GSSAPIAuthentication option is set to yes by default. This option is only useful in an environment where Kerberos authentication is used. Having this feature on slows down the authentication procedure.
The UseDNS option is also enabled by default and instructs the SSH server to lookup the remote hostname and check with DNS that the hostname maps back to the same IP address (Reverse DNS Lookup). Although this option has some security benefits, it also involves a significant performance penalty. Set this to no if client connections are slow. 

To give an example on Reverse DNS lookups, assume you're connecting from a client with the 8.8.8.8 ip address. The SSH server will lookup the PTR record for the 8.8.8.8.in-addr.arpa domain which would result in dns.google. In turn, this result resolves back to 8.8.8.8.  The reverse DNS database of the Internet is rooted in the .arpa top-level domain.  

$ dig -x 8.8.8.8
;; ANSWER SECTION:
8.8.8.8.in-addr.arpa.    76082    IN    PTR    dns.google.

$ dig dns.google
;; ANSWER SECTION:
dns.google.        824    IN    A    8.8.4.4
dns.google.        824    IN    A    8.8.8.8
The MaxSessions option specifies the maximum number of sessions that can be opened from one IP address simultaneously. You might need to increase this option beyond the default value of 10.
Connection Keepalive Options
The TCPKeepAlive option is used to monitor whether the client is still available.
This option is by default enabled and sends a keepalive probe packet with the ACK flag to the client after a certain amount of time. If a reply is received, the SSH server can assume that the connection is still up and running.
The ClientAliveInterval option sets an interval in seconds after which the server sends a packet to the client if no activity has been detected. The ClientAliveCountMax parameter specifies how many of these should be sent. So if the ClientAliveInterval is set to 30 and the ClientAliveCountMax to 10, inactive connections are kept alive for about 5 minutes.

The equivalent client side options are ServerAliveInterval and ServerAliveCountMax, useful if you cannot change the configuration of the SSH server.

Configuring Key-Based Authentication with Passphrases
By default, password authentication is allowed on RHEL 8 SSH servers. You can disable password authentication and allow public/private key-based authentication only by setting the PasswordAuthentication option to no.
When using key-based authentication you can set a passphrase which makes the key pair stronger. In case an intruder has access to the private key he would also need to know the passphrase before being able to use the key.
Without further configuration the use of passphrases would mean that users have to enter the passphrase every time before a connection can be created, which is inconvenient. To work around this we can cache the passphrase for a session:

Execute the ssh-agent /bin/bash command to start the agent for the current (Bash) shell.
Execute ssh-add to add the passphrase for the current user's private key. The key is now cached.
Connect to the remote server, you'll notice you do not need to enter the passphrase.

Copying and synchronizing files securely over SSH
scp is a program for copying files securely between computers using the SSH protocol.
The basic usage is as follows:  

To copy a local file to a remote host:  scp localfile remote_host:remote_path  
To copy a remote file to a local path: scp remote_host:remote_file localpath  
To copy entire directory trees, add the -r option: scp -r remote_host:path/directory .

Rsync, which stands for “remote sync”, is a remote and local file synchronization tool. It uses an algorithm that minimizes the amount of data copied by only moving the portions of files that have changed. The basic syntax is similar to that of scp: rsync source destination.

rsync -anvzP --progress remote_host:/path/to/directory/ /some/local/path

The -a option is a combination flag, it stands for "archive" and syncs recursively and preserves symbolic links, special and device files, modification times, group, owner, and permissions. You could use -r to only sync recursively instead.
The -n flag is the same as the --dry-run option and allows you to check results before actually running the synchronization. You need the -v flag (verbose) to get the appropriate output to verify.
The -z option can reduce network transfer by adding compression.
The -P flag combines the --progress and --partial options, it gives you a progress bar and allows you to resume interrupted transfers.
Finally, you can use the -A flag to preserve Access Control Lists, and the -X flag to preserve SELinux context labels.

Notice the traling slash / at the end of the first argument in the example command. This is necessary to include the contents of the source path. Without the trailing slash, directory would be created inside /some/local/path.




Introduction to Bash Shell Scripting
Joeri JM Smissaert — Fri, 25 Sep 2020 00:00:00 GMT
{{< image src="/img/redhat-8-logo.png" alt="Red Hat logo" position="center" >}}
{{< image src="/img/bash_logo.png" alt="Bash logo" position="center" style="width:150px">}}
Core Elements
A shell script is a list of sequentially executed commands with optional scripting logic to allow code to be executed under specific conditions only. Starting a script from the parent shell opens a subshell from where the commands in the script are executed. These commands can be interpreted in different ways, to make it clear how they should be interpreted the shebang is used on the first line of the script: #!/bin/bash, which would call and execute the script in a bash subshell. 
The below script asks you for a path and stores the path in the DIR variable, then changes directory to the DIR value and prints the current working directory.
#!/bin/bash
# MyComment

echo Provide a path to a directory:
read DIR
cd $DIR
pwd
exit 0
When you execute this script, notice how your current working directory hasn't changed after the script has executed. This is because the script executes in a subshell of the parent shell from where you invoked the script.
At the end of the above script an exit 0 statement is included. An exit statement tells the parent script whether the scipt was successful, a 0 means it was successfull, while anything else means a problem was encountered. 
A script needs to be executable. The most common way to make a script executable is by applying the execute permission to it. The script can also be executed as an argument to the bash command, e.g. bash myscript.sh. 
You can store a script anywhere you like, but if it's stored outide of the $PATH you need to execute it with a ./ in front: ./myscript.sh.
Variables and Input
Scripts typically aren't a list of sequential commands, they can work with variables and input to be more flexible.
Positional Parameters
When starting a script, an argument can be used. Arguments are anything you put behind the command while starting the script, e.g. useradd lisa where the command is useradd and the argument is lisa. In a script, the first variable is referred to as $1, the second as $2 and so on. 
#!/bin/bash
# Run this script with a few arguments

echo The first argument is $1
echo The 2nd argument is $2
echo The 3rd argument is $3
Run the above script with a few arguments, and it will make sense:
./script 1 2 3 4
You'll notice the 4th argument, being 4 isn't echoed. We can work around that by making the script more flexible using a conditional for loop instead of echoeing each argument one after the other:
#!/bin/bash
# Run this script with a few arguments

echo You have entered $# arguments.

for i in "$@"
  do echo $i
done

exit 0
$# is a counter that shows how many arguments were used when starting the script.
$@ refers to all arguments used when starting the script. 
In the above script, the condition is for i in "$@", which means "for each argument in the list of arguments". I'll cover more on for loops later, but what this script basically does is loop through the list of arguments ($@) and echo each one (do echo $i). 
Variables
Variables are labels that refer to a specific location in memory which contains a specific value. They can be defined statically or dynamically. Variables are defined by using the = sign directly after the uppercase name, followed by the value. You should never use spaces when defining variables:
MYVAR=value, this would be a statically defined variable.
There are two solutions for defining variables dynamically:

Using read in the script to ask the user for input. IT stops the script so input can be processed and stored in a variable:  [joeri@Ryzen7 ~]$ read NAME
joeri
[joeri@Ryzen7 ~]$ echo $NAME
joeri

Using command substitution where you assign the result of a specific command to a variable. For example: TODAY=$(date +%d-%m-%y).
You enclose the command whose result you want to use between parentheses and preceed that with a $ sign.  [joeri@Ryzen7 ~]$ TODAY=$(date +%d-%m-%y)
[joeri@Ryzen7 ~]$ echo $TODAY
31-10-20


Conditional Loops
Conditional loops are executed only if a certain condition is true. I'll cover the most often used conditional loops in this section.
if ... then ... else
This construction is common to evaluate specific conditions and are often used together with the test command. Have a look at the man page of test for a complete overview of all the functionality.
Let's look at an example:
#!/bin/bash
# MyComment

if [ -z $1 ]
then
  echo No value provided
fi
The  -z test command checks if the length of a string is zero (man test). If that is true, then "No value provided" will be echoed to the screen.
The above script will only provide output if you run it without any argument.
Below is another example using multiple test commands:
#!/bin/bash
# Run this script with one argument.
# Find out if the argument is a file or a directory

if [ -f $1 ]
then
  echo "$1 is a file"
elif [ -d $1 ]
then
  echo "$1 is a directory"
else
  echo "Not sure what $1 is...."
fi

exit 0
|| and &&
Instead of writing full if ... then statements we can use logical operators. || is a logical OR and will execute the second part of the statement only if the first part is not true. && is a logical AND, and will execute the second part of the statement only if the first part is true.
"true" is the state where a command exits with a 0. 
[ -z $1 ] && echo no argument provided
ping -c 1 192.168.1.256 || echo node does not exist
For ... do ... done
The for conditional loop provides a solution for processing ranges of data. It always starts with for followed by the condition, then do followed by the commands to be executed when the condition is true, and finally closed with done.
In the below example the COUNTER variable is initialized with a value of 10, if the value is greater than or equal to 0 we substract 1. As long as this condition is true we then echo the value of COUNTER:
#!/bin/bash
#

for (( COUNTER=10; COUNTER>=0; COUNTER--))
do
  echo $COUNTER
done
exit 0
We can also define a range by specifying the first number followed by two dots and closing with the last number in the range:
[joeri@Ryzen7 ~]$ for i in {85..90}; do ping -c 1 192.168.100.$i >/dev/null && echo 192.168.100.$i is UP; done
192.168.100.88 is UP
With for i in each of the numbers in the range is assigned to the variable i. For each of those values the ping -c 1 command is executed, and output is redirected to /dev/null since we don't need it. Based on the exit status of the ping command, exit 0 or true, the part behind the logical operator && is executed.
While and until
The while statement is useful if you want to do something as long as a condition is true. Its counterpart is until which keeps the iteration open as long as the condition is false, or until the condition is true.
The below script initializes the COUNTER value with a value of 0 and while the value is less than 11 we echo the value and increase the value with 1:
#!/bin/bash
#

COUNTER=0

while [ $COUNTER -lt 11 ]; do
  echo The counter is $COUNTER
  (( COUNTER=COUNTER+1 ))
done
Below we echo the value of COUNTER and increase its value with 1 until the value is equal to 11. At that point we break out of the loop:
#!/bin/bash
#

COUNTER=0

until [ $COUNTER = 11 ]; do
  echo The counter is $COUNTER
  (( COUNTER=COUNTER+1 ))
done
Case
The case statement is used to evaluate a number of expected values, you define very specific argument that you expect followed by the command that needs to be executed if that argument was used.  
The generic syntax is case item-to-evaluate in, followed by a list of all possible values that need to be evaluated. Each item is closed with a ). Then follows a list of commands that are executed if the specific argument was used, the commands are closed with a double semicolon, ;;.   
The evaluations in case are performed in order. Then the first match is made, the case statement will not evaluate anything else. Whitin the evaluaten, wildcard-like patterns can be used. For example *), which is a "catchall" statement.
#!/bin/bash

echo -n "Enter the name of a country: "
read COUNTRY

echo -n "The official language of $COUNTRY is "

case $COUNTRY in

  Lithuania)
    echo -n "Lithuanian"
    ;;

  Romania | Moldova)
    echo -n "Romanian"
    ;;

  Italy | "San Marino" | Switzerland | "Vatican City")
    echo -n "Italian"
    ;;

  *)
    echo -n "unknown"
    ;;
esac
Script debugging
If a script does not do what you expect it to do, try starting it as an argument to the bash -x command. This will show you line by line what the script is trying to do and will show specific errors if it does not work. 
[joeri@Ryzen7 ~]$ bash -x lang.sh 
+ echo -n 'Enter the name of a country: '
Enter the name of a country: + read COUNTRY
Germany
+ echo -n 'The official language of Germany is '
The official language of Germany is + case $COUNTRY in
+ echo -n unknown
unknown



Troubleshooting Boot Issues
Joeri JM Smissaert — Fri, 18 Sep 2020 00:00:00 GMT
{{< image src="/img/redhat-8-logo.png" alt="Red Hat logo" position="center" >}}
The RHEL8 Boot Procedure
In order to fix boot issues we need to be able to judge in which phase of the boot procedure the issue occurs so we can apply appropriate means to fix it.
The following steps summarize the boot procedure:

POST - The machine is powered on, the Power-On-Self-Test executes and hardware required to start the system is initialized.
Boot device selection - From UEFI or BIOS, a bootable device is located.
Loading the boot loader - From the bootable device a boot loader is located.
Loading the kernel - The kernel is loaded together with the initramfs. The initramfs contains kernel modules required to boot as well as initials scripts to proceed to the next stage of booting.
Starting /sbin/init - The first process is loaded, /sbin/init, which is a symlink to Systemd. The udev daemon is loaded to take care of further hardware initialization. This all happens from initramfs.
Process initrd.target - The Systemd process executes all units from the initrd.target, preparing a minimal operating environment from where the root file system on disk is mounted onto the /sysroot directory.
Switch to root file system - The system switches to the root file system on disk and loads the Systemd process from disk.
Running the default target - Systemd looks for the default target to execute and runs all of its units.

The below table summarizes where a specific phase is configured and what you can do to troubleshoot if something goes wrong.
{{}}
Phase | Configuration | Fix
-----|----|-----
POST | Hardware Configuration, BIOS, UEFI | Replace Hardware
Boot Device | BIOS/UEFI configuration or boot menu | Replace hardware or use rescue system
Boot Loader | grub2-install and edits to /etc/defaults/grub | GRUB Boot menu, edits to /etc/defaults/grub followed by grub2-mkconfigKernel | Edits to GRUB config and /etc/dracut.conf | GRUB Boot menu, edits to /etc/defaults/grub followed by grub2-mkconfig/sbin/init | Compiled into initramfs | init= kernel boot argument, rd.break kernel boot argument, recreate initramfs
initrd.target | Compiled into initramfs | recreate initramfs
Root file system | Edits to /etc/fstab | Edits to /etc/fstabDefault Target | systemctl set-default | Start rescue.target as a kernel boot argument
{{


}}
Passing Kernel Boot Arguments
The GRUB boot prompt offers a way to stop the boot procedure and pass specific options to the kernel.
When you see the GRUB2 menu, type e to enter a mode where you can edit commands and scroll down to the section that begins with linux ($root)/vmlinuz. This line tells GRUB how to start a kernel and looks similar to this:
linux ($root)/vmlinuz-4.18.0-193.19.1.el8_2.x86_64 root=/dev/mapper/cl-root ro crash kernel-auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet
Additional boot arguments need to be added to the end of this line.
The rhgb and quiet boot options hide boot messages, we can remove these in order to see what's happening when we boot the machine.
Once you made the necessary changes, press CTRL+X to start the kernel. Note that this change is not persistent, to make them persistent we must modify the content of /etc/default/grub and use grub2-mkconfig -o /boot/grub2/grub.cf to apply the change.
Starting a Troubleshooting Target
In the GRUB boot prompt we can use several options to allow us to fix our issue:

rd.break - Stops the boot procedure in the initramfs phase. This option is useful if you don't have the root password.
init=/sbin/bash - A shell will be started immediately after loading the kernel and initrd.target.
systemd.unit=emergency.target - Enters a mode that loads the bare minimum of required Systemd units, it requires a root password.
systemd.unit=rescue.target - Starts more Systemd units to bring up a more complete operational mode. 

Using a Rescue Disk
The default rescue image for RHEL is on the installation disk. When booting from the installation disk you'll see a Troubleshooting menu item which presents you with the following options:

Install RHEL in Basic Graphics Mode - This option reinstalls the machine. You should not use it unless a normal installation does not work and you need basic graphics mode.
Rescue a RHEL System - This options prompts you to press Enter to start the installation, but only loads a rescue system. It does not overwrite the current configuration. The Rescue System will try to find an installed Linux system and mount it on /mnt/sysimage. If a valid installation was found and mounted you can press Enter twice to access the rescue shell. At this point we can switch to the root file system on disk to access all tools we need to repair the system: chroot /mnt/sysimage
Run a Memory Test - If you encounter memory errors this tool allows you to mark bad memory chips so you can boot your machine normally.
Boot from Local Drive - If you cannot boot from GRUB on your usual boot device try this option. It offers a boot loader that will try to load the OS from your hard disk.

Reinstalling Grub Using a Rescue Disk
One of the most common reasons to start a rescue disk is if the GRUB2 boot loader breaks. Once you have access to your machine using the rescue disk, reinstalling GRUB2 is a two step process:

Make sure you switch to the root file system on disk: chroot /mnt/sysroot
Use grub2-install followed by the name of the device on which you want to reinstall GRUB2, i.e. grub2-install /dev/sda

Recreating Initramfs Using a Rescue Disk
You know there is a problem with initramfs when you never see the root file system getting mounted on the root directory and don't see any Systemd unit files being started when analyzing the boot procedure.
To repair the initramfs image after booting into the rescue environment you can use the dracut command. dracut --force overwrites the existing initramfs and creates a new initramfs image for the currently loaded kernel. There is also the /etc/dracut.conf configuration file you can use to include specific options while re-creating initramfs. The dracut configuration itself is dispersed over several locations:

/usr/lib/dracut/dracut.conf.d/ - Contains the system default configuration files
/etc/dracut.conf.d/ - Contains custom dracut configuration files
/etc/dracut.conf - The master configuration file

Recovering from File System Issues
When there is a misconfiguration in the file system mounts the boot procedure may end with the "Give root password for maintenance" message. If a device does not exist or there's an error in the UUID, for example, Systemd waits to see if the device comes back online by itself. When that doesn't happen, the "Give root password for maintenance" message appears.
After entering the root password, issue the journalctl -xb command to see if relevant messages providing information about what is wrong are written to the journal. If the problem is indeed file system oriented we need to make sure the root file system is mountend with read/write rights, analyze what's wrong in /etc/fstab and fix that: mount -o remount,rw /
Resetting the Root Password
When the root password is lost, the only way to reset it is to boot into minimal mode which allows you to login without using a password:

Pass the rd.break boot argument to the kernel
Boot the system
The boot procedure stops after loading initramfs and before mounting the root file system.
Re-mount the root file system on disk to get read/write access to the system image: mount -o remount,rw /sysroot
Make the contents of the /sysroot directory the new root directory: chroot /sysroot
Use the passwd  command to set the new password.
Load the SELinux policy: load_policy -i
Set the correct SELinux contect type to /etc/shadow: chcon -t shadow_t /etc/shadow
Reboot by issuing the exit command twice. Use the new root password at the next boot.


An alternative to applying the SELinux context to /etc/shadow is to create the /.autorelabel file which forces SELinux to restore labels set on the entire file system the next time the system is booted.




Managing Systemd Targets and Working with GRUB2
Joeri JM Smissaert — Fri, 11 Sep 2020 00:00:00 GMT
{{< image src="/img/redhat-8-logo.png" alt="Red Hat logo" position="center" >}}
Managing Systemd Targets
A Systemd target is a group of units belonging together, some of these targets can be used to define the state a system should boot in. These targets can be isolated and have the AllowIsolate property in their [Unit] section.
Four targets can be used to boot into:

emergency.target : A minimal number of units are started.
rescue.target : A fully operation Linux system without nonessential services.
multi-user.target : The default target commonly used on servers, starts everything needed for full system functionality.
graphical.target : Starts all units needed for full system functionality as well as a graphical interface.

A target configuration consists of two parts, the target unit file and the "wants" directory that contains references to all unit files that need to be loaded when entering that specific target. They can also have other targets as dependencies, specified in the target unit file.
[root@server1 ~]# systemctl cat multi-user.target 
# /usr/lib/systemd/system/multi-user.target
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Multi-User System
Documentation=man:systemd.special(7)
Requires=basic.target
Conflicts=rescue.service rescue.target
After=basic.target rescue.service rescue.target
AllowIsolate=yes
The target unit doesn't contain much, it defines what it requires and which services and targets it can't coexist with. The After statement in the [Unit] sections also defines load ordering. It does not contain any information about units that it "wants".
Understanding Wants
Wants define which units should start when booting or starting a specific target.
Wants are created when enabling units using systemd enable, this happens by creating a symbolic link in the /etc/systemd/system directory. This directory contains a subdirectory for every target, which in turn contains "wants" as symbolic links to specific services that should be started:
[root@server1 ~]# ls -l /etc/systemd/system/multi-user.target.wants/
total 0
lrwxrwxrwx. 1 root root 35 Sep 26 17:46 atd.service -> /usr/lib/systemd/system/atd.service
lrwxrwxrwx. 1 root root 38 Sep 26 17:44 auditd.service -> /usr/lib/systemd/system/auditd.service
lrwxrwxrwx. 1 root root 44 Sep 26 17:46 avahi-daemon.service -> /usr/lib/systemd/system/avahi-daemon.service
lrwxrwxrwx. 1 root root 39 Sep 26 17:45 chronyd.service -> /usr/lib/systemd/system/chronyd.service
lrwxrwxrwx. 1 root root 37 Sep 26 17:44 crond.service -> /usr/lib/systemd/system/crond.service
...
The [Install] section in a service unit file specifies the target it is "wanted" by. Enabling the service creates a symbolic link in that targets' "wants" directory, making sure it starts when that target is booted into or started.
[root@server1 ~]# systemctl cat httpd.service 
...
[Install]
WantedBy=multi-user.target

[root@server1 ~]# systemctl enable httpd
Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service → /usr/lib/systemd/system/httpd.service.
Isolating Targets
To get a list of all targets that are currently loaded, we can use the systemctl --type=target command. This shows all currently active targets. The systemctl --type=target --all command also shows inactivate targets.
[root@server1 ~]# systemctl --type=target
UNIT                   LOAD   ACTIVE SUB    DESCRIPTION                
basic.target           loaded active active Basic System               
cryptsetup.target      loaded active active Local Encrypted Volumes    
getty.target           loaded active active Login Prompts              
graphical.target       loaded active active Graphical Interface        
local-fs-pre.target    loaded active active Local File Systems (Pre)   
local-fs.target        loaded active active Local File Systems         
multi-user.target      loaded active active Multi-User System          
network-online.target  loaded active active Network is Online          
network.target         loaded active active Network                    
nfs-client.target      loaded active active NFS client services        
nss-user-lookup.target loaded active active User and Group Name Lookups
paths.target           loaded active active Paths                      
remote-fs-pre.target   loaded active active Remote File Systems (Pre)  
remote-fs.target       loaded active active Remote File Systems        
rpc_pipefs.target      loaded active active rpc_pipefs.target          
rpcbind.target         loaded active active RPC Port Mapper            
slices.target          loaded active active Slices                     
sockets.target         loaded active active Sockets                    
sound.target           loaded active active Sound Card                 
sshd-keygen.target     loaded active active sshd-keygen.target         
swap.target            loaded active active Swap                       
sysinit.target         loaded active active System Initialization      
timers.target          loaded active active Timers                     

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

23 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
Some of these targets can be isolated, they can be started to define the state of the machine and these are also the targets that can be set as the default target. They roughly correspond to the following System V runlevels:
{{}}
Target | Runlevel
-----|----
poweroff.target | runlevel 0
rescue.target | runlevel 1
multi-user.target | runlevel 3
graphical.target | runlevel 5
reboot.target | runlevel 6
{{
}}
As mentioned earlier, targets that can be isolated have the AllowIsolate property in their [Unit] section:
[root@server1 system]# grep Isolate *.target
anaconda.target:AllowIsolate=yes
ctrl-alt-del.target:AllowIsolate=yes
default.target:AllowIsolate=yes
emergency.target:AllowIsolate=yes
exit.target:AllowIsolate=yes
graphical.target:AllowIsolate=yes
halt.target:AllowIsolate=yes
initrd-switch-root.target:AllowIsolate=yes
initrd.target:AllowIsolate=yes
kexec.target:AllowIsolate=yes
multi-user.target:AllowIsolate=yes
poweroff.target:AllowIsolate=yes
reboot.target:AllowIsolate=yes
rescue.target:AllowIsolate=yes
runlevel0.target:AllowIsolate=yes
runlevel1.target:AllowIsolate=yes
runlevel2.target:AllowIsolate=yes
runlevel3.target:AllowIsolate=yes
runlevel4.target:AllowIsolate=yes
runlevel5.target:AllowIsolate=yes
runlevel6.target:AllowIsolate=yes
system-update.target:AllowIsolate=yes
To switch the current state of your machine to either one of these targets, use the systemctl isolate command:
systemctl isolate rescue.target
systemctl isolate reboot.target
We can set a default ttarget using the systemctl set-default command, or check the current default target using the systemctl get-default command. Notice how the existing symlink is removed and a new one is created for default.target:
[root@server1 system]# systemctl get-default 
graphical.target

[root@server1 system]# systemctl set-default multi-user.target 
Removed /etc/systemd/system/default.target.
Created symlink /etc/systemd/system/default.target → /usr/lib/systemd/system/multi-user.target.
Working with GRUB2
The GRUB2 bootloader makes sure we can boot Linux, it's installed in the boot sector of the hard drive and loads a Linux kernel and initramfs.
The initramfs contains a mini file system, mounted during boot, from where kernel modules load that are needed during the rest of the boot process, e.g. LVM modules.
We apply changes to GRUB2 by editing the /etc/default/grub file and we pass boot arguments to the kernel by editing the GRUB_CMDLINE_LINUX line:
[root@server1 system]# cat /etc/default/grub 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
The GRUB_TIMEOUT parameter defines how long GRUB2 waits before proceeding with the boot procedure. During this time you can press e to make changes to the configuration, just as you would by editing the /etc/default/grub file. 
Removing the rhgb and quiet boot options would allow you to see the output of the boot procedure on screen.
After making changes to /etc/default/grub the relevant GRUB file on the /boot partition needs to be regenerated. On a BIOS system this file is located in /boot/grub2/grub.cfg, while on a UEFI system the file is located in /boot/efi/EFI/redhat/grub.cfg. To regenerate these files, we issue the grub2-mkconfig command and redirect its output to either one of these files:
grub2-mkconfig -o /boot/grub2/grub.cfg
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg



Basic Kernel Management
Joeri JM Smissaert — Tue, 01 Sep 2020 00:00:00 GMT
{{< image src="/img/redhat-8-logo.png" alt="Red Hat logo" position="center" >}}
The Role of the Linux Kernel
The Linux kernel is the layer between the user who works with Linux from a shell environment and the available hardware. It manages the I/O instructions received from software and translates it to CPU instructions. The kernel also handles essential operating system tasks like the scheduler to make sure that any processes started on the OS are handled by the CPU.
OS tasks that are handled by the kernel are implemented by using different kernel threads. You can easily indentify them with a command like ps aux, the kernel threads are listed between square brackets:
[root@server1 ~]# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.5 180072 10364 ?        Ss   09:50   0:01 /usr/lib/syst
root           2  0.0  0.0      0     0 ?        S    09:50   0:00 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   09:50   0:00 [rcu_gp]
root           4  0.0  0.0      0     0 ?        I<   09:50   0:00 [rcu_par_gp]
root           6  0.0  0.0      0     0 ?        I<   09:50   0:00 [kworker/0:0H
root           8  0.0  0.0      0     0 ?        I<   09:50   0:00 [mm_percpu_wq
root           9  0.0  0.0      0     0 ?        S    09:50   0:00 [ksoftirqd/0]
root          10  0.0  0.0      0     0 ?        I    09:50   0:00 [rcu_sched]
The kernel also handles hardware initialization, making sure hardware can be used. To do so, drivers must be loaded and since the kernel is modular these drivers are loaded as kernel modules.
Hardware manufacturers do not always provide open source drivers, in this case the alternative would be to use closed source drivers. This is not always ideal, a badly functioning driver can crash the entire kernel. If this happens on an open source driver, the Linux community would jump in to debug and fix the problem which cannot be done on a closed source driver. A closed source or proprietary driver may however provide additional functionality not available in the open source equivalent. A kernel that is using closed source drivers is known as a tainted kernel.
Analyzing What the Kernel is Doing
A few different tools are provided by the Linux operating system to help check what the kernel is doing:

dmesg
The /proc pseudo file system
The uname and hostnamectl utility

When you require detailed information about kernel activity, you can use the dmesg command. This prints the content of the kernel ring buffer, an area of memory where the kernel keeps the recent log messages. Each entry in the output starts with a time indicator that shows the specific second the event was logged, relative to the start of the kernel.
An alternative to dmesg is journalctl - -dmesg or journalctl -k. These commands show a clock time indicator.
[root@server1 ~]# dmesg | head
[    0.000000] Linux version 4.18.0-193.19.1.el8_2.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP Mon Sep 14 14:37:00 UTC 2020
[    0.000000] Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-193.19.1.el8_2.x86_64 root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet
[    0.000000] x86/fpu: x87 FPU will use FXSAVE
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007ffdcfff] usable
[    0.000000] BIOS-e820: [mem 0x000000007ffdd000-0x000000007fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved


[root@server1 ~]# journalctl -k | head
-- Logs begin at Fri 2020-10-02 09:50:11 +04, end at Fri 2020-10-02 10:53:20 +04. --
Oct 02 09:50:11 server1.example.local kernel: Linux version 4.18.0-193.19.1.el8_2.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP Mon Sep 14 14:37:00 UTC 2020
Oct 02 09:50:11 server1.example.local kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-193.19.1.el8_2.x86_64 root=/dev/mapper/cl-root ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet
Oct 02 09:50:11 server1.example.local kernel: x86/fpu: x87 FPU will use FXSAVE
Oct 02 09:50:11 server1.example.local kernel: BIOS-provided physical RAM map:
Oct 02 09:50:11 server1.example.local kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Oct 02 09:50:11 server1.example.local kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
Oct 02 09:50:11 server1.example.local kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
Oct 02 09:50:11 server1.example.local kernel: BIOS-e820: [mem 0x0000000000100000-0x000000007ffdcfff] usable
Oct 02 09:50:11 server1.example.local kernel: BIOS-e820: [mem 0x000000007ffdd000-0x000000007fffffff] reserved
Many of the performance related commands or tools we use grab their information from the /proc file system. It contains detailed status information about what is happening on the machine. The /proc directory contains Process ID subdirectories which contain information about the particular process. The directory also contains status files, i.e. /proc/partitions or /proc/meminfo:
[root@server1 ~]# cat /proc/1146/status
Name:    kvdo0:cpuQ0
Umask:    0000
State:    S (sleeping)
Tgid:    1146
Ngid:    0
Pid:    1146
PPid:    2
...

[root@server1 ~]# cat /proc/partitions 
major minor  #blocks  name

  11        0    8038400 sr0
   8       64    5242880 sde
   8       48    5242880 sdd
   8        0   26214400 sda
   8        1    1048576 sda1
   8        2   25164800 sda2
   8       32    5242880 sdc
   8       16    5242880 sdb
   8       17     102400 sdb1
   8       18     921600 sdb2
   8       19    2097152 sdb3
   8       20    2120687 sdb4
...



[root@server1 ~]# cat /proc/meminfo 
MemTotal:        1870616 kB
MemFree:           87464 kB
MemAvailable:     184724 kB
...
We can change kernel performance parameters during run time by writing values to the /proc/sys pseudo file system. You can apply the changes permanently by writing the parameters to /etc/sysctl.conf. To see what parameters are currently in use, issue the systctl -a command.
[root@server1 ~]# sysctl -a | grep ip_forward
net.ipv4.ip_forward = 0

[root@server1 ~]# echo "1" > /proc/sys/net/ipv4/ip_forward
[root@server1 ~]# sysctl -a | grep ip_forward
net.ipv4.ip_forward = 1


[root@server1 ~]# echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf
[root@server1 ~]# reboot


[root@server1 ~]# sysctl -a | grep ip_forward
net.ipv4.ip_forward = 1
Another useful command would be uname and hostnamectl, it gives different kinds of information about the OS:
[root@server1 ~]# uname -a
Linux server1.example.local 4.18.0-193.19.1.el8_2.x86_64 #1 SMP Mon Sep 14 14:37:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[root@server1 ~]# uname -r
4.18.0-193.19.1.el8_2.x86_64

[root@server1 ~]# hostnamectl status
   Static hostname: server1.example.local
         Icon name: computer-vm
           Chassis: vm
        Machine ID: e40db12b26ec4e9bb6a6f295f6d4d83e
           Boot ID: 5441210211394c5098724e9b89426cb2
    Virtualization: kvm
  Operating System: CentOS Linux 8 (Core)
       CPE OS Name: cpe:/o:centos:centos:8
            Kernel: Linux 4.18.0-193.19.1.el8_2.x86_64
      Architecture: x86-64
Lastly, you can cat the distribution release verion:
[root@server1 ~]# cat /etc/redhat-release 
CentOS Linux release 8.2.2004 (Core)
Working with Kernel Modules
Since the release of Linux kernel 2.0 kernels are no longer compiled but modular. A modular kernel consists of a relatively small kernel core and provides driver support through modules that are loaded when they are required. Modules implement specific kernel functionality, they are not limited to loading hardware drivers alone. For example, file system support is also loaded as kernel modules.
Understanding Hardware Initialization
The loading of drivers is an automated process:

The kernel probes available hardware during boot.
When a hardware component is detected, the systemd-udevd process loads the appropriate driver and makes the device available.
systemd-udevd reads the rules in /usr/lib/udev/rules.d/. These are system-provided rules that should not be modified.
systemd-udevd reads custom rules from the /etc/udev/rules.d directory, if available.
Required kernel modules have been loaded and the status of associated hardware is written to the sysfs file system on /sys. This pseudo file system tracks hardware-related settings.

The systemd-udevd process continuously monitors for plugging and unplugging of hardware devices. You can see this in action when plugging/unplugging an usb or other block device while the udevadm monitor command is running:
[root@server1 ~]# udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[7080.543250] change   /devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sr0 (block)
UDEV  [7080.558849] change   /devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sr0 (block)
KERNEL[7080.578292] change   /devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sr0 (block)
UDEV  [7080.746283] change   /devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sr0 (block)
Managing Kernel Modules
Although loading of drivers happens automatically when they are required, there might be occasions where you need to manually load the appropriate kernel module.
To list all currently used kernel modules we use the lsmod command:
[root@server1 ~]# lsmod | head
Module                  Size  Used by
binfmt_misc            20480  1
nls_utf8               16384  1
isofs                  45056  1
fuse                  131072  3
uinput                 20480  1
xt_CHECKSUM            16384  1
ipt_MASQUERADE         16384  3
xt_conntrack           16384  1
ipt_REJECT             16384  2
modinfo provides more information about a specific kernel module, including two interesting sections: the alias and parms.
The alias refers to an alternative name that can be used to address the module and, the parms section refer to parameters that can be set while loading the module.
[root@server1 ~]# modinfo e1000
filename:       /lib/modules/4.18.0-193.19.1.el8_2.x86_64/kernel/drivers/net/ethernet/intel/e1000/e1000.ko.xz
version:        7.3.21-k8-NAPI
license:        GPL
description:    Intel(R) PRO/1000 Network Driver
author:         Intel Corporation, <linux.nics@intel.com>
rhelversion:    8.2
srcversion:     9DFB28D9833DABBB7757EDD
alias:          pci:v00008086d00002E6Esv*sd*bc*sc*i*
...
depends:        
intree:         Y
name:           e1000
vermagic:       4.18.0-193.19.1.el8_2.x86_64 SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         CentOS Linux kernel signing key
sig_key:        4C:02:86:8D:9E:A5:E0:4D:A9:C5:DF:8B:D7:28:EA:05:AF:C6:2A:6D
sig_hashalgo:   sha256
signature:      65:B3:87:34:C5:6F:E5:26:A7:41:90:2C:BB:20:04:54:6E:93:44:2A:
        86:73:D7:FF:FD:12:D3:17:74:EB:4B:9B:9C:FB:19:3F:D8:6A:16:10:
        0D:72:69:CA:63:B2:2E:63:A9:B4:84:94:0D:4B:C4:94:FC:E6:48:CC:
        95:DB:99:65:BC:6F:57:1C:F2:C5:CF:F0:BE:F2:8B:63:11:8F:43:C1:
        8C:1C:D3:03:6B:BC:76:0E:18:06:76:F1:C1:CF:72:84:04:92:07:A7:
        C4:59:4B:7B:72:86:CD:EB:A8:C5:EF:D9:39:FD:B0:38:1A:E3:49:18:
        04:88:39:8D:B9:98:D3:5E:EA:0C:CA:B7:44:51:64:F8:7F:CA:01:75:
        9A:48:DD:E9:2E:E1:38:60:C6:33:37:1A:81:79:B1:22:63:16:5B:42:
        DF:E2:08:9B:B4:47:47:9E:9A:69:5D:62:E9:9E:72:A3:7D:D0:E0:B0:
        51:24:EA:AD:B1:0B:08:67:63:89:17:19:9A:DF:13:82:FB:C2:DA:32:
        97:AA:07:C4:75:A5:6A:A1:E4:AF:D3:64:04:45:24:3F:40:81:21:12:
        99:11:54:2C:04:0C:86:98:56:79:C9:34:EC:B9:96:4F:52:BE:A4:CC:
        0A:3D:0F:78:5B:0E:1A:E3:7A:57:45:FA:B3:80:EF:B0:2E:75:8F:8B:
        FE:71:A1:74:63:DC:B2:7E:29:AD:87:4B:6E:AF:66:F7:81:34:1E:0B:
        7D:02:71:93:20:01:A7:9B:08:5F:AD:8C:EA:F5:E4:1E:4A:D1:AF:90:
        CE:23:9A:65:5B:F7:DE:94:3C:DF:6F:5C:15:51:62:D1:64:05:B3:8A:
        9A:F4:83:3C:C4:31:E4:EE:A5:6C:0D:56:96:DC:F1:00:53:91:78:BD:
        D4:20:03:A1:59:07:58:16:B0:8D:7B:19:E6:6A:A3:31:81:7E:31:ED:
        77:66:58:B0:F5:68:4E:A0:FA:5C:8B:56:40:4A:BB:77:E3:E3:13:62:
        1B:E5:5C:13
parm:           TxDescriptors:Number of transmit descriptors (array of int)
parm:           RxDescriptors:Number of receive descriptors (array of int)
parm:           Speed:Speed setting (array of int)
parm:           Duplex:Duplex setting (array of int)
parm:           AutoNeg:Advertised auto-negotiation setting (array of int)
parm:           FlowControl:Flow Control setting (array of int)
parm:           XsumRX:Disable or enable Receive Checksum offload (array of int)
parm:           TxIntDelay:Transmit Interrupt Delay (array of int)
parm:           TxAbsIntDelay:Transmit Absolute Interrupt Delay (array of int)
parm:           RxIntDelay:Receive Interrupt Delay (array of int)
parm:           RxAbsIntDelay:Receive Absolute Interrupt Delay (array of int)
parm:           InterruptThrottleRate:Interrupt Throttling Rate (array of int)
parm:           SmartPowerDownEnable:Enable PHY smart power down (array of int)
parm:           copybreak:Maximum size of packet that is copied to a new buffer on receive (uint)
parm:           debug:Debug level (0=none,...,16=all) (int)
To manually load and unload modules we use the modprobe and modprobe -r commands.
The modprobe command automatically loads any dependencies. 
Checking Driver Availability for Hardware Devices
To check if a particular device is supported and thus has a module loaded you can use the lspci -k command. 
If there are any devices for which no kernel module was loaded you're likely dealing with an unsupported device.
[root@server1 ~]# lspci -k
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
    Subsystem: Red Hat, Inc. QEMU Virtual Machine
00:01.0 VGA compatible controller: Red Hat, Inc. Virtio GPU (rev 01)
    Subsystem: Red Hat, Inc. Device 1100
    Kernel driver in use: virtio-pci
...
Managing Kernel Module Parameters
You may want to load kernel modules with specific parameters you've discovered using the modinfo command. To do so, specify the name of the parameter and its value in the modprobe command:
[root@server1 ~]# modprobe cdrom debug=1
[root@server1 ~]#
To make this persistent, you can add an entry to /etc/modprobe.conf or create a file in the /etc/modprobe.d/ directory where the name of the file matches the module name and the content specifies the parameters you want to set:
[root@server1 modprobe.d]# pwd
/etc/modprobe.d
[root@server1 modprobe.d]# cat cdrom.conf 
options cdrom debug=1
Upgrading the Linux Kernel
When upgrading the Linux kernel a new version of the kernel is installed next to the current version and will be used by default.
The kernel files for the last four kernels installed will be kept in /boot. The GRUB2 boot loader automatically picks up all kernels found in this directory, allowing you to select an older kernel at boot time in case the newly installed kernel doesn't boot correctly.
To install a new version of the kernel, issue the yum upgrade kernel or yum install kernel command.



Advanced Storage: Virtual Data Optimizer
Joeri JM Smissaert — Thu, 20 Aug 2020 00:00:00 GMT
{{< image src="/img/redhat-8-logo.png" alt="Red Hat logo" position="center" >}}
Virtual Data Optimizer is a storage solution developed to reduce disk space usage on block devices by applying deduplication features. VDO creates volumes on top of any existing block device from where you either create an XFS file system, or use the volume as a Physical Volume in an LVM setup.
VDO uses three common technologies:

Zero-block elimination to filter out data blocks that contain only zeros.
Deduplication of redundant data blocks.
Compression when the kvdo module compresses data blocks.

Typical usage cases for VDO are host platforms for containers and virtual machines or cloud block storage. 
Commonly, a logical size of up to 10 times the physical size is used for these types of environments. 
Setting up VDO
To use VDO the underlying block devices must have a minimal size of 4GiB and the vdo and kmod-kvdo packages must be installed. 
We create the VDO device using the vdo create command, specify a name using the --name= option, and we can specify the logical size using the --vdoLogicalSize= option. e.g. vdo create --name=myvdo1 --vdoLogicalSize=1T /dev/sdb
Once the device is created, we can put an XFS file system on top of it:
mkfs.xfs -K /dev/mapper/myvdo1
The -K option prevents unused blocks from being discarded immediately, making the command much faster.
At this point we issue the udevadm settle command to ensure device nodes have been created succesfully.
To persistently mount the VDO file system using the /etc/fstab file we must include the following mount options:
x-systemd.requires=vdo.service,discard
This makes sure the vdo service is loaded before trying to mount the file system. 
An alternative method to persistently mount the VDO file system is to use the example systemd mount unit found in /usr/share/doc/vdo/examples/systemd.
Copy it to /etc/systemc/system/mountpointname.mount and edit the following lines:
name = 
What = 
Where =
The Unit file name must correspond to the name, What and Where values.
Make sure to enable and start the moutn at boot: systemctl enable --now mountpointname.mount
Example
[root@server1 ~]# vdo create --name=vdo1 --device=/dev/sdb --vdoLogicalSize=1T
Creating VDO vdo1
      The VDO volume can address 2 GB in 1 data slab.
      It can grow to address at most 16 TB of physical storage in 8192 slabs.
      If a larger maximum size might be needed, use bigger slabs.
Starting VDO vdo1
Starting compression on VDO vdo1
VDO instance 0 volume is ready at /dev/mapper/vdo1


[root@server1 ~]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0   25G  0 disk 
├─sda1        8:1    0    1G  0 part /boot
└─sda2        8:2    0   24G  0 part 
  ├─cl-root 253:0    0   22G  0 lvm  /
  └─cl-swap 253:1    0  2.1G  0 lvm  [SWAP]
sdb           8:16   0    5G  0 disk 
└─vdo1      253:2    0    1T  0 vdo  
sdc           8:32   0    5G  0 disk 

[root@server1 ~]# mkfs.xfs -K /dev/mapper/vdo1 
meta-data=/dev/mapper/vdo1       isize=512    agcount=4, agsize=67108864 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=268435456, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=131072, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

[root@server1 ~]# udevadm settle

[root@server1 ~]# cp /usr/share/doc/vdo/examples/systemd/VDO.mount.example /etc/systemd/system/vdo1.mount
[root@server1 ~]# vim /etc/systemd/system/vdo1.mount 
....

[root@server1 ~]# cat /etc/systemd/system/vdo1.mount 
[Unit]
Description = Mount filesystem that lives on VDO
name = vdo1.mount
Requires = vdo.service systemd-remount-fs.service
After = multi-user.target
Conflicts = umount.target

[Mount]
What = /dev/mapper/vdo1
Where = /vdo1
Type = xfs
Options = discard

[Install]
WantedBy = multi-user.target


[root@server1 ~]# systemctl enable --now vdo1.mount

[root@server1 ~]# vdostats --human-readable 
Device                    Size      Used Available Use% Space saving%
/dev/mapper/vdo1          5.0G      3.0G      2.0G  60%           99%

[root@server1 ~]# df -h /vdo1/
Filesystem        Size  Used Avail Use% Mounted on
/dev/mapper/vdo1  1.0T  7.2G 1017G   1% /vdo1

[root@server1 ~]# reboot



Advanced Storage: Configuring Stratis
Joeri JM Smissaert — Mon, 17 Aug 2020 00:00:00 GMT
{{< image src="/img/redhat-8-logo.png" alt="Red Hat logo" position="center" >}}
Stratis, created as an answer to Btrfs and ZFS by Red Hat, is a volume-managing file system that introduces advanced storage features like:

Thin-provisioning: The file system presents itself to users as much bigger than it really is. Useful in virtualized environments.
Snapshots: Allows users to backup the current state of the file system and makes it easy to revert to a previous state.
Cache tier: A Ceph storage feature that ensures data is stored physically closer to the Ceph client, making data access faster.
Programmatic API: Storage can be configured and modified through API access, particularly useful in cloud environments.
Monitoring and repair: Stratis has built-in features to monitor and repair the file system, compared to traditional file systems which would rely on tools like fsck. 

Stratis Architecture
The lowest layer in the Stratis architecture is the pool, which is comparable to an LVM volume group. The pool represents all available storage and consists of one or more storage devices (referred to as blockdev). These block devices can be of any type, including LVM devices but not partitions, but cannot be thin-provisioned as Stratis creates volumes that are thing provisioned themselves. Stratis creates a /stratis/poolname directory for each pool, this directory contains links to devices that represent the file systems in the pool. 
From the Stratis pool file systems are created which live in a volume on top of the pool. A pool can contain one or more file systems. Stratis only works with XFS file systems and these are integrated within the Stratis volume: You should not reformat or reconfigure XFS file systems that are managed by Stratis. 
The file systems are thin-provisioned, they don't have a fixed size and grow automatically as more data is added to the file system.
Creating and Mounting Stratis Storage
To create Stratis storage, we need to create a pool from a block device and add a file system on top of the pool. Block devices need to be 1GiB at a minimum. Note that a Stratis file systems occupies a minimum of 527MiB even if no data has been added. 
Let's make sure we have the stratis-cli and stratisd packages installed, then start and enable the stratisd daemon:
[root@server1 ~]# yum install stratis-cli stratisd
...
[root@server1 ~]# systemctl enable --now stratisd
[root@server1 ~]# systemctl status stratisd
● stratisd.service - A daemon that manages a pool of block devices to create flexible file systems
   Loaded: loaded (/usr/lib/systemd/system/stratisd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-08-19 12:25:50 EDT; 4s ago
We create a pool from one of the available block devices. Make sure the block device does not contain any file system (use blkid -p /dev/sdx) or partition table, if so we wipe them with the wipefs command, e.g wipefs --all /dev/sdx.
[root@server1 ~]# stratis pool create mypool1 /dev/sda
[root@server1 ~]# stratis pool list
Name     Total Physical Size  Total Physical Used
mypool1               10 GiB               52 MiB
Next, we create the myfs1 file system on top of the mypool1 pool:
[root@server1 ~]# stratis fs create mypool1 myfs1
[root@server1 ~]# stratis fs list
Pool Name  Name   Used     Created            Device                  UUID                            
mypool1    myfs1  545 MiB  Aug 19 2020 12:28  /stratis/mypool1/myfs1  ffdbb3a131f6421c990f69aa8d87c6aa
[root@server1 ~]#
To peristently mount a Stratis file system, the UUID must be used in the /etc/fstab file and the mount option x-systemd.requires=stratisd.service must be specified to ensure that Systemd waits to activate this device until the stratisd service is loaded:
[root@server1 ~]# blkid -p /stratis/mypool1/myfs1
/stratis/mypool1/myfs1: UUID="ffdbb3a1-31f6-421c-990f-69aa8d87c6aa" TYPE="xfs" USAGE="filesystem"

[root@server1 ~]# mkdir /mnt/myfs1
[root@server1 ~]# vim /etc/fstab
...
UUID=ffdbb3a1-31f6-421c-990f-69aa8d87c6aa       /mnt/myfs1      xfs     defaults,x-systemd.requires=stratisd.service    0 0
...
[root@server1 ~]# 
[root@server1 ~]# mount -a
[root@server1 ~]# reboot
Managing Stratis
Traditional Linux tools cannot handle thin-provisioned volumes, we need to use the Stratis specific tools:

stratis blockdev: Shows information about all block devices.
stratis pool: Shows information about Stratis pools.
stratis fs: Shows information about file systems.

You can use tab-completion on the above commands to reveal specific options. 
Expanding and Renaming a Pool and File System
We can add a block device to a pool to expand the storage capacity of the pool using the stratis pool add-data poolname blockdevice command:
[root@server1 ~]# stratis pool list
Name     Total Physical Size  Total Physical Used
mypool1               10 GiB              597 MiB

[root@server1 ~]# stratis pool add-data mypool1 /dev/sdb

[root@server1 ~]# stratis pool list
Name     Total Physical Size  Total Physical Used
mypool1               15 GiB              601 MiB
Destroying a Pool and File System
To destroy a pool and file system, we need to unmount the file system first. Then use the stratis fs destroy poolname fsname command, followed by the stratis pool destroy poolname command:
[root@server1 ~]# stratis fs list
Pool Name  Name   Used     Created            Device                  UUID                            
mypool1    myfs1  545 MiB  Aug 19 2020 12:28  /stratis/mypool1/myfs1  ffdbb3a131f6421c990f69aa8d87c6aa

[root@server1 ~]# umount /stratis/mypool1/myfs1
[root@server1 ~]# stratis fs destroy mypool1 myfs1

[root@server1 ~]# stratis fs list
Pool Name  Name  Used  Created  Device  UUID

[root@server1 ~]# stratis pool list
Name     Total Physical Size  Total Physical Used
mypool1               15 GiB               56 MiB

[root@server1 ~]# stratis pool destroy mypool1 

[root@server1 ~]# stratis pool list
Name  Total Physical Size  Total Physical Used
[root@server1 ~]#
Creating and Accessing a Stratis Snapshot
In Stratis, a snapshot is a regular Stratis file system created as a copy of another Stratis file system. The snapshot initially contains the same file content as the original file system, but can change as the snapshot is modified. Whatever changes you make to the snapshot will not be reflected in the original file system. 
To create a Stratis snapshot, use stratis fs snapshot poolname fsname snapshotname.
To access the snapshot, mount it as a regular file system from the /stratis/my-pool/ directory: mount /stratis/poolname/snapshotname mount-point
[root@server1 ~]# stratis pool create mypool1 /dev/sda
[root@server1 ~]# stratis pool add-data mypool1 /dev/sdb
[root@server1 ~]# stratis pool list
Name     Total Physical Size  Total Physical Used
mypool1               15 GiB               56 MiB

[root@server1 ~]# stratis fs create mypool1 myfs1
[root@server1 ~]# stratis fs list
Pool Name  Name   Used     Created            Device                  UUID                            
mypool1    myfs1  545 MiB  Aug 19 2020 13:16  /stratis/mypool1/myfs1  d9e0c47f26e44e0b8990a6aa7546d0f7

[root@server1 ~]# stratis fs snapshot mypool1 myfs1 myfs1snapshot
[root@server1 ~]# stratis fs list
Pool Name  Name           Used     Created            Device                          UUID                            
mypool1    myfs1          545 MiB  Aug 19 2020 13:16  /stratis/mypool1/myfs1          d9e0c47f26e44e0b8990a6aa7546d0f7
mypool1    myfs1snapshot  545 MiB  Aug 19 2020 13:17  /stratis/mypool1/myfs1snapshot  b2fb662124a4424c9d21429012fcfdc4

[root@server1 ~]# mkdir -p /mnt/myfs1snapshot
[root@server1 ~]# mount /stratis/mypool1/myfs1snapshot /mnt/myfs1snapshot/
[root@server1 ~]# umount /mnt/myfs1snapshot
[root@server1 ~]# mount /stratis/mypool1/myfs1 /mnt/myfs1
[root@server1 ~]#
Reverting a Stratis File System to a Previous Snapshot
It's a good idea to backup the current file system before reverting to a previous snapshot:
[root@server1 ~]# stratis fs snapshot mypool1 myfs1 myfs1snapshot2
[root@server1 ~]#
Next, we unmount and remove the original file system:
[root@server1 ~]# umount /mnt/myfs1
[root@server1 ~]# stratis fs destroy mypool1 myfs1
[root@server1 ~]#
We create a copy of a previous snapshot which we wish to restore, under the name of the original file system:
[root@server1 ~]# stratis fs list
Pool Name  Name            Used     Created            Device                           UUID                            
mypool1    myfs1snapshot2  545 MiB  Aug 19 2020 13:23  /stratis/mypool1/myfs1snapshot2  f54d88f686d64acd94c3a7d73dac92f5
mypool1    myfs1snapshot   545 MiB  Aug 19 2020 13:17  /stratis/mypool1/myfs1snapshot   b2fb662124a4424c9d21429012fcfdc4

[root@server1 ~]# stratis fs snapshot mypool1 myfs1snapshot myfs1
[root@server1 ~]# stratis fs list
Pool Name  Name            Used     Created            Device                           UUID                            
mypool1    myfs1snapshot2  545 MiB  Aug 19 2020 13:23  /stratis/mypool1/myfs1snapshot2  f54d88f686d64acd94c3a7d73dac92f5
mypool1    myfs1           545 MiB  Aug 19 2020 13:31  /stratis/mypool1/myfs1           82f75da64c744079b1c2ae51792812a0
mypool1    myfs1snapshot   545 MiB  Aug 19 2020 13:17  /stratis/mypool1/myfs1snapshot   b2fb662124a4424c9d21429012fcfdc4
We mount the snapshot, now accessible with the same name as the original file system:
[root@server1 ~]# mount /stratis/mypool1/myfs1 /mnt/myfs1
[root@server1 ~]#
Removing a Stratis Snapshot
We remove a Stratis snapshot by unmounting it first if required, then using the stratis fs destroy poolname snapshotname command.
[root@server1 ~]# stratis fs destroy mypool1 myfs1snapshot2
[root@server1 ~]#