Find and Locate to search files in Linux System

Finding by Name
The most obvious way of searching for files is by name.

         find -name "query"

This will be case sensitive, meaning a search for "file" is different than a search for "File".

To find a file by name, but ignore the case of the query, type:

          find -iname "query"

If you want to find all files that don't adhere to a specific pattern, you can invert the search with "-not" or "!". If you use "!", you must escape the character so that bash does not try to interpret it before find can act:

          find -not -name "query_to_avoid"

Or

          find \! -name "query_to_avoid"

Finding by Type
You can specify the type of files you want to find with the "-type" parameter. It works like this:

          find -type type_descriptor query

Some of the most common descriptors that you can use to specify the type of file are here:

           f: regular file

           d: directory

           l: symbolic link

           c: character devices

           b: block devices

For instance, if we wanted to find all of the character devices on our system, we could issue this command:

find / -type c
/dev/parport0
/dev/snd/seq
/dev/snd/timer
/dev/autofs
/dev/cpu/microcode
/dev/vcsa7
/dev/vcs7
/dev/vcsa6
/dev/vcs6
/dev/vcsa5
/dev/vcs5
/dev/vcsa4
. . .
We can search for all files that end in ".conf" like this:

        find / -type f -name "*.conf"

/var/lib/ucf/cache/:etc:rsyslog.d:50-default.conf
/usr/share/base-files/nsswitch.conf
/usr/share/initramfs-tools/event-driven/upstart-jobs/mountall.conf
/usr/share/rsyslog/50-default.conf
/usr/share/adduser/adduser.conf
/usr/share/davfs2/davfs2.conf
/usr/share/debconf/debconf.conf
/usr/share/doc/apt-utils/examples/apt-ftparchive.conf
. . .

Filtering by Time and Size
Find gives you a variety of ways to filter results by size and time.

by Size

You can filter by size with the use of the "-size" parameter.

We add a suffix on the end of our value that specifies how we are counting. These are some popular options:

c: bytes

k: Kilobytes

M: Megabytes

G: Gigabytes

b: 512-byte blocks

To find all files that are exactly 50 bytes, type:
find / -size 50c

To find all files less than 50 bytes, we can use this form instead:
find / -size -50c

To Find all files more than 700 Megabytes, we can use this command:

find / -size +700M


By Time

Linux stores time data about access times, modification times, and change times.

Access Time: Last time a file was read or written to.

Modification Time: Last time the contents of the file were modified.

Change Time: Last time the file's inode meta-data was changed.

We can use these with the "-atime", "-mtime", and "-ctime" parameters. These can use the plus and minus symbols to specify greater than or less than, like we did with size.

The value of this parameter specifies how many days ago you'd like to search.

To find files that have a modification time of a day ago, type:

find / -mtime 1
If we want files that were accessed in less than a day ago, we can type:

find / -atime -1
To get files that last had their meta information changed more than 3 days ago, type:

find / -ctime +3
There are also some companion parameters we can use to specify minutes instead of days:

find / -mmin -1

This will give the files that have been modified type the system in the last minute.

Find can also do comparisons against a reference file and return those that are newer:

find / -newer myfile

Finding by Owner and Permissions
You can also search for files by the file owner or group owner.

You do this by using the "-user" and "-group" parameters respectively. Find a file that is owned by the "syslog" user by entering:

find / -user syslog

Similarly, we can specify files owned by the "shadow" group by typing:
find / -group shadow
We can also search for files with specific permissions.

If we want to match an exact set of permissions, we use this form:
find / -perm 644

This will match files with exactly the permissions specified.

If we want to specify anything with at least those permissions, you can use this form:
find / -perm -644

This will match any files that have additional permissions. A file with permissions of "744" would be matched in this instance.

Filtering by Depth
For this section, we will create a directory structure in a temporary directory. It will contain three levels of directories, with ten directories at the first level. Each directory (including the temp directory) will contain ten files and ten subdirectories.

Make this structure by issuing the following commands:

cd
mkdir -p ~/test/level1dir{1..10}/level2dir{1..10}/level3dir{1..10}
touch ~/test/{file{1..10},level1dir{1..10}/{file{1..10},level2dir{1..10}/{file{1..10},level3dir{1..10}/file{1..10}}}}
cd ~/test
Feel free to check out the directory structures with ls and cd to get a handle on how things are organized. When you are finished, return to the test directory:

cd ~/test
We will work on how to return specific files from this structure. Let's try an example with just a regular name search first, for comparison:

find -name file1
./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
./level1dir7/level2dir8/level3dir6/file1
./level1dir7/level2dir8/level3dir5/file1
./level1dir7/level2dir8/file1
. . .
There are a lot of results. If we pipe the output into a counter, we can see that there are 1111 total results:

find -name file1 | wc -l
1111
This is probably too many results to be useful to you in most circumstances. Let's try to narrow it down.

You can specify the maximum depth of the search under the top-level search directory:

find -maxdepth num -name query

To find "file1" only in the "level1" directories and above, you can specify a max depth of 2 (1 for the top-level directory, and 1 for the level1 directories):

find -maxdepth 2 -name file1
./level1dir7/file1
./level1dir1/file1
./level1dir3/file1
./level1dir8/file1
./level1dir6/file1
./file1
./level1dir2/file1
./level1dir9/file1
./level1dir4/file1
./level1dir5/file1
./level1dir10/file1
That is a much more manageable list.

You can also specify a minimum directory if you know that all of the files exist past a certain point under the current directory:

find -mindepth num -name query
We can use this to find only the files at the end of the directory branches:

find -mindepth 4 -name file
./level1dir7/level2dir8/level3dir9/file1
./level1dir7/level2dir8/level3dir3/file1
./level1dir7/level2dir8/level3dir4/file1
./level1dir7/level2dir8/level3dir1/file1
./level1dir7/level2dir8/level3dir8/file1
./level1dir7/level2dir8/level3dir7/file1
./level1dir7/level2dir8/level3dir2/file1
. . .
Again, because of our branching directory structure, this will return a large number of results (1000).

You can combine the min and max depth parameters to focus in on a narrow range:

find -mindepth 2 -maxdepth 3 -name file
./level1dir7/level2dir8/file1
./level1dir7/level2dir5/file1
./level1dir7/level2dir7/file1
./level1dir7/level2dir2/file1
./level1dir7/level2dir10/file1
./level1dir7/level2dir6/file1
./level1dir7/level2dir3/file1
./level1dir7/level2dir4/file1
./level1dir7/file1
. . .
Executing and Combining Find Commands
You can execute an arbitrary helper command on everything that find matches by using the "-exec" parameter. This is called like this:

find find_parameters -exec command_and_params {} \;
The "{}" is used as a placeholder for the files that find matches. The "\;" is used so that find knows where the command ends.

For instance, we could find the files in the previous section that had "644" permissions and modify them to have "664" permissions:

cd ~/test
find . -type f -perm 644 -exec chmod 664 {} \;
We could then change the directory permissions like this:

find . -type d -perm 755 -exec chmod 700 {} \;
If you want to chain different results together, you can use the "-and" or "-or" commands. The "-and" is assumed if omitted.

find . -name file1 -or -name file9 


Find Files Using Locate:
An alternative to using find is the locate command. This command is often quicker and can search the entire file system with ease.

You can install the command with apt-get:

sudo apt-get update
sudo apt-get install mlocate

The reason locate is faster than find is because it relies on a database of the files on the filesystem.

The database is usually updated once a day with a cron script, but you can update it manually by typing:

sudo updatedb
Run this command now. Remember, the database must always be up-to-date if you want to find recently acquired or created files.

To find files with locate, simply use this syntax:
locate query

You can filter the output in some ways.
For instance, to only return files containing the query itself, instead of returning every file that has the query in the directories leading to it, you can use the "-b" for only searching the "basename":

locate -b query

To have locate only return results that still exist in the filesystem (that were not remove between the last "updatedb" call and the current "locate" call), use the "-e" flag:

locate -e query

To see statistics about the information that locate has cataloged, use the "-S" option:
locate -S

Database /var/lib/mlocate/mlocate.db:
    3,315 directories
    37,228 files
    1,504,439 bytes in file names
    594,851 bytes used to store database

Tomcat log rotation

1. Create this file
/etc/logrotate.d/tomcat  

2. Copy the following contents into the above file

/oracle/mm/prod-tomcat-1/logs/catalina.out {  
 copytruncate  
 daily  
 rotate 7  
 compress  
 missingok  
 size 5M  
}  

About the above configuration:

Make sure that the path /oracle/mm/prod-tomcat-1/logs/catalina.out above is adjusted to point to your tomcat’s catalina.out

daily - rotates the catalina.out daily
rotate – keeps at most 7 log files
compress – compresses the rotated files
size – rotates if the size of catalina.out is bigger than 5M

copytruncate – truncates the original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one, It can be used when some program can not be told to close its logfile and thus might continue writing (appending) to the previous log file forever. Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost. When this option is used, the create option will have no effect, as the old log file stays in place.

You don’t need to do anything else.

How it works:

Every night the cron daemon runs jobs listed in the /etc/cron.daily/ directory

This triggers the /etc/cron.daily/logrotate file which is generally shipped with linux installations. It runs the command “/usr/sbin/logrotate /etc/logrotate.conf“

The /etc/logrotate.conf includes all scripts in the /etc/logrotate.d/ directory.
This triggers the /etc/logrotate.d/tomcat file that you wrote in the previous step.

Run logrotate manually

Run the following command to run the cron job manually:

/usr/sbin/logrotate /etc/logrotate.conf  

Is it completely safe?

See the description of copytruncate method above. There is a slight chance of a small amount of logging data loss between copy and truncate steps, usually it is acceptable but sometimes its not.

More logrotate options

To see all logrotate options on your system, see the manual:

man logrotate  

CORS : Cross-Origin Resource Sharing

The future of the web is cross-domain, not same-origin. As CORS continues the spirit of the open web by bringing API access to all.

The same-origin policy is an important security concept implemented by web browsers to prevent Javascript code from making requests against a different origin (e.g., different domain) than the one from which it was served.

Cross-Origin Resource Sharing (CORS) is a specification that enables truly open access across domain-boundaries. The spec defines a set of headers that allow the browser and server to communicate about which requests are (and are not) allowed. CORS is a technique for relaxing the same-origin policy, allowing Javascript on a web page to consume a REST API served from a different origin.

Simple implementation:

In the simplest scenario, cross-origin communications starts with a client making a GET, POST, or HEAD request against a resource on the server. In this scenario, the content type of a POST request is limited to application/x-www-form-urlencoded, multipart/form-data, or text/plain. The request includes an Origin header that indicates the origin of the client code.

The server will consider the request's Origin and either allow or disallow the request. If the server allows the request, then it will respond with the requested resource and an Access-Control-Allow-Origin header in the response. This header will indicate to the client which client origins will be allowed to access the resource. Assuming that the Access-Control-Allow-Origin header matches the request's Origin, the browser will allow the request.

On the other hand, if Access-Control-Allow-Origin is missing in the response or if it doesn't match the request's Origin, the browser will disallow the request.

For simple CORS requests, the server only needs to add the following header to its response:

Access-Control-Allow-Origin: *

It differs to implement CORS for specific platforms. Suppose for tomcat a minimal CORS configuration as:


<filter>
  <filter-name>CorsFilter</filter-name>
  <filter-class>org.apache.catalina.filters.CorsFilter</filter-class>
</filter>
<filter-mapping>
  <filter-name>CorsFilter</filter-name>
  <url-pattern>/*</url-pattern>
</filter-mapping>




N.B: Bear in mind that CORS is not about providing server-side security. The Origin request header is produced by the browser and the server has no direct means to verify it.

Concurrency &. Parallelism

For the case of multi threaded programs we often use these terms, concurrency and parallelism.

Multiple tasks are in progress at the same time refers to concurrency. An application may process one task at at time (sequentially) or work on multiple tasks at the same time (concurrently).

Each task is broken in to sub tasks which can be processed in parallel. It is related to how an application handles each individual task. An application may process the task serially from start to end, or split the task up into subtasks which can be completed in parallel.

• An application can be concurrent, but not parallel. This means that it processes more than one task at the same time, but the tasks are not broken down into subtasks.
• An application can also be parallel but not concurrent. This means that the application only works on one task at a time, and this task is broken down into subtasks which can be processed in parallel.
• Additionally, an application can be neither concurrent nor parallel. This means that it works on only one task at a time, and the task is never broken down into subtasks for parallel execution.
• Finally, an application can also be both concurrent and parallel, in that it both works on multiple tasks at the same time, and also breaks each task down into subtasks for parallel execution. However, some of the benefits of concurrency and parallelism may be lost in this scenario, as the CPUs in the computer are already kept reasonably busy with either concurrency or parallelism alone. Combining it may lead to only a small performance gain or even performance loss. We should analyze and measure deeply before we adopt a concurrent parallel model blindly.

Cloud: IaaS, PaaS, SaaS

Cloud computing is the phrase used to describe different scenarios in which computing resource is delivered as a service over a network connection (usually, this is the internet). Cloud computing is therefore a type of computing that relies on sharing a pool of physical and/or virtual resources, rather than deploying local or personal hardware and software.

IaaS:

Infrastructure as a Service (IaaS) is one of the three fundamental service models of cloud computing alongside Platform as a Service (PaaS) and Software as a Service (SaaS). As with all cloud computing services it provides access to computing resource in a virtualised environment, “the Cloud”, across a public connection, usually the internet. In the case of IaaS the computing resource provided is specifically that of virtualised hardware, in other words, computing infrastructure. The definition includes such offerings as virtual server space, network connections, bandwidth, IP addresses and load balancers. Physically, the pool of hardware resource is pulled from a multitude of servers and networks usually distributed across numerous data centers, all of which the cloud provider is responsible for maintaining. The client, on the other hand, is given access to the virtualised components in order to build their own IT platforms.

In common with the other two forms of cloud hosting, IaaS can be utilised by enterprise customers to create cost effective and easily scalable IT solutions where the complexities and expenses of managing the underlying hardware are outsourced to the cloud provider. If the scale of a business customer’s operations fluctuate, or they are looking to expand, they can tap into the cloud resource as and when they need it rather than purchase, install and integrate hardware themselves.

PaaS:

Platform as a Service, often simply referred to as PaaS, is a category of cloud computing that provides a platform and environment to allow developers to build applications and services over the internet. PaaS services are hosted in the cloud and accessed by users simply via their web browser. 

Platform as a Service allows users to create software applications using tools supplied by the provider. PaaS services can consist of preconfigured features that customers can subscribe to; they can choose to include the features that meet their requirements while discarding those that do not. Consequently, packages can vary from offering simple point-and-click frameworks where no client side hosting expertise is required to supplying the infrastructure options for advanced development.

The infrastructure and applications are managed for customers and support is available. Services are constantly updated, with existing features upgraded and additional features added. PaaS providers can assist developers from the conception of their original ideas to the creation of applications, and through to testing and deployment. This is all achieved in a managed mechanism.

As with most cloud offerings, PaaS services are generally paid for on a subscription basis with clients ultimately paying just for what they use. Clients also benefit from the economies of scale that arise from the sharing of the underlying physical infrastructure between users, and that results in lower costs.

Below are some of the features that can be included with a PaaS offering:

• Operating system
• Server-side scripting environment
• Database management system
• Server Software
• Support
• Storage
• Network access
• Tools for design and development
• Hosting

Software developers, web developers and businesses can benefit from PaaS. Whether building an application which they are planning to offer over the internet or software to be sold out of the box, software developers may take advantage of a PaaS solution. For example, web developers can use individual PaaS environments at every stage of the process to develop, test and ultimately host their websites. However, businesses that are developing their own internal software can also utilise Platform as a Service, particularly to create distinct ring-fenced development and testing environments.

SaaS?

SaaS, or Software as a Service, describes any cloud service where consumers are able to access software applications over the internet. The applications are hosted in “the cloud” and can be used for a wide range of tasks for both individuals and organisations. Google, Twitter, Facebook and Flickr are all examples of SaaS, with users able to access the services via any internet enabled device. Enterprise users are able to use applications for a range of needs, including accounting and invoicing, tracking sales, planning, performance monitoring and communications (including webmail and instant messaging).

SaaS is often referred to as software-on-demand and utilising it is akin to renting software rather than buying it. With traditional software applications you would purchase the software upfront as a package and then install it onto your computer. The software’s licence may also limit the number of users and/or devices where the software can be deployed. Software as a Service users, however, subscribe to the software rather than purchase it, usually on a monthly basis. Applications are purchased and used online with files saved in the cloud rather than on individual computers.