Analysis of very large gzip/bzip2 compressed log files streamed


You can stream in PHP in the process the data very large log files, are several GB large, easily analyze with the help of Stream compression filter.

Thus the file without it can fully load into memory to unpack and be analyzed when importing the file.

This way you can search while reading the log file after any places in your code:

$file = 'X:/file.log.bz2';
$searchString = '2017-01-01';

if(file_exists($file)){
   $handle = fopen($file, 'r');
   stream_filter_append($handle, 'bzip2.decompress');
   while(!feof($handle)){
      $line = fgets($handle);
      if(strpos($line, $searchString) !== false){
         echo $line;
      }
   }
   fclose($handle);
}