Abstract: This paper integrates Octagon and Square Search
pattern (OCTSS) motion estimation algorithm into H.264/AVC
(Advanced Video Coding) video codec in Adaptive Group of Pictures
(AGOP) mode. AGOP structure is computed based on scene change
in the video sequence. Octagon and square search pattern block-based
motion estimation method is implemented in inter-prediction process
of H.264/AVC. Both these methods reduce bit rate and computational
complexity while maintaining the quality of the video sequence
respectively. Experiments are conducted for different types of video
sequence. The results substantially proved that the bit rate,
computation time and PSNR gain achieved by the proposed method
is better than the existing H.264/AVC with fixed GOP and AGOP.
With a marginal gain in quality of 0.28dB and average gain in bitrate
of 132.87kbps, the proposed method reduces the average computation
time by 27.31 minutes when compared to the existing state-of-art
H.264/AVC video codec.